Trigger your import or export job

Last update:
Created :
Written by Thomas Speekenbrink

Overview: 

Automation with Nebula DataBridge webhooks. Nebula DataBridge now has three methods available to start your import or export job.

  • Manual
  • Scheduled
  • Triggered

How to:

On the ‘PROFILE SETTINGS’ page of the Nebula DataBridge wizard there are 3 methods available to start your import or export job:

  1. MANUAL, according to the profile configuration you can manually select a compatible import file in the browser to start processing in a background process; or start an export with a single button click.
  2. SCHEDULED, automatically start a job at the selected days and time. In case of an import a new file will be retrieved from your selected storage location.
  3. TRIGGERED, work will be done according to the profile configuration when a request is made to a specific secret URL. There are 2 options that you need to enable separately with the toggle button:
    1. Triggered jobs, allows you to create and run a triggered job in a background process by calling a URL. This is a long running process and the execution is the same as with SCHEDULED jobs. Calling a triggered job is limited to once per 15 minutes on each profile, more calls to the same trigger URL will fail; this fair-use-policy is maintained to keep te load on Nebula DataBridge and SuperOffice CRM Online acceptable.
    2. Instant jobs, allows you to create and run an instant job directly in the browser by calling a URL with parameters. Because the work is done directly in the browser request this is only suitable for limited work; for example if you just want to export a single record to a file for further processing. Please carefully read the information below if you want to use this advanced functionality.

 

Triggered.PNG

 

When you select the TRIGGERED scheduling method you will be presented with the following options:

  • Triggered jobs enabled: switch to ‘Yes’ in order to create and run a triggered job in a background process by calling a URL.
  • Instant jobs enabled: switch to ‘Yes’ in order to create and run an instant job directly in the browser by calling a URL with parameters.
  • Secret access key: this key is similar to what is called an API key. Together with the profile ID this acts as a username and password for the webhook. In case you need to renew it press the GENERATE button and SAVE at the bottom of the page.

Triggered jobs

When triggered jobs are enabled and you make a request to your Trigger/Now URL, for example:

https://trigger-databridge.infobridgeuniverse.com/Trigger/Now/253?key=[Your_secret_access_key]

A JSON result will be returned that you can optionally parse in your integration/application:

{"Success":true,"Message":"OK"}

Your profile will be tagged to start a new job as soon as possible, this can take up to 1 minute before it actually gets created and processed. If successful you will find the new job in the jobs overview of the profile where you can also view the log.

NOTE: The only way we can currently offer and allow unlimited triggers without additional cost is with a fair use policy of one trigger every 15 minutes per profile.

For processing changes faster we have developed the instant sync feature which processes 1 or a small amount of changes immediately, see below

Instant jobs

When instant jobs are enabled and you make a request to your Instant/Run URL, for example:

https://trigger-databridge.infobridgeuniverse.com/Instant/Run/253?key=[Your_secret_access_key]

Processing is started immediately in the browser request and a JSON result will be returned when done that you can optionally parse in your integration/application:

{"Success":true,"Message":"OK"}

Logs of the instant requests are written to a single instant job entity per profile which you can access on the JOBS overview page by the VIEW INSTANT JOB button. Here you see cumulative counters and you will be able to access the logs.

Without additional parameters the job will be processed according to the profile settings as configured; meaning that the import is done with the original and potentially large import file; and the export is based on the full originally selected SuperOffice selection.

This means that without extra control there won’t be much benefits against triggered jobs, this is where the extra parameters come in which you can add to the request URL:

General engine parameters

Parameter

Description

ShowLog (true or false)

Show the log output in the browser when done.

Verbose (true or false)

Enable full detailed logging.

ExportFile (filename)

Override the file to export by name.

 

FTP specific parameters

Parameter

Description

FTPFilePath (filepath)

Override the file to import by file path when using FTP.

 

SuperOffice specific parameters

Parameter

Description

ContactId

Primary key of the company (named 'contact' in the database) to process, SuperOffice 'cuid' variable.

PersonId

Primary key of the person to process, SuperOffice 'atid' variable.

AppointmentId

Primary key of the appointment to process, SuperOffice 'baid' variable.

DocumentId

Primary key of the document to process, SuperOffice 'doid' variable.

ProjectId

Primary key of the project to process, SuperOffice 'prid' variable.

SaleId

Primary key of the sale to process, SuperOffice 'said' variable.

QuoteLineId

Primary key of the quote line to process.

 

Best practise:

By default you receive a JSON result whether the request was successful, this is good for production and saves bandwidth. But when setting up an integration and doing some tests it is very useful to append ‘&ShowLog=true&Verbose=true’ to your request URL; this will output the full Nebula DataBridge log and all steps taken to your browser when done so you can exactly see what happened.

Example 1:

If you have configured your profile to import data in SuperOffice from an FTP location and want to run the import targeting a different import file your integration has prepared:

https://trigger-databridge.infobridgeuniverse.com/Instant/Run/253?key=05c3f4fe-08c0-4b58-ba91-d7cd34be9489&FTPFilePath=/DataBridgeImport/Contact1337.csv

The ‘FTPFilePath’ parameter makes sure another file is imported than configured in the profile. This file must be compatible and have the same columns as configured or it will fail!

Example 2:

If you have configured your profile to export companies from SuperOffice to Dropbox and want to export a single specific company to a specific file in Dropbox:

https://trigger-databridge.infobridgeuniverse.com/Instant/Run/253?key=05c3f4fe-08c0-4b58-ba91-d7cd34be9489&ExportFile=Contact1337.csv&ContactId=1337

This makes sure only one record from SuperOffice (the company with ID 1337) is exported to the requested file (Contact1337.csv in your Dropbox folder). Please make sure your SuperOffice selection chosen in the profile wizard matches the type of record you want to export!

Instant jobs limitations

The instant jobs are very powerful when building an integration, but because everything is executed on the webserver we have to deal with timeouts and limited capacity. Therefore the following limitations apply:

  • Limit of lines per instant request: 10
    • When 10 lines are processed the instant job will be aborted, this will be visible in the logs and the job will return as successful.
  • Limit of import file size in instant requests: 2MB (2.000.000 Bytes)
    • When you try to load an import file above 2MB in an instant job the job will fail with an error.
  • Limit of lines per profile when in trial license: 100
    • Without a license you can try instant jobs with a limit of 100 lines in total per profile, when the limit is reached in trial the jobs will fail with an error.
  • Instant jobs will ignore the notification settings from the PROFILE SETTINGS page, these are only for the other scheduling methods running full jobs in a background process.
  • Instant jobs have a limited list of superoffice fields to select from when exporting a single entity, these are marked with an * in the 'Customize fields' list on the CONFIGURE SOURCE page, most of these exclusions have to do with selections.