commercetools Connector Extend Configuration

Credential / Secrets Configuration

In order for the commercetools-connector to connect to external systems, it requires access to their credentials and these must be stored in a secure location. The commercetools-connector provides a way for new credential storage services to be added and this is covered in the customisation guide. To configure your credentials, follow the Credential / Secret Setup Guide.

Common Configuration

commercetools Connector Listener Configuration

Before configuring the actual listeners, let's first look at the commercetools-connector settings as these will impact how listeners are initiated within the commercetools-connector.

1fluent-connect:
2  message-listener-mode: sqs
3  batch-listener-mode: sqs
4  available-listeners:
5    - "sqs"
6    - "kafka"

Listener Configuration

It is possible to have any number of listeners running and each must have its own configuration like the sample below.

1listeners:
2    # listener configurion block 
3    listener-id:
4      name: "ENV_VARIABLE"
5      pool-size: 20
6      prefix: "my-prefix"
7      shutdown-timeout: 300
8      visibility-timeout: 120
9      poll-wait: 20
10      dlq-suffix: "-dlq"
11      retry: 5
12      type: sqs   

Job Scheduler Configuration

The commercetools-connector doesn't have an internal scheduler, and it depends on external systems to trigger an HTTP request for a job to be executed as described below.

1PUT https://<domain>/api/v1/fluent-connect/scheduler/add/<fluent-account>/<fluent-retailer>/<job-name>

All job requests require 3 parameters:

• fluent-account: Fluent account name
• fluent-retailer: Fluent retailer id
• job-name: The name of the handler that will execute the job

For example:

1PUT https://localhost:8080/api/v1/fluent-connect/scheduler/add/cnctdev/1/batch-inventory-sync

The execution of a job is performed asynchronously and the HTTP request returns 200 OK.

All jobs in the commercetools-connector share the same pre-execution and post-execution steps:

• A job cannot have multiple instances running simultaneously
• Retrieves the job configuration from Fluent settings
• Calculate a date range (from-to datetime) the job may use during its execution
• Execute the actual job
• Persist the calculated date range as part of the job configuration, so it can be utilised on the next execution
• commercetools-connector runs at UTC time.

There are 2 kinds of configuration for a job:

• Runtime configuration
• Kept at Fluent as settings
• The configuration key of a job is determined dynamically by taking the `config-prefix` and appending the job name at the end. With the template below, config-prefix once configured would have the value of `fc.connect.my-sample-connector.data-sync` for example. Combining this with the job name, the final configuration key would be `fc.connect.my-sample-connector.data-sync.batch-inventory-sync`.

`1 job-scheduler: 2 config-prefix: "fc.connect.<connector-name>.data-sync"`

Every job gets a pre-calculated date range and this is provided by the commercetools-connector to any job handler as part of the job context. This date range allows the job to work with data in delta mode (if required), only fetching/processing items that have changed since the last execution of the job.

Below is an example of a configuration with no previous execution and some properties in `props`. The date `previousEndDate` is not mandatory. Whenever there is no valid date set at `previousEndDate`, the commercetools-connector considers the job never ran before and the start date used by the job is EPOCH time. When configuring the job for the first time, if there is a need that its first execution uses a specific start date for the calculated date range then set the desired date at `previousEndDate`.

1{
2    "previousEndDate": "",
3    "props": {
4        "fluentNetworkRef": "BASE_67"
5    }
6}

The date range works in the following way:

• First execution (with no date set at `previousEndDate`) - From: Epoch time, To: Now
• First execution (with a date set at `previousEndDate`) - From: date set at `previousEndDate`, To: Now
• Any subsequent execution - From: Last execution end datetime (`previousEndDate`), To: now

If the commercetools-connector runs a daily job with the configuration above, the result of the first execution would be as shown below. As the job starts executing, `previousEndDate` is set to the server’s current datetime (in UTC). The calculated date range used by the job is also saved at `lastRun.param.start` and `lastRun.param.end` - note the correlation of `lastRun.param.end` and `previousEndDate`.

1{
2  "previousEndDate": "2022-09-22T03:00:00",
3  "props": {
4    "fluentNetworkRef": "BASE_67"
5  },
6  "lastRun": {
7    "param": {
8      "start": "1970-01-01T00:00:00",
9      "end": "2022-09-22T03:00:00",
10      "props": {
11        "fluentNetworkRef": "BASE_67"
12      }
13    },
14    "jobStart": "2022-09-22T03:00:00",
15    "jobsEnd": "2022-09-22T03:05:00",
16    "status": "SUCCESSFUL"
17  }
18}

The next day the job runs again, the current value of `previousEndDate` is used as the start date `lastRun.param.start` and a new date are set for both `lastRun.param.end` and `previousEndDate`.

1{
2  "previousEndDate": "2022-09-23T02:00:00",
3  "props": {
4    "fluentNetworkRef": "BASE_67"
5  },
6  "lastRun": {
7    "param": {
8      "start": "2022-09-22T03:00:00",
9      "end": "2022-09-23T02:00:00",
10      "props": {
11        "fluentNetworkRef": "BASE_67"
12      }
13    },
14    "jobStart": "2022-09-23T02:00:00",
15    "jobsEnd": "2022-09-23T02:05:00",
16    "status": "SUCCESSFUL"
17  }
18}

It is also possible for a user to override the normal job operation described above by providing a custom date range. These user-specified dates in UTC are only used once and will not be taken into account for subsequent executions.

1{
2  "customDateRange": {
3    "start": "2022-04-22T00:00:00",
4    "end": "2022-04-23T00:00:00"
5  },
6  "previousEndDate": "2022-09-23T02:00:00",
7  "props": {
8    "fluentNetworkRef": "BASE_67"
9  },
10  "lastRun": {
11    "param": {
12      "start": "2022-09-22T03:00:00",
13      "end": "2022-09-23T02:00:00",
14      "props": {
15        "fluentNetworkRef": "BASE_67"
16      }
17    },
18    "jobStart": "2022-09-23T02:00:00",
19    "jobsEnd": "2022-09-23T02:05:00",
20    "status": "SUCCESSFUL"
21  }
22}

After the job runs with the custom dates, the custom range is removed and `previousEndDate` remains as it previously was so the job resumes its normal operation.

1{
2  "previousEndDate": "2022-09-23T02:00:00",
3  "props": {
4    "fluentNetworkRef": "BASE_67"
5  },
6  "lastRun": {
7    "param": {
8      "start": "2022-04-22T00:00:00",
9      "end": "2022-04-23T00:00:00",
10      "props": {
11        "fluentNetworkRef": "BASE_67"
12      }
13    },
14    "jobStart": "2022-09-24T02:00:00",
15    "jobsEnd": "2022-09-24T02:05:00",
16    "status": "SUCCESSFUL"
17  }
18}

While the job is running, a temporary property called `executionStart` is added to indicate the job is running, and it is removed when the job completes.

1{
2  "previousEndDate": "2022-09-23T02:00:00",
3  "executionStart" : "2022-09-24T02:00:00",
4  "props": {
5    "fluentNetworkRef": "BASE_67"
6  },
7  "lastRun": {
8    "param": {
9      "start": "2022-04-22T00:00:00",
10      "end": "2022-04-23T00:00:00",
11      "props": {
12        "fluentNetworkRef": "BASE_67"
13      }
14    },
15    "jobStart": "2022-09-24T02:00:00",
16    "jobsEnd": "2022-09-24T02:05:00",
17    "status": "SUCCESSFUL"
18  }
19}

• Build configuration
  • Kept part of the deployable artifact (yaml file properties, for example)
  • Every Connect commercetools-connector handler has access to optional configuration (props) that are part of the route setup. This is covered in more detail in the Handler Configuration of the customisation guide.

1routes:
2    - name: "fc.connect.batch-inventory-sync"
3      handler: "BatchInventorySyncJob"
4      props:
5        page-size: 500

General configuration

At a minimum, the commercetools-connector expects the following configuration.

1connector-name: commercetools
2configuration:
3    base-path-key: "fc.connect.<connector-name>"
4    account-mapping-key: "fc.connect.<connector-name>.account-mapping"
5    general-key: "fc.connect.<connector-name>.general"

Account Mapping Example

1{
2    "fluent": {
3        "accountId": "FCTDEV",
4        "retailerId": "100"
5    },
6    "externalAccount": {
7        "projectKey": "my-project-qa",
8        "uid": "123"
9    }
10}

General Settings Example

1{
2    "productCatalogue": {
3        "entityRef": "PC:MASTER:100",
4        "entitySubType": "MASTER"
5    }
6}