commercetools Connector Extend Configuration

Feature

Changed on:

31 Jan 2024

Overview

The commercetools-connector expects a minimum of 4 configuration files:

application-connector.yml - Project and Connector specific configuration.
application.yml - Core Connect commercetools-connector configuration. These are mostly default configurations and core settings.
bootstrap.yml - Server/Container configuration. This file includes settings for all different environments and -follows the spring profile standards.
logback-spring.xml - Log configuration also supports different configurations based on spring profiles.

It is possible to have dedicated configuration files for each environment. This is often used to override a common setting for a particular environment. To create such a file, follow SpringBoot's format having the additional desired spring profile name at the end of the application.yml file, for example: application-dev.yml.

Detailed Technical Description

Credential / Secrets Configuration

In order for the commercetools-connector to connect to external systems, it requires access to their credentials and these must be stored in a secure location. The commercetools-connector provides a way for new credential storage services to be added and this is covered in the customisation guide. To configure your credentials, follow the Credential / Secret Setup Guide.

Common Configuration

commercetools Connector Listener Configuration

Before configuring the actual listeners, let's first look at the commercetools-connector settings as these will impact how listeners are initiated within the commercetools-connector.

Property	Required	Description
message-listener-mode	yes	Sets the commercetools-connector message listener type. This controls what kind of listener is used for the commercetools-connector message queue only, and external listeners are not impacted by this setting.
batch-listener-mode	yes	Sets the commercetools-connector batch listener type. This controls what kind of listener is used for the commercetools-connector batch queue.
available-listeners	yes	Lists of all available listeners for the commercetools-connector. If a listener is not listed here, the commercetools-connector will not consider it.

1fluent-connect:
2  message-listener-mode: sqs
3  batch-listener-mode: sqs
4  available-listeners:
5    - "sqs"
6    - "kafka"

Language: java

Name: Example

Description:

[Warning: empty required content area]

Listener Configuration

It is possible to have any number of listeners running and each must have its own configuration like the sample below.

1listeners:
2    # listener configurion block 
3    listener-id:
4      name: "ENV_VARIABLE"
5      pool-size: 20
6      prefix: "my-prefix"
7      shutdown-timeout: 300
8      visibility-timeout: 120
9      poll-wait: 20
10      dlq-suffix: "-dlq"
11      retry: 5
12      type: sqs

Language: java

Name: Example

Description:

[Warning: empty required content area]

Property	Required	Description	Default Value
listener-id	yes	Actual id of the listener. This must match the id defined at the class when using @ListenerInfo.	n/a
name	yes	Name of the environment variable that will contain the real queue name.	Empty String
type	yes	Type of the listener. This is based on a list of supported listeners of the Connect commercetools-connector.	NOT_DEFINED
fluent-account	yes	Fluent account name. This is required and restricts the processing of messages to services and configuration for said account.	n/a
pool-size	no	Number of threads available to process messages from the queue. Determines how many messages can be processed in parallel	10
prefix	no	Prefix given to the threads processing messages.	default
shutdown-timeout	no	Wait time in seconds to allow in-flight threads to gracefully finish processing before forcing them to quit	300
visibility-timeout	no	Queue visibility timeout in seconds - Messages read from the queue and kept invisible to other listeners while the node that took the message processes it. If the message is not deleted before the timeout the message is made available again for processing - retry in case of errors.	120
poll-wait	no	Listener wait time in seconds to read as many items from the queue as possible.	10
dlq-suffix	no	Dead letter queue suffix. DLQ is a generated name from the queue name with the suffix appended to the end.	-dlq
retry	no	Queue retry policy. How many times the message is put for retry on the queue before it goes to DLQ.	2
max-allowed-pull-size	no	Max number of allowed messages to pull from the queue.	10
props	no	Additional listener configuration. Connect commercetools-connector will read all configuration set under props and have them stored as a Map.	n/a

Job Scheduler Configuration

The commercetools-connector doesn't have an internal scheduler, and it depends on external systems to trigger an HTTP request for a job to be executed as described below.

1PUT https://<domain>/api/v1/fluent-connect/scheduler/add/<fluent-account>/<fluent-retailer>/<job-name>

Language: plain_text

Name: Trigger HTTP request

Description:

[Warning: empty required content area]

All job requests require 3 parameters:

fluent-account: Fluent account name
fluent-retailer: Fluent retailer id
job-name: The name of the handler that will execute the job

For example:

1PUT https://localhost:8080/api/v1/fluent-connect/scheduler/add/cnctdev/1/batch-inventory-sync

Language: plain_text

Name: Example

Description:

[Warning: empty required content area]

The execution of a job is performed asynchronously and the HTTP request returns 200 OK.

All jobs in the commercetools-connector share the same pre-execution and post-execution steps:

A job cannot have multiple instances running simultaneously
Retrieves the job configuration from Fluent settings
Calculate a date range (from-to datetime) the job may use during its execution
Execute the actual job
Persist the calculated date range as part of the job configuration, so it can be utilised on the next execution
commercetools-connector runs at UTC time.

There are 2 kinds of configuration for a job:

Runtime configuration
Kept at Fluent as settings
The configuration key of a job is determined dynamically by taking the
```
`config-prefix`
```
and appending the job name at the end. With the template below, config-prefix once configured would have the value of
```
`fc.connect.my-sample-connector.data-sync`
```
for example. Combining this with the job name, the final configuration key would be
```
`fc.connect.my-sample-connector.data-sync.batch-inventory-sync`
```
.

`1 job-scheduler: 2 config-prefix: "fc.connect.<connector-name>.data-sync"`

Every job gets a pre-calculated date range and this is provided by the commercetools-connector to any job handler as part of the job context. This date range allows the job to work with data in delta mode (if required), only fetching/processing items that have changed since the last execution of the job.

Below is an example of a configuration with no previous execution and some properties in

`props`

. The date

`previousEndDate`

is not mandatory. Whenever there is no valid date set at

`previousEndDate`

, the commercetools-connector considers the job never ran before and the start date used by the job is EPOCH time. When configuring the job for the first time, if there is a need that its first execution uses a specific start date for the calculated date range then set the desired date at

`previousEndDate`

1{
2    "previousEndDate": "",
3    "props": {
4        "fluentNetworkRef": "BASE_67"
5    }
6}

Language: json

Name: Example

Description:

[Warning: empty required content area]

The date range works in the following way:

First execution (with no date set at
```
`previousEndDate`
```
) - From: Epoch time, To: Now
First execution (with a date set at
```
`previousEndDate`
```
) - From: date set at
```
`previousEndDate`
```
, To: Now
Any subsequent execution - From: Last execution end datetime (
```
`previousEndDate`
```
), To: now

If the commercetools-connector runs a daily job with the configuration above, the result of the first execution would be as shown below. As the job starts executing,

`previousEndDate`

is set to the server’s current datetime (in UTC). The calculated date range used by the job is also saved at

`lastRun.param.start`

and

`lastRun.param.end`

- note the correlation of

`lastRun.param.end`

and

`previousEndDate`

1{
2  "previousEndDate": "2022-09-22T03:00:00",
3  "props": {
4    "fluentNetworkRef": "BASE_67"
5  },
6  "lastRun": {
7    "param": {
8      "start": "1970-01-01T00:00:00",
9      "end": "2022-09-22T03:00:00",
10      "props": {
11        "fluentNetworkRef": "BASE_67"
12      }
13    },
14    "jobStart": "2022-09-22T03:00:00",
15    "jobsEnd": "2022-09-22T03:05:00",
16    "status": "SUCCESSFUL"
17  }
18}

Language: json

Name: Example

Description:

[Warning: empty required content area]

The next day the job runs again, the current value of

`previousEndDate`

is used as the start date

`lastRun.param.start`

and a new date are set for both

`lastRun.param.end`

and

`previousEndDate`

1{
2  "previousEndDate": "2022-09-23T02:00:00",
3  "props": {
4    "fluentNetworkRef": "BASE_67"
5  },
6  "lastRun": {
7    "param": {
8      "start": "2022-09-22T03:00:00",
9      "end": "2022-09-23T02:00:00",
10      "props": {
11        "fluentNetworkRef": "BASE_67"
12      }
13    },
14    "jobStart": "2022-09-23T02:00:00",
15    "jobsEnd": "2022-09-23T02:05:00",
16    "status": "SUCCESSFUL"
17  }
18}

Language: json

Name: Example

Description:

[Warning: empty required content area]

It is also possible for a user to override the normal job operation described above by providing a custom date range. These user-specified dates in UTC are only used once and will not be taken into account for subsequent executions.

1{
2  "customDateRange": {
3    "start": "2022-04-22T00:00:00",
4    "end": "2022-04-23T00:00:00"
5  },
6  "previousEndDate": "2022-09-23T02:00:00",
7  "props": {
8    "fluentNetworkRef": "BASE_67"
9  },
10  "lastRun": {
11    "param": {
12      "start": "2022-09-22T03:00:00",
13      "end": "2022-09-23T02:00:00",
14      "props": {
15        "fluentNetworkRef": "BASE_67"
16      }
17    },
18    "jobStart": "2022-09-23T02:00:00",
19    "jobsEnd": "2022-09-23T02:05:00",
20    "status": "SUCCESSFUL"
21  }
22}

Language: json

Name: Example

Description:

[Warning: empty required content area]

After the job runs with the custom dates, the custom range is removed and

`previousEndDate`

remains as it previously was so the job resumes its normal operation.

1{
2  "previousEndDate": "2022-09-23T02:00:00",
3  "props": {
4    "fluentNetworkRef": "BASE_67"
5  },
6  "lastRun": {
7    "param": {
8      "start": "2022-04-22T00:00:00",
9      "end": "2022-04-23T00:00:00",
10      "props": {
11        "fluentNetworkRef": "BASE_67"
12      }
13    },
14    "jobStart": "2022-09-24T02:00:00",
15    "jobsEnd": "2022-09-24T02:05:00",
16    "status": "SUCCESSFUL"
17  }
18}

Language: json

Name: Example

Description:

[Warning: empty required content area]

While the job is running, a temporary property called

`executionStart`

is added to indicate the job is running, and it is removed when the job completes.

1{
2  "previousEndDate": "2022-09-23T02:00:00",
3  "executionStart" : "2022-09-24T02:00:00",
4  "props": {
5    "fluentNetworkRef": "BASE_67"
6  },
7  "lastRun": {
8    "param": {
9      "start": "2022-04-22T00:00:00",
10      "end": "2022-04-23T00:00:00",
11      "props": {
12        "fluentNetworkRef": "BASE_67"
13      }
14    },
15    "jobStart": "2022-09-24T02:00:00",
16    "jobsEnd": "2022-09-24T02:05:00",
17    "status": "SUCCESSFUL"
18  }
19}

Language: json

Name: Example

Description:

[Warning: empty required content area]

Property	Description
customDateRange	Optional date range a user can set to override the jobs current 'previousEndDate'. This is only used once and gets removed after a successful execution.
customDateRange.start	Custom user date range start datetime.
customDateRange.end	Custom user date range end datetime.
previousEndDate	Used to keep track of the last execution date. This drives how the commercetools-connector calculates the next date range execution. Whatever the value is set here, this is used as the next start date. If this is missing or invalid, epoch is as the start date. The end date is always the current time the job is executed.
props	Optional job settings.
lastRun	Keeps a copy of the last execution parameters, date range used as well as job execution times.
lastRun.param.start	Last execution date range start.
lastRun.param.end	Last execution date range end.
lastRun.param.props	Last execution props.
lastRun.jobStart	Last execution date time the job start running.
lastRun.jobsEnd	Last execution date time the job finished running.
lastRun.status	Last execution status of how the job finished: FAILED / SUCCESSFUL.

Build configuration
- Kept part of the deployable artifact (yaml file properties, for example)
- Every Connect commercetools-connector handler has access to optional configuration (props) that are part of the route setup. This is covered in more detail in the Handler Configuration of the customisation guide.

1routes:
2    - name: "fc.connect.batch-inventory-sync"
3      handler: "BatchInventorySyncJob"
4      props:
5        page-size: 500

Language: xml

Name: Example

Description:

[Warning: empty required content area]

General configuration

At a minimum, the commercetools-connector expects the following configuration.

1connector-name: commercetools
2configuration:
3    base-path-key: "fc.connect.<connector-name>"
4    account-mapping-key: "fc.connect.<connector-name>.account-mapping"
5    general-key: "fc.connect.<connector-name>.general"

Language: java

Name: Example

Description:

[Warning: empty required content area]

Property	Description
connector-name	This is the name of the connector and this property is used to both name the connector as well as build the path of other configuration keys. Keep characters limited to 'a-z' and '-'.
base-path-key	Used to fetch all configurations of a Fluent account-retailer based on path. Any custom configuration must adhere to the format otherwise it will not be found.
account-mapping-key	Maps a Fluent account-retailer to an external system. The identification of the external system must be unique and it is possible to be a composite value.
general-key	Any general settings that don't quite fit in any category.
log-level-key	Global level setting key to control the log level (DEBUG, INFO, WARN, ERROR) of the commercetools-connector. Changes to this property will take effect on the first configuration refresh in the commercetools-connector.

Account Mapping Example

1{
2    "fluent": {
3        "accountId": "FCTDEV",
4        "retailerId": "100"
5    },
6    "externalAccount": {
7        "projectKey": "my-project-qa",
8        "uid": "123"
9    }
10}

Language: json

Name: Example

Description:

[Warning: empty required content area]

General Settings Example

1{
2    "productCatalogue": {
3        "entityRef": "PC:MASTER:100",
4        "entitySubType": "MASTER"
5    }
6}

Language: json

Name: Example

Description:

[Warning: empty required content area]

commercetools Connector Extend Configuration

Overview

Detailed Technical Description

Credential / Secrets Configuration

Common Configuration

commercetools Connector Listener Configuration

Listener Configuration

Job Scheduler Configuration

General configuration

Building Blocks

By Type

Helpful Resources

Quick Links