Webhook Retries and Inline Events

How a Webhook Retry Is Built

• An event context starts with an event from outside the current event context, for example:
  • An event that an external system sends through the Event API
  • The `CREATE` event of an entity created with a GraphQL mutation
  • A scheduled event, for example from the `ScheduleEvent` rule
  • An event for another root entity or another Retailer
• This event is the initial event of the event context
• An event that a rule sends without a schedule for the same root entity is an inline event. It’s processed in the same event context, so one event context can run rulesets of several entities of that root entity, for example an Order and its Fulfillments
• A rule that sends a webhook, such as `SendWebhook`, produces a webhook action. The action is queued and executed after all rulesets of the event context have run
• When the request of the webhook action fails with a network error, the Orchestration Engine posts a retry event. As with a Manual Event with Run-Once Workflow, the retry event carries its own workflow with a single ruleset that contains only a webhook rule, so the other rules of the original ruleset aren’t run again. By default, this is the built-in platform webhook rule. With `preserve-event-details`, it’s the rule that produced the webhook action
• The `fc.rubix.webhook.retry.event` setting decides which event the retry event is built from:
  • `webhook-event`: the webhook event, which is the event that triggered the ruleset in which the webhook action was produced
  • `initial-event`: the initial event of the event context

What Each Value Changes

Workflow Patterns

Pattern 1: Webhook in the Ruleset That Starts the Event Context

• The ruleset that produces the webhook action is triggered by the initial event, such as an event from an external system, a `CREATE` event, a scheduled event, or an event for another root entity or Retailer

• The `NotifyOrderReceived` ruleset of an Order contains `SendWebhook` and is triggered by an event that the e-commerce platform sends through the Event API

• None. The webhook event is the initial event, so both values give the same retry

• No action needed

Pattern 2: Inline Event on the Same Entity

• A ruleset sends an inline event to another ruleset of the same entity, and that ruleset produces the webhook action
• A common case is a ruleset that changes the status of an entity and then sends an inline event to a notification ruleset

• The `ConfirmPayment` ruleset of an Order changes the Order status and sends an inline event to the `NotifyOrderConfirmed` ruleset, which contains `SendWebhook`

• `webhook-event`: the retry repeats the `NotifyOrderConfirmed` call. After the final retry, `NotifyOrderConfirmed Exception` is raised for the Order
• `initial-event`: the retry has the event name and attributes of `ConfirmPayment`, so a receiving system that reads the event name or the attributes that `ConfirmPayment` added to the inline event gets a different call. The entity is the same, so the receiving system is likely to accept the retry, and the failure goes unnoticed. After the final retry, `ConfirmPayment Exception` is raised for the Order

• Set `webhook-event`

Pattern 3: Inline Events Across Entities

• A ruleset of one entity sends inline events to related entities of the same root entity, and the webhook action is produced in a ruleset of a related entity

• The `PaymentCaptured` ruleset of an Order sends an inline event to the `NotifyWarehouse` ruleset of each Fulfillment, which calls the warehouse management system
• A Virtual Catalog ruleset updates its Virtual Positions with inline events, and the `NotifySalesChannel` ruleset of each Virtual Position sends the new quantity to a sales channel
  Note: A webhook for each Virtual Position is a high-volume pattern. See When To Use a Webhook

• `webhook-event`: the retry is about the same Fulfillment or Virtual Position, with the attributes of the webhook event, including the quantity
• `initial-event`: the retry is about the entity of the initial event, which is the Order instead of the Fulfillment, or the Virtual Catalog instead of the Virtual Position, without the position reference and quantity. When the receiving system responds to the retry, the Fulfillment or quantity update isn’t sent again. After the final retry, the Exception event is raised for the Order or the Virtual Catalog

• Set `webhook-event`. This pattern has the highest impact, especially for webhooks that send inventory or fulfillment updates to external systems

Pattern 4: Custom or Reference Webhook Rule

• The webhook is sent by a custom rule that uses the `postWebhook` action, in any of the other patterns
• The webhook is sent by the reference `SendWebhook` rule, which posts the event under the name from its webhook setting

• The `NotifyErp` ruleset of an Order contains a custom `SendOrderToErp` rule that builds its own payload and posts it with the `postWebhook` action

• The `fc.rubix.sdk.webhook.retry.method` workflow setting decides which rule sends the retry, and `fc.rubix.webhook.retry.event` decides which event the retry is built from
• Without `preserve-event-details`, the retry is sent by the built-in platform webhook rule, without the logic of the custom rule. For the reference `SendWebhook` rule, the retry is posted under the name of the retry event instead of the name from the webhook setting
• With `preserve-event-details` and `initial-event`, the rule runs on the initial event, so it builds its payload from that event

• Check that the rule can run more than once. Then add `preserve-event-details` to the workflow and set `webhook-event` for the account

Pattern 5: Webhook Isolated With a Scheduled Event

• A ruleset sends a scheduled event, for example with the `ScheduleEvent` rule and a short delay, to the ruleset that produces the webhook action, so that ruleset starts its own event context

• The `ConfirmPayment` ruleset of an Order schedules the `NotifyOrderConfirmed` event instead of sending it as an inline event

• None. The scheduled event is the initial event of the new event context, as in Pattern 1

• No action needed. With `webhook-event`, the scheduled event isn’t required for correct retries

Choosing a Value

• Your workflows handle failed webhooks with `<InitialRulesetName> Exception` rulesets, and you can’t add `<WebhookRulesetName> Exception` rulesets yet
• The receiving system or your monitoring relies on retries that carry the initial event

Before You Switch

• Find the webhook actions whose webhook event is an inline event. For each ruleset that produces a webhook action, check whether another ruleset sends an event with the name of that ruleset, for the same entity type and root entity, without a schedule. For example, `SendEvent`, or a rule that sends events to all Fulfillments of an Order. A ruleset that can also be triggered by an initial event still counts
• Check the Exception rulesets. If a workflow contains `<InitialRulesetName> Exception` rulesets that handle these webhooks, add a `<WebhookRulesetName> Exception` ruleset for the entity type of the webhook ruleset. Add triggers for every status that the entity can be in when the final retry fails. With exponential back-off, this is more than a day after the original call
• Check your monitoring. After the switch, Send Webhook audit events and Exception events are on the entity of the webhook event, not on the entity of the initial event. Update queries that look for them by entity. See Orchestration & Audit Events
• Check your own resend logic. If your workflows or integrations resend failed webhooks themselves, make sure that the receiving system can handle a call that the platform retry has already delivered
• For custom webhook rules and the reference `SendWebhook` rule, add `preserve-event-details` to the workflow. See `fc.rubix.sdk.webhook.retry.method`
• Switch a non-production account first and test your webhooks, as described in Verify the Retry

Switch and Verify

Check the Current Value

1query getWebhookRetryEventSetting($name: [String!], $context: [String!]) {
2  settings(name: $name, context: $context, first: 1) {
3    edges {
4      node {
5        id
6        name
7        value
8        context
9        contextId
10      }
11    }
12  }
13}

1{
2  "name": ["fc.rubix.webhook.retry.event"],
3  "context": ["ACCOUNT"]
4}

1{
2  "data": {
3    "settings": {
4      "edges": [
5        {
6          "node": {
7            "id": "5001",
8            "name": "fc.rubix.webhook.retry.event",
9            "value": "initial-event",
10            "context": "ACCOUNT",
11            "contextId": 0
12          }
13        }
14      ]
15    }
16  }
17}

Update the Setting

• Set the value to `webhook-event` with the `updateSetting` mutation. See the update example in `fc.rubix.webhook.retry.event`
• If the query returns no setting, the account already uses `webhook-event`, and no update is needed
• Allow up to 5 minutes for the change to take effect. It applies to event contexts that start after that

Verify the Retry

• On a non-production account, choose a webhook action whose webhook event is an inline event, as in Pattern 2 or 3. Point the webhook to an endpoint that doesn’t respond within the read timeout of 10 seconds, and trigger the workflow
• In the Event API, check that the retry event has the name and entity of the webhook event
• After the final retry, check that the `<WebhookRulesetName> Exception` event is raised for the entity of the webhook event
• If the workflow sets `fc.rubix.sdk.webhook.retry.schedule` to `exponential`, the final retry and the Exception event come more than a day after the original call. Test on a workflow without this setting, or plan for the delay

Roll Back

• To return to the previous behavior, set the value to `initial-event`. Allow up to 5 minutes for the change to take effect. Event contexts that start after that are retried with the initial event again