How do Inventory Feeds work

Pre-Requisites

This document assumes you are knowledgeable and aware of the following subjects:

• GraphQL API
• Fluent Big Inventory
• Global Inventory
  • Specifically Virtual Catalogues and Inventory Catalogues
• Amazon Web Services and AWS Simple Storage Solution

Inventory Feed Components

Inventory Feeds is an inventory availability data export tool. They allow users to create a programmatic export of selected inventory data, ready to be consumed by any inventory-aware system.

Inventory Feeds contain two technical components:

• Inventory Feed
• Inventory Feed Run

Inventory Feed

The Inventory Feed component is the “parent” data modal of the Inventory Feed. This is where an Inventory Feed is configured and set up. 

At the Inventory Feed level, you can define information such as:

• Export Schedule
• Target destination
• Data filters

Inventory Feed Run

Under an Inventory Feed, there is a one-to-many relationship to Inventory Feed Runs. Each Run is an individual execution, exporting a new data file based on the configuration of the parent Inventory Feed.

Each Inventory Feed Run contains information such as:

• Status of the run (success / failed)
• Total amount of exported inventory items
• Type of run (incremental or full)

Inventory Feed Runs are created by the Fluent Big Inventory system based on the settings configured at the Inventory Feed parent level. However, there is a capability to manually create and trigger Inventory Feed Runs under existing Inventory Feeds.

Inventory Feed Data Flow

Inventory Feeds are designed to export tailored views of inventory availability into your own ecosystem. Once the data has been exported to your ecosystem it is in an easy-to-use format ready to be shared with any other systems that benefit from inventory availability.

High-Level Diagram

Data Flow

Inventory Feeds are designed to provide data in a format that can be “owned” by the user. Fluent Big Inventory is designed for configurability and extensibility, and this solution enables our Customers and Partners to best integrate the data from Fluent into a target storage location where it can then be integrated into any target system.

How it works

The Inventory Feeds service will trigger a run based on the parent configuration, or 5 minutes by default. When a run processes, it will generate an output of the inventory availability from Global Inventory. This generated view of inventory will be saved into an S3 bucket within the Fluent technology stack.

When configuring an Inventory Feed it is required to configure a Customer or Partner owned AWS S3 bucket as a target destination. Once the target destination bucket is defined, the Inventory Feed service will set up an AWS cross-account replication service. This service will inspect the S3 bucket within the Fluent tech stack, and whenever it identifies any changes or new files added to the Fluent S3 bucket, the cross-account replication service will immediately start to replicate the changes across to the target destination bucket.

At this point, the target S3 bucket will have an exact copy of the inventory availability data generated by the Inventory Feed. This data can then be used to provide a view of inventory availability. Additionally, this also provides a historic cache of inventory availability snapshots all the way back to the first initial run of the Inventory Feed.

Types of Inventory Feed Runs

Inventory Runs have two types. Incremental and Full. Every Inventory Feed has a standard way of deciding what type of run to produce next. However, there is also the ability to force a run via API of a different type as needed.

Full Feed Run

An Inventory Feed Run of type Full will export everything within the Inventory Feeds configured filter. This file will be the starting point for any integration/use of inventory availability data from the Inventory Feed. A full run will always be triggered the first time an Inventory Feed needs to run, as the Inventory Feed service will always need to start with a known foundation of data to understand where an incremental run can export from.

Incremental Feed Run

An Inventory Feed Run of type Incremental will export only what has changed since the last time an Inventory Feed Run has occurred. The amount of data exported here will be determined by how many changes have happened within the configured filter and how often that Inventory Feed is configured to run. For example, if an Inventory Feed is configured to run once every minute, the amount of incremental changes will be small compared to an Inventory Feed that is configured to run once every hour.

File Format

The inventory availability data exported as part of an Inventory Feed will be in a Parquet file format. We have chosen to export in a Parquet file as it provides a range of benefits. 

Some of these benefits are:

• Open-source - All customers will be able to use and integrate these file types into their existing architectures
• Schema evolution - Parquet files are designed to evolve. Being column-oriented storage (instead of row-based) and self-describing, Parquet files are designed to evolve, which we expect to happen as we increase the feature set of Inventory Feeds
• Performance - Parquet files are designed for querying directly and as a result perform faster than other types of data file such as CSV

Parquet File Structure

Inventory Feed Data Model