Implement Custom Data Pipelines

This article provides a guide on how to set up your own custom data pipelines to sync Ordergroove data using Ordergroove REST APIs. It covers fetching data, handling pagination, and scheduling ETL jobs.

Architecture Template

Ordergroove recommends the following steps for architecting the ingestion:

1. Schedule a job that triggers an Ordergroove data sync.
2. Configure a separate thread for each endpoint (subscriptions, items, orders, etc.) and call the API with updated_start set to the start time of the last sync as a cutoff timestamp.
3. Page through all results using cursor-based pagination until all new data is retrieved.
4. Load the data into the warehouse, upserting into destination tables.
5. Store the start time of this job in your system to use as an update timestamp filter for the next run.

API Access and Prerequisites

Before you can sync data, make sure you have the proper API access:

1. API Key: Ordergroove uses API key authentication. See Authentication for more information.
2. Permissions: Your API key must have the “Bulk” scope/permissions to read data for syncing. Test your key with a simple call (e.g., list customers) to verify access. If you receive 401/403 errors, check that the key is correct and has read permissions.
3. Testing Environment: If you have a staging Ordergroove environment, test your pipelines against it first.
4. Data Model Understanding: Familiarize yourself with the JSON structure of each endpoint by reviewing the Ordergroove API reference or making sample calls. Understanding these fields helps in designing your warehouse schema (e.g. you might have separate tables for orders and order_items, linked by an order ID).
5. Logging and Monitoring: Implement logging for each sync run, including the number of records fetched and timestamps. This helps in troubleshooting if something goes wrong.

Reference — API Endpoints and Capabilities

Below is a reference table of the main Ordergroove API endpoints for data sync, indicating which support incremental fetch using the updated_start filter and which support cursor-based pagination (API v2):

Notes:

• For Products and Payments: since Ordergroove keeps a copy of what’s already in your ecommerce system, you should not need to sync this data.
• All “Yes” entries for cursor-based pagination assume you include the X-OG-API-VERSION: 2 header on your requests. Without that, the API may default to an older page-index method (not recommended).

Use Incremental Fetch Logic with updated_start

Incremental sync means fetching only records that have changed (created or updated) since your last sync run, rather than pulling all data every time. Using incremental fetch allows you to run the job faster and costs less in terms of time and resources. Ordergroove’s list endpoints support filtering by an update timestamp — except for the Payments and Products list endpoints. The key query parameter is updated_start, which accepts a datetime (CST timezone) and returns all objects updated on or after that timestamp.

How to use updated_start: For each API resource, specify ?updated_start=<datetime> in the request. For example, to fetch subscriptions updated since January 1, 2025:

GET /subscriptions/?updated_start=2025-01-01T12:00:00

This returns all subscriptions whose updated date is 2025-01-01 (CST) or later. On initial creation, an object’s updated timestamp is set to its created date, so newly created records are captured by the filter as well. Typically, you would store the start time of the last sync in your system and use that as the updated_start on the next run.

Exceptions — Products, Payments, Offer Profiles, and One-Time Discounts

These endpoints do not support incremental fetch, so you will need to do a full refresh.

This is due to a combination of practical usage patterns and data volume considerations:

• Products and Payments are typically sourced directly from the merchant’s commerce platform (e.g., Shopify, Salesforce, etc.) and are often available in the merchant’s primary system of record. Most merchants do not rely on Ordergroove as the authoritative source for this data.
• Offer Profiles and One-Time Discounts are relatively lightweight datasets, making it feasible to retrieve the full set on each sync.

Support for incremental sync on these endpoints may be considered in the future, but for now full refresh is the recommended approach.

Initial Sync and Full Refresh

To set up an initial sync (or any full refresh), omit the updated_start parameter to fetch all records from an endpoint. This retrieves the entire history/dataset for that resource. In practice, the API call is the same as incremental — e.g. GET /orders/ without any date filters returns all orders.

If you experience longer response times (e.g. >20s), you might perform the initial load in segments using updated_start and updated_end with yearly or monthly ranges to reduce response time. Otherwise, using pagination (discussed below) is sufficient. After the initial load, switch to using updated_start with the last updated timestamp from your warehouse to only get new or changed records going forward.

Pagination (Cursor-Based)

For data syncing, you should only use cursor-based pagination with REST APIs. This ensures stable paging even as data changes. By default, a list request returns 10 results, but you can control the page size up to 100 records using the page_size parameter. When there are more records to fetch, the response includes a next URL (and a previous URL if applicable) containing a cursor parameter. To get the next page, make a request to the next URL provided. This cursor is an opaque pointer the server uses to maintain your position in the dataset.

You can read more on cursor-based pagination in the API reference.

Do not use page-index based pagination (e.g. ?page=2) — it is deprecated and can lead to inconsistencies. Always use the cursor method with API v2. Cursor pagination ensures you won’t miss or duplicate records if new updates arrive during paging.

ETL Scheduling and Automation

We recommend scheduling an incremental daily data sync between 12PM CST and 8PM CST, when most order placement activity has completed. This helps ensure you capture a full day of data. If fresher data is needed, you can schedule it twice a day.

Many teams use orchestrators like Airflow or simple cron jobs to run incremental syncs.

Handling Deleted Data (API Limitations)

In certain cases, Ordergroove will delete future, unprocessed Order and Item records from the system. This happens when a customer takes an action that changes the lifecycle of a subscription or an order, e.g.:

• A customer cancels their subscription, and any upcoming orders that haven’t yet been processed are removed.
• A subscription’s next order date is skipped or delayed, causing previously scheduled (but not yet fulfilled) items or orders to be deleted.

While these deletions are expected and part of normal subscription management behavior, a key limitation of the Ordergroove REST API is that deleted Order and Item records will no longer appear in list API responses, and there’s no tombstone flag to indicate removal. This means if you’re syncing data via API polling, these deletions won’t be reflected in your warehouse unless you capture them separately through webhooks. Support for syncing deleted records is on the roadmap and expected by the end of 2025.

Solution — Webhooks for Order and Item deletions: We recommend subscribing to the order.delete and item.remove webhook events in addition to your API sync to account for these deletions.

• order.delete: Notifies you when an order is deleted. Upon receiving this event, mark that order as deleted in your system.
• item.remove: Notifies you when an item is deleted. Use this to delete or flag the item record in your database.

By capturing webhook payloads, you can perform a soft delete or flagging of records in your warehouse that are no longer active. This approach complements the incremental API sync: the API keeps your data updated for new and modified records, while webhooks inform you of removals that the next API poll might not catch.

Best Practices

If you’re building a custom data pipeline with Ordergroove’s APIs, keep these tips in mind to ensure reliable, efficient syncing:

• Use cursor-based pagination
• Do incremental fetch with timestamps
• Implement upsert logic
• Test with a smaller time window
• Implement logging and monitoring
• Do not make concurrent requests for the same object type
• Do not assume deleted data is visible via the API
• Be aware of rate limits