> ## Documentation Index > Fetch the complete documentation index at: https://docs.edges.run/llms.txt > Use this file to discover all available pages before exploring further. # Incremental Sync > Fetch only new data since the last retrieval with sync_mode incremental on async and scheduled runs. Both **Async** and **Schedule** modes support an optional **incremental sync** mechanism. Instead of retrieving the full dataset on every run, you can set `sync_mode: "incremental"` inside the `parameters` object to fetch **only new data** since the last retrieval. Incremental sync is only available on [engagement-free actions](/v1/identities/engagement#included-actions) and requires an **Engagement Identity** (`type: "engagement"`). Standard identities cannot use this feature. ### How It Works 1. **First run (initial full sync):** When `sync_mode` is set to `incremental` and no previous data has been retrieved, the system performs a full sync limited by `max_results` (either the value you set or the action's default). 2. **Subsequent runs (incremental):** Once a first retrieval has been completed, the next runs automatically switch to incremental mode and return only newly available data. With each continue or scheduled iteration, the system fetches up to `max_results` new items. If more new items were created than the `max_results` limit between two iterations, items beyond that limit will be **lost** and cannot be recovered. This happens because data is always fetched **from the most recent to the oldest**. To minimize this risk, choose a `max_results` value and schedule frequency that match your expected data volume. ### Using Incremental Sync with Async Mode In async mode, you trigger incremental updates manually by **continuing** a completed run: ```bash theme={null} curl -X POST "https://api.edges.run/v1/runs/{run_uid}/continue" \ -H "Accept: application/json" \ -H "X-API-Key: " \ -H "Content-Type: application/json" ``` Each call to [`POST /v1/runs/{run_uid}/continue`](/v1/api/runs/continue) resets the run and fetches up to `max_results` new items. The launch response returns both `run_id` and `run_uid`. Keep the **`run_uid`** — that is what every subsequent call takes. ### Example: Async Incremental Sync ```bash theme={null} # 1. Launch an async run with incremental sync curl -X POST "https://api.edges.run/v1/actions/linkedin-extract-connections/run/async" \ -H "Accept: application/json" \ -H "X-API-Key: " \ -H "Content-Type: application/json" \ -d '{ "inputs": [{}], "callback": { "url": "https://yourdomain.com/callback" }, "parameters": { "sync_mode": "incremental" } }' # 2. Once the run completes, continue it later to get new data curl -X POST "https://api.edges.run/v1/runs/{run_uid}/continue" \ -H "Accept: application/json" \ -H "X-API-Key: " \ -H "Content-Type: application/json" ``` ### Using Incremental Sync with Schedule Mode In schedule mode, incremental sync is automatic: each scheduled iteration fetches only the new data since the last execution. No manual continue call is needed. In scheduled mode, if the previous run is still in progress when the next iteration is triggered, that iteration will be **skipped**. The following iteration will trigger the update instead. ### Example: Scheduled Incremental Sync ```bash theme={null} # Schedule a recurring incremental sync every day at 9am curl -X POST "https://api.edges.run/v1/actions/linkedin-extract-messages/run/schedule" \ -H "Accept: application/json" \ -H "X-API-Key: " \ -H "Content-Type: application/json" \ -d '{ "inputs": [{ "linkedin_thread_url": "https://www.linkedin.com/messaging/thread//" }], "callback": { "url": "https://yourdomain.com/callback" }, "cron": "0 9 * * *", "timezone": "Europe/Paris", "parameters": { "sync_mode": "incremental" } }' ``` ### Supported Actions The following actions currently support incremental sync: | Action | Description | | -------------------------------------------------------------------------------------- | ------------------------------- | | [`linkedin-extract-connections`](/v1/api/actions/linkedin-extract-connections) | Extract LinkedIn connections | | [`linkedin-extract-messages`](/v1/api/actions/linkedin-extract-messages) | Extract LinkedIn messages | | [`linkedin-extract-profile-viewers`](/v1/api/actions/linkedin-extract-profile-viewers) | Extract profile viewers | | [`linkedin-extract-page-followers`](/v1/api/actions/linkedin-extract-page-followers) | Extract LinkedIn page followers | ### Continuable Run Statuses A run can only be continued if its status is one of: * `BLOCKED` * `STOPPED` * `FAILED` * `PARTIAL_SUCCEEDED` * `SUCCEEDED` When continued, the run status and all related inputs are reset to `SCHEDULED`. ### Limitations * **Engagement Identities only.** A standard identity cannot run incremental sync. * **No backfill.** Data is fetched newest-first, so anything that overflowed `max_results` between two iterations is not picked up later — see the warning above. * **Schedules cannot be edited.** Changing `sync_mode` on an existing schedule means cancelling and recreating it, which is what Part 2 below covers. * **Overlapping executions are skipped.** If the previous run is still going when the next is due, that iteration does not run. ### Resume vs continue after a failed run A run can be continued when its status is `FAILED` (among others listed above). That fetches the **next delta** — it does not retry what failed. **Resume** is the other option, but it accepts `BLOCKED` runs only; anything else returns `400`. See [Resume vs. continue](/v1/runs/recover-blocked-run#resume-vs-continue). ## Migrating existing schedules This guide shows how to use the **scheduled run endpoint** for [Extract LinkedIn Connections](/v1/api/actions/linkedin-extract-connections-schedule) with **incremental sync** via Engagement Identities, and how to migrate existing schedules to incremental mode using the `/actions/linkedin-extract-connections/run/schedule` route. Incremental mode fetches **only new data** since the last retrieval instead of the full dataset on every run. It is available only on some [engagement-free actions](/v1/identities/engagement#included-actions) and requires an Engagement Identity. *** ### Prerequisites * [Engagement Identities](/v1/identities/engagement) enabled in your workspace and at least one Engagement Identity created with a connected LinkedIn account * API key with access to runs, schedules, and identities *** ### Shared setup (reuse in your code) The examples below use a single base URL and a small request helper so headers and error handling stay consistent. Define these once and reuse them across Part 1 and Part 2. ```javascript theme={null} const EDGES_API_BASE = 'https://api.edges.run/v1'; async function edgesRequest(apiKey, url, { method = 'GET', body } = {}) { const res = await fetch(url, { method, headers: { 'X-API-Key': apiKey, 'Content-Type': 'application/json' }, ...(body != null && { body: JSON.stringify(body) }), }); if (!res.ok) { const err = await res.json().catch(() => ({})); throw new Error(err.message ?? `Request failed: ${res.status}`); } return res.json(); } ``` *** ### Part 1: Implementing Incremental Mode #### 1.1 Create a new scheduled run in incremental mode To schedule **Extract LinkedIn Connections** in incremental mode, send `sync_mode: "incremental"` in the `parameters` object and use an Engagement Identity in `identity_ids`. **Behavior:** * **First run:** Full sync (capped by `max_results`). * **Next runs:** Only new connections since the last run. No need to call continue; each scheduled execution is automatically incremental. In schedule mode, if the previous run is still in progress when the next iteration is due, that iteration is **skipped**. The following one will run the incremental update. **JavaScript example: create a scheduled incremental Extract Connections run** ```javascript theme={null} // Uses EDGES_API_BASE and edgesRequest from the shared setup above. async function createIncrementalExtractConnectionsSchedule(apiKey, options) { const { identityId, callbackUrl, cron = '0 9 * * *', // daily at 9am timezone = 'Europe/Paris', maxResults, } = options; return edgesRequest(apiKey, `${EDGES_API_BASE}/actions/linkedin-extract-connections/run/schedule`, { method: 'POST', body: { inputs: [{}], callback: { url: callbackUrl }, identity_ids: [identityId], cron, timezone, parameters: { sync_mode: 'incremental', ...(maxResults != null && { max_results: maxResults }), }, }, }); } // Usage: ensure identityId is an Engagement Identity (type === 'engagement') // const schedule = await createIncrementalExtractConnectionsSchedule(process.env.EDGES_API_KEY, { // identityId: 'id_xxx', // callbackUrl: 'https://yourdomain.com/webhooks/edges', // }); ``` #### 1.2 Async mode: continue an incremental run If you use **async** instead of schedule, you trigger each incremental update by calling the continue endpoint after the previous run has finished. **JavaScript example: run async and continue later (Extract Connections)** ```javascript theme={null} // Uses EDGES_API_BASE and edgesRequest from the shared setup above. async function runExtractConnectionsIncrementalAsync(apiKey, options) { return edgesRequest(apiKey, `${EDGES_API_BASE}/actions/linkedin-extract-connections/run/async`, { method: 'POST', body: { inputs: [{}], callback: { url: options.callbackUrl }, identity_ids: [options.identityId], parameters: { sync_mode: 'incremental', ...(options.maxResults != null && { max_results: options.maxResults }), }, }, }); } async function continueIncrementalRun(apiKey, runUid) { return edgesRequest(apiKey, `${EDGES_API_BASE}/runs/${runUid}/continue`, { method: 'POST' }); } ``` *** ### Part 2: Migrating full-mode schedules to incremental Schedules cannot be edited in place — there is no update endpoint, only `pause`, `resume` and `cancel`. Switching an existing schedule to incremental means cancelling it and creating a replacement. **A schedule does not return everything needed to recreate it.** `GET /schedules/{scheduled_run_uid}` returns `cron`, `timezone`, `parameters`, `status` and the scheduling timestamps — but **not** the `inputs`, the `callback` or the `identity_ids` the schedule runs with. Cancel first and that configuration is gone. Recover it **before** cancelling anything: inputs and callback come from a past run of the schedule, and the identity has to come from your own records. #### 2.1 What you can recover, and from where | Needed to recreate | Where it comes from | | -------------------------------- | ------------------------------------------------------------------------------------------- | | `cron`, `timezone`, `parameters` | `GET /schedules/{scheduled_run_uid}` | | `callback` | A past run of the schedule — `GET /runs` returns `callback` and `scheduled_run_uid` per run | | `inputs` | [`GET /runs/{run_uid}/inputs`](/v1/api/runs/inputs) — the `input_data` of each input | | `identity_ids` | **Not exposed by the API.** You must supply it | Because the identity is not recoverable, a migration cannot be fully automatic. The script below collects everything it can, tells you what is missing, and only then migrates — with the identity mapping you provide. #### 2.2 List the schedules to migrate ```javascript theme={null} // Uses EDGES_API_BASE and edgesRequest from the shared setup above. async function listSchedules(apiKey, actionName) { const schedules = []; let offset = 0; const limit = 100; for (;;) { const params = new URLSearchParams({ action_name: actionName, status: 'ACTIVE', limit: String(limit), offset: String(offset), }); const page = await edgesRequest(apiKey, `${EDGES_API_BASE}/schedules?${params}`); const items = Array.isArray(page) ? page : (page.data ?? []); schedules.push(...items); if (items.length < limit) return schedules; offset += limit; } } ``` Each schedule is identified by its `uid`. That is the value the endpoints below call `scheduled_run_uid`. A schedule whose `parameters.sync_mode` is already `incremental` needs no migration — filter those out before you go any further. #### 2.3 Recover the inputs and callback from a past run Runs carry the `scheduled_run_uid` that produced them, so the most recent run of a schedule is where its inputs and callback survive. ```javascript theme={null} // Uses EDGES_API_BASE and edgesRequest from the shared setup above. /** Most recent run produced by a given schedule, or null if it never ran. */ async function findRunForSchedule(apiKey, actionName, scheduleUid) { let offset = 0; const limit = 100; for (;;) { const params = new URLSearchParams({ action_name: actionName, limit: String(limit), offset: String(offset), }); const page = await edgesRequest(apiKey, `${EDGES_API_BASE}/runs?${params}`); const items = Array.isArray(page) ? page : (page.data ?? []); const match = items.find((r) => r.scheduled_run_uid === scheduleUid); if (match) return match; if (items.length < limit) return null; offset += limit; } } async function getRunInputs(apiKey, runUid) { const inputs = []; let offset = 0; const limit = 100; for (;;) { const page = await edgesRequest( apiKey, `${EDGES_API_BASE}/runs/${runUid}/inputs?limit=${limit}&offset=${offset}` ); const items = Array.isArray(page) ? page : (page.data ?? []); inputs.push(...items.map((i) => i.input_data)); if (items.length < limit) return inputs; offset += limit; } } ``` A schedule that has never executed has no run to recover from. Those have to be rebuilt from your own records, or recreated by hand. #### 2.4 Supply the identities `identity_ids` is not returned anywhere, so map each schedule to the Engagement Identity it should run on. List your identities to get their UIDs: ```javascript theme={null} // Uses EDGES_API_BASE and edgesRequest from the shared setup above. async function getEngagementIdentities(apiKey) { const found = []; let page = 1; for (;;) { const res = await edgesRequest(apiKey, `${EDGES_API_BASE}/identities?page=${page}`); const items = Array.isArray(res) ? res : (res.data ?? []); if (items.length === 0) return found; found.push(...items.filter((i) => i.type === 'engagement')); page += 1; } } ``` Then build the mapping yourself — from your own database, or by hand for a small number of schedules: ```javascript theme={null} // scheduleUid -> the Engagement Identity UID the new schedule should use const IDENTITY_BY_SCHEDULE = { 'schedule-uid-1': 'engagement-identity-uid-a', 'schedule-uid-2': 'engagement-identity-uid-b', }; ``` #### 2.5 Dry run, then migrate Collect everything for every schedule and report on it **before** cancelling a single one. A schedule missing its inputs, callback or identity is skipped rather than recreated half-configured. ```javascript theme={null} // Uses EDGES_API_BASE, edgesRequest and the helpers above. const ACTION = 'linkedin-extract-connections'; async function planMigration(apiKey, identityBySchedule) { const schedules = await listSchedules(apiKey, ACTION); const plan = []; for (const schedule of schedules) { if (schedule.parameters?.sync_mode === 'incremental') continue; const identityId = identityBySchedule[schedule.uid]; const run = await findRunForSchedule(apiKey, ACTION, schedule.uid); const inputs = run ? await getRunInputs(apiKey, run.uid) : null; const missing = [ !identityId && 'identity_ids', !run && 'no past run — inputs and callback unavailable', run && !run.callback && 'callback', inputs && inputs.length === 0 && 'inputs', ].filter(Boolean); plan.push({ scheduleUid: schedule.uid, ready: missing.length === 0, missing, body: { inputs, callback: run?.callback, identity_ids: identityId ? [identityId] : undefined, cron: schedule.cron, timezone: schedule.timezone, parameters: { ...schedule.parameters, sync_mode: 'incremental' }, }, }); } return plan; } ``` Review the plan. Only when every schedule you care about reads `ready: true` should you run the migration: ```javascript theme={null} async function migrate(apiKey, plan) { const results = []; for (const item of plan) { if (!item.ready) { results.push({ scheduleUid: item.scheduleUid, skipped: item.missing }); continue; } try { await edgesRequest(apiKey, `${EDGES_API_BASE}/schedules/${item.scheduleUid}/cancel`, { method: 'POST', }); const created = await edgesRequest( apiKey, `${EDGES_API_BASE}/actions/${ACTION}/run/schedule`, { method: 'POST', body: item.body } ); results.push({ scheduleUid: item.scheduleUid, newScheduleUid: created.uid }); } catch (e) { // The old schedule may already be cancelled — item.body has everything // needed to recreate it, so keep it. results.push({ scheduleUid: item.scheduleUid, error: e instanceof Error ? e.message : String(e), recreateWith: item.body, }); } } return results; } const plan = await planMigration(process.env.EDGES_API_KEY, IDENTITY_BY_SCHEDULE); console.table(plan.map(({ scheduleUid, ready, missing }) => ({ scheduleUid, ready, missing: missing.join(', ') }))); // Inspect the table, then: // const results = await migrate(process.env.EDGES_API_KEY, plan); ``` Set a low `max_results` on the first incremental schedule. The first execution is a full sync, so a large value there fetches history you probably don't want, at full cost. The migration runs **sequentially** on purpose. Cancelling and recreating schedules in parallel makes the identity-management rate limit the thing that fails, mid-migration, with some schedules cancelled and not yet recreated. ### Summary | Goal | Approach | | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **New incremental schedule** | `POST /v1/actions/linkedin-extract-connections/run/schedule` with `parameters.sync_mode: "incremental"` and an Engagement Identity in `identity_ids`. | | **Async incremental** | Same `parameters` on async run; then call `POST /v1/runs/{run_uid}/continue` for each subsequent fetch. | | **Existing full-mode schedules → incremental** | List schedules (`GET /v1/schedules?action_name=linkedin-extract-connections`), filter by Engagement Identity if needed, then for each: GET schedule → cancel → recreate with `sync_mode: "incremental"`. | ## Related Endpoint reference for fetching the next delta. What incremental sync requires, and how they are billed. Creating, pausing and cancelling scheduled runs.