How to merge datasets and remove duplicate rows from many Apify runs
Short answer
Pick the datasets, choose the fields that make two rows the same (a URL, a place ID, a name plus address), and normalise them so www, trailing slashes, case and phone formats don't hide duplicates. Rowmeld on Apify does this at $0.0002 per unique row; duplicates are free, and scheduled runs return only rows you have never received.
- Pick the datasets to merge in the form (earlier runs' results), or pass their IDs through the API.
- Set dedupeFields: url, placeId, website, or name + address. Leave it empty to remove only identical rows.
- Keep smartMatching on so https://www.acme.com/ and acme.com, or two phone formats, count as one.
- Turn on fillBlanks to copy a phone or email from a duplicate into the row you keep; use excludeDatasetIds to drop rows already in your CRM export.
- Chain it after a scraper with monitor on and a monitorKey: each run passes on only rows it has never sent.
Copy-paste code
Python (pip install apify-client): two runs in, one unique list out
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
runs = [r for r in client.runs().list(limit=2, desc=True).items] # your last two runs
run = client.actor("cybermax/merge-dedupe-datasets").call(run_input={
"datasetIds": [r["defaultDatasetId"] for r in runs],
"dedupeFields": ["url"],
"fillBlanks": True,
})
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(len(rows), "unique rows")Chain after any scraper (Apify Integrations -> Run Actor): only new rows
{
"datasetIds": ["{{resource.defaultDatasetId}}"],
"dedupeFields": ["url"],
"monitor": true,
"monitorKey": "my-leads"
}No code: n8n, Make or Zapier
n8n: HTTP Request node (or the Apify node, action "Run an Actor and get dataset")
Method: POST
URL: https://api.apify.com/v2/acts/cybermax~merge-dedupe-datasets/run-sync-get-dataset-items
Auth: Header Auth -> Authorization: Bearer <your Apify token>
Body: JSON {"datasetIds": ["ID1", "ID2"], "dedupeFields": ["website"]}
Output: one n8n item per unique row, ready for Google Sheets, Airtable or HubSpot
Make: Apify app -> "Run an Actor" (Actor: cybermax/merge-dedupe-datasets, wait for finish = yes)
-> Apify "Get Dataset Items" (dataset = the run's defaultDatasetId) -> your next module
Zapier: Apify app -> action "Run Actor" (cybermax/merge-dedupe-datasets, input JSON as above)
-> trigger "Finished Actor Run" -> "Fetch Dataset Items" -> your app
Runs longer than 5 minutes: start with POST /v2/acts/cybermax~merge-dedupe-datasets/runs and read the dataset when the run finishes.From an AI agent (MCP)
# Any MCP client (Claude Desktop, Claude Code, Cursor, VS Code): Apify's hosted MCP server
{
"mcpServers": {
"merge-dedupe-datasets": {
"command": "npx",
"args": ["-y", "mcp-remote", "https://mcp.apify.com/?tools=cybermax/merge-dedupe-datasets",
"--header", "Authorization: Bearer ${APIFY_TOKEN}"],
"env": { "APIFY_TOKEN": "<your Apify token>" }
}
}
}
# The agent sees one tool, cybermax--merge-dedupe-datasets, with the Actor's input schema.| name | url | city | phone |
|---|---|---|---|
| Blue Bottle Coffee | https://bluebottlecoffee.com/ | Oakland | +1 510-653-3394 (filled from a duplicate) |
| Stumptown Coffee | https://www.stumptowncoffee.com | Portland | +1 503-230-7702 |
| Intelligentsia | https://intelligentsia.com | Chicago | +1 312-253-0594 (filled from a duplicate) |
| Counter Culture | https://counterculturecoffee.com/ | Durham |
Source: Rowmeld (Apify Actor cybermax/merge-dedupe-datasets) README, real Apify run on 10 Nov 2026
Why exact matching misses duplicates
The same company shows up as https://www.acme.com/ in one run and acme.com in another, or with two phone formats. Smart matching ignores case and spaces and normalises URLs and phone numbers before comparing, so these count as one row.
Dataset access with limited permissions
Apify Actors now run with limited permissions by default and can read your datasets only when they are passed through the dataset picker. Rowmeld uses that picker, so a merge of your own runs works without extra setup.
Only rows you don't have yet
Put your CRM export in excludeDatasetIds to drop leads you already have, or turn on monitor mode so scheduled runs write only rows never sent before; you pay only for those.
Good to know
Price on Apify: $0.0002 per unique row written ($0.20 per 1,000); duplicates, excluded and already-sent rows are free, with no start fee. Apify's free plan includes $5 of usage each month.
Rowmeld only reads the datasets you pick and writes a new dataset; your originals are not changed.
Cartlark: Every product, price and stock level
$0.0008 per product, no start feeMerging product runs from many stores? Cartlark scrapes every product, price and stock level of any Shopify or WooCommerce store at $0.0008 per product.
Runs on your own Apify account, pay per result; failed results are not charged.Questions
How big can the datasets be?
Millions of rows: rows stream through and only a short fingerprint of each key is kept. Fill blanks keeps whole rows in memory, about 75,000 unique rows at 512 MB.
Which copy is kept?
The first (earliest dataset and row) by default, or the last with keep set to last, for example the newest price per product.
Can I merge on nested fields?
Yes. Use dots, for example location.city.
Can an AI agent use it?
Yes. Apify's MCP server exposes it as cybermax--merge-dedupe-datasets for Claude, Cursor and other MCP clients.
Free: the Free API Starter Kit
Copy-paste calls for 4 APIs with free daily tiers (CVE priority, VIN recalls, US duty rates, email checks), then 5 short emails on KEV/EPSS, MCP and free data.