How to convert a whole website to Markdown for RAG, with llms.txt

Updated 2026-11-10 · Data and API builders · by CyberMax

Copy-paste code

Python (pip install apify-client): docs site to RAG chunks

Copyimport os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("cybermax/website-to-markdown").call(run_input={
    "startUrls": ["https://docs.apify.com/academy"],
    "includePaths": ["/academy*"],
    "maxPages": 200,
    "outputMode": "chunks",      # one row per chunk
    "chunkSize": 2000, "chunkOverlap": 200,
})
for c in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(c["chunkId"], c["heading"], c["chunkTokens"])  # embed c["text"], keep c["url"] as the source

kv = client.key_value_store(run["defaultKeyValueStoreId"])
open("llms.txt", "w").write(kv.get_record("LLMS-TXT")["value"])

LangChain (Python): load the run straight into Documents

Copyfrom langchain_community.document_loaders import ApifyDatasetLoader
from langchain_core.documents import Document

loader = ApifyDatasetLoader(
    dataset_id=run["defaultDatasetId"],
    dataset_mapping_function=lambda d: Document(
        page_content=d["text"], metadata={"source": d["url"], "heading": d["heading"]}),
)
docs = loader.load()  # the chunks are already heading-aware: embed them as they are

JavaScript / Node (npm i apify-client): pages as Markdown files

Copyimport { ApifyClient } from 'apify-client';
import { writeFileSync } from 'node:fs';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('cybermax/website-to-markdown').call({
  startUrls: ['https://docs.example.com'], maxPages: 300,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((p, i) => writeFileSync(`page-${i}.md`, `<!-- ${p.url} -->\n${p.markdown}`));

No code: n8n, Make or Zapier

Copyn8n: HTTP Request node (or the Apify node, action "Run an Actor and get dataset")
Method:  POST
URL:     https://api.apify.com/v2/acts/cybermax~website-to-markdown/run-sync-get-dataset-items
Auth:    Header Auth  ->  Authorization: Bearer <your Apify token>
Body:    JSON  {"startUrls": ["https://docs.yourproduct.com"], "monitor": true, "outputMode": "chunks"}
Output:  one n8n item per result row

Make:    Apify app -> "Run an Actor" (Actor: cybermax/website-to-markdown, wait for finish = yes)
         -> Apify "Get Dataset Items" (dataset = the run's defaultDatasetId) -> your next module
Zapier:  Apify app -> action "Run Actor" (cybermax/website-to-markdown, input JSON as above)
         -> trigger "Finished Actor Run" -> "Fetch Dataset Items" -> your app
Runs longer than 5 minutes: start with POST /v2/acts/cybermax~website-to-markdown/runs and read the dataset when the run finishes.

Fresh RAG index: Schedule trigger (daily 03:00) -> the request above -> your vector store node (Pinecone, Qdrant, Supabase) "Upsert" by chunkId -> delete chunks whose url is in the OUTPUT record's removedSinceLastRun.

From an AI agent (MCP)

Copy# Any MCP client (Claude Desktop, Claude Code, Cursor, VS Code): Apify's hosted MCP server
{
  "mcpServers": {
    "website-to-markdown": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://mcp.apify.com/?tools=cybermax/website-to-markdown",
               "--header", "Authorization: Bearer ${APIFY_TOKEN}"],
      "env": { "APIFY_TOKEN": "<your Apify token>" }
    }
  }
}
# The agent sees one tool, cybermax--website-to-markdown, with the Actor's input schema.
# Ask: "read docs.example.com as Markdown and summarise how authentication works".

Source: Docsieve (Apify Actor cybermax/website-to-markdown) README, real Apify run on 10 Nov 2026

Why clean Markdown matters for RAG

Raw HTML wastes tokens on navigation, footers and scripts, and naive text extraction loses code blocks and tables. Main-content Markdown keeps the structure the model needs, and heading-aware chunks (with a heading path such as Guide > Install > Linux) retrieve far better than fixed-size slices of a page.

Generate llms.txt and llms-full.txt for your site

Every run writes LLMS-TXT (site name, summary and a link list grouped by section) and LLMS-FULL-TXT (every page's Markdown with its source URL) to the run's storage, free. Publish them at your site root so AI assistants and AI search can read your site cleanly.

Give an AI agent a whole site to read

Through Apify's MCP server an agent can call Docsieve to read a docs site as Markdown before answering, alongside the other CyberMax web data tools (company domains, contacts, tech stack, trends).

Good to know

Price on Apify: $0.001 per page converted ($1 per 1,000), no start fee; a 500-page docs site is $0.50. llms.txt files are free, and thin, noindex and robots-blocked pages are skipped and not charged. Apify's free plan includes $5 of usage each month.

Docsieve reads the HTML the server sends and does not render JavaScript; most docs platforms (Docusaurus, GitBook, MkDocs, Mintlify, WordPress, server-rendered Next.js) work as is.

Questions

Related answers