Guide

Website sources

Point the crawler at a public site and it becomes a data source you can search, re-sync and include in any collection. This page explains what it fetches, in what order, and how to control it.

How pages are discovered

The crawler starts from your URL and works sitemap-first. It reads robots.txt and honours it, then fetches the sitemap, including nested sitemap indexes, and visits the pages in scope. Links found on pages only fill the gaps: they add pages the sitemap did not list, up to the link depth you set. Every sitemap page in scope is included regardless of depth.

Before any of that it looks for llms.txt at the site root. Sites that publish one get their curated page list used directly.

Markdown first

For each page the crawler tries to get Markdown before HTML: a .md twin of the URL, then the same URL with Accept: text/markdown. Documentation platforms increasingly serve these, and the result is far cleaner than parsed HTML. When only HTML is available, the main content is kept, navigation, footers and other repeated parts are dropped, and the rest is converted to Markdown. Page metadata, layout markup and "copy page" buttons are removed.

The Markdown is chunked along its headings, and each chunk is prefixed with its heading path, so both rankers know where a passage sits. That path comes back with every search hit.

Options

Look before you crawl

The console checks a URL before starting, and you can call the same check yourself. It fetches a handful of URLs, indexes nothing, and reports whether the address is allowed, whether it is reachable, what robots.txt says, how many pages the sitemap lists, whether llms.txt exists, and whether the pages look JavaScript-only.

curl https://api.hayfork.dev/crawl/check \
  -H "Authorization: Bearer $HAYFORK_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com/"}'

Starting a crawl from the API

Requires the ingest scope and a verified email.

curl https://api.hayfork.dev/crawl \
  -H "Authorization: Bearer $HAYFORK_KEY" -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com/",
    "include_paths": ["/docs/"],
    "page_limit": 500,
    "max_depth": 2,
    "collection_id": "COLLECTION_ID"
  }'

# poll the job
curl https://api.hayfork.dev/crawl/JOB_ID -H "Authorization: Bearer $HAYFORK_KEY"

The job reports status, pages_crawled, page_limit and any error. Statuses are queued, crawling, completed and failed. A crawl that is interrupted, for example by maintenance, is marked failed with an "interrupted" message; POST /crawl/JOB_ID/retry resumes it with its remaining page budget and skips pages already indexed. A running crawl stops on POST /crawl/JOB_ID/cancel, and the pages it already indexed stay.

Keeping a source current

Press Sync on a website source in Data sources, or post to /crawl with its data_source_id. The saved settings for page limit and scope are shown in the list and reused. A re-crawl refreshes existing pages in place rather than duplicating them, and re-syncs do not count against your document limit.

The circular-arrow button, or POST /data-sources/ID/reindex, re-embeds what is already stored without fetching anything. It is only needed after an embedding model change on a self-hosted deployment.

What the crawler refuses

Because customers point it at arbitrary URLs, the crawler checks every request, including sitemap and llms.txt fetches and each redirect hop. It refuses:

• Anything other than http and https, and URLs with embedded credentials.
• Private, loopback, link-local and other non-public addresses, including public hostnames that point to one, and the special addresses cloud providers use for machine metadata.
• Bare or internal-looking hostnames such as localhost, *.internal, *.local and *.svc.
• Responses over 5 MB and redirect chains longer than five hops.

A refused start URL returns 400 with the reason. A page blocked mid-crawl is recorded as a failed document with the reason, so the crawl continues.