Crawler

A crawler is the content source that feeds documents into an engine’s index.

A crawler is a content source attached to an engine. It produces the documents that end up in the search index — either by fetching pages on a schedule, or by accepting documents you push through the API. An engine can have one or more crawlers (for example, a website crawler plus a feed for product data).

Every API path that writes to the index takes a crawlerId: Index Management uses /index/{crawlerId}/documents to push, patch, or delete documents, and the queue endpoint at /crawler/{crawlerId}/paths adds URLs to a fetch queue. Crawlers themselves are created and configured in MyCludo under Crawlers.

See Pushing Content for the end-to-end workflow.