Gratis starten

Fabrikken-Crawler

Last updated 29 October 2026.

Fabrikken-Crawler is the robot that collects content for chatbots built on fabrikken.ai. When an organisation creates a chatbot with us and chooses a website as a data source, the crawler visits that website and the documents its pages link to, so the chatbot can answer from the public content.

How to recognise us

The crawler always sends this User-Agent:

Mozilla/5.0 (Fabrikken-Crawler/1.0; +https://fabrikken.ai/crawler-info)

If you are unsure whether a visit really came from us, write to kontakt@fabrikken.ai and we will confirm it.

What we fetch

  • Only publicly accessible pages and documents (HTML, PDF and Office files) from websites our customers have chosen themselves, plus documents those pages link to directly.
  • We never log in, never fill in forms and never try to bypass logins, captchas or other access controls. If we meet a security check, we skip the page.
  • We fetch one page at a time with a pause in between. A website is typically revisited daily or weekly, and we use conditional requests so unchanged content is not downloaded again.
  • Pages that require JavaScript may be rendered in a headless browser. It uses the same User-Agent.

What the content is used for

The content is used only to answer questions in the chatbot of the customer who chose the website. We do not sell it, share it with other customers or train AI models on it. Data is processed in the EU, see our sub-processors. If a page disappears from your website, or you block it, it is removed from the chatbot's knowledge at the next visit.

How to control our access

We respect robots.txt for every page and document. You can block us entirely:

User-agent: Fabrikken-Crawler
Disallow: /

Or only from parts of your site, e.g. /internal/. Rules for "User-agent: *" also apply to us unless you have a group specifically for Fabrikken-Crawler. If robots.txt cannot be fetched, we treat it as having no restrictions.

Contact

If you have questions, experience load from the crawler, or want to give us access to material behind a security check, write to kontakt@fabrikken.ai. We usually reply within one working day.