Lunadesk · AI Powered Helpdesk
Sign in (opens app.lunadesk.ai)Start free trial (opens app.lunadesk.ai)
Crawler information

LunadeskBot

LunadeskBot is the crawler Lunadesk uses to read public web pages that its customers asked it to learn from.

Why is LunadeskBot visiting my site?

A Lunadesk customer added pages of this website as a knowledge source for their support assistant. LunadeskBot reads those pages so the assistant can answer questions using the content of those pages.

We only crawl the URLs and patterns that customer configured, on a single domain.

How to identify LunadeskBot

Every request from LunadeskBot carries exactly this User-Agent:

LunadeskBot/1.0 (+https://lunadesk.ai/bot)

The token to use in robots.txt is LunadeskBot.

What LunadeskBot does and doesn’t do

What it does

  • Reads and respects robots.txt, including Crawl-delay. robots.txt is re-read at most once every 24 hours.
  • Respects <meta name="robots" content="noindex"> and the X-Robots-Tag header.
  • Reads sitemap.xml once a day to find pages, and uses lastmod to avoid useless visits.
  • Polite rate: one request at a time per website, at least 1 second apart (longer if your Crawl-delay asks for it).
  • Visits the same page automatically at most once a day. Unchanged pages are visited less and less often, down to once a week.
  • Uses conditional requests (If-None-Match / If-Modified-Since), so a page that has not changed costs you almost nothing.
  • If your server has a temporary problem (timeout, 5xx), it retries a page at most 3 times, 30 minutes apart, then goes back to its normal pace.
  • Only reads public pages. It never logs in, never submits forms and never bypasses paywalls.
  • Never visits private or internal network addresses.

What it doesn’t do

  • Does not download PDFs or other files. It only notes that they exist.
  • Does not use your content to train AI models. The text is only used as a reference by the support assistant of the one customer who added your site.

How to control or block LunadeskBot

Add one of these rules to the robots.txt file at the root of your website.

# Block LunadeskBot entirely
User-agent: LunadeskBot
Disallow: /
# Block one section
User-agent: LunadeskBot
Disallow: /private/
# Slow it down (seconds between two requests)
User-agent: LunadeskBot
Crawl-delay: 10

Changes are picked up within 24 hours, because robots.txt is re-read at most once a day.

Questions about LunadeskBot

Short answers for website owners and administrators.

Contact

Questions or concerns about LunadeskBot? Email bot@lunadesk.ai.