Site Crawler Check screenshot

Site Crawler Check

Resolve DNS records, read robots.txt, or forward-confirm a crawler's IP by reverse DNS, from the server side.

Privacy First
0 Runs
Updated 2026-11-07
Runs on the tool server
No account needed
Read-only checks
Private addresses are refused

What this site check does

Three server-side questions: what DNS records a host publishes, what its robots.txt actually allows, and whether an address that claims to be a crawler really belongs to that crawler. A browser cannot read those records or fetch a cross-origin robots.txt reliably, so the check runs on the server and says so.

How to use it

Choose DNS, robots or spider. DNS lists the A, AAAA, CNAME, MX, TXT and NS answers and names the record types that returned nothing. Robots fetches /robots.txt over https first, falls back to http, and applies the longest matching allow or disallow rule for the user agent you name. Spider reverses the address, resolves the resulting name forward, and calls a crawler verified only when the same address comes back.

Privacy and safety

The server makes the lookup or fetch, so the page does not claim to run locally. Private and reserved addresses are refused, redirects are not followed, and a robots.txt read is capped at 128 KB.

This page asks three questions of a server rather than of a page: what its DNS answers, what its robots.txt says for a chosen user agent, and whether an address that claims to be a crawler really belongs to that crawler.

The DNS mode resolves A, AAAA, CNAME, MX, TXT and NS records with a five-second resolver budget and reports, per record type, either the answers or the reason none came back. The robots mode fetches robots.txt over https and then http, reads at most 128 KB, parses the groups and evaluates the rule that matches the root path.

The spider mode is a forward-confirmed reverse lookup: it resolves the address to a name, resolves that name forward again, and checks the original address is among the answers. It then names which known crawler domains the reverse name belongs to, if any.

What each mode reads

DNS answers are the resolver's reply, not a historical record and not a propagation check. An empty record type is reported with the resolver's own error name, so a timeout and a genuine no-answer are distinguishable.

The robots mode returns the URL it fetched, the status code, whether the file was present, the parsed groups and the sitemap lines. A missing file is reported as absent with the note that everything is then allowed, rather than as an error.

  • DNS: A, AAAA, CNAME, MX, TXT and NS
  • robots.txt: 128 KB cap, https then http
  • Spider: reverse lookup then forward confirmation

What the verdict cannot prove

A robots.txt parse follows the file as written; it does not fetch the sitemap, crawl the paths or know which crawler will obey. The page evaluates one user agent against the root path, which is a useful default rather than every URL on the site.

A forward-confirmed reverse lookup is evidence, not proof of identity: it shows the address is consistent with a name in a crawler's domain. Spoofing DNS is possible, and a claim about a crawler should be read alongside the request's behaviour and the operator's published ranges.

How to use the three modes

Choose the DNS, robots or spider mode, type the host or address, and press Run; private addresses are refused before any check starts. DNS reports all six record types with the resolver's own reason where nothing came back, robots shows the fetched URL, status and parsed groups, and spider shows the reverse name with its forward confirmation.

Run the same host in more than one mode when the question spans them: a DNS answer explains a resolution failure, while the spider verdict answers whether the address belongs to a known crawler domain. Treat every verdict as the current reading, not a cached history.

What to Expect

Best for

  • Seeing which records a host publishes right now
  • Reading a robots.txt file and its sitemap lines
  • Checking whether an address forward-confirms to a crawler domain

Not ideal for

  • Proving an address is a specific crawler
  • Crawling a site or fetching a sitemap
  • A DNS propagation or history check

What this tool keeps

  • Each record type separately, including the types that returned nothing
  • The exact robots.txt URL and status code
  • The reverse name and the forward answers behind a spider verdict

What may need cleanup

  • A resolver timeout, which is reported as unavailable rather than as an empty record
  • The robots verdict for the root path, which may not hold for a deeper URL
  • The crawler-domain match, which should be treated as one piece of evidence

Common errors

  • Reading a no-answer as a missing DNS record when the resolver timed out
  • Expecting the robots mode to fetch the sitemap it lists
  • Treating a forward-confirmed name as proof that the request was legitimate

Example use cases

  • Diagnosing why a domain does not resolve
  • Checking a robots.txt before a launch
  • Investigating a crawler claim in a server log

Sample input

The host example.com checked in DNS mode, then in robots mode for the user agent Googlebot.

Sample output

A record table for six types, then a robots report with the fetched URL, status code, parsed groups and sitemap lines.

Who this is for

  • Site owners debugging DNS
  • SEO engineers checking a robots file
  • Server administrators investigating crawler traffic

Frequently asked questions

What three questions does the site crawler check ask?

What a domain's DNS answers, what its robots.txt says for a chosen user agent, and whether a claimed crawler address really belongs to that crawler.

Does it prove an address is a specific crawler?

No. The reverse and forward name match should be treated as one piece of evidence, not proof.

Does it crawl the site?

No. It reads the DNS and robots.txt answers only, and does not fetch a sitemap or walk pages.

Developer program

Build custom tools for your store

Join the SDK program to create useful ecommerce utilities for your store.