CiteVisorAudit and CiteVisorResearch / Defined scopes

One request.
A defined scope.

CiteVisorAudit runs a one-time technical inspection when a visitor submits a public URL. It does not collect a training corpus, sample AI product interfaces or promise search rankings. CiteVisorResearch is a separate crawler that reads only robots.txt and llms.txt for our published research.

What the crawler requests

The audit uses GET requests to inspect the submitted page, its public robots.txt, a sitemap and optional llms.txt. If the default sitemap is unavailable, it may try the first alternative sitemap declared in robots.txt. It follows up to five public HTTP or HTTPS redirects per resource, validates each destination, sets no Cookie header and does not execute page JavaScript or bypass access controls.

The user agent is CiteVisorAudit/1.0 (+https://citevisor.com/bot). Only parsed technical observations are retained in a private report. Response bodies are used during the inspection and are not saved.

Frequency and limits

Default limits are ten submissions per visitor per hour and 500 submissions across the service per UTC day. A normalized URL result is reused for 60 minutes to avoid another crawl when possible. Operators may lower these limits or pause audits. Each resource has a 12-second timeout, with a 50-second total audit budget and response size limits.

The audit is limited to one submitted page and its supporting files. It does not recursively crawl a website, send model requests or perform ongoing monitoring.

Website owner opt-out

If you operate a website and want its host excluded from this audit, contact contact@citevisor.com with the domain and enough information to establish ownership. The operator maintains a host blocklist and can add the host and its subdomains after reviewing the request.

We do not claim that this process is an instant self-service opt-out. Do not use this tool to submit a URL you have no right to inspect or a URL containing confidential information.

Research crawler: CiteVisorResearch

For our 2026 robots.txt study, the CiteVisorResearch crawler requests two files from each domain on a public top-sites list: /robots.txt, and /llms.txt only where that robots.txt allows our crawler. It does not request pages, follow links, run scripts or send cookies, and it stops reading a file after 512 KiB.

The user agent is CiteVisorResearch/1.0 (+https://citevisor.com/bot) and its robots.txt token is CiteVisorResearch. Requests to one host run one at a time, at least a second apart, and no more than ten requests start per second across all hosts. Each file has a 12-second limit, and a failed request is retried once, at least 15 minutes later. Each crawl is a one-off decision; we add its dates to this page. So far: two test runs over the top 200 domains on 2026-10-22 between 23:01 and 23:10 UTC, and the Tranco top 10,000 from 2026-10-22 23:12 to 2026-10-23 00:15 UTC, with 55 of those domains requested once more from 2026-10-23 00:25 to 2026-10-23 00:26 UTC after a fix to how our client read compressed answers.

To keep /llms.txt out of future crawls, add a group for our token. robots.txt itself is always read, because that is how a crawler learns your rules.

User-agent: CiteVisorResearch
Disallow: /llms.txt

To exclude a domain entirely, or to have its row removed from the published CSV, email contact@citevisor.com with the domain. We add it to the crawler's exclusion list, remove the row and note the removal on the study page.

What a result means

Checks use the initial HTML and HTTP signals from a single request. They do not establish rendered browser content, actual AI crawler access, indexing, citations or rankings. Training and search permissions are described separately.

See the privacy notice for report retention and deletion and the terms of use for acceptable use.