What this tool does
This checker looks at two public text files sites place at well-known locations: security.txt, which explains how to report a vulnerability, and robots.txt, which gives instructions to crawlers. It checks whether they exist and reviews their most important fields or directives.
security.txt: a standardized reporting channel
Before RFC 9116, a security researcher who found a flaw on a site often had to hunt for a valid contact address — a generic contact form, a barely-monitored inbox, or nothing at all. security.txt solves this by standardizing a single location (/.well-known/security.txt) where a dedicated security contact can be found. This tool also checks the legacy root fallback (/security.txt), still recognized by many tools even though the RFC now recommends only the .well-known location.
Why the expiry date matters as much as the contact
A security.txt file left untouched for five years, pointing at an email address that no longer exists, is worse than useless — it creates a false impression of an active channel. RFC 9116 makes the Expires field mandatory precisely to prevent that: a file with an expired date signals to automated tools (and attentive researchers) that the information might no longer be reliable, without anyone having to check every field by hand.
robots.txt: instructions for crawlers, not a protection mechanism
robots.txt tells well-behaved crawlers (search engines, archiving tools) which parts of a site they may access — but it's a voluntary convention, not a security mechanism. A robots.txt that blocks /admin/ protects nothing: it just tells polite crawlers to stay away, while publicly revealing to anyone reading the file that the path exists. Real protection for a sensitive area is always authentication, never robots.txt.
The classic forgotten Disallow: / trap
The single most common cause of a site vanishing entirely and inexplicably from search results is almost never an algorithmic penalty — it's a Disallow: / left active after a staging deployment, never removed when the site went live. This kind of mistake can go unnoticed for weeks, with the site working perfectly for human visitors while its search traffic quietly collapses. This tool flags this case specifically instead of just dumping the raw file contents.