The first thing I check on every external web application assessment is the robots.txt file. Not because I want to respect the crawling directives. Because the file is a map of everything the organization does not want found.
robots.txt was designed in 1994 as a voluntary protocol for telling well-behaved crawlers which paths to skip. It was never a security mechanism. It has no access control. It provides no authentication. It is a publicly accessible plaintext file that lists, explicitly, the paths the site owner considers sensitive.
And yet, organizations routinely use it to “hide” admin panels, internal tools, staging environments, API documentation, and backup directories.
I have found more initial footholds through robots.txt than through any automated scanner.
What I have found
On one engagement, a financial services company’s robots.txt contained:
Disallow: /admin-portal/
Disallow: /api/v2/docs/
Disallow: /staging/
Disallow: /backup/
Disallow: /internal/reports/
Five paths. The admin portal had a login page with default credentials. The API documentation described every endpoint, including unauthenticated ones. The staging environment was a copy of production with test data that included real customer records. The backup directory contained a database dump. The internal reports path had access control, but the directory listing was enabled, revealing report filenames that disclosed business-sensitive information.
None of these paths appeared in the site’s navigation, public links, or JavaScript bundles. The only reason I found them was that someone listed them in robots.txt to keep Google from indexing them.
This is not an isolated case. I see it on roughly 40% of the external assessments I run. The robots.txt file is the organization’s confession of where the sensitive things live.
The information disclosure problem
The security issue is not that robots.txt exists. It is that organizations treat “Disallow” as “deny access” when it actually means “please do not look here.”
Every malicious actor, every pentester, every bug bounty hunter checks robots.txt first. Automated tools like Nuclei, Burp Suite, and even basic recon scripts parse it and add every Disallow path to their target list. The file does not deter attackers. It guides them.
The information leaked by robots.txt goes beyond direct paths. The structure of the paths reveals the technology stack, internal naming conventions, and organizational structure. A path like /wp-admin/ confirms WordPress. /jenkins/ confirms a CI/CD tool. /grafana/ confirms monitoring infrastructure. /api/v3/ tells me there is a versioned API and v1 and v2 might still be accessible.
Disallow entries that reference environment-specific paths (/staging/, /dev/, /qa/) reveal that non-production environments exist on the same domain or server. These environments typically have weaker security controls and are prime targets.
The bot management angle
The robots.txt problem has gotten worse with the rise of AI scrapers. As I wrote about previously, 51% of web traffic is now automated. AI crawlers from OpenAI, Anthropic, Google, and dozens of smaller operators check robots.txt for scraping permissions. Organizations are adding increasingly detailed robots.txt files to manage AI crawler access.
This creates a new class of information disclosure. When a media company adds:
User-agent: GPTBot
Disallow: /premium-content/
Disallow: /subscriber-only/
Disallow: /archive/paywalled/
They are telling every bot, including malicious ones, exactly where the premium content lives and what the URL structure looks like. Some of these paths may have weak paywall enforcement that can be bypassed with a direct request or a modified user agent.
The tension is real. Organizations need robots.txt to manage crawler behavior. But every directive they add is intelligence for attackers.
What I recommend
Audit your robots.txt now. Read it as an attacker would. Every Disallow path is a target. If any of those paths rely on obscurity rather than proper authentication and access control, you have a vulnerability.
Do not list sensitive paths. If a path should not be accessed by unauthorized users, enforce that with authentication, not with a robots.txt directive. Remove admin panels, API documentation, staging environments, and backup directories from robots.txt entirely. Either protect them with access control or remove them from the public-facing server.
Use authentication, not obscurity. Admin panels should require authentication. API documentation should be behind access control or, better, not hosted on the public-facing domain at all. Staging environments should be on separate infrastructure with network-level restrictions. Backups should never be accessible from a web server.
Use robots.txt only for non-sensitive crawl management. Legitimate uses of robots.txt include preventing crawlers from overloading dynamic search pages, avoiding duplicate content indexing, and managing crawl budgets. These are SEO and performance concerns, not security concerns. If a path appears in your robots.txt for security reasons, the security is already broken.
Monitor robots.txt changes. I have seen developers add Disallow entries as a quick fix when they deploy something they realize should not be public. “Just add it to robots.txt until we fix it properly.” That temporary fix becomes permanent, and the path remains accessible. Monitor robots.txt for changes and treat any new Disallow entry as a potential security issue to investigate.
Automate it
Every pentester knows to check robots.txt. Few do it systematically across an entire asset inventory.
I maintain a script that fetches robots.txt from every domain and subdomain in a client’s scope. It parses the Disallow entries, checks which paths return 200 status codes, fingerprints the technologies behind them, and generates a report of exposed sensitive paths.
This single script, run monthly, has found more initial access vectors than any vulnerability scanner I use. It takes five minutes to run and consistently produces actionable findings.
The oldest, simplest file on the web is still one of the most useful recon sources around. Check yours today.