← Back to Challenges

🤖 Challenge 3: The Robot's Secret

Every website can have a special file that tells search engine "robots" which pages are off-limits...

📖 The Story

You're investigating a suspicious website and you suspect the site owner has hidden some secret pages. In the world of web security, there's a publicly readable "gentleman's agreement" file that websites use to tell search engine crawlers "don't index these pages." But here's the thing — the file is readable by anyone... If you know its name, you might just discover the hidden pages!
📝 Someone's Blog Posts | About | Contact

Welcome to My Blog!

Lots of interesting articles here.

But there are some pages I don't want anyone to find~

I already told the search engine robots to stay away, so nobody will discover them... right?

🛠️ About robots.txt

  • robots.txt is a text file placed at a website's root directory
  • It tells search engine crawlers which pages NOT to index
  • But it's just a "gentleman's agreement" — anyone can read the file directly!
  • Try adding /robots.txt to the end of the website URL in your browser
💡 Need a hint? Click to reveal

Type http://localhost:3000/robots.txt in your browser's address bar. See the Disallow path? Try visiting that path!