What It Is
robots.txt is a plain text file placed at the main address of a site, such as example.com/robots.txt. It gives instructions to search engine crawlers (the automated programs that read websites) about which folders or pages to skip. Think of it as a sign on certain doors saying "staff only." Reputable crawlers like Google’s follow these rules, though it does not physically lock anything.
Why It Matters for Your Business
Used well, it keeps crawlers away from pages that do not need attention, like admin areas or internal search results. Used badly, it can hide an entire site. A common accident happens when a site moves from a staging copy to the live address and still carries a rule blocking everything, so new pages stop appearing in search.
Good to Know
Common mistake: using robots.txt to keep a page out of search results. Blocking a page from crawling does not reliably remove it from Google, and it can still appear if other sites link to it. To keep a page out of results, use a "noindex" setting on the page instead, and use a password for anything truly private.