{"id":660,"date":"2026-10-26T03:49:42","date_gmt":"2026-10-26T03:49:42","guid":{"rendered":"https:\/\/sitefitters.com\/learn\/glossary\/robots-txt\/"},"modified":"2026-10-26T03:49:42","modified_gmt":"2026-10-26T03:49:42","slug":"robots-txt","status":"publish","type":"page","link":"https:\/\/sitefitters.com\/fr\/learn\/glossary\/robots-txt\/","title":{"rendered":"robots.txt"},"content":{"rendered":"
Learn<\/a> \/ Glossary<\/a> \/ SEO & Local Search<\/a><\/div>

robots.txt Explained.<\/em><\/h1>
In one sentence<\/small>robots.txt is a small text file at the root of a website that tells search engine crawlers which parts of the site they should or should not visit.<\/div><\/div><\/section>

What It Is<\/h2>

robots.txt is a plain text file placed at the main address of a site, such as example.com\/robots.txt. It gives instructions to search engine crawlers (the automated programs that read websites) about which folders or pages to skip. Think of it as a sign on certain doors saying "staff only." Reputable crawlers like Google’s follow these rules, though it does not physically lock anything.<\/p>

Why It Matters for Your Business<\/h2>

Used well, it keeps crawlers away from pages that do not need attention, like admin areas or internal search results. Used badly, it can hide an entire site. A common accident happens when a site moves from a staging copy to the live address and still carries a rule blocking everything, so new pages stop appearing in search.<\/p>

Good to Know<\/h2>

Common mistake: using robots.txt to keep a page out of search results. Blocking a page from crawling does not reliably remove it from Google, and it can still appear if other sites link to it. To keep a page out of results, use a "noindex" setting on the page instead, and use a password for anything truly private.<\/p>

Related terms<\/h4>
Crawling<\/a>Indexing<\/a>XML sitemap<\/a>Technical SEO<\/a>Staging site<\/a><\/div><\/div>