Sauce · Fundamentals
One text file, five lines long, can hide your whole website from Google.
Quick answer
robots.txt is a small text file at your website address followed by /robots.txt. It tells search crawlers which parts of your site to skip. One wrong line, Disallow: /, tells them to skip everything. This line is often left on by accident after a site is built, because the build tool blocks the site while it is being worked on and nobody turns it back. It is a request, not a lock, and it is not security. Check yours in two minutes.
Is your trade still open in your city?We take one client per trade, per city. The call starts with that check, not a pitch.
Book a free assessmentYour site was fine last month. Now it is gone from Google.
No penalty email. No warning. The pages still load for you. But search your business name and the site is missing, or search your trade and town and it never was there.
Before anything else, check one file. It is the fastest thing on this whole list to rule out, and it is the one that hides sites most often.
What robots.txt is
Every website can have a plain text file at the top level. You reach it by typing your website address and adding /robots.txt. Try it now with your own site.
It is a note to search crawlers. It says, in a simple format, which folders and pages to visit and which to leave alone. Crawlers read it first, before anything else on the site.
Three things to be clear on
A public text file at yoursite.com/robots.txt. Anyone can read it. It has a few lines: which crawler it is talking to, then Allow and Disallow rules for paths on the site.
It manages crawl traffic. It keeps well-behaved crawlers out of parts of the site you do not want them spending time on, like an internal search results page or a login area. Google follows it.
It is not a lock. It does not stop a page from being indexed if another site links to it. It is not security, since the file is public and lists the paths you are trying to hide. And it does not remove a page that is already in Google.
The ways it quietly breaks a local site
Each of these is common. Each is a one-line fix once you find it.
Disallow: / with nothing after the slash. That means block every path. Site builders and staging tools add this so a half-built site does not get indexed. It is meant to be removed at launch. It often is not.Disallow: /services/ blocks every page under services. Someone added it years ago to hide one draft, and it now hides the whole section. Read every Disallow line and ask what lives in that folder today.noindex line in robots.txt is ignored. To keep a page out of Google, the page itself needs a noindex tag, and that page must not be blocked by robots.txt, or Google never sees the tag.robots.txt controls crawling. A noindex tag controls indexing. They are different tools for different jobs.
robots.txt or noindex: which do you want?
This mix-up costs people real pages. Here is the split.
- You want to save crawl effort on pages that do not need visiting, like an on-site search page or a filtered list with endless combinations.
- You do not care whether the URL ever shows in results, only that Google should not spend time fetching it.
- You are keeping a whole staging site or admin area out of a crawler's path.
- You have a real page that must not appear in Google results: a thank-you page, an old promo, a duplicate.
- Add
<meta name="robots" content="noindex">to that page, and make sure robots.txt is not blocking it. - Once Google recrawls and sees the tag, the page drops out of results.
Go deeper
Which part is your problem?
Each of these goes deeper on one piece.
Google Search Essentials, in plain words
The three things Google needs before any of this matters. A ten-minute self-check.
See the check › Piece 02The 2026 local ranking factors
Once the site is visible, this is what moves the map. The sourced weights.
See the breakdown › Piece 03How to measure local visibility
How to tell whether a change actually helped, across your whole service area.
See the method ›Check yours in five minutes
Open the file
Type your website address, then /robots.txt, in the browser bar. Read every line. If you see Disallow: / on its own, that is the problem.
Check every Disallow line
For each one, open a page that lives in that folder. If it is a page you want customers to find, that line has to change.
Make sure your style and script folders are not blocked
If any Disallow line points at a folder with CSS or JavaScript, remove it. Google needs to load the page fully.
Use the robots.txt report in Search Console
If you have Google Search Console, its robots.txt report shows the file Google last read and flags errors. It also lets you ask Google to fetch a fresh copy.
See where you actually show up first.Before you pay us a dollar, we run a free 49-point geo-grid scan of your service area. You keep the results either way.
Get the free scanWhat this does not mean
It does not mean robots.txt is bad. Used right, it is useful. It keeps crawlers off pages that waste their time.
It does not mean you need one. A small site with nothing to hide can skip it. Google will just crawl everything it can reach.
It does not mean this is the only thing that hides a site. A wrong canonical tag, a lost link, or a site-wide noindex can do it too. This is only the first and fastest thing to rule out.
Open your robots.txt now. Two minutes. If the first real line is Disallow: /, you just found why the phone went quiet.
What does robots.txt do?
It tells search crawlers which parts of a site to crawl and which to skip. It manages crawl traffic. Google reads it before crawling anything else on the site and follows its rules.
Can robots.txt hide my page from Google?
Not reliably. Blocking a page in robots.txt stops Google from crawling it, but the URL can still appear in results if other pages link to it, usually with no description. To keep a page out of results, use a noindex tag on the page itself and do not block that page in robots.txt.
Why did my whole site disappear from Google after a redesign?
A common cause is a Disallow: / line left in robots.txt from the build or staging environment. Site builders block the site while it is being worked on and the block is meant to be removed at launch. Check yoursite.com/robots.txt first. A site-wide noindex tag left on by the same process can do it too.
Should I block anything in robots.txt?
Usually only pages that have no value to a searcher and waste crawl effort: an on-site search results page, faceted filter URLs with endless combinations, a login or admin area. Do not block your CSS or JavaScript folders, because Google needs them to render the page properly.
What is the difference between robots.txt and a noindex tag?
robots.txt controls crawling: whether Google fetches the page at all. A noindex tag controls indexing: whether a page Google has fetched is allowed to appear in results. If you want a page kept out of Google, you need the noindex tag, and the page must stay crawlable so Google can see it.
Sources
- Google Search Central, “Introduction to robots.txt”, developers.google.com: what the file is for, and that it is not a mechanism for keeping a page out of Google.
- Google Search Central, “How Google interprets the robots.txt specification”: rule syntax,
AllowandDisallow, and that anoindexdirective in robots.txt is not supported. - Google Search Central, “Block Search indexing with noindex”: the correct way to keep a page out of results, and why the page must not be blocked by robots.txt.
- Google Search Central, “Test your robots.txt with the robots.txt report”: the Search Console report that shows the file Google last fetched and any errors.
- Google Search Central, “Google Search Essentials”: a page must be crawlable and not blocked to appear in Search.
Next step
Find out whether your trade is still open in your city
One client per trade, per city. The call starts with that check, not a pitch.