Optimize your WordPress robots.txt file for improved SEO and user experience. Learn how to manage search engine crawlers, block unwanted bots, and ensure your site's key pages are prioritized in search results.
You block a page in robots.txt, and weeks later it still turns up in Google, URL and all. That catches a lot of site owners off guard. It isn’t a bug. The robots.txt file just doesn’t do the job most people think it does.
So here’s the honest version: how the file works on WordPress, what it actually controls, what it doesn’t, and how to set it up without quietly hurting your own SEO.
What is the robots.txt file?
robots.txt is a plain text file that lives at the root of your site, at yoursite.com/robots.txt. It tells crawlers like Googlebot which URLs they’re allowed to request. That’s the whole job. It manages crawling, not indexing, and that difference is the part that trips everyone up (more on it below).
One thing to know first if you’re on WordPress: you probably don’t have a physical robots.txt file at all. WordPress serves a virtual one for you. Load /robots.txt on a fresh install and you’ll see something close to this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpBecause it’s virtual, you won’t find it in your files over FTP. To change it you either drop a real robots.txt into the site root (a physical file overrides the virtual one), edit it through an SEO plugin, or hook the robots_txt filter in code.
How does the robots.txt file work?
When a crawler shows up, it looks for /robots.txt first. If it finds the file, it follows the rules. If there’s no file, it assumes everything is fair game.
The two directives you’ll reach for most are User-agent and Disallow. User-agent names the bot a rule applies to, and the wildcard * means all of them. Disallow names a path you don’t want crawled. The default WordPress block above tells every bot to skip /wp-admin/ but still allow /wp-admin/admin-ajax.php, since plenty of themes and plugins depend on it.
One rule worth burning in: don’t block your CSS and JavaScript, and don’t disallow /wp-includes/ or your wp-content assets. Old tutorials still tell you to. Google renders your pages like a browser does, and if it can’t fetch those files it sees a broken layout and can rank you lower for it. Leave them crawlable.
Crawling is not indexing, and that changes everything
Here’s the part the old advice gets wrong. robots.txt controls crawling, not indexing. Blocking a URL does not keep it out of Google.
If another site links to a page you’ve disallowed, Google can still index that URL, sometimes with the anchor text from those links, even though it never read the page itself. The page then shows up in results with no description, which looks worse than not blocking it at all.
Want a page truly out of search? Use a noindex meta tag or an X-Robots-Tag header, and let Google crawl the page so it can actually see that instruction. Blocking it in robots.txt does the opposite of what you want. Google doesn’t even support a noindex line inside robots.txt, so don’t reach for that either.
So what is robots.txt good for? Crawl budget and noise. Keep bots out of endless filtered URLs, internal search result pages, and admin paths so they spend their time on pages that matter. That’s the real win, and it’s a smaller, humbler job than most posts make it sound.
That said, here are the practical rules for setting yours up well on WordPress.
1. Know what actually needs blocking
Start by being clear on what you’re trying to keep bots out of, and why. As a rule, you want everything with real content crawled: blog posts, product pages, landing pages. What you can safely block is the plumbing, admin endpoints, internal search result URLs, and thin duplicate paths that just waste crawl budget.
And to say it one more time, blocking is about crawl efficiency, not hiding a page from search. If your goal is to keep a page out of Google, that’s a noindex job, not a robots.txt one.
2. Use a generator if you’re not comfortable editing by hand
You don’t have to write the syntax from memory. A generator asks which paths you want to block and hands you clean rules to paste in. Our own Robots.txt Generator by DopeThemes does exactly that. Whatever tool you use, read the output before you ship it. One stray Disallow: / takes your whole site off the map.
3. Block unwanted bots and crawlers
Beyond paths, you can tell specific bots to stay out entirely, which is handy for aggressive scrapers chewing through your bandwidth. Add this to your robots.txt:
User-agent: [bot name]
Disallow: /To block the SemrushBot crawler, for example:
User-agent: SemrushBot
Disallow: /Worth being honest here: this only stops the polite bots. robots.txt is a request, not a wall. Bad actors ignore it, so for anything you genuinely need to stop, block it at the server or firewall level instead.
4. Use the robots meta tag for granular control
When you need per-page control, and especially when you actually want a page out of the index, the robots meta tag is the right tool. It sits in the head of the page itself:
<?php
<meta name="robots" content="[directive]">The [directive] can be one of the following:
index: allows search engines to index the pagenoindex: keeps the page out of the indexfollow: lets search engines follow links on the pagenofollow: tells them not to follow links on the page
To keep a page out of search, use:
<?php
<meta name="robots" content="noindex">Remember the earlier point though: for this to work, the page has to stay crawlable in robots.txt. Block it there and Google never sees the noindex.
5. Test before you trust it
After any change, check it. In Google Search Console, open the robots.txt report. It replaced the old standalone robots.txt Tester, which Google retired. The report shows you the version Google last fetched, flags errors, and lets you request a fresh crawl when you’ve just made a change. Read it once a page, and you’ll catch a bad rule before it costs you traffic.
The short version
robots.txt is a small, useful tool doing a smaller job than the hype suggests. It shapes how bots crawl your site. It does not decide what Google indexes. Keep your admin plumbing out, keep your CSS and JS in, and reach for noindex when you actually want a page gone. Set it, test it in Search Console, then get back to the work that actually moves rankings.