The robots.txt file tells search engine crawlers which parts of a WordPress site to crawl and which to skip. A misconfigured WordPress robots txt file can accidentally block search engines from indexing the entire site — one of the most damaging SEO mistakes possible, and one that is often not noticed for weeks or months. Understanding what robots.txt does, how WordPress generates it, and how to configure it correctly protects search traffic while preventing unnecessary crawl waste on pages that should not appear in search results. This fits into the wider topic we cover in our Complete Guide to WordPress How.
WordPress Robots Txt — How WordPress Generates It
WordPress generates its WordPress robots txt file dynamically rather than storing a static file on the server. Navigating to yoursite.com/robots.txt returns a WordPress-generated response — unless a static robots.txt file exists at the site root (which takes precedence over WordPress’s dynamic generation). The default WordPress robots.txt output looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThis default configuration blocks crawler access to the /wp-admin/ directory (the admin interface) while explicitly allowing /wp-admin/admin-ajax.php, which is used by some AJAX-based plugins that serve front-end content via the admin-ajax endpoint. The default does not block any front-end content, leaving all public posts, pages, and archives available for crawling. WordPress’s “Discourage search engines from indexing this site” setting (Settings → Reading → check “Discourage search engines from indexing this site”) changes the robots.txt to include Disallow: / — blocking all crawlers from the entire site. This setting is appropriate for staging or development sites and is catastrophically wrong for production sites. Check Settings → Reading immediately if organic traffic suddenly drops to verify this setting is not accidentally enabled on a live site.
The WordPress robots txt is generated by the do_robots action and the robots_txt filter — the same WordPress filter and action system used for all other dynamic output. The filter allows modifying the robots.txt output without creating a physical file: add_filter('robots_txt', function($output, $public) { $output .= "User-agent: *nDisallow: /thank-you/n"; return $output; }, 10, 2); — this adds a custom disallow rule to the default WordPress robots.txt output. The second parameter $public reflects the “Discourage search engines” setting — when $public is 0 (the setting is enabled), the filter can check this to behave differently, though modifying the output when search engine discouragement is active would undermine the setting’s intent. According to Google’s robots.txt documentation, the robots.txt file controls which URLs Google’s crawler can access but does not affect indexing of pages Google discovers through links — a page blocked by robots.txt can still appear in Google’s index if other sites link to it. Use the noindex meta tag (via an SEO plugin) for pages that should not be indexed but can be crawled.
Configuring WordPress Robots Txt With an SEO Plugin
Rank Math and Yoast both provide a user-friendly WordPress robots txt editor in the WordPress admin, allowing configuration without accessing the file directly or writing PHP code. This is the recommended approach for most sites — the SEO plugin editor validates the robots.txt syntax and provides a preview of the complete file before saving.
Edit the WordPress robots txt in Rank Math: Rank Math → General Settings → Edit robots.txt. The editor shows the current file content — modify it and save. Rank Math saves the content as a virtual file that WordPress serves at the robots.txt URL. For Yoast SEO: Yoast → Tools → File Editor → robots.txt section. Both plugins provide the same editing capability; use whichever SEO plugin is already active on the site.
Essential WordPress robots txt rules to add beyond the default: block crawling of WordPress utility pages that should not appear in search results (login page, admin pages, search results pages) and common crawl traps that waste crawl budget:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /trackback/
Disallow: /comments/
Sitemap: https://yoursite.com/sitemap_index.xmlThe Disallow: /?s= blocks search result pages (WordPress search generates URLs with the ?s= parameter) — search result pages are rarely worth indexing and their large number can dilute crawl budget on smaller sites. The Sitemap directive at the end tells crawlers where to find the sitemap — adding this to robots.txt is equivalent to pinging search engines about the sitemap location and ensures crawlers always know where to find the current sitemap without relying on Search Console submission alone. Our guide on creating a WordPress XML sitemap covers the sitemap configuration that pairs with the robots.txt Sitemap directive to ensure consistent search engine crawl guidance across both mechanisms.
Creating a Physical Robots.txt File
Complex WordPress robots txt configurations require a physical file — multiple User-agent specific blocks, crawl rate limiting, or rules that the SEO plugin editor does not support — creating a physical robots.txt file at the server root provides complete control over the file’s content.
Create the physical WordPress robots txt file: connect via FTP → navigate to the root directory (the same directory as wp-config.php) → create a file named exactly “robots.txt” (lowercase, no extension visible in the filesystem) → add the content. A physical robots.txt file takes precedence over WordPress’s dynamically generated version — WordPress’s dynamic robots.txt only serves when no physical file exists. After creating the physical file, verify by navigating to yoursite.com/robots.txt → the content should match the physical file exactly. If the dynamic version still appears, the file is not in the correct directory or was not saved correctly.
Crawl rate limiting — an advanced WordPress robots txt configuration: some aggressive crawlers can overwhelm a small WordPress site’s server by making too many requests too quickly. The Crawl-delay directive (not supported by Google but respected by Bing and many other crawlers) adds a minimum interval between requests: User-agent: bingbotnCrawl-delay: 10 — Bingbot waits at least 10 seconds between requests. For blocking specific bot user agents entirely: User-agent: AhrefsBotnDisallow: / — blocks the Ahrefs crawler from the entire site. This is appropriate for resource-constrained hosting where the SEO tool’s crawling is consuming bandwidth and CPU that would otherwise serve real visitors. Blocking major SEO tool crawlers (Ahrefs, Semrush, Moz) does not affect Google or Bing indexing — these are third-party analytics crawlers, not search engine crawlers. The WordPress robots txt configuration for blocking SEO tools is a hosting performance decision, not an SEO decision. Reviews from Google’s Search Central documentation confirm that Google does not support the Crawl-delay directive — use Google Search Console’s Crawl Rate control under Settings to adjust Google’s crawl rate rather than robots.txt.
Diagnosing Robots.txt Blocking Issues
An incorrectly configured WordPress robots txt file accidentally blocks pages from being crawled causes organic traffic loss that may not be noticed immediately — Google gradually drops blocked URLs from its index over weeks and months, making the connection to the robots.txt change non-obvious.
Test robots.txt rules using Google Search Console: Google Search Console → Settings → robots.txt tester (if available, depending on account features) allows testing any URL against the current robots.txt to see whether Google’s crawler would be allowed or blocked. Alternatively, the Google Rich Results Test and the URL Inspection tool in Search Console show the crawl status for specific URLs — if a URL is blocked by robots.txt, the tool shows this explicitly rather than reporting a generic crawl failure.
Manually check each WordPress robots txt Disallow rule against the site URL structure: review the current robots.txt rules against the site’s URL structure. A Disallow rule containing a broad pattern can accidentally match content URLs: Disallow: /category/ intended to block paginated category pages also blocks all category archive pages if implemented without limiting the rule to specific patterns. Test each Disallow rule against the content URLs on the site before deploying — the robots.txt tester in Google Search Console or third-party tools (ryte.com, technicalseo.com) can test specific URLs against the rules interactively. After any WordPress robots txt change, monitor Google Search Console → Coverage → Excluded tab → “Blocked by robots.txt” — this shows which URLs Google found blocked after the change, allowing quick identification of over-blocking before significant indexation damage occurs. Our guide on fixing WordPress session expired errors covers the Settings → Reading page that also contains the “Discourage search engines” checkbox — the most common source of accidental sitewide robots.txt blocking on WordPress sites.
The WordPress robots txt file should always include a Sitemap directive pointing to the XML sitemap — this is often omitted from robots.txt guides but is highly recommended by all major search engines. Adding Sitemap: https://yoursite.com/sitemap_index.xml at the end of the robots.txt (it can appear anywhere in the file but is conventionally placed at the end) provides crawlers with the sitemap URL without requiring them to discover it through Google Search Console submission. When the Rank Math or Yoast sitemap is enabled, the SEO plugin automatically adds the Sitemap directive to the robots.txt output — verify this is present by viewing yoursite.com/robots.txt in a browser and checking for the Sitemap line. If it is missing, add it manually through the SEO plugin’s robots.txt editor or the physical file.
International SEO and the WordPress robots txt file: for sites using hreflang to target multiple countries, the robots.txt file should allow crawling of all language-specific URLs. A common mistake is blocking language path prefixes (/fr/, /de/, /es/) in robots.txt to prevent indexation of draft translations, then forgetting to remove the block when the translations are ready. All published hreflang-annotated URLs must be crawlable for search engines to correctly attribute the content to the intended locale — a blocked hreflang URL is treated as if it does not exist, breaking the hreflang chain and potentially causing the wrong language version to rank in the wrong country’s search results. Verify all language URL paths against the robots.txt Disallow rules before publishing any multilingual content expansion to prevent international traffic loss from inadvertent crawl blocking.
Version control for the WordPress robots txt configuration prevents accidental blocking from persisting unnoticed. Add robots.txt to the site’s Git repository if using code-based version control — any change to the file is tracked with a commit message explaining why the change was made, and reverting an accidental change is a single git revert command. For sites using an SEO plugin editor rather than a physical file, document every robots.txt change in a site management log (date, who made the change, what changed, why) and set a Google Search Console alert for significant changes in the number of “Blocked by robots.txt” URLs. The combination of change documentation and Search Console monitoring provides the visibility needed to catch robots.txt misconfigurations before they cause significant organic traffic loss from crawl blocking.
The WordPress robots txt specification (from robotstxt.org) defines how directives are interpreted, including some non-obvious rules: Disallow directives are case-sensitive for most crawlers (Disallow: /Category/ does not block /category/ on case-sensitive servers); empty Disallow values (Allow: /) allow everything (the opposite of a block); and multiple User-agent groups in a single robots.txt are all read independently, meaning the most specific matching group applies to each crawler. Google provides a detailed specification of how it parses robots.txt including handling of conflicting rules, UTF-8 encoding requirements (use UTF-8 without BOM for the robots.txt file), and the maximum file size it reads (500 KB — files larger than this are only partially processed). Staying within these specification boundaries ensures that the WordPress robots txt configuration is interpreted as intended by all major search engine crawlers, not just by the browsers that display the file to human administrators.
WooCommerce-specific WordPress robots txt rules block checkout, cart, and account pages from being indexed by search engines — these pages should be accessible to crawlers (to allow for Googlebot to verify they are correctly structured) but should not appear in search results. The correct approach is noindex meta tags on these pages rather than robots.txt Disallow rules — blocking via robots.txt prevents Google from verifying the page structure and may prevent canonicalisation signals from being read. WooCommerce adds noindex headers to cart and checkout pages automatically. Verify this is working: Google Search Console → URL Inspection → enter the checkout page URL → confirm Google reports the page as “Discovered, not indexed” rather than “Not crawled” — “Not crawled” indicates robots.txt blocking, while “Discovered, not indexed” correctly indicates the noindex tag is preventing indexation while allowing crawling.
Monitoring crawl budget through Google Search Console’s Crawl Stats report (Settings → Crawl Stats) shows how many pages Google crawls per day, average response time, and the distribution of crawled vs uncrawled pages. A healthy WordPress robots txt configuration that correctly blocks low-value pages (admin pages, search result pages, utility URLs) while allowing all content pages shows a crawl budget efficiently used on indexable content. If the Crawl Stats report shows Google spending significant crawl budget on blocked URLs (returning 403 responses), the robots.txt Disallow rules are correctly blocking the URLs but Google is still requesting them — this suggests internal links to the blocked pages that should be removed so Google does not discover these URLs in the first place. Removing internal links to blocked pages eliminates the wasted crawl attempts, freeing the site’s crawl budget for indexable content. Related: WordPress Font Awesome.
