How to Stop Search Engines From Indexing Your WordPress Site (3 Easy Methods)

Hey there! Are you concerned about Google and other search engines indexing pages on your WordPress site that you want to keep private? Maybe you have a site under development, a staging site you don‘t want publicly accessible, or membership content that should be hidden from non-logged-in users.

Navi.

Don‘t worry, you‘re not alone. In fact, a recent study found that over 20% of WordPress sites have at least one page that is unintentionally being indexed by search engines when it shouldn‘t be. This can lead to duplicate content issues, SEO problems, and even legal concerns in some cases.

Fortunately, there are several ways you can stop search engines from crawling and indexing all or parts of your WordPress site. In this guide, we‘ll walk you through three easy methods with step-by-step instructions:

Using WordPress‘ built-in "Discourage search engines" setting
Noindexing individual pages and posts with an SEO plugin
Password protecting your entire site

We‘ll also compare the pros and cons of each method so you can choose the best option for your specific needs.

First though, let‘s cover a little background on how search engines discover and index content, and why you might want to prevent that in certain cases.

How Search Engine Crawling and Indexing Works

Search engines like Google use automated programs called "crawlers" or "spiders" to continuously scan the web for new and updated content. These crawlers discover pages by following links from one page to another.

When a crawler finds a new page, it processes and analyzes the content and adds it to the search engine‘s index. This is a massive database of all the discovered web pages that can appear in search results.

However, there are some situations where you may not want your WordPress pages to show up in search results, such as:

A development or staging site that‘s not ready for public viewing
A private internal site for company use only
Membership pages that should only be accessible to logged-in users
Duplicate versions of pages that could cause SEO issues
Legal documents or content with sensitive information

So how can you tell search engines that they shouldn‘t index certain pages or your entire site? That‘s where the following methods come in.

Method 1: Using WordPress‘ Built-in "Discourage Search Engines" Setting

Did you know WordPress has a built-in option to discourage search engines from indexing your site? Yep, it‘s tucked away in the Reading settings.

Here‘s how to enable it:

From your WordPress dashboard, go to Settings > Reading.
Scroll down to the "Search Engine Visibility" section.
Check the box next to "Discourage search engines from indexing this site."
Click the "Save Changes" button.

When you enable this setting, WordPress does two things:

Adds a noindex meta tag to your site‘s <head> section telling search engines not to index your pages:

<meta name=‘robots‘ content=‘noindex,follow‘ />

Modifies your site‘s robots.txt file to disallow all crawling:

User-agent: *
Disallow: /

This method is quick and easy to implement – just check a box and you‘re done. However, it‘s not foolproof for a few reasons:

It relies on search engines honoring your request not to index your content. Most will, but some may ignore it.
It applies sitewide, so you can‘t selectively choose which pages to noindex. It‘s all or nothing.
If your site was already indexed before you enabled this setting, it can take some time for search engines to recrawl your pages and remove them from the index.
It doesn‘t actually prevent access to your pages, it just tells search engines not to include them in results. The pages are still publicly accessible to anyone with the URL.

So while this method can be a quick fix in some cases, for more granular control and complete blocking of access, you‘ll want to use method 2 or 3 below.

Method 2: Noindexing Individual Pages and Posts With an SEO Plugin

If you want to noindex specific pages or posts rather than your entire site, an SEO plugin is the way to go. Most popular WordPress SEO plugins like Yoast SEO, All in One SEO Pack, and Rank Math make it easy to noindex individual pieces of content.

Here‘s how the process works in Yoast SEO:

Edit the post or page you want to noindex.
Scroll down to the Yoast SEO meta box and click the Advanced tab.
Under "Allow search engines to show this Post in search results?", select "No".
Update or publish the post/page.

Yoast will add this meta robots tag to the page‘s HTML:

<meta name="robots" content="noindex" />

This tells search engines not to index this specific page/post, while still allowing them to crawl it and follow any links to other pages.

Some other options you have with the robots meta tag:

noindex,nofollow: Tells search engines not to index the page or follow any links on the page. Use this to completely isolate a page.
noindex,follow: Tells search engines not to index the page but they can still follow links on the page to discover other content.
index,nofollow: Allows indexing of the page but tells search engines not to follow or pass authority through links on the page.

Besides the robots meta tag, most SEO plugins also give you control over what shows up in search results for your pages via other meta tags like the meta description, SEO title, and social media appearance.

The main benefit of using an SEO plugin to noindex content is the granular control on a page-by-page basis and the ease of implementation. You don‘t have to edit any code, just toggle an option in the WordPress editor.

However, like WordPress‘ built-in option, this method doesn‘t fully block access to the page, it just removes it from search results.

Also, depending on the SEO plugin you use, it may take some time for search engines to recrawl the page and process the noindex directive, especially if the page was already indexed before.

Method 3: Password Protecting Your Entire WordPress Site

Looking for a surefire way to prevent search engines AND regular visitors from accessing any part of your WordPress site? Password protection is the solution.

There are two main ways to implement password protection on a WordPress site:

Via your web hosting control panel (like cPanel)
Using a WordPress plugin

Password Protecting via cPanel

If your WordPress hosting uses cPanel, you can password protect your entire site with just a few clicks. Here‘s how:

Log in to your cPanel dashboard
Find the "Directory Privacy" option under the Files section and click it
Find the directory for your WordPress site (usually public_html) and click the Edit button
Check the box for "Password protect this directory"
Enter a name for the protected directory (can be anything)
Click Save
On the next page, create a username and password that will be required to access the site
Click "Save" again

Now when anyone tries to access your WordPress site, they‘ll be prompted to enter the username and password you just created. This includes both human visitors and search engine bots.

Until the correct credentials are entered, your site is completely inaccessible, so there‘s no way for search engines to crawl or index any of the content.

To remove the password protection, simply edit the directory in cPanel again and uncheck the "Password protect this directory" box.

Password Protecting with a Plugin

Don‘t have cPanel or prefer to handle password protection within WordPress? There are several free and paid plugins that can lock down your entire site:

Password Protected
SiteWide Password Protection
Password Protect WordPress
Simple Password Protect

For this example, we‘ll use the free Password Protected plugin.

From your WordPress dashboard, go to Plugins > Add New
Search for "Password Protected"
Install and activate the plugin
Go to Settings > Password Protected
Check the "Enabled" box
Enter a password in the "New Password" field
Customize the protection message if desired
Click "Save Changes"

Like the cPanel method, this will require a password to access any part of your WordPress site, both for regular visitors and search engine crawlers.

In the plugin settings, you can choose to allow logged-in users or administrators to bypass the password prompt, as well as allow access to RSS feeds or certain IP addresses.

The main advantage of using a plugin vs cPanel is the ability to customize permissions and password protect only certain parts of your site if desired.

However, the cPanel method may be simpler and faster to implement sitewide password protection.

Pros and Cons of Each Method

To help you decide which method is right for your needs, here‘s a quick comparison table:

Method	Pros	Cons
Discourage search engines setting	Quick and easy to enable sitewide	Not 100% effective, search engines may still index pages
Noindex with SEO plugin	Granular control, can noindex specific pages	Doesn‘t fully block access, just removes from search results
Password protection – cPanel	Complete lockdown of entire site, blocks all access	All or nothing, can‘t customize permissions
Password protection – plugin	Can customize permissions, password protect specific areas	More setup involved than cPanel method

For most scenarios, we recommend using an SEO plugin to noindex specific pages that you want to keep out of search results, combined with password protection (either via cPanel or a plugin) for any content that needs to be completely blocked from public access.

The "discourage search engines" setting can work in a pinch for a development or staging site, but it‘s not a long-term solution.

Tips and Best Practices for Blocking Search Engine Indexing

Before we wrap up, here are a few tips and best practices to keep in mind when blocking indexing on your WordPress site:

Use robots.txt carefully: Blocking crawlers in robots.txt is not always reliable, as some search engines may ignore it. It‘s better to use meta tags like noindex and nofollow.
Understand the difference between noindex and nofollow: Noindex tells search engines not to include a page in search results, while nofollow tells them not to crawl links on a page.
Consider your specific needs: Do you need to noindex your entire site or just certain pages? Is it sufficient to keep pages out of search results or do you need to completely block public access with a password?
Avoid using "discourage search engines" long-term: It‘s not a foolproof solution and can cause issues if accidentally left enabled.
Regularly audit your site‘s indexing: Use tools like Google Search Console and Screaming Frog to check which pages are indexed and identify any unintended indexing.
Don‘t forget to remove noindex and password protection when ready: When you‘re ready for pages to be indexed again, disable any blocking measures you previously implemented.

Frequently Asked Questions

Before we close out this guide, let‘s address a few common questions about blocking search engine indexing in WordPress:

Can I noindex my entire site except for a few pages?

Yes, you can use an SEO plugin to manually noindex all pages except the ones you want indexed. However, this can be time-consuming for larger sites. In that case, it‘s usually better to noindex the entire site using the "discourage search engines" setting or robots.txt, then use the plugin to manually remove the noindex tag on pages you want indexed.

Will noindexed pages still appear in Google Search Console?

Yes, noindexed pages will still show up in Google Search Console as "Excluded" pages. This is normal and doesn‘t mean the page is indexed in search results. You can use the Coverage report to see all pages that Google has crawled but not indexed due to a noindex directive.

I password protected my site but it still shows up in search results. Why?

If your site was already indexed by search engines before you implemented password protection, those pages may still linger in search results for a while. Search engines will eventually recrawl the pages, encounter the password prompt, and remove them from the index. To speed up the process, you can try manually requesting removal via Google Search Console.

Wrapping Up

Whew, that was a lot of information! We hope this guide has armed you with the knowledge and tools you need to keep unwanted pages out of search engine indexes.

In most cases, using an SEO plugin to noindex specific pages, or implementing password protection for sensitive content, is the way to go.

The main thing is to consider your unique scenario and choose the method (or combination of methods) that gives you the right balance of control, security, and simplicity.

Now go forth and exercise your right to not be indexed! As always, if you have any questions or tips of your own to share, leave a comment below.