My site is not indexed, what do I do?

In the past 5+ years I have seen this question many times. The first thing I always check is the homepage source code, right click on the page and view source. Look in the head section for the word NOINDEX, you can just use the find function. if you see that then you need to get it removed.

Next area to check is the domain.com/robots.txt file to nee is you are preventing the search engines from crawling your site. Look for:

User-agent: *
Disallow: /

The “User-agent: *” means this section applies to all robots. The “Disallow: /” tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

  • robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
  • the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use.

So don’t try to use /robots.txt to hide information.

Conclusion:

When you are seeing your site not indexed, these are usually the two factors. I have seen little tiny to major 100 million page sites get de-indexed from these two factors. These two forms of preventing a page form being indexed is a huge issue that has cost many a job and many companies money.

Comments are closed.