fbpx

Focus on Doing what you do best and let us market your business using social media marketing

Robots.txt Noindex Removal & What It Means

At the beginning of July 2019, Google released a statement saying that they would no longer support the Noindex directive and it would retire all code that handles unsupported & unpublished rules.

So what is Noindex & how did it work?

The Noindex directive simply prevented pages from being indexed, combined with the disallow directive – it meant that pages were not crawled or indexed – therefore creating crawl efficiency.

This was a supported feature within robots.txt for over 10 years!

Why did Google cancel the support of Noindex?

Gary Illyes, from Google, explained that ‘after running analysis around the use of noindex in robots.txt, they found that a high number of sites were hurting themselves.’ He also stated that the update ‘is for the better ecosystem & those who used it correctly will find better ways to achieve the same thing.’

So what are the better ways?

In their official blog explaining it, Google outline several options that would help achieve the same noindex

  1. Noindex Robots Meta Tag: This is the most effective to remove URLs from the index but still allow crawling. These tags are supported in both the HTTP response headers & HTML & are achieve by adding a meta robots noindex directive on the actual page
  2. 404 & 410 HTTP status codes: These status codes basically inform Search Engines that they pages do not exist, which will lead to them being dropped from the index – once they have been crawled.
  3. Password Protection: Prevent Google from accessing a page by hiding it behind a login – this will generally lead to it being removed from the index.
  4. Disallow in Robots.txt: Blocking a page from being crawled will generally prevent it from being indexed, as Search Engines are only able to index pages they know about. Even though a page may be indexed due to links pointing to it – Google will aim to make the pages less visible in the SERPs.
  5. Search Console Remove URL Tool: The URL removal tool in GSC is a quick and easy way to temporarily remove a URL from the SERPs.

Next Steps?

Identify & Monitor your NoIndex Robots.txt pages ahead of the September 1st update. Simply do a crawl, using your chosen crawler and check which of your pages are currently being noindexed and how: through header, meta tags or robots.txt.

From there you are determine what changes to make, but you best hurry, as it is now a month to the deadline and Google are emailing all webmasters via Google Search Console to tell them to remove all noindex statements from their robots.txt file.

For now, Googlebots are obeying the noindex directive but only until September 1st.

Not sure where to begin? Have no idea if your site is affected by this? Get in touch and we will do a free audit on your site to determine if your site will be affected and how we can help to update.

Leave a Comment

Your email address will not be published. Required fields are marked *

9

0

It's Friday. We have all been there. The motivation starts to lag during the day & your ever-growing list seems to get more tasks added to it and nothing disappears! This #fightprocrastinationday we will be sharing our top tips to remain productive & smash that to-do list even if it is a Friday!Watch our #stories throughout the day to find our tips! πŸ‘†πŸ‘†πŸ‘† . . . . . . . . . #socialmedia #awareness #productivity #freelancer #entrepreneur #smallbusiness #lists #bulletjournal #digitalmarketing

5

0

#TuesdayThoughts . . You put all the work in & #Google is still not playing fair. This quote from Wendy Piersall says it all - unless people love visiting you then Google won't even look at your direction.What do you think? Let us know in the comments below πŸ‘‡πŸ‘‡πŸ‘‡πŸ‘‡ . . . . . . . . #marketingquotes #instaquotes #smallbusiness #agencylife #meme #ashford #kentdigitalagency

5

0

Do you think #branding is important? . . . Most #smallbusinesses don't think about branding holistically - instead they think branding is just a logo and colours. In fact it is so much more. It is the whole customer experience - from your #socialmedia to your #website, even your #customerservice!! Click the #linkinbio to read more about why having a strong branding strategy is important! πŸ‘†πŸ‘†πŸ‘†πŸ‘† . . . . . . . . . . . #blog #branding #design #smallbusiness #tuesdaythoughts #digitalmarketing #ashfordkent #ashforddigitalagency #agencylife #webdesign

8

0

#ThursdayThoughts . . As we wind down the short week into the weekend, this quote from Jay Baer got us thinking about how important #socialmedia is for expanding your reach beyond your traditional marketing audience. Your content may be perfect but with social media you can send that content further than before!Do you think #socialmedia is important? Or is it a frilly add-on? . . . . . . . . #thoughts #inspirationalquote #marketing #digitalmarketing #socialmediamarketing #contentmarketing #ashfordkent #kent #digtalmarketingagency

7

0

Do you trust #socialmedia? . . . Trust in social media platforms have fallen by 60%, on #Facebook alone that trust fell by 66%. This is the time for #smallbusinesses to make a stand and start building communities on their pages. Let's not just push sales - let's start conversations!Click the #linkinbio for our latest blog where we talk social media #trends, how users are changing the way they use social media & what that means for #business! πŸ‘†πŸ‘†πŸ‘†πŸ‘†πŸ‘† . . . . . . . . #agencylife #blog #socialmedia #socialmediastrategy #communities #smallbusiness #digitalmarketingagency #ashfordagency #ashfordkent #webtexlimited #wednesdaywisdom

1

0

Over 3.5 billion people are now using #socialmedia in one way or another. That is 46% of the total global population 🀯That is why #socialmedia needs to become a key part of your strategy! Do you have a solid social media #strategy for your business?Click the link above to get in touch for a tailored solution for your business πŸ‘†πŸ‘†πŸ‘†πŸ‘†πŸ‘†πŸ‘† . . . . . . . . . #socialmedia #agency #digitalmarketing #ashfordkent #kent #smallbusiness #strategy #smallbusinessadvice

7

0

Everyday we are bombarded by #advertising! . . But did you know that #programmatic ads now make up 62% of all digital display ad buying? The use of #AI to create more targeted campaigns is increasing to get more #ROI.Do you use programmatic ads or are you thinking about implementing this important strategy? . . . . . . . . #TipsTuesday #digitalmarketing #digitalads #advertising #adman #smallbusinessadvice #ashfordkent #kent #digitalmarketingagency #contentmarketing #digitalmarketingstrategy #strategy

4

0

Visuals. Videos. Voice . . . 3 important elements that will bring your business closer to your customers. As #technology progresses faster and faster, smartphone advances into better cameras, allowing users to share images on the go! How important are images in your #contentmarketing strategy? Do you think about them as much as your actual content or are they a last minute thought? . . . . . . . . . #TipsTuesday #agencylife #socialmedia #contentmarketing #entrepreneur #smallbusiness #digitalmarketingagency #ashfordkent #kent #digitalmarketing #smallbusinesstips

10

0

#TipsTuesday . . Hashtags are an important part of #socialmedia and how they resonate with your brand. Social media is about creating a community rather than just pushing products. Do you agree? . . . . . #socialmediamarketing #digitalmarketing #agencylife #ashfordkent #ashford #smallbusinessadvice

Contact Us

  • Suite 31 Letraset Building
  • Wotton Road Kingsnorth Industrial Estate
  • Ashford Kent TN23 6LN
  • 01233-222239

Latest News

Scroll to Top