fbpx

Focus on Doing what you do best and let us market your business using social media marketing

Robots.txt Noindex Removal & What It Means

At the beginning of July 2019, Google released a statement saying that they would no longer support the Noindex directive and it would retire all code that handles unsupported & unpublished rules.

So what is Noindex & how did it work?

The Noindex directive simply prevented pages from being indexed, combined with the disallow directive – it meant that pages were not crawled or indexed – therefore creating crawl efficiency.

This was a supported feature within robots.txt for over 10 years!

Why did Google cancel the support of Noindex?

Gary Illyes, from Google, explained that ‘after running analysis around the use of noindex in robots.txt, they found that a high number of sites were hurting themselves.’ He also stated that the update ‘is for the better ecosystem & those who used it correctly will find better ways to achieve the same thing.’

So what are the better ways?

In their official blog explaining it, Google outline several options that would help achieve the same noindex

  1. Noindex Robots Meta Tag: This is the most effective to remove URLs from the index but still allow crawling. These tags are supported in both the HTTP response headers & HTML & are achieve by adding a meta robots noindex directive on the actual page
  2. 404 & 410 HTTP status codes: These status codes basically inform Search Engines that they pages do not exist, which will lead to them being dropped from the index – once they have been crawled.
  3. Password Protection: Prevent Google from accessing a page by hiding it behind a login – this will generally lead to it being removed from the index.
  4. Disallow in Robots.txt: Blocking a page from being crawled will generally prevent it from being indexed, as Search Engines are only able to index pages they know about. Even though a page may be indexed due to links pointing to it – Google will aim to make the pages less visible in the SERPs.
  5. Search Console Remove URL Tool: The URL removal tool in GSC is a quick and easy way to temporarily remove a URL from the SERPs.

Next Steps?

Identify & Monitor your NoIndex Robots.txt pages ahead of the September 1st update. Simply do a crawl, using your chosen crawler and check which of your pages are currently being noindexed and how: through header, meta tags or robots.txt.

From there you are determine what changes to make, but you best hurry, as it is now a month to the deadline and Google are emailing all webmasters via Google Search Console to tell them to remove all noindex statements from their robots.txt file.

For now, Googlebots are obeying the noindex directive but only until September 1st.

Not sure where to begin? Have no idea if your site is affected by this? Get in touch and we will do a free audit on your site to determine if your site will be affected and how we can help to update.

Leave a Comment

Your email address will not be published. Required fields are marked *

2

0

Avoid The Burnout . . . Burnout is a key topic in recent days with 1 in 6 workers experiencing a #mentalhealth problem in any given week! How do you cope with #stress or do you just try keep busy to ignore it? . . . . . . #WorkLifeWeek #MentalHealthAwareness #AgencyLife #entrepreneur #smallbusiness #Stress #digitalmarketing

2

0

#TuesdayThoughts . . . When thinking about your brand what comes to mind? Do you stand out against your competitors or are your blending in? Set your brand apart with a tailored #digital strategy that includes #socialmedia, #SEO & #webdesign.What are you waiting for? Click the #linkinbio and get in touch today . . . . . . #agencylife #smallbusiness #digitalmarketing #strategy #ashfordkent #entrepreneur #branding #socialmedia #contentmarketing

9

1

#MondayMotivation . . . It should not be a balancing act to try manage work & life. This #WorkLifeWeek we will be looking at why it is important for employers and employees to manage expectations and stopping those from reaching burnout.How do you avoid work burnout? . . . . . . . #worklifeweek #worklifebalance #agencylife #mentalhealth #mentalhealthawareness #digitalmarketing #socialmedia #smallbusiness #entrepreneur

4

0

#MondayMotivation . . . There is only 99 days until the end of the year! Where has 2019 gone? So we are asking, now that we are in double figures, what are your goals to smash the end of the year? How will you change in Q4? Will you land a new client or will you focus on making your business offering stronger?Let us know in the comments below! . . . . . . . . . . #mondaymotivation #agencylife #smallbusiness #entrepreneur #digitalmarketing #agencykent #smallbusiness #AshfordKent

7

0

#SmallBusinessTips . . . . Is your #GoogleBusiness listing the best it can be? Do you update it if something happens? Do you advertise specials or post photos? Most #smallbusinesses don't understand the importance of these listings and often think of it as the last minute thing - almost like every other listing post on directory websites.Not sure where to start or if your listing is effective? Get in touch with our amazing team & we can help take your #GMB listing from zero to hero & watch your local area start finding your business instead of your competitors! ๐Ÿ‘†๐Ÿ‘†๐Ÿ‘†๐Ÿ‘† Click the Link! ๐Ÿ‘†๐Ÿ‘†๐Ÿ‘†๐Ÿ‘† . . . . . . . . . . . #thursdaythoughts #smallbusinessadvice #smallbusinessowner #google #seo #searchengineoptimization #entrepreneur #digitalmarketing #webdeveloper #agencylife #developers #socialmedia

6

0

#ThursdayThoughts . . . #coding, #computerscience - it's all just for nerds who enjoy sitting in dark rooms right? Well in recent years, the rise in data science, computer programming and app development - a qualification or skills in any #STEM subject makes you one of the coolest people (and most in demand) people out there! As university students head back for either their first year or final year, what are your thoughts on #STEM? Do you wish you had studied something like this back when you did uni? Or are you learning new skills now as the #digitalrevolution evolves? Let us know your thoughts below . . . . . . . . . . . . . #Smallbusiness #womenintech #technicalseo #stem #digitalmarketing #agencyinkent #seo #coding #htmlcssphp #developers #engineers

9

0

It's Friday. We have all been there. The motivation starts to lag during the day & your ever-growing list seems to get more tasks added to it and nothing disappears! This #fightprocrastinationday we will be sharing our top tips to remain productive & smash that to-do list even if it is a Friday!Watch our #stories throughout the day to find our tips! ๐Ÿ‘†๐Ÿ‘†๐Ÿ‘† . . . . . . . . . #socialmedia #awareness #productivity #freelancer #entrepreneur #smallbusiness #lists #bulletjournal #digitalmarketing

5

0

#TuesdayThoughts . . You put all the work in & #Google is still not playing fair. This quote from Wendy Piersall says it all - unless people love visiting you then Google won't even look at your direction.What do you think? Let us know in the comments below ๐Ÿ‘‡๐Ÿ‘‡๐Ÿ‘‡๐Ÿ‘‡ . . . . . . . . #marketingquotes #instaquotes #smallbusiness #agencylife #meme #ashford #kentdigitalagency

5

0

Do you think #branding is important? . . . Most #smallbusinesses don't think about branding holistically - instead they think branding is just a logo and colours. In fact it is so much more. It is the whole customer experience - from your #socialmedia to your #website, even your #customerservice!! Click the #linkinbio to read more about why having a strong branding strategy is important! ๐Ÿ‘†๐Ÿ‘†๐Ÿ‘†๐Ÿ‘† . . . . . . . . . . . #blog #branding #design #smallbusiness #tuesdaythoughts #digitalmarketing #ashfordkent #ashforddigitalagency #agencylife #webdesign

Contact Us

  • Suite 31 Letraset Building
  • Wotton Road Kingsnorth Industrial Estate
  • Ashford Kent TN23 6LN
  • 01233-222239

Latest News


Important
This site makes use of cookies which may contain tracking information about visitors. By continuing to browse this site you agree to our use of cookies.

Scroll to Top