fbpx

Focus on Doing what you do best and let us market your business using social media marketing

Robots.txt Noindex Removal & What It Means

At the beginning of July 2019, Google released a statement saying that they would no longer support the Noindex directive and it would retire all code that handles unsupported & unpublished rules.

So what is Noindex & how did it work?

The Noindex directive simply prevented pages from being indexed, combined with the disallow directive – it meant that pages were not crawled or indexed – therefore creating crawl efficiency.

This was a supported feature within robots.txt for over 10 years!

Why did Google cancel the support of Noindex?

Gary Illyes, from Google, explained that ‘after running analysis around the use of noindex in robots.txt, they found that a high number of sites were hurting themselves.’ He also stated that the update ‘is for the better ecosystem & those who used it correctly will find better ways to achieve the same thing.’

So what are the better ways?

In their official blog explaining it, Google outline several options that would help achieve the same noindex

  1. Noindex Robots Meta Tag: This is the most effective to remove URLs from the index but still allow crawling. These tags are supported in both the HTTP response headers & HTML & are achieve by adding a meta robots noindex directive on the actual page
  2. 404 & 410 HTTP status codes: These status codes basically inform Search Engines that they pages do not exist, which will lead to them being dropped from the index – once they have been crawled.
  3. Password Protection: Prevent Google from accessing a page by hiding it behind a login – this will generally lead to it being removed from the index.
  4. Disallow in Robots.txt: Blocking a page from being crawled will generally prevent it from being indexed, as Search Engines are only able to index pages they know about. Even though a page may be indexed due to links pointing to it – Google will aim to make the pages less visible in the SERPs.
  5. Search Console Remove URL Tool: The URL removal tool in GSC is a quick and easy way to temporarily remove a URL from the SERPs.

Next Steps?

Identify & Monitor your NoIndex Robots.txt pages ahead of the September 1st update. Simply do a crawl, using your chosen crawler and check which of your pages are currently being noindexed and how: through header, meta tags or robots.txt.

From there you are determine what changes to make, but you best hurry, as it is now a month to the deadline and Google are emailing all webmasters via Google Search Console to tell them to remove all noindex statements from their robots.txt file.

For now, Googlebots are obeying the noindex directive but only until September 1st.

Not sure where to begin? Have no idea if your site is affected by this? Get in touch and we will do a free audit on your site to determine if your site will be affected and how we can help to update.

Leave a Comment

Your email address will not be published. Required fields are marked *

8

0

#YouTube is everywhere, and it isn't just for watching cat videos! It is becoming a vital part of the #marketingmix. No matter what your product or service is - incorporating video into your strategy is a must! Read our free magazine ☝☝☝☝#linkinbio . . . . . . . #videomarketing #video #youtube #digitalmarketing #magazine #issuu #socialmedia #digitalstrategy #smallbusiness #localbusiness

7

0

#TuesdayThoughts . . Have you updated your #websites robots.txt file? From September 1st, #Google will be retiring all code that handles unsupported and unpublished rules in robots.txt - this includes the use of the #noindex directive.Want to know more? Click the #linkinbio to read our latest #blog 💻💻 . . . . . . . . . . . . . #blog #digitalmarketing #searchengineoptimization #seo #websites #optimisation #robots #noindex #disallow #coding #technicalseo

7

0

🤓#ThursdayThoughts... Nearly 25 years ago, the internet was mearly a gimmick, you did some research and then you were done. Nowadays, we use the #internet for shopping, socialising, ordering food and more. With so much data passing through - how do you spend your time & avoid the #digitalburnout? . . . . . . . #thursdaythoughts #socialmedia #digitalmarketing #agencylife #instagram #digtal #entrepreneur #smallbusiness #agencykent

8

0

🤔Thought of the day🤔How long should your content be? That is almost the same as asking "How long is a piece of string". #Content should not be prepped on the basis of a word count. In order to #rank on #Google and other #SERPs, you need to answering the #searchintent first. If that takes 50 words or if that takes 2000 words, as long as you are answering the question then you will rank & obtain plausible #backlinks.How do you plan your content & #blogs? . . . . . . . . . . . . . #webdesign #contentmarketing #agencylife #smallbusiness #blogs #blogger #digitalmarketing #agencykent #webtexlimited #entrepreneur #tuesdaythoughts #quoteoftheday #QOTD #research #wordcount #searchintent #google #algorithm

5

0

For 67 years #NelsonMandela fought for the human rights of South African Citizens and remained a humble man. This #MandelaDay what will you be doing for #67minutes? To all our South African Customers, we hope you have a wonderful day, giving back to those in need. . . . . . . . . #MandelaDay #HumanRights #Charity #SouthAfrica #Awareness #Volunteer #agency #digitalmarketing

12

4

📆#WorldEmojiDay📆 #Emojis were originally created in 1999 as a way to save on data when messaging / paging 📲(remember that!?) others. It wasn't until 2011 when it became widely adopted on all devices ⌚📱💻⌨🖥. In 7 short years, emoji has become the fastest growing language & is seen as a huge boost in #marketingstrategies! What is your favourite emoji to use? Drop it 👇👇👇 . . . . . . . . . . . #worldemojiday #socialmedia #engagement #agencylife #innovation #digitalmarketing #digitallanguage #awarenessdays #agencykent #contentmarketing

5

2

#SmallBusinesses should embrace #socialmedia as a testing ground for innovation and new ideas instead of moving away from it. From building excitement over new products to getting your customers engaged in your community. #SocialMedia should be all about community not followers! . . . . . . . . #engagement #socialmedia #agencylife #smallbusiness #digitalmarketingagency #agencyinkent #ashfordkent #entrepreneur

16

2

Monday means one thing: #MarketingMondayWe have been scouring the news channels and sifting through #CWC2019 Final news to bring you the top stories that have caught our eye today: ✅The National Lottery get real in their newest campaign showing that anyone can win ✅High streets suffer another slump in June as footfall declines by 2.9% ✅Camden Town Brewery launch their biggest campaign yet: #FreshAsHellWhat are you reading today? How do you keep on top of #marketing and #branding news stories? Let us know below 👇 . . . . . . . . #agencylife #marketing #digitalmarketing #smallbusiness #marketingmonday #mondaymotivation #ashford #kent #entrepreneur #socialmedia #seo #searchengineoptimization

8

0

With #Google announcing a core update to its #algorithm and #Facebook announcing changes to their business pages, are you keeping up with the times or does your #smallbusiness need an update?⠀⠀⠀⠀⠀⠀⠀⠀⠀ Our team of experts are ready to bring your business into the future without any hassle!⠀⠀⠀⠀⠀⠀⠀⠀⠀ Click the #link in bio to get in touch. (Plus there are some amazing downloads in there too ;) )⠀⠀⠀⠀⠀⠀⠀⠀⠀ .⠀⠀⠀⠀⠀⠀⠀⠀⠀ .⠀⠀⠀⠀⠀⠀⠀⠀⠀ .⠀⠀⠀⠀⠀⠀⠀⠀⠀ .⠀⠀⠀⠀⠀⠀⠀⠀⠀ .⠀⠀⠀⠀⠀⠀⠀⠀⠀ .⠀⠀⠀⠀⠀⠀⠀⠀⠀ .⠀⠀⠀⠀⠀⠀⠀⠀⠀ #dontbeadodo #onlinemarketing #digitalmarketing #viralmarketing #socialmediamarketing #contentcreation #socialmediamanagement #emailmarketing #facebookadvertising #instagramadvertising #branding #brandbuilding #graphicdesign #seo #sem #searchengineoptimisation #searchmarketing #leadgeneration #leadgen #newleads #b2bmarketing #b2b #seomarketing #googlemarketing #google

Contact Us

  • Suite 31 Letraset Building
  • Wotton Road Kingsnorth Industrial Estate
  • Ashford Kent TN23 6LN
  • 01233-222239

Latest News

Scroll to Top