Crawling & Robots

Google Notifying Webmasters To Remove Noindex From Robots.txt Files

Now that Google is completely removing support for the noindex directive in robots.txt files, Google is sending out notifications to those that have such directives. On the morning of Jul 29, a number of folks in the SEO community began to recieve notifications from Google Search Console with the subject line, “remove “noindex” statements from …

Google Notifying Webmasters To Remove Noindex From Robots.txt Files Read More »

Google Adds JavaScript SEO Basics To Its Search Developer’s Guide

On July 18, Google added a JavaScript SEO basics section to its Search developer’s guide. It includes general descriptions of how Google processes JavaScript as well as some best practices. Not only is there an overview of how Googlebot crawls, renderers and indexes JavaScript web apps, the guide provides basic tips, which are accompanied by links to …

Google Adds JavaScript SEO Basics To Its Search Developer’s Guide Read More »

Bing Says It Is Improving Web Crawler Efficiency

The Principal program manager for Bing Webmaster Tools, Fabrice Canel has come out with an update on his and his team’s efforts to improve the efficiency of BingBot, the company’s webcrawler. Canel had a talk at SMX Advanced back in June where he announced an 18-month effort to improve BingBot, and this update is the follow up …

Bing Says It Is Improving Web Crawler Efficiency Read More »

Google Explains What “Crawl Budget” Means For Webmasters

A blog post, written by Gary Illyes, has been written, called What Crawl Budget Means for Googlebot, and explains what crawl budget is, how crawl rate limits work, and what crawl demand and what factors impact a site’s crawl budget. Gary explained that, for most sites,  crawl budget is something that we wouldn’t normally have to …

Google Explains What “Crawl Budget” Means For Webmasters Read More »

Google To Change Smartphone User-Agent Of Googlebot From iPhone To Android On April 18th

On the Google Webmaster blog, Google has announced that on April 18, 2016, they’ll be updating their user-agent for the smartphone Googlebot crawler.  The user agent is going to be moving away being an iPhone user-agent to an Android user-agent.  Even with this move, Google says that there should be no impact on 99% of all web …

Google To Change Smartphone User-Agent Of Googlebot From iPhone To Android On April 18th Read More »