{"id":1615,"date":"2019-07-08T14:36:06","date_gmt":"2019-07-08T10:36:06","guid":{"rendered":"https:\/\/blog.dezigneronline.net\/?p=1615"},"modified":"2019-07-08T14:37:48","modified_gmt":"2019-07-08T10:37:48","slug":"google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt","status":"publish","type":"post","link":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/","title":{"rendered":"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt"},"content":{"rendered":"\n<p>Google announced on July 02, 2019 that they are not going to support crawl-delay, nofollow, and noindex directives in robots.txt file.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\"><p>While open-sourcing our parser library, we analyzed the usage of robots.txt rules. In particular, we focused on rules unsupported by the internet draft, such as crawl-delay, nofollow, and noindex. Since these rules were never documented by Google, naturally, their usage in relation to Googlebot is very low. Digging further, we saw their usage was contradicted by other rules in all but 0.001% of all robots.txt files on the internet. These mistakes hurt websites&#8217; presence in Google&#8217;s search results in ways we don\u2019t think webmasters intended.<\/p><cite>Google<\/cite><\/blockquote>\n\n\n\n<p>There are some alternative ways to use the proper directives.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\"><p><strong><a href=\"https:\/\/support.google.com\/webmasters\/answer\/93710\">Noindex<\/a>\u00a0in robots meta tags:<\/strong>\u00a0Supported both in the HTTP response headers and in HTML, the noindex directive is the most effective way to remove URLs from the index when crawling is allowed.<\/p><p><strong><a href=\"https:\/\/en.wikipedia.org\/wiki\/HTTP_404\">404 and 410 HTTP status codes<\/a>:<\/strong>\u00a0Both status codes mean that the page does not exist, which will drop such URLs from Google&#8217;s index once they&#8217;re crawled and processed.<\/p><p><strong>Password protection:<\/strong>\u00a0Unless markup is used to indicate\u00a0<a href=\"https:\/\/developers.google.com\/search\/docs\/data-types\/paywalled-content\">subscription or paywalled content<\/a>, hiding a page behind a login will generally remove it from Google&#8217;s index.<\/p><p><strong>Disallow in robots.txt:\u00a0<\/strong>Search engines can only index pages that they know about, so blocking the page from being crawled usually means its content won\u2019t be indexed.\u00a0 While the search engine may also index a URL based on links from other pages, without seeing the content itself, we aim to make such pages less visible in the future.<\/p><p><strong><a href=\"https:\/\/support.google.com\/webmasters\/answer\/1663419\">Search Console Remove URL tool<\/a>:<\/strong>\u00a0The tool is a quick and easy method to remove a URL temporarily from Google&#8217;s search results.<\/p><cite>Google<\/cite><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>Google announced on July 02, 2019 that they are not going to&hellip;<\/p>\n","protected":false},"author":1,"featured_media":1616,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[147,189],"tags":[216,215,214],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v19.2.1 (Yoast SEO v21.8) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt - Web Design Dubai<\/title>\n<meta name=\"description\" content=\"Google announced on July 02, 2019 that they are not going to support crawl-delay, nofollow, and noindex directives in robots.txt file.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt\" \/>\n<meta property=\"og:description\" content=\"Google announced on July 02, 2019 that they are not going to support crawl-delay, nofollow, and noindex directives in robots.txt file.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/\" \/>\n<meta property=\"og:site_name\" content=\"Umair Hashmi\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/m.umair.hashmi\" \/>\n<meta property=\"article:author\" content=\"M.Umair.Hashmi\" \/>\n<meta property=\"article:published_time\" content=\"2019-07-08T10:36:06+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-07-08T10:37:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dezigneronline.net\/blog\/wp-content\/uploads\/2019\/07\/google-robots-noindex-nofollow.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"935\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Umair Hashmi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@UmairHashmi\" \/>\n<meta name=\"twitter:site\" content=\"@umairhashmi\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Umair Hashmi\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":[\"Article\",\"NewsArticle\"],\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/\"},\"author\":{\"name\":\"Umair Hashmi\",\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/person\/d3fb8d27a63cee89b22d31c87cb1ba5c\"},\"headline\":\"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt\",\"datePublished\":\"2019-07-08T10:36:06+00:00\",\"dateModified\":\"2019-07-08T10:37:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/\"},\"wordCount\":299,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#organization\"},\"keywords\":[\"nofollow\",\"noindex\",\"robots\"],\"articleSection\":[\"News\",\"SEO\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/#respond\"]}],\"copyrightYear\":\"2019\",\"copyrightHolder\":{\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/\",\"url\":\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/\",\"name\":\"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt - Web Design Dubai\",\"isPartOf\":{\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#website\"},\"datePublished\":\"2019-07-08T10:36:06+00:00\",\"dateModified\":\"2019-07-08T10:37:48+00:00\",\"description\":\"Google announced on July 02, 2019 that they are not going to support crawl-delay, nofollow, and noindex directives in robots.txt file.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.dezigneronline.net\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#website\",\"url\":\"https:\/\/www.dezigneronline.net\/blog\/\",\"name\":\"Umair Hashmi\",\"description\":\"Digital Marketer &amp; UI\/UX Designer\",\"publisher\":{\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dezigneronline.net\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#organization\",\"name\":\"Umair Hashmi\",\"url\":\"https:\/\/www.dezigneronline.net\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.dezigneronline.net\/blog\/wp-content\/uploads\/2022\/09\/android-chrome-512x512-1.png\",\"contentUrl\":\"https:\/\/www.dezigneronline.net\/blog\/wp-content\/uploads\/2022\/09\/android-chrome-512x512-1.png\",\"width\":512,\"height\":512,\"caption\":\"Umair Hashmi\"},\"image\":{\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/m.umair.hashmi\",\"https:\/\/twitter.com\/umairhashmi\",\"https:\/\/www.instagram.com\/m.umair.hashmi\",\"https:\/\/www.linkedin.com\/in\/umairhashmi\",\"https:\/\/www.pinterest.com\/umairhashmi\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/person\/d3fb8d27a63cee89b22d31c87cb1ba5c\",\"name\":\"Umair Hashmi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/7cfcbae520bdde6c9354158b64eb26e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/7cfcbae520bdde6c9354158b64eb26e4?s=96&d=mm&r=g\",\"caption\":\"Umair Hashmi\"},\"description\":\"Digital Marketer &amp; UI\/UX Designer\",\"sameAs\":[\"https:\/\/www.dezigneronline.net\/blog\",\"M.Umair.Hashmi\",\"umair.dxb\",\"umairhashmi\",\"https:\/\/twitter.com\/UmairHashmi\"],\"url\":\"https:\/\/www.dezigneronline.net\/blog\/author\/umair\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt - Web Design Dubai","description":"Google announced on July 02, 2019 that they are not going to support crawl-delay, nofollow, and noindex directives in robots.txt file.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/","og_locale":"en_US","og_type":"article","og_title":"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt","og_description":"Google announced on July 02, 2019 that they are not going to support crawl-delay, nofollow, and noindex directives in robots.txt file.","og_url":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/","og_site_name":"Umair Hashmi","article_publisher":"https:\/\/www.facebook.com\/m.umair.hashmi","article_author":"M.Umair.Hashmi","article_published_time":"2019-07-08T10:36:06+00:00","article_modified_time":"2019-07-08T10:37:48+00:00","og_image":[{"width":1280,"height":935,"url":"https:\/\/www.dezigneronline.net\/blog\/wp-content\/uploads\/2019\/07\/google-robots-noindex-nofollow.png","type":"image\/png"}],"author":"Umair Hashmi","twitter_card":"summary_large_image","twitter_creator":"@UmairHashmi","twitter_site":"@umairhashmi","twitter_misc":{"Written by":"Umair Hashmi"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["Article","NewsArticle"],"@id":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/#article","isPartOf":{"@id":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/"},"author":{"name":"Umair Hashmi","@id":"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/person\/d3fb8d27a63cee89b22d31c87cb1ba5c"},"headline":"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt","datePublished":"2019-07-08T10:36:06+00:00","dateModified":"2019-07-08T10:37:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/"},"wordCount":299,"commentCount":0,"publisher":{"@id":"https:\/\/www.dezigneronline.net\/blog\/#organization"},"keywords":["nofollow","noindex","robots"],"articleSection":["News","SEO"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/#respond"]}],"copyrightYear":"2019","copyrightHolder":{"@id":"https:\/\/www.dezigneronline.net\/blog\/#organization"}},{"@type":"WebPage","@id":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/","url":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/","name":"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt - Web Design Dubai","isPartOf":{"@id":"https:\/\/www.dezigneronline.net\/blog\/#website"},"datePublished":"2019-07-08T10:36:06+00:00","dateModified":"2019-07-08T10:37:48+00:00","description":"Google announced on July 02, 2019 that they are not going to support crawl-delay, nofollow, and noindex directives in robots.txt file.","breadcrumb":{"@id":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dezigneronline.net\/blog\/google-is-going-to-stop-supporting-crawl-delay-nofollow-and-noindex-in-robots-txt\/amp\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.dezigneronline.net\/blog\/"},{"@type":"ListItem","position":2,"name":"Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt"}]},{"@type":"WebSite","@id":"https:\/\/www.dezigneronline.net\/blog\/#website","url":"https:\/\/www.dezigneronline.net\/blog\/","name":"Umair Hashmi","description":"Digital Marketer &amp; UI\/UX Designer","publisher":{"@id":"https:\/\/www.dezigneronline.net\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dezigneronline.net\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.dezigneronline.net\/blog\/#organization","name":"Umair Hashmi","url":"https:\/\/www.dezigneronline.net\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.dezigneronline.net\/blog\/wp-content\/uploads\/2022\/09\/android-chrome-512x512-1.png","contentUrl":"https:\/\/www.dezigneronline.net\/blog\/wp-content\/uploads\/2022\/09\/android-chrome-512x512-1.png","width":512,"height":512,"caption":"Umair Hashmi"},"image":{"@id":"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/m.umair.hashmi","https:\/\/twitter.com\/umairhashmi","https:\/\/www.instagram.com\/m.umair.hashmi","https:\/\/www.linkedin.com\/in\/umairhashmi","https:\/\/www.pinterest.com\/umairhashmi\/"]},{"@type":"Person","@id":"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/person\/d3fb8d27a63cee89b22d31c87cb1ba5c","name":"Umair Hashmi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dezigneronline.net\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/7cfcbae520bdde6c9354158b64eb26e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/7cfcbae520bdde6c9354158b64eb26e4?s=96&d=mm&r=g","caption":"Umair Hashmi"},"description":"Digital Marketer &amp; UI\/UX Designer","sameAs":["https:\/\/www.dezigneronline.net\/blog","M.Umair.Hashmi","umair.dxb","umairhashmi","https:\/\/twitter.com\/UmairHashmi"],"url":"https:\/\/www.dezigneronline.net\/blog\/author\/umair\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/posts\/1615"}],"collection":[{"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/comments?post=1615"}],"version-history":[{"count":0,"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/posts\/1615\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/media\/1616"}],"wp:attachment":[{"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/media?parent=1615"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/categories?post=1615"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dezigneronline.net\/blog\/wp-json\/wp\/v2\/tags?post=1615"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}