{"version":"1.0","provider_name":"Blog","provider_url":"https:\/\/elementor.com\/blog","author_name":"Itamar Haim","author_url":"https:\/\/elementor.com\/blog\/author\/itamarha\/","title":"The Ultimate Guide to Robots.txt Disallow: How to (and How Not to) Block Search Engines","type":"rich","width":600,"height":338,"html":"<blockquote class=\"wp-embedded-content\" data-secret=\"877isfSYhQ\"><a href=\"https:\/\/elementor.com\/blog\/robots-txt-disallow\/\">The Ultimate Guide to Robots.txt Disallow: How to (and How Not to) Block Search Engines<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/elementor.com\/blog\/robots-txt-disallow\/embed\/#?secret=877isfSYhQ\" width=\"600\" height=\"338\" title=\"&#8220;The Ultimate Guide to Robots.txt Disallow: How to (and How Not to) Block Search Engines&#8221; &#8212; Blog\" data-secret=\"877isfSYhQ\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script>\n\/*! This file is auto-generated *\/\n!function(d,l){\"use strict\";l.querySelector&&d.addEventListener&&\"undefined\"!=typeof URL&&(d.wp=d.wp||{},d.wp.receiveEmbedMessage||(d.wp.receiveEmbedMessage=function(e){var t=e.data;if((t||t.secret||t.message||t.value)&&!\/[^a-zA-Z0-9]\/.test(t.secret)){for(var s,r,n,a=l.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),o=l.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),c=new RegExp(\"^https?:$\",\"i\"),i=0;i<o.length;i++)o[i].style.display=\"none\";for(i=0;i<a.length;i++)s=a[i],e.source===s.contentWindow&&(s.removeAttribute(\"style\"),\"height\"===t.message?(1e3<(r=parseInt(t.value,10))?r=1e3:~~r<200&&(r=200),s.height=r):\"link\"===t.message&&(r=new URL(s.getAttribute(\"src\")),n=new URL(t.value),c.test(n.protocol))&&n.host===r.host&&l.activeElement===s&&(d.top.location.href=t.value))}},d.addEventListener(\"message\",d.wp.receiveEmbedMessage,!1),l.addEventListener(\"DOMContentLoaded\",function(){for(var e,t,s=l.querySelectorAll(\"iframe.wp-embedded-content\"),r=0;r<s.length;r++)(t=(e=s[r]).getAttribute(\"data-secret\"))||(t=Math.random().toString(36).substring(2,12),e.src+=\"#?secret=\"+t,e.setAttribute(\"data-secret\",t)),e.contentWindow.postMessage({message:\"ready\",secret:t},\"*\")},!1)))}(window,document);\n\/\/# sourceURL=https:\/\/elementor.com\/blog\/wp-includes\/js\/wp-embed.min.js\n<\/script>\n","thumbnail_url":"https:\/\/elementor.com\/blog\/wp-content\/uploads\/2025\/09\/imgi_25_imgi_19_WordPress-Website-Maintenance_1200x628-1-4.jpeg","thumbnail_width":1200,"thumbnail_height":628,"description":"Every website has a hidden \"doorman\" that greets search engine crawlers. This doorman operates 24\/7, holding a simple set of instructions that tell bots like Googlebot where they are and are not allowed to go. This instruction file is robots.txt, and its most powerful and misunderstood command is Disallow."}