robots.txt. It might look like a simple text file, but it’s your first line of communication with search engines. Getting it right is crucial for good SEO. Getting it wrong can make your site invisible.
That’s why we built this free Robots.txt Generator. It takes the guesswork out of the equation, helping you create a perfectly formatted file in seconds. This guide will walk you through everything you need to know—what robots.txt is, why it matters, and how to use our tool to take control of your site’s SEO.
noindex meta tag on the page itself.?sessionid=...)Disallow rule for these directories in your robots.txt file is a simple and effective first step to keep them out of the public eye.
User-agent: This specifies which crawler the rule applies to. You can target all bots or specific ones.
User-agent: * (The asterisk is a wildcard that means “all crawlers”)User-agent: Googlebot (This rule applies only to Google’s main crawler)User-agent: Bingbot (This rule applies only to Bing’s crawler)Disallow: This tells the user-agent not to crawl a specific URL path.
Disallow: /wp-admin/ (Blocks the entire WordPress admin directory)Disallow: /private-page.html (Blocks a single page)Disallow: / (Blocks the entire website. Be very careful with this one!)Allow: This directive, primarily used by Google, lets you create an exception to a Disallow rule. For example, if you want to block an entire directory except for one specific file inside it.
Disallow: /media/Allow: /media/public-image.jpg (This would allow Googlebot to access public-image.jpg while still blocking everything else in the /media/ folder).Sitemap: This specifies the absolute URL of your sitemap.
Sitemap: https://www.yourdomain.com/sitemap.xmlUser-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.yourdomain.com/sitemap.xml
admin-ajax.php, which is sometimes needed for a site’s functionality to be rendered correctly. It also points them to the sitemap.
/wp-content/themes/, /wp-content/plugins/) was common practice. Do not do this today. Google needs to see your site exactly as a user does. If you block CSS and JS files, it can’t render the page properly. This can severely hurt your rankings, especially for sites built with advanced visual tools like Elementor, where rendering is everything.Disallow: /) will tell every search engine to ignore your entire website. It’s a simple mistake that can completely de-index your site./Photo/ and /photo/ are two different directories. If your folder is named Photo, Disallow: /photo/ will do nothing.User-agent: *). You can choose to allow all or block all. For most websites, “Allow All” is the right starting point.yourdomain.com/sitemap.xml) and paste it into the sitemap field. If you’re using an SEO plugin like Yoast or Rank Math, it generates this for you automatically.Disallow rules. We’ve pre-filled some common ones for WordPress (/wp-admin/, /wp-includes/). You can add any other directory you want to block, like /staging/ or /private/.https://www.yourdomain.com/robots.txt
For those of you building with WordPress and Elementor, you have a few easy options:
Tools > File Editor and paste the text from our generator directly. This is the easiest method.public_html folder (or your site’s root directory), and create a new file named robots.txt. Paste the generated text and save.
Stop wasting time optimizing images by hand. Our plugin does it automatically, making your site faster and freeing you up to focus on what matters most.