CrawlReady

Free tool · No signup

robots.txt generator

A robots.txt is a small text file that sits at the top of your website (yoursite.com/robots.txt). It tells search engines and AI crawlers which pages they're allowed to read. Get one line wrong and you can accidentally block ChatGPT, Perplexity or Google from your entire site — it's one of the most common issues our scanner finds. This tool builds the file for you, with the AI crawlers handled correctly.

Your sitemap lists every page on your site. If you have one, paste its full address — it helps crawlers find everything.

One per line. Pages crawlers should not visit — like your admin area or shopping cart. Most small sites only list one or two.

Asks crawlers to wait this many seconds between visits. Almost every site leaves this blank.

These are the crawlers behind ChatGPT, Claude, Perplexity and other AI tools. We recommend Allow for all of them — blocking them means AI can't read or recommend your site.


          

            
            
          

          

        

      

    

  


  

    

      
Putting it on your site

      
Three steps, about five minutes.

      
    
        
  1. Copy or download the file above. If you downloaded it, it's already named robots.txt — keep that exact name, all lowercase.
    2. 
        
  2. Upload it to the very top level of your website — the same place your homepage lives. Use your hosting company's file manager, an FTP app, or your website builder's file upload.
    3. 
        
  3. Check it worked: open yoursite.com/robots.txt in your browser. If you see the text, you're done.
    4. 
      

      

        Don't want to do it yourself? Send this to your web person:


        Hi — could you please upload the attached robots.txt file to the root of our website, so it loads at yoursite.com/robots.txt? It controls which crawlers (including AI crawlers like GPTBot) can read the site. Thanks!


        
      

    

  


  

    

      
Go further

      
Is the rest of your site AI-ready?

      
robots.txt is one of six checks in a full CrawlReady scan — crawler access, structure, metadata, content, FAQ signals and performance.

      

        Try the free scan
        Make an llms.txt next