Video Downloader for Facebook, Linkedin, Dailymotion, and 31+ more Websites ! Facebook-Online Video Downloader

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


Video Downloader for Facebook, Linkedin, Dailymotion, and 31+ more Websites ! Facebook-Online Video Downloader

About Robots.txt Generator

Robots.txt is a file which is placed in the root folder of your website. It helps “search engine’s” (SE) in indexing your website more effectively.

Search engines such as Google, Yahoo, and Bing etc. Uses website robots or crawlers which review all the data on your site.

There can be parts of your site that you don’t want them to crawl’s by search engines. To include in user search results like admin, about us pages etc. You can add these pages to this file for clearly ignored.

Robots.txt data use somewhat which is called the Robots “Exclusion Protocol”. This Robots.txt generator tool will easily produce this file. You have to just give the inputs of pages which you wish to be excluded.

Online Robots.txt generator tool by Small Seo can meet your requirement for making Robots files. You can create Robots.txt code for following search engine bots free with our best tool:

  1. Google
  2. MSN Search   
  3. Yahoo
  4. Ask or Teoma 
  5. Giga Blast      
  6. DMOZ Checker         
  7. Nutch
  8. Yandex
  9. Alexa/Wayback          
  10. Baidu 
  11. Naver 
  12. Bing

Robots.txt Generator tool for search bots of google, yahoo, bing, baidu, yandex, moz nutch, alexa/wayback, naver, msn and PicSearch by free seo small tools.