bulk email server

Results 1 to 8 of 8

Share Thread: What is Robots.txt?

  1. #1
    Senior Member
    Join Date
    Sep 2017
    Posts
    109
    Feedback Score
    0

    What is Robots.txt?

    VPS Hosting
    Hello Friends,

    Please tell me what is robots.txt.

  2. #2
    Senior Member
    Join Date
    Jul 2017
    Location
    Forum
    Posts
    862
    Feedback Score
    0
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting | $1 VPS Hosting
    Cheap Dedicated Servers | Servers Starting from $29/MO

  3. #3
    Senior Member
    Join Date
    Nov 2016
    Posts
    131
    Feedback Score
    0
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.


    17 Seater Tempo Traveller in Delhi

  4. #4
    Member
    Join Date
    Jan 2018
    Posts
    37
    Feedback Score
    0
    robots.txt is a text file which contains the instruction for the search engine related the crawling of web pages. The file lists webpages that are allowed and disallowed from search engine crawling.

  5. #5
    Junior Member
    Join Date
    Feb 2018
    Posts
    24
    Feedback Score
    0
    A robots.txt file consists of one or more rules. Each rule blocks (or or allows) access for a given crawler to a specified file path in that website.

    Here is a simple robots.txt file with two rules, explained below:

    # Rule 1
    User-agent: Googlebot
    Disallow: /nogooglebot/

    # Rule 2
    User-agent: *
    Allow: /

    Sitemap: http://www.example.com/sitemap.xml

  6. #6
    Member
    Join Date
    Feb 2018
    Posts
    51
    Feedback Score
    0
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  7. #7
    Junior Member
    Join Date
    Apr 2018
    Posts
    4
    Feedback Score
    0
    Robots.txt file is primarily used to specify which parts of your website should be crawled by spiders or web crawlers.

  8. #8
    Member
    Join Date
    Oct 2017
    Location
    New Delhi, India
    Posts
    36
    Feedback Score
    0
    Basically robot.txt is text file which gives necessary information to crawlers to crawl the webpages they are allowed, it also prevents their crawling access to such files they are not allowed

 

 

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •