bulk email server

Results 1 to 9 of 9

Share Thread: What is Robots.txt?

  1. #1
    Senior Member
    Join Date
    Sep 2017
    Posts
    109
    Feedback Score
    0

    What is Robots.txt?

    VPS Hosting
    Hello Friends,

    Please tell me what is robots.txt.

  2. #2
    Member
    Join Date
    Jul 2017
    Posts
    57
    Feedback Score
    0
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites. Not all robots cooperate with the standard; email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard can be used in conjunction with Sitemaps, a robot inclusion standard for websites.

  3. #3
    Senior Member
    Join Date
    Jul 2017
    Location
    Forum
    Posts
    821
    Feedback Score
    0
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting | $1 VPS Hosting
    Cheap Dedicated Servers | Servers Starting from $29/MO

  4. #4
    Member
    Join Date
    Oct 2017
    Posts
    31
    Feedback Score
    0
    Quote Originally Posted by glstechserve View Post
    Hello Friends,

    Please tell me what is robots.txt.
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

  5. #5
    Member
    Join Date
    Oct 2017
    Posts
    33
    Feedback Score
    0
    Robots exclusion standard. From Wikipedia, the free encyclopedia. Jump to: navigation, search. The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

  6. #6
    Senior Member
    Join Date
    Nov 2017
    Location
    Mumbai, India
    Posts
    100
    Feedback Score
    0
    Robots.txt is the primary file which any search engine looks for when they lands on your website. It helps them to understand which part of the website need to be crawled and which one should not..
    EWEBAC - SEO Company in Mumbai. Offers result oriented SEO Services for SME's. Also visit for more seo tips & tricks.

  7. #7
    Senior Member
    Join Date
    Jun 2018
    Location
    Pakistan
    Posts
    155
    Feedback Score
    0
    robots.txt is a notepad file which contains the instructions of website owner for web crawler to not visit, crawl and index the site or link.

  8. #8
    Member
    Join Date
    Oct 2017
    Location
    New Delhi, India
    Posts
    36
    Feedback Score
    0
    thanks for making it clear but how to create it? Is I have to ask developer to do it or can I also do it being a digital marketer without coding language?

  9. #9
    Member
    Join Date
    Oct 2017
    Location
    Delhi, India
    Posts
    46
    Feedback Score
    0
    thanks for sharing

 

 

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •