Breaking

Saturday, 27 April 2019

Use Custom Robots.txt With Your Blog

Use Custom Robots.txt With Your Blog

Custom Robots.tx
Custom Robots.tx

Use Custom Robots.txt

  If you want to know what is a robots.txt file? And how to use Custom Robots.txt with your blog, you can find the full information by reading this article.

  It is very important for SEO of your blog. it may be completely remove from your blog search engine due to its misuse. Follow the guide to know more the benefits and use of Robots.txt File.

  The direct connection of Robots.txt File is linked to your blog's indexing and crawl rate. Generally, all search engines have their own robots or bots which crawl your blog and the links in it and index it in the search engine. Because of which, whatever you update in the blog, it also shows the whole update in the search engine. What will crawl in your blog, it's controlling by robot.txt?



What is Robot.txt

  Robots.txt is a text file with some simple code, which gives instruction to web crawlers, which page of your blog to crawl and index or not.

  This file is saved in any website or blog server because search engine bots can be managed. It is used mainly to prevent crawling any non-important web pages (Demo page, Archive, Labels, etc).

  By the robots.txt file, you have the authority of your blog, which page you want to show in the search engine and what page you want to hide. The search engine scans the robots.txt file before crawling any web page.

Also, read it to know more about How To Become a Successful Blogger

  By default Blogger's robots.txt file looks like this, as you can see in the screenshot.

  To see the Custom Robots file, you can type slash (/) after the homepage URL of your blog and type robots.txt and press Enter.
Custom Robots.txt
Use Custom Robots.txt With Your Blog

What does robots.txt code mean?

User-agent: Mediapartners-Google

  This code is for Google AdSense robots, which governs what kind of Adsense Ads shows on your blog and website.

User-agent: *

  User-agent: * This code means all search engine bots can crawl our blog. Asterisk (*) means all robot (Googlebot, Yahoo, etc).

Disallow: / search

  This means that the search engine will not crawl where 'search' keyword will appear after your domain name.

  https://www.onlineak.net/search/label/Blogging
This link is the label name (Menu) of our blog. Similarly, the number of labels in our blog will not be indexed.

Allow:/

  This code helps you index by crawling the homepage of your blog.

Sitemap

  A sitemap gives the search engine information about all the existing and new posts in our blog. That means whenever the web crawler scans our robots.txt file, it will get information from all the posts we have published through the sitemap.



How to Add Robots.txt File to Blogger

  To add a custom robots.txt file to Blogger, first of all, you can login to Blogger and come to the Dashboard.

 - You can now go to Settings and click on the Search Preferences option.

 -In this page, you will find a section of Crawlers and indexing, where there will be an option of Custom robots.txt [Disabled]. To enable this, you have to click on Edit and tick on Yes and setup Robots.txt file.

 - After this, copy the code below and paste it in the box of Custom robots.txt and click on Save Changes.

 - Copy Robots.txt code

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: / search

Allow: /

Sitemap: http://example.com/sitemap.xml

  Instead of http://example.com/sitemap.xml, you should use your blog's sitemap in the Sitemap.
Custom Robots.txt
Use Custom Robots.txt With Your Blog
   I have explained the complete process of Use Custom Robots.txt With Your Blog, by following it, you can also add a robots.txt file to your blog.

  If you want to hide any post or page from here, then you should have proper knowledge of robots.txt, because of its misuse, your blog's SEO can be completely destroyed. Do not tamper with its settings without knowledge.

  If you have any questions related to 'Use Custom Robots.txt With Your Blog' then you can ask in comment. And if you like this post, then share it with your friends on social media.

Also, read it to know more about On Page SEO Techniques

No comments:

Post a Comment