What Is Robots.txt?

For a search engine to keep their listings up to date, and present the most accurate search engine results, they perform an action known as a ‘crawl’. This is essentially sending a ‘bot’ (sometimes known as a ‘spider’) out to crawl the internet. The bot will then find new pages, updated pages or pages it did not previously know to exist. The end result of the crawl is that the search engine results page is updated, and all of the pages found on the last crawl are now included. It’s simply a method of finding sites on the internet.

However, there may be some instances where you have a website page you do not want included in search engine results. For example, you may be in the process of building a page, and do not want it listed in search engine results until it is completed. In these instances, you need to use a file known as robots.txt to tell a search engine bot to ignore your chosen pages within your website.

Robots.txt is basically a way of telling a search engine “don’t come in here, please”. When a bot finds a robots.txt file, it will ‘read’ it and will usually ignore all the URLs contained within. Therefore pages within the file do not appear in search results. It isn’t a failsafe; robots.txt is a request for bots to ignore the page, rather than a complete block, but most bots will obey the information found within the file. Some “nasty” bots may actually ignore your robots.txt file and index everything they find. However, for the nice bots, when you are ready for the page to be included in a search engine, you simply modify your robots.txt file and remove the URL of the designated page.

Why Is SEO Important?

Think back to the days before the internet, when everyone looked for business listings in the Yellow Pages or a similar hard copy directory. Now, imagine you’re trying to find a plumber. You go to the “P” page, flick through, and only see three plumbers listed.

It’d be a massive advantage for the three plumbers, wouldn’t it? To effectively be all that potential customers see. Well, that’s the power of SEO.

When someone uses a search engine, they type in a set of words to bring up results that are relevant to them. Once the applicable websites appear on the results page, they are rated in terms of ‘relevancy’. Any website which has correctly used search engine optimization will be judged by Google to be ‘relevant’. So if you have correctly used SEO, your website will appear somewhere near the top of the search engine results page – hopefully in the top three.

Why is that so important? Well, studies have shown that the vast majority of those using search engines only click on websites listed in the top three results of any search engine results page. This is where it compares to being only one of three businesses listed in an old-style paper directory like the Yellow Pages. Master SEO, and your website will appear in high in search results, in the slots (one to three) that users click. That naturally leads on to more people visiting your website, and that in turn means more business, more customers, and ultimately more money.