For any search engine, there are always three steps involved: Crawling, Indexing and Ranking. Also, those steps need to be in such sequence. Among all of them, ranking is perhaps the most controversial to the SEO communities.
Let’s begin with crawling. Basically, each search engine will have its own spiders that are going around the web, and discovering new URLs and content. Search engines have very advanced programs that allow spiders to follow the links in order to find new content or pages.
First, spiders will go to your website, and then they will look at the robots.txt file if you have one, it is recommended that you setup one, because not all files or folders that you want crawlers or spiders to be looked at. To setup the restriction, you can do it by using robots.txt.
After search engines have crawled URLs and page, they will generate a copy of each page, which is Cache, and then will pass it to index. The index is basically sets of database that contain copies of millions of pages in such a manner that allow them accessible quickly when search certain keyword strings. That is, when we enter a keyword phrase to search engine, we actually are not searching the web, that would be impossible; even it is possible, it will take very long time. Instead, we are searching the index of the search engines, so we are able to get the millions of result within seconds.
The final piece of search engine is ranking. It is the most important aspect of any search engine. Each search engine has its own unique algorithm to rank the web pages or websites based on keyword phrases. When a users type in a keyword phrase to search, the strings will be passed through their search engine system which implements very complex algorithms, and then return most relevant results to the user. It sounds very complex, the whole thing only takes about few seconds to complete. Google perhaps has the most complex algorithms that are designed to rank web pages, and they are constantly updating their algorithms. A lot of black-hat gurus are trying to beat the system, but few of them are actually know the way. Even if they found the loopholes, those loopholes are usually closed up pretty quickly by Google.
In summary, the above are three steps on how most search engines work: crawling, indexing, and ranking. Now, you should have basic understanding of each one. If you have any question regarding any of them, please feel free to leave a comment or contact us.