Search engines used for (Search Engine Optimization) have two major functions: crawling and building an index, and providing search users with a ranked list of the websites they’ve determined are the most relevant.
A Network of Stops
Each stop is a unique document (usually a web page, but sometimes a PDF, JPG, or other file). The search engines need a way to “crawl” the entire city and find all the stops along the way, so they use the best path available—links.
- Crawling and indexing the billions of documents, pages, files, news, videos, and media on the World Wide Web.
- Providing answers to user queries, most frequently through lists of relevant pages that they’ve retrieved and ranked for relevancy (Search Engine Optimization).
The Link Structure
Links allow the search engines’ automated robots, called “crawlers” or “spiders,” to reach the many billions of interconnected documents on the web. Once the engines find these pages, they decipher the code from them and store selected pieces in massive databases, to be recalled later when needed for a search query.
To accomplish the monumental task of holding billions of pages that can be accessed in a fraction of a second, the search engine companies have constructed data centers all over the world for better Search Engine Optimization.
These monstrous storage facilities hold thousands of machines processing large quantities of information very quickly. When a person performs a search at any of the major engines, they demand results instantaneously; even a one- or two-second delay can cause dissatisfaction, so the engines work hard to provide answers as fast as possible.
Relevance and Popularity
For Search Engine Optimization, relevance means more than finding a page with the right words. In the early days of the web, search engines did not go much further than this simplistic step, and search results were of limited value. Over the years, smart engineers have devised better ways to match results to searchers’ queries. Today, hundreds of factors influence relevance.
Popularity and relevance aren’t determined manually. Instead, the engines employ mathematical equations (algorithms) to sort the wheat from the chaff (relevance), and then to rank the wheat in order of quality (popularity).
To perform better in search engine listings, your most important content should be in HTML text format. Images, Flash files, Java applets, and other non-text content are often ignored or devalued by search engine crawlers, despite advances in crawling technology.
The easiest way to ensure that the words and phrases you display to your visitors are visible to search engines is to place them in the HTML text on the page. However, more advanced methods are available for those who demand greater formatting or visual display styles.
Crawlable Link Structures
Just as search engines need to see content in order to list pages in their massive keyword-based indexes, they also need to see links in order to find the content in the first place. A crawlable link structure—one that lets the crawlers browse the pathways of a website—is vital to them finding all of the pages on a website.
Hundreds of thousands of sites make the critical mistake of structuring their navigation in ways that search engines cannot access, hindering their ability to get pages listed in the search engines’ indexes.
Keyword Usage and Targeting
Keywords are fundamental to the search process. They are the building blocks of language and of search. In fact, the entire science of information retrieval (including web-based search engines like Google) is based on keywords.
As the engines crawl and index the contents of pages around the web, they keep track of those pages in keyword-based indexes rather than storing 25 billion web pages all in one database. Millions and millions of smaller databases, each centered on a particular keyword term or phrase, allow the engines to retrieve the data they need in a mere fraction of a second.
Obviously, if you want your page to have a chance of ranking in the search results for “dog,” it’s wise to make sure the word “dog” is part of the crawlable content of your document.
Keywords dominate how we communicate our search intent and interact with the engines. When we enter words to search for, the engine matches pages to retrieve based on the words we entered.
The order of the words (“pandas juggling” vs. “juggling pandas”), spelling, punctuation, and capitalization provide additional information that the engines use to help retrieve the right pages and rank them.
Search engines measure how keywords are used on pages to help determine the relevance of a particular document to a query. One of the best ways to optimize a page’s Search Engine Optimization rankings is to ensure that the keywords you want to rank for are prominently used in titles, text, and metadata.
Generally speaking, as you make your keywords more specific, you narrow the competition for search results, and improve your changes of achieving a higher ranking.