Search engines direct visitors to your website.
So, how do these search engines actually work and how do they present the information to the visitor doing the search?
As complex as it looks, the concept is pretty straightforward really . The underlying algorithms’ however are super complex.
There are basically two parts to a search engine. The first is the robot; also known as a crawler or spider. The second is the UI (user interface)… the UI is where the visitor types their search query into, and views the results of that query or search phrase.
Google Search Results
Search Engines use their crawlers or spiders to read the contents of your web site, then store and rank that information in their database. This is called indexing a website. When you submit your website to a search engine, or for best results entice the spider to visit your site by an external link on another site, or by a link embedded in a web 2.0 or social media site. If you’ve successfully seduced the search engines’ robot, the search engine spider/robot will begin crawling through your web sites pages… and if they’ve been linked together correctly, it will index your entire site.
Sites not linked correctly end up with only a few of its pages available in the search results.
Linking pages correctly is an important web design strategy, and results in the best SEO possible – when it comes to SEO and search engines, every extra effort goes a long way. Often it can be the difference between a page 10 listing (no traffic / no visitors) and a page 1 listing (lots of traffic / lots of visitors).
Technically speaking a ‘spider’ is an automated program used by the search engine to travel the web and spider (index) new and changing content. Spiders visit the web site, read that content; like the Meta tags, text, picture captions and follow the links the site points to. The spider then returns home with all that indexed information and dumps it into a central repository, where the data is taken apart, analyzed, ranked and indexed.
Spiders periodically return to a site and check for any updated information. The frequency of re-spidering is determined by the programers of the search engine and is often dependent on the type of site, niche or category its filed under.
A spider is similar to a book, where as it contains the table of contents, the actual content and the links and references for all the websites it finds during its search. Spiders index million and millions of pages each day.
There are only a handful of BIG search engines; Bing (formerly MSN), Yahoo and the grand daddy if them all… Google. Google owns in the range of 75% to 85% of all searches performed. So always focus your SEO (search Engine Optimization) efforts on Google first, for other two and the hundreds of thousands of other lessor search engines, will quickly follow in its footsteps.
When you ask a search engine to locate information by typing keywords or keyphrases into its search query form, it quickly searches through its own index – the index it created from all the data that its spiders gathered – not by actually searching the Web.
Different search engines produce different search results… because every search engine uses a different algorithm to spider (index) and search through its indices.
There you have it Search engines 101. Spiders, crawlers and robots, oh my!