3. Understanding Search Engines

How do search engines work?

Although search engines are basically just navigators of the internet, it would be impossible to imagine the internet without them. Due to the sheer size and depth of the World Wide Web, this very wealth of information becomes meaningless or unusable without search engines. Search engines basically perform three tasks: 

1) They search the World Wide Web using specific words or phrases.

2) They index the results they find and their locations.

3) They provide a user access to this index and the ability to search it using combinations of words or phrases of their choosing.

Today search engines perform billions of queries every given day and are constantly in a process of evolving and reinventing their services to balance the needs of websites, users, and advertisers.

It all starts with something called spiders (also called crawlers) and how appropriate, having in mind that the internet is called the World Wide Web. Spiders are inter-bots or robots composed of software or coding. Just like real spiders, internet spiders do something called web crawling or just crawling, to make huge lists of what is found where on the internet. Crawling is the act of browsing, indexing and copying done by spiders. They start with a heavily used server or popular website (a busy part of the web) and crawl their way outwards to every link found on that website or server, behaving pretty much like real spiders in the way they spread.

Interesting fact: Google started out with just four spiders who would read 600 kilobytes per second, a laughable figure for today. This shows us how much the World Wide Web has grown.

Although Google was the first to come up with spiders, their use has spread across the search engine community and the quest to make newer, faster and more creative spiders is always on.

Spiders perform indexing of everything they ‘read’, using, among other things, something called meta tags which are the descriptions of the keywords or phrases under which the website owner would like his or her website to be indexed. They help the spiders understand what the page is about and where it belongs in an index. Webmasters might use words or descriptions in the meta tags which are inaccurate or reference popular topics and keywords which their site might not deal with. To protect against this, spiders are designed to correlate the meta tags with the content found on the website and come up with their own understanding of where the page belongs on an index.

The next big organ in the anatomy of a search engine is the algorithms it uses. Algorithms are computer coding or rather software that is designed to take the words you put into the search bar and provide you with the most relevant results from the indexes that the spiders have made. They are the ‘answer-makers’ that make a search engine an answering machine. They depend on signals or clues from the websites that are indexed to come up with rankings of search results.

These signals vary from how fresh the content on a website is, to how much it has been visited recently and the region that the user asking the question belongs to. Google alone is known to use around 200 kinds of signals in helping its algorithms create search listings every time you press that ‘search’ button on your computer.

How do people interact with search engines?

The key to designing and improving any product or service is to understand the human end of it or how real people interact with it. The best way to design better car interiors, for example, is to understand how people interact and use the interiors of their cars. Similarly, if one is to design a marketing strategy geared around SEO and search engine marketing, it begins with understanding how people interact with search engines. You can only give users what they want if you know what they want, what they are already getting and how you can improve it.

Search engine facts and figures

Let us start by understanding some basic facts and figures about search engines and people, so that we know where we stand today. The number of searches on Google per second is more than 40,000. There are 130 trillion indexed pages and this number is constantly growing. This shows how the reliance and dominance of search engine activity over other internet activities is growing. Data for the August 2016 show that Google holds the highest market share of all the desktop searches, with over 79%, followed by Bing (9.9%) and Yahoo! (8.34%). For mobile searches, Google’s share is even larger, with over 90%.

Studies by UserCentric clearly demonstrate that users pay more attention to the organic results as opposed to the paid or sponsored results, and they are much more likely to click through to organic results than sponsored results. However, interestingly enough, sponsored results and advertisements receive equal and in some cases more attention when they appear above the organic results.

Another finding that the same study made was that users pay little attention to what is contained at the bottom of the page compared to what is contained at the top. Similarly, the keywords and descriptions that are texted in bold letters receive significantly more attention and click through rates when compared to simple text, especially if it is found further down the page.

The queries that users make on search engines can typically be divided into three categories:

• Action searches or ‘Do’ queries – e.g. Convert dollars to Euros or download a file.
• Navigation searches or ‘Go’ queries – e.g. Users looking for a particular website like a band’s fan page or news website.
• Information searches or ‘Know’ queries – e.g. What year did the French Revolution start?

In addition to these queries, with the overwhelming advent of e-commerce, a further fourth category can be made which can be called:

• Commercial searches or ‘Buy and Sell’ queries – e.g. Buy used iPhone or sell a car.

Search engine marketing

Search engine marketing or SEM is a type of internet marketing concerned with promoting websites and increasing their visibility or Search Engine Results Pages (SERP) by either optimizing the content and structure of the websites themselves and how they respond to search engines, or by using advertising and purchasing sponsored results. The end goal is to get higher rankings in search result pages, achieve better visibility and gain more new traffic whilst retaining old users. It has become a huge market in itself. In fact, it is estimated that brands and agencies in the US will spend more than $65 billion in 2016 for search engine optimization (Source). And the figure will be growing in the next four years.

Google AdWords, Baidu and Bing Ads are the biggest vendors of search engine marketing online. Advertising agencies have begun to specialize in SEM as it was the fastest growing form of internet marketing and even overtook some traditional forms of advertising. Although most campaigns are carried out through vendors and agencies, self-serve solutions and SEM tools are also available online. The term SEM has come to cover the wide range of search engine related marketing activities that include the management of sponsored listings at search engines, the submission of websites to directories and the designing of marketing strategies for businesses, individuals and organizations that operate through the web.

Search engine marketing includes:

• Organic or natural results – These results are improved through the process of search engine optimization, which is the topic of this book.
• Paid results – In this case advertising programs are used to promote the website in the search engine result pages.

Therefore, search engine marketing includes a series of tasks through which the website can be promoted, regardless if those are organic or paid results. Here are some of the most important tasks in this process.

1) Keyword analysis

This involves analyzing keywords to discover the relevant keywords and phrases, which generate the most traffic for one’s product or service. Since you want your website to be associated with certain keywords, which are most likely to be used by the potential customers, you need to focus on analysis and how keywords can help you optimize the website.

2) Site popularity

This is a measure of the weight or presence, that a website carries on different search engines. It is measured by something we mentioned earlier above called search engine saturation, which is an indicator of how much of a particular website has been indexed by spiders and the amount of backlinks that a website has.

3) Web analytic and HTML tools

This has to do with the backend of the website, or the part that is relatively invisible to the users. It ranges from web analytics tools that measure, analyze and log file analyzers to complicated ones like page tagging, which uses a JavaScript or an image to follow the actions of a user on a page. HTML validators and spider-simulators are tools that optimize the hidden areas of a site for possible issues and those relating to usability. They make sure that a page conforms to W3C code standards.

4) Sponsored results

This has to do with buying advertisements or buying places on search engine pages to get higher up views in the sponsored results section and typically involving Pay-Per-Click (PPC) solutions offered by search engines.

Why search engine marketing matters?

The internet gives e-commerce a unique quality over other traditional forms of business and that is the way it is centered on creative, new and entertaining web content and low start up costs. It only takes a great idea and a little money to start earning money on the internet.

The key to having a successful internet business is attracting more and more visitors to one’s website and that can only happen with great content that is made as accessible as possible to as wide a population of users as possible. All internet businesses and websites are always struggling for maximum visibility.

Here is why SEM matters so much nowadays.

Firstly, everyone uses search engines, as we have talked about earlier, to such an extent that it is quickly becoming the most popular activity on the internet. That makes it the prime battleground for the attention and resources of internet users. SEM is what takes your website out of your hands and in front of the eyes of the largest internet community – search engine users.

Internet users have become accustomed to not thinking of a website or business as legitimate unless it shows up on search engines. A website cannot expect to have the legitimacy factor unless it is engaged in SEM and visibility on search engines.

SEM allows you the freedom to do whatever you wish with your page since your website can become self-supporting through the revenue generated by traffic. You do not even have to sell anything. You could run a blog or a website about a topic or provide any service you wish without worrying about supporting the website.

SEM and SEO are the forms of advertising that are available to everyone, and you use them to promote your website on your own, or you can hire an agency or professional to assist you in this endeavor. The marketing and optimization perpetuates itself if done properly and regularly enough, as long as your content remains on the web in some location.

Lastly, and this might be an awful reason but it is an important one, if you are an internet business, organization, interest group or public individual, you can be sure that all your competitors will be involved in SEM and SEO. So any hopes of survival in today’s aggressive internet arena cannot be entertained without thorough work regarding SEM.