SEO: "seek, and ye shall find." (Luke 11)

Anyone remember WebCrawler? Infoseek? HotBot? AltaVista? Lycos? As long as we’ve had search engines, we’ve had SEO (= search engine optimisation). And as Microsoft announces a UK roadshow to launch its new search platform, Bing, as a serious rival to Google, SEO is firmly established as part of the digital marketing landscape.

Indeed SEO or 'natural search' is a whole separate industry these days. It can be defined as: helping site owners to optimise their ranking in Google and other search engines when users search by certain relevant “keywords”.

The evolving role of the practice of SEO is currently a hot topic in digital marketing circles. Some are even suggesting its days are numbered as the search engines (especially Google) get wise to sharp practices to manipulate rankings.

To clarify: we are not talking here about paid search (= pay per click) e.g. Google AdWords, where one ‘buys’ keywords (via a bidding process) and Google displays one’s clickable text-only ad as a sponsored link in the Google Results page. (see the results at the top and right hand side of the Google Results Page.) The 'opposite' of paid search is organic or natural search which is the subject of this post.

Why does SEO matter? Look at the facts: 90% of all web sessions start with a search engine (Google, Yahoo!, Live Search, Ask Jeeves etc.) This figure has been increasing year on year as more and more people give up on bookmarking sites (far less actually remembering urls!) and rely on Google to find the site for them. So if you have a website, and someone 'Googles' the thing you sell, you want to appear (or be ‘ranked’) near the top of the listings and certainly on the first page. If you are not, people may never find your site and you won’t be very successful. In fact you risk being invisible, lonely and broke (to misquote Cory Doctorow). Received wisdom is that 62% of searchers click on a result within the first page of results, and approx 28% of all searchers click on results within page 2 or 3. Readers of a mathematical bent will notice that this leaves only 10% i.e. hardly anyone looks beyond page 3. (This raises the question: how many pages of search results do users really require? - which could be the topic of another blog post or indeed a session at an SEO conference!)

One more fact: Google handles between 75% and 86% (UK: Hitwise April 2009) of all search traffic depending on which country you look at; so for the purposes of this post when we say “Google” we mean “Search Engine”. Unhealthy? Maybe, but realistic currently.

Suppose I have a website which sells Golf equipment. Keywords of interest to me might be: Golf equipment, Golf gear, Golf accessories, Golf clubs, Golf bags, Golf clothes, Golf apparel etc etc.

I might wish to consider optimising my site for any or all of these “keywords” (which are often than not more than one actual word). One can achieve high rankings more easily for niche keywords: If you search Google for “Golf Equipment” you will get about 37,700,000 results. (in 0.25 seconds!) At 10 results per page that gives us approx 3,770,000 pages of results. If however one lives in the West Midlands of England and is looking for a professional to stuff your dead cat, and you search by “Taxidermists in Wolverhampton” one gets only 16,900 results but this is still 1,690 pages. Only a few sites make it onto page 1 and many excellent sites languish invisibly on page 500+.

So SEO is, these days, a major digital marketing discipline; but is it legitimate? i.e. both ethical and good business?

Let’s just have a look at the agendas of the various parties involved:

• Google: wants its search engine to be good at its job; i.e. it wants as many users as possible to have the best possible experience; i.e. to find a selection of suitable sites as quickly and as painlessly as possible. It particularly wants to direct users to site/s with well written and presented content that best deliver what they were looking for; in pursuit of this it tries to interpret their intention (which is not always exactly what they type in to the Google search box). If it succeeds, it will attract more traffic and make more money out of AdWords (its pay- per-click product).

• The Site owner: wants high ranking in search results whenever a user types in a keyword which it deems relevant to its business; i.e. they want to maximize quality and quantity of traffic to their site.

• The SEO consultancy/ professional: wants to do a good job for the Site owner i.e. get the site up the Google rankings, driving more traffic, pleasing the client and thus getting more optimization briefs and making more money.

Good site design and legitimate SEO helps everyone: users who search by a given keyword find a relevant site which delivers what they were looking for and provides a good experience. The SEO professional and the design/build agency should work together to make the site visible to search engines which pleases them since they are helping their customers (=searchers). The Google spiders (or crawlers) look at millions of web pages and assess their relevance to a particular search (yes, it’s amazing I know). It’s absolutely OK to help them by a little judicious signposting (like putting up a sign in the street outside your shop; effectively the site is ‘putting its best foot forward’) . Also, if you can get plenty of inbound links (i.e. other sites publishing links to your site) Google likes this: it suggests your site must be of high quality. It’s OK to solicit these links (e.g. if you are a florist, you might ask local chocolate shops if they want to link to your site). So far so good.

The problems occur when SEO practitioners and/ or site owner Webmasters attempt to deceive the search engine spiders (and therefore ultimately the user) by employing a range of techniques to manipulate the Google rankings (including any or all of: spamdexing, cloaking, doorway pages, keyword stuffing, invisible text, deals with link farms +++).

Ever since Search was invented, unscrupulous SEO experts (known as ‘black hat’ operators) have been trying to manipulate it, i.e. to get people to visit sites under false pretences, taking them to sites which were not what they were looking for; the result being an unfair advantage to the site owner and/or confused/ dissatisfied visitors, a scenario which benefits no-one except the black hat guys! If they succeed, they achieve an inappropriately high Google ranking and ultimately can even take the user to a site that wasn’t what they were looking for. Of course it isn’t always as cut and dried as this: in many cases, the user would be equally satisfied by any one of a number of search results, and techniques may be used which give a good site a ‘helping hand’ in the listings. There are grey areas: like how many times can one legitimately include a given keyword in one’s content: 3 times in a page makes the site relevant, three times in a sentence looks sinister and may lead to Google penalizing or even delisting your site.

Of course, there is one obvious question which we haven’t addressed here (space doesn’t allow): who decides which SEO techniques are OK and which are ‘black hat’? which can get your site demoted in the listings or even banned? Of course currently it’s Google who decides. 'Baddies' clearly operate in this space so some sanctions are necessary; but should a single company (even our buddies at Google) be given so much power?

Effectively the black hat SEO specialists are engaged in an ever-shifting battle of wits with the search engines and the (secret) Google algorithm is certainly much more sophisticated at picking up dubious SEO techniques than it was 10 years ago. eg. stuffing in loads of invisible keywords used to be much more prevalent (and effective!). Even legitimate SEO has evolved with the Search Engines. Metatags (invisible to users but viewed by the spiders) used to be very important; less so today. Indeed, if you read a book on SEO dated before 2004 (say) I respectfully suggest you may be wasting your time!

As part of an ongoing mission to fight spam and improve user experience, Google is apparently implementing a series of changes to the algorithm (it doesn't announce these). In the future, as Google moves towards behaviour/ intent based search, it is possible that each person who conducts a search for a particular term will get different results based on their interests, search history and even their location. As a result, SEO will need to evolve. One possibility is that link building will become far less important in the future of SEO because Google will determine the ‘value’ (= relevance) of a website based on how visitors engage with it. This would imply that the ultimate goal of site owners should be to provide compelling content that entices visitors to read, share, bookmark, and so on.

Another possibility is that search engines will in future provide user-controlled rankings. Users might have the chance to vote for sites they like and sites will get ranked based on such votes. (The model would be similar to Digg and Reddit). Of course, search engines will need to ensure that votes are genuine in order to prevent black hat SEO specialists from manipulating the results.

Google and the other search engines are raising the bar in SEO. Initially, this will make it harder for SEO professionals but the end result must be good. Spammers and 'black hat's will have more difficulty succeeding in their unscrupulous efforts and search engine users will be provided with content that is more relevant.

As always, the aim of webmasters and SEO professionals should be to appeal to humans not machines. Your site should certainly be optimized (to give you a fighting chance in the competitive marketplace) but your priority should always be to meet human needs and provide solutions. Considering the direction that SEO is going, human actions and behaviour will ultimately determine rankings. Your SEO success will depend on your ability to engage people through great content and shrewd social media marketing. This is as it should be.

The internet has transformed our lives: at work, at home and increasingly on the move. The Web is becoming richer and more complex every day. To cope with the mind boggling amount of data it offers, we need increasingly sophisticated tools to make sense of it: to 'organise the world’s information'. Unscrupulous operators will always try to manipulate the search engine spiders (or crawlers) to gain a higher ranking but they must be prevented from doing this; it’s in the public interest, as well as that of Google, Yahoo!, Ask Jeeves, Live Search/ Bing and not least genuine SEO professionals, that the user can find the 'right' site as rapidly as possible every time.

If one believes that Advertising (assisted by agents as 'paid advocates') i.e. promoting the benefits of a product or service to the target audience by all appropriate media, is a legitimate activity in a free society, then it follows that SEO is OK and indeed desirable: it helps honest merchants reach their willing audiences. Indeed, Google doesn’t want to kill SEO; and even if it ceases to be such a dominant player in the search world, other search engines are likely to want to work with ‘honest’ SEO professionals rather than against them.

So current rumours of the imminent death of SEO are much exaggerated. As long as search engines publish free lists of ‘relevant’ sites, SEO will exist. Naturally (sorry) like everything in the digital world, SEO will need to evolve and adapt. It will do so. Intent and behaviour-based search will grow. Social search (e.g. via Twitter) will enable users to ask questions to aggregated groups of real people in real time. Not so easy for site owners to optimize! Watch this space.