Get Indexed by Googles Googlebot Right Away, the Right Way

Everyone in the online world knows extremely well that the most sought after traffic to ones site comes from a Google search. Folks, 80% of searches on the internet are done in Google.

In theory, it is simple - if you have something interesting to someone else, if you build a website with the honest to goodness goal to provide something useful for someone else, that someone else will find you. That is also how the creators of Google describe their main goal, to more or less have a great repository of information, and help people of our planet find useful stuff.

In practice, it is not that simple. It is not that simple because there are thousands, possibly even millions of sites like yours, because you might be running a very honest online business, selling some very useful product, but do not have unheard of, exceptionally grand content. If your site is listed on page 265 of a search results set, be sure you will never get any visitors that way.

Unlike Yahoo and others, who rely on human involvement, Google does everything through automation. Websites are indexed (or crawled, or spidered - all terms refer to the same process) by their indexing software called Googlebot. Googlebot looks at websites daily, and rules programmed into the software decide which of your pages make it into the main Google index and which dont. After your site was indexed, whether it was submitted for indexing by a human or the robot just stumbled upon it, your pages are ranked, so Google knows on which page of a search to put your site on, and on what search phrases should your site even be part of the result search.

The Googlebot is very smart and works really well. Keep in mind however, that is just a piece of software, a very sophisticated one, but its just a computer program. Consequently, it has a set of algorithms (rules) it uses to index web site content (information), a set of capabilities (as I said before, Googlebot is really intelligent) and a set of limitations. As such, there is an impressive number of ways in which one can trip up the Googlebot and make it impossible for it to index your content. Alternately, the Googlebot can index your site well, and then people will find it when searching for words it contains.

This article will try to teach you all the basics necessary to achieve consistency and persistency in Google, starting with the very basic step: getting indexed by Googlebot, Googles indexing robot.

1. Read Googles own Webmaster Guidelines

The people behind Google seem to have two main things down to a science: One, most of their algorithms (rules) are so secret, that all us non-Google employees do is speculate. Two, their guidelines are very simple, direct and precise. Following their guidelines will never hurt your sites ranking. Disregarding their guidelines can and probably will hurt you in the long run. So go to http://www.google.com/webmasters/guidelines.html and read what Google has to say about itself.

2. Have text links.

Make every single page on your site accessible via a text-based link, as opposed to Javascript, Flash, DHTML (Dynamic HTML), etc. Googlebots native language is text.Google says: Make a site with a clear hierarchy and text links. Every page should be reachable from at least one static text link.

This is probably the number one key to your sites existence in Google. Googlebot is actually a robotic, browser-like software, based on the venerable Lynx browser. The reasoning behind this approach is that the creators are trying to get as close as possible to emulating human browsing, making sure your website is actually human friendly. Consequently, by downloading Lynx on your computer and looking at your site through Lynx (http://lynx.isc.org), you will see more or less exactly the information Googlebot can read and index and the links Googlebot can follow. You will also see HTML errors on your pages and places where a robot would be stuck and could not reach the rest of your site.

I know it is very unfair to those of us who understand and love the potential of websites built completely in Flash, or other engines. However, until the nice folks who run Google figure out a good way to crawl inside a Flash file and extract the appropriate information, we are stuck with standard HTML.

This is not to say that you cannot make your site really pretty and fill it with Java Script and Flash eye candy. But you must have regular text and standard text links. Usually you can achieve the desired effect by having extra navigation menus based on standard text links.

3. Avoid frames.

Avoid frames at all cost. If you must use them (for example to make someone elses page look like its part of your site), do not use them on your front page.

Frames are like the plague, they sneak up on you. It is incredibly easy to lose Googlebots tracks inside a badly formatted frameset. You might hear that some of the robots, including Googles Googlebot and Yahoos Slurp are quickly gaining capabilities to go inside frames properly. My philosophy is, until a feature becomes ubiquitous, if youre uncertain, leave it in the closet.

4. Keep the number of links on a given page less than 100.

This comes straight from Googles Webmaster Guidelines: Keep the links on a given page to a reasonable number (fewer than 100).

This looks more like a suggestion and I am not 100% sure if you get penalized in any way or if Googlebot just stops reading your links after 100. I can however tell you from personal experience that I tried a page with 700 links and it seemed fine. Then one day I tried to view the page from my Blackberry PDA and I got this strange error message saying my page is illegally formatted. After I split the page into several ones with 80 links each, the pages worked on the PDA also.

Who cares about the Blackberry Well, if youre reading this and your goal is to get visitors, then your main concern should be not to alienate anyone. Remember, today more than ever, people use different devices and different software to access the web. Every visitor is a potential customer. Every employee at a major US lawfirm and many other corporate people use a Blackberry.

Lastly, why would you need that many links on one page anyway Lets say, for example, that you specialize in promotional products - corporate branded gifts, such as pens, caps, mints and other products (called sometimes premiums) imprinted with ones logo. Your name is John Doe, and you decided to name your company JDPromos (not very imaginative, but will do for our examples). You would want to have every item in your catalog as a text link, so every item gets indexed as a link and as a keyword. Also, those who run forums, ezines, blogs, might want to have standard links to their articles, as the software they use might create dynamic links, invisible to certain robots.

5. Give every page a meaningful title.

Give every single page on the site a complete and meaningful title. This is also directly from Googles Webmaster Guidelines. See Rule #1.

Incidentally, for those who are fascinated by the debates on the death of the Meta Tags, the