Do you have a Web site with many pages of content? Are you having trouble with Google visiting and indexing all of your pages? If so, this is a common problem.  I see this all the time. There are certain industries that lend themselves to this issue on a regular basis. Real estate is one that comes to mind where a broker site can have several thousand listings under their own listings and tens of thousands of listings in the MLS. Regardless of your industry if you want to maximize the number of pages that Google indexes you need to LINK TO THOSE PAGES.

I know that sounds very basic but I have long stopped being surprised when I come across Web sites that block Google from indexing their own site. Typically, this happens because when the site was designed there was very little time devoted to information architecture. Without a plan for moving information across the Web site either through visitor interaction or search bots – things break down once the site gets beyond a certain size.

How do you stop Google from fully indexing your Web site? By having two few incoming links, improperly targeted incoming links, an internal linking structure that places pages too many steps from the entry page or a combination of all the above.  For the sake of this article I am discounting instances where Web developers accidentally block the search engines through improper use of robots.txt, meta tags, error pages etc.

Try looking at the following when analyzing why Google will not index more of your Web site:

• If you have a paginated list of pages (a page of links or content with pages 2 through 20 linked to at the bottom) try increasing the size of each page. Instead of having 20 links to products – increase this to 50 links to products. You can then reduce the number of pages in the list.
• Instead of having a deep site architecture with ever deeper categories flatten this out with fewer categories and more pages within each category.  Here is a deep link structure – “home/kitchen/utensils/silverware/forks.html”. It is going to be difficult to have pages at the ‘forks.html’ level be consistently indexed – and have a strong internal Page Rank. Reduce this structure to something like, “kitchen/forks.html”.
• If there are sections of the site that are not being indexed or being indexed poorly you should point external links to some of these pages in a tactical manner. Watch your link text and spread the links around in a natural manner for best effect.
• Depth & frequency of linking to pages on the domain – it is important to maintain as many deep links as possible when starting your online marketing or making major changes to a Web site.  Deep links are very important to indexing more files and increasing Page Rank.
• Sometimes the reason for poor indexing is the server where your Web site is being hosted is slow. This will directly impact the rate of crawl by the search bots. Your Web site must be on a fast server. At least it must be on a server that is not slow or giving the occasional server error.  In 2010 Google has said they will be adding page load time to their list of on-site elements that determine search result placement. It’s time to fix your slow site.