The Invisible Site Blues

One of the things rarely mentioned to a potential site owner (especially not mentioned by the site developer) is that many shop online sites have difficulty being indexed by search engines. And they're not the only ones. Some sites, in an attempt to utilize the latest and greatest of Internet technology also end up with invisible sites.

What do I mean by "invisible sites?" These are sites that seem to contain no content to search engine spiders.

How does this happen? Well, typically this can be a problem for sites that are pulling the site content from a database. Many of these types of sites have only a few actual pages, which are really just templates with place holders for the information to appear. The page is actually created "on-the-fly" based on the actions of the site visitor. For instance, when a site visitor comes to the main page and clicks on "products" a list of products is pulled from a database and the information populates the template page. Often these sites have the "?" character in the URL. If you look at the address bar of your browser you might see something like this: http://www.somedomain.com/default.asp?products=1234.

The problem with that URL is that the search engine spiders cannot and will not read beyond the "?" in the URL. So no matter how many products there actually might be, the spider can't see anything more than the static content on the default.asp page.

Now, compound that problem by placing these pages inside of frames. (Don't laugh you wouldn't believe how often I run across this.) Every search engine has stated that they have difficulty indexing a site in frames - so why do we still use them? Well, for some sites frames are really the only way to bring in content from another site and have that content appear to be a part of their own site. For instance, you might have a Yahoo shopping site that you want to integrate with your current site. The use of frames allows you to bring in that storefront and still have your main site navigation available.

If the frame is used only when someone clicks on "Shop Online" at your site there shouldn't be much of a problem because all of the other pages at your site will get indexed. The problem occurs when the frames are the main page of the site so none of the pages get indexed.

If frames alone are not enough of a problem, try JavaScript frames. Search engines do not read JavaScript - they have no real reason to. So a JavaScript frame offers even less possibility that any of your pages will get indexed.

Is your situation hopeless if you have one of these sites? The good news is, no. There are a couple of solutions. If your site is not extremely large, it can be optimized to ensure that all of the important pages (if not all of the pages) will get indexed. If your site is extremely large or, so dynamic that simple optimization won't benefit it, we have a different solution.

We propose that you have a smaller, more focused static HTML site. This would require a new domain name and hosting services. This site should be optimized to rank well with the search engines and contain a good overview of the real "meat and potatoes" of what your company is selling. This site would send site visitors to your main site for more detailed information or to buy online. For huge, database driven sites that are not ranking well with the search engines, this may be their only option short of redesigning their site.

Understand, however, that this second site is not a "doorway" page. It should have 8 or more pages of real content. We recommend focusing on your products and leave the "about us" and investment information as links to the main site.

Do all database driven sites have problems getting indexed by the search engines? Nope. If done correctly there's no reason why a database driven site cannot be indexed. However, to do it correctly means that you have to have more HTML templates than the problematic sites OR you have to setup fixes to remove the "?" character from the URL. These "fixes" generally have to be set at the server for the server to interpret the URL and resolve it to a static URL without the "?"

Okay, that was confusing wasn't it. Here's what I'm trying to say: Depending on the programming language, there are ways to tell the server that http://www.somedomain.com/default.asp?product=1234 should read http://www.somedomain.com/Product1234.htm. Whether or not that type of resolution can be done at your server, depends on your hosting company and what permissions they will allow you.

If your hosting company will not allow you to set that type of conversion for the URL (it all depends on the operating system of the server, the programming language, etc.), then you need to create more template pages - a template page for each group of products or individual product that will automatically populate the page when the page loads. So instead of having a link to default.asp?product=1234, you would link to Product1234.htm, which is a real page that loads only Product 1234 information. This means that you will need a page for every product and product group. The good news is that the pages will update automatically as your make changes in the database. The bad news is that you will have to create new pages if you add new products.

Will search engines get upset if you have this second site? I don't see why they would. It's not like your first site is taking up much room in their database. I think they realize that they have a problem with certain sites and will tolerate reasonable workarounds. They will get upset, however, if you create multiple sites of identical content - all of which do not have indexing problems. This clutters up their database forcing them to purge some of those sites - or ban all of them.

What's the best way to ensure that my site doesn't get banned? Let an expert do it. Web-Kare, LLP has been optimizing websites for 5 years now. We have never had a site get banned and we don't ever plan on getting a site banned. We don't believe in "tricking" the search engines. We want to help the search engines produce the most relevant sites to a query and help our client's sites get the proper exposure for targeted traffic. The "do-it-yourselfer" who picks up on tips and techniques shared via family and friends will find himself either not getting the results he had hoped for, or getting his site banned. If you're a business owner, you have to ask yourself, "What business am I in? Am I in the Internet marketing business or am I in (whatever business you may have)?" How much time can you afford to spend on Internet marketing; researching keywords, keeping up with search engine changes, monitoring site results, etc.?

Want to know how your site is doing with search engines? Request our free site evaluation.