There are a lot of things going on, both out here on the open forum and back behind the scenes. Most of them doing the best we can do to attract those spiders. Revenue from our generous donors and our forum advertisers allowed us to add a powerful optimization script to the forums, back in mid-May. This script, called vBSEO, is a God-send for us. We have spent hours optimizing the site for search engine crawls and vBSEO has complimented those efforts handsomely. Take a look at the image below -
First off, look at all the red lines. When RPM posted this thread, he titled it
Running alive with spiders. And look at all the places where these spiders are going to see that phrase, when they crawl the page.
It's in the title bar of the browser, the URL of this page is rewritten to include the phrase, there is a bold title in a custom relevant replacement showing the phrase, the phrase is in the text body of the relevant replacement, the bold phrase appears again in the breadcrumb navigation bits, and then it finally appears again in the thread title bar. If you scroll through the thread, you'll see each post has the thread title added to it.
Now look at the blue lines and you will see the first few characters of RPM's original post also appear in the text of the relevant replacement.
This is not an excellent example of how effective we can be with providing the search engine spiders and bots with food to chew on and digest, but think about some of the more technical thread titles in the article forums. When a spider lands on this page, there is no doubt this page is all about spiders running alive around here.
Then we move to the back room, where there are a lot more things going on.
Every time a new thread is posted, the forum 'pings' the search engines to let them know we have some new material for them to read and index. We're currently pinging 19 different search engines.
Every 24 hours, we automatically generate a sitemap of the forum. Once the sitemap is generated, we ping the major search engines to let them know we have a new map available for them to download and we provide them with a reminder of where they will find the sitemap stored, here on this site.
We then log the activity of the search engine bots, to be sure we are succeeding in luring them here.
We run a tiny bit of code that Google tracks, showing us how many people have visited the site each day. We track how many people come here directly, either from a bookmark in their browser or from typing the forum's URL into the address bar of their browser. We track how many people arrived here by clicking a link on another site. We also track how many people got here by using the search engines.
As of this morning, our direct traffic made up 39.18% of the total. Traffic from referring sites accounted for 23.04% of the total. And the traffic from the search engines accounted for 37.78% of the total.
Of that search engine traffic, 85.25% of it came from Google, 4.98% came from Yahoo, 3.42% came from AOL, 2.81% came from MSN Live Search, etc.
If you do some searches on some phrases, you'll see how well we are ranking against other sites. Search
T-Bucket and on Google, we're ranked numbers 4 & 5 of 255,000 results. Search
t-bucket and we're number 7 of 222,000 results. Search
T-Bucket Forum and we're numbers 1 & 2 of 44,200 results. Search
T-Bucket Forums and we're numbers 1 & 2 of 38,900 results, with links to the forums also coming in at numbers 3,4,5,7,8 & 9. Search
jag rear end and we're number 9 of 421,000 results.
The ranking on the search engines is critical for us. Remember, 37.78% of our traffic gets here from the search engines. The effort to make the site more attractive to the search engine spiders is what takes up the lion's share of my time on the forums. At all cost, we want the forums to be as user-friendly as we can make them. But I spend a considerable amount of time each week, streamlining all of the forum files I want the spiders to crawl and blocking them out of files that will not help our indexing. We try to prepare a keyword smorgasboard for the spiders to dine on and then we turn on all the neon signs that say 'Diner' and 'Eats', so the spiders know they can get their fill here.
We use a metric called a 'long tail' to track how many search engine visits we get as a result of how many keywords being searched. Actually, 'long tail' is a bit of a misnomer, as we want the number to get smaller with the progression of time. On 17 May, our long tail metric was 7.03 and today it is down to 2.46. That is excellent progress from where I'm sitting.
In the first 19 months of operation, we had 439 members and 8,771 posts. In the 5 months since, we now have 771 members and 19,081 posts. (And I just removed 40 unconfirmed members from the database a week ago.) A lot of these new members have arrived as a result of the links you have all provided on other sites, but as we continue to increase our long tail, the search engines are sending us a lot of new traffic, as well.
So, there's the long story on why I love seeing all the spiders around here.
I really wish I had more time to participate on the forums themselves, but my time is better spent making this site as comfortable as I can make it, for both the search engine bots and for our 'organic' traffic as well. The results of those efforts speak for themselves, when you look at all the growing statistics around here. Thumbs up to all you for inviting friends to join us and for providing links on other sites. Let me remind you, we opened this site in the hopes of seeing as many as 250 registered members, and now look.
Total guests that have visited the forum in the last 24 hours: 473 
The most guests online over 24 hours was 1,141, 03-31-2008 at 12:29 PM.
