SEO Information

Search Bots, Crawlers, and Spiders


If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/


MORE RESOURCES:

Over One Million Placements Achieved by USA SEO Pros
PR.com (press release), NY - 51 minutes ago
Las Vegas, NV, September 05, 2008 --(PR.com)-- USA SEO Pros has successfully achieved 1194195 firs page placements on the major search engine as of August 1 ...
USA SEO Pros Expands Services ro Include Image Gallery and Video ... Newswire Today (press release)
all 2 news articles


Ethical SEO Marketing Company Unlimited Web Solutions Expands ...
PR Web (press release), WA - 51 minutes ago
Unlimited Web Solutions, a leading SEO and Internet marketing company, has announced the expansion of their facilities, pricing structure and website. ...
Search3w Named As Finalist In The Top Seo 30 Awards PR Web (press release)
all 2 news articles


ABC News

Who cares about Chrome? The SEO mindset.
Techworld.com, UK - 13 hours ago
How many browser wars are we on now? I make it at three and counting, each one won by achieving a different objective. Browser war one, the war to control ...
Video: Tech Test: Google Chrome Lacks Polish AssociatedPress
Google: Chrome browser no Trojan horse TelephonyOnline
Review : Google Chrome gets people talking Hit Search
E-consultancy - WebProNews
all 3,411 news articles


SEO 'should incorporate accessibility'
Epiphany Search News, UK - 4 hours ago
It is a good idea for marketers to consider website accessibility when undertaking search engine optimisation (SEO), according to one expert. ...


SEO Expert Cindy Krum to Participate in the 2008 Search Engine ...
PRarticle.com (press release), Canada - 10 hours ago
Monday, August 18th, Cindy Krum will be a panelist on the Mobile SEO: Death of the .mobi panel. This session explores how search marketers can use one ...


Search Engine Journal

5+ SEO Text Analyzers for SEO Diagnostics & Copywriting
Search Engine Journal - 9 hours ago
Both with on-page SEO diagnostics and SEO copywriting, we often need to perform various types of text analysis. Here is the collection of tools that should ...


Best Syndication

SEO New York Enhances Your Business Activity on Web
Best Syndication, CA - 14 hours ago
SEO New York specializes in search engine optimization (SEO) services which will help your company’s website rank in the list of top ten websites in the ...


SEO and Web Accessibility Come Hand in Hand
Search Engine Journal - 18 hours ago
By implementing accessibility standards, you put an end to questionable SEO issues. Include descriptive alternative text and title for content-sensitive ...


How Do You Do SEO for a Site that is Under Construction?
Search Engine Roundtable - 17 hours ago
Sometimes it's better to do SEO for a site before it's launched. Such is the case for a High Rankings Forum member who is looking to build up his site's ...


Direct Traffic Media

SEO spending 'to reach £330m'
Epiphany Search News, UK - Sep 3, 2008
UK spending on search engine optimisation (SEO) will reach £330 million this year, up 32 per cent compared with 2007. This is according to E-consultancy's ...
UK search marketing spend 'to reach £2.75bn in 2008' DirectNews
Search expenditure to rise by 24 per cent PrecisionMarketing
SEM expenditure 'to rise 24% in 2008' Internet Marketing News
Business Feet - Digital Response Media
all 10 news articles

SEO - Google News

home | site map
© 2006