I'm sure we're all familiar with seeing site crawlers from search engines such as Google and Yahoo seeking to index our posts. Something I noticed, though, is that while Google, Yahoo, and Bing crawlers are referred to as [Bot], China's Baidu search engine is tagged [Spider]. Does anyone here know why that is?

I think I had this conversation with you on steam silver. You found out that Baidu is China's equivalent to Google.
Also something about a white rabbit.

Nope. And actually, there was one I found a while ago called "Alexa[Bot]" or something out of the ordinary. I think a spider does the same thing as a bot

Spiders craw.

nojo34 wrote:I think I had this conversation with you on steam silver. You found out that Baidu is China's equivalent to Google.
Also something about a white rabbit.

I wanted to ask the question here, since there are other forumers that have been here longer that may know the root of this.

I believe- This is from a brief browsing of the ever so reliable wikipedia- That a Web Crawler is specifically used to make and update index lists of the site they visit- This allows search engines to access this data and draw from this index to find Key words that a search engine uses.

But the difference between a spider, a bot, and a Crawler particularly? Its beyond me, but if you understand some Website building lingo it may be apparent:

Spider Scan and judge you website content ,Crawler crawl you content and index it in there searches according to Inbound links both techniques helps google to judge importance of website in terms of ranking ,robot.txt file is that file which is included into web coding so that google not consider it in crawling or to hide page from google crawling

Interestingly enough- these 'web crawlers' are why the deep web exist- Bots and Spiders are unable to access certain sites leading to many sites and pages being hidden to the regular user.

I decided to do a quick test I copied this

mathwhiz9 wrote:Nope. And actually, there was one I found a while ago called "Alexa[Bot]" or something out of the ordinary. I think a spider does the same thing as a bot

and google searched it. You'd expect that, since the bots are always here, they might've databases it? Apparently not. It came up with some other links, none of them relating to subterfuge. I then decided to randomly look up some forumers. Searching Aclonicy lead me to the failed L1 tournament, and Roadkeihl took me to the lore competition. (Don't have links for either) Anyway, my point is. I believe that the bots don't fully go through all the posts, as this may be very extensive. Instead, they catalogue the thread titles (like Aclonicys L1 cup and Roadys Lore) and put those into their files.

As for the spider, I'm as confused as JD.

I've noticed it only finds the titles too because I've searched mathwhiz and it finds the picture about your birthday, math.

aclonicy wrote:I've noticed it only finds the titles too because I've searched mathwhiz and it finds the picture about your birthday, math.

Yeah, i remember searching myself once too

That's strange- all I get is job sites.