I’m sick of it. Slurp is Yahoo’s piece of crap robot. It’s the one sucking more of your bandwidth than MSN or Google put together. Check your logs, you’ll see.
Why do people let them? Because of the promise of a bunch of targeted traffic from a “big search engine!” Woooo. BFD.
I’ve had it with that “search engine.” I’ve banned Slurp from chewing huge chunks of my bandwidth every month for nothing in return. If any other site did what they do, they’d be getting sued left and right and banned by everyone as a scraping spammer.
Their results are horrific. They always have been. They are one of the biggest reasons Google is so popular. Google is the antithesis of Yahoo in every way that matters.
Yahoo’s answer to competition is to immediately go in the opposite direction. In this case, Google loves blogs, so Yahoo and their “engineers” fabricate a brilliant plan: “Let’s hate blogs!”
In a study, my own official study, of over 1500 sites in our network, all blogs, Yahoo has the least to do with traffic numbers of all three engines. It ain’t just me getting ignored. If you have a blog, you are NOT getting the kind of traffic from Yahoo that Google is probably sending you.
Content be damned!
“Google likes content, so we hate it” seems to be the prevailing wisdom over there.
If the Yahoos at Yahoo are bending you over each month for nothing, not even the most obvious rankings for your site, do what I did:
User-agent: slurp
disallow: /
That’s the best search engine optimization trick you’ll learn all week. Get rid of the dead weight. They don’t want people reading content any place else but on Yahoo.com - YOU are their competition, why should they reward you when they are trying to act like they never meant to be a search engine in the first place?
You can’t fire me, I quit!
Update 6/6/07
Now I am really puzzled. Some readers have done some searches today to see if I really rank that poorly in Yahoo.
Check out what they found (And remember that Yahoo has only sent me 262 visitors this month.)
Blog Marketing Tips #3 - #4
Go figure. You’d think I’d be rolling in visitors and knocking Google because I don’t have the same rankings over there. But if the traffic isn’t there, it isn’t there.
Don't
Miss Out On Free Traffic!
Subscribe to the FTR RSS feed
or our
email list so you don't miss out on real, traffic driving tips from Jack
Humphrey! Thanks for visiting!
{ 5 trackbacks }
{ 20 comments… read them below or add one }
Rick Butts 06.06.07 at 2:13 am
Wow Jack!
In a thousand years I never imagined that anyone who was interested in traffic, or organic search results would even think of disallowing /slurp - but I can’t argue with anything you are saying - especially about blogs!
There is no question that their search results display has for a long time been just to awful and cluttered to use by anyone other than anyone who had never heard of Google.
I keep hearing that Yahoo.com has more traffic than any other website - yet I cannot fathom this - why does anyone even use them who is not already tied to their email? (which sucks, too.)
When they came out with their own syndicated PPC - I thought they would pull their heads out of their butt and fix their terrible contextual advertising system - talk about antiquated!
But they only changed the name from Overture to Yahoo Search - and the interface but it’s so cumbersome I don’t even bother.
I don’t know whether to tell others about your post here - or wait to see if you still feel the same way in a month or two.
But I know and respect you as a real student of this game - and I know you read your logs - so I guess I’m gonna have to act on this one.
You are a very brave lad!
Rick Butts
David Ledoux 06.06.07 at 4:44 am
Wow, holey moley!
This is the most interesting thing I have read in days online…I have a 1 month old blog and Slurp is pounding it daily, you’re so absolutely right!
Kudos on a fine piece of CRITICAL THINKING! Keep sticking it to the ‘man’ Jack!
Brandon Hall 06.06.07 at 8:09 am
Forget Yahoo. They are a complete waste of time, energy and effort. Google rules and there is a reason for it. Yahoo and MSN are like the b**tard stepchildren who never learn their lesson.
Jack Humphrey 06.06.07 at 11:02 am
If I change my mind it will be for one reason only and that will be when Yahoo becomes an actual search engine.
They are polluted completely. No one should trust a thing in their rankings because of their complete disregard for what a search engine should be.
All search engines are self serving to some degree (they are businesses after all) - but Yahoo has gone too far for too long in the direction of using their directory of “natural results” (which are neither natural nor “results”) to put money in their pockets.
A result is an outcome. An answer to a problem or query. Go to Yahoo and try to solve a problem or get a definitive answer to something using their “engine.”
You will get results, don’t get me wrong, but all too often you will get very poor results. Results that make no sense if you know the industry you are doing searches in. Sites that should be there don’t show up. Irrelevant spammy sites DO show up in many searches. They seem to have no rhyme or reason and absolutely no control ever the quality of sites they “choose” to list.
I could get a guy on Elance to program something more relevant than Yahoo for $3000. I’m not kidding. They don’t use any kind of serious technology to produce their results - it’s like a 13 year old’s programming class project. Yet the whole world seems to bow to this mighty search engine. No one will talk about them like this but me and a handful of others.
It’s ridiculous.
People don’t always know what to think about Google, but one thing is for sure, they are the most trusted results based on a very fair ranking algorithm and we feel as though we always know where we stand with them.
Hey - we are businesses too. If a website of any kind doesn’t prove to be worth the time we spend trying to get traffic from it, we axe them.
Yahoo is just a website. Nothing more. It doesn’t produce results for me. What business sense does it make to continue letting them Slurp my bandwidth every month for nothing?
What sense does it make to continually try to make them see their results in my category are totally irrelevant without the FTR in them?
None at all. They don’t care. They are on a wholly different mission than a real search engine should be.
Yahoo is irrelevant in the search engine industry.
Bill Wardell 06.06.07 at 1:31 pm
Hi Jack,
It’s amazing how much of a difference a year makes when it comes to SEO’s techniques and strategies of the this bizarre SE online world…
Remember last year in our coaching program what you were talkking about and then doing, and wondered if Google would ever catch on….
Thanks for Honesty then and Now,
Bill Wardell
Nathan Anderson 06.06.07 at 1:43 pm
Boy, SOMEBODY didn’t have their morning coffee!
You’re a brave man, Jack. And now you see why my primary content isn’t placed on blogs…
MSN has it’s own tarring-and-feathering due. Live.com is so polluted with msn.com and microsoft.com URLs, it’s hard to find real information! My statistical analysis shows beyond any reasonable doubt that they manually manipulate their results to give their network sites enormous preference… That’s right: direct manipulation of their “natural” results in order to send people to their own sites. Typical for this greediest of corps.
Harry Pickett 06.07.07 at 1:25 pm
Jack,
I was amazed when I started reading this and then verified it for myself! This is why FTR is such great reading and a great resource for the most up to date info on what’s really going on “behind the scenes” and I really appreciate it.
Harry
El Yanqui 06.08.07 at 1:04 am
Wow, that was a great read! I have a new blog and I’m checking the logs now. I haven’t ever gotten anyone from Yahoo and I really don’t know of anyone who uses them that isn’t saddled with their subpar mail.
Jack Norell 06.08.07 at 4:35 am
I’ll jump on the bandwagon, almost. Yahoo! sends me 12% of my traffic, but that is low quality: Lots of fast pageviews and then off the site. Nearly no conversions.
MSN does a little better, but I think that’s because that search is now defaulted on a lot of PCs when shipped. In other words, I’m getting the clueless ones who can’t figure a webpage quickly.
WebStractions 06.08.07 at 5:35 am
This has been going on for years. I remember back in 2004 when Yahoo launched SiteMatch and the free crawl gobbled up tons of bandwidth, but the pages just disappeared into a black hole. Day in and day out, Slurp came knocking, ate your goodies and left without even a ‘howdy-doo’.
One thing Yahoo is doing right now, and it really sticks in my craw, is their newest selective page indexing directive by introducing a new class attribute of NOCONTENT. Not only is this method ill-conceived, but it is about the most stupidest thing I have seen Yahoo do to date.
I digress. It would not be totally encompassing if there were mention of Yahoo’s other crawler:
User-agent: Yahoo-MMCrawler
Disallow: /
Images will chew up more bandwidth than your Html pages will.
Chris Lang 06.08.07 at 8:36 pm
Let me play devil’s advocate here and maybe upset a few people.
#1 Low traffic from Yahoo! You can’t compare apples to oranges. In a tech based website you can’t expect much traffic from Y! Most of your SE based visitors are going to use Google. How many people interested in website promotion do you think would not use Google? Also in any market Google controls 60% or more of the searches.
#2 If Slurp seems to pound your site it is probably because you have RSS feeds that are embedded in MyYahoo pages. RSS feeds get crawled every day by not only Y! but the RSS online readers do a full crawl every day!
#3 If you ban slurp will your feeds on readers MyYahoo pages be updated? I would guess not. While MyYahoo does not qualify as a RSS reader because it only shows the link tag in feeds we don’t have any feed reader stats on MyYahoo! However if you look in your log files you can see how many subscribers you have in the agent tag.
I would love to hear the other side of this devil’s advodate coin I just threw in the mix! - Chris Lang
Stephen Spry 06.09.07 at 12:53 am
Jack, I encourage EVERYONE to look at their log files. These figures are from my awstats figures for one of my sites in May… and I’m appalled by it!
Yahoo Slurp - 29274+2618 HITS - 395.71 MB bandwidth
MSNBot - 13963+1722 HITS - 207.43 MB
Googlebot- 12539+9 HITS - 188.87 MB
(the + figure is # of hits on robots.txt file)
However, my visitors from each search engines during May tell another story:
Yahoo! - 2,407 visits
MSN Search - 1,726 visits
Google - 57,576 visits
So far in June for EIGHT days I have:
Yahoo Slurp, 12269+1109 hits for 165.14 MB = 431 visits
MSNBot, 4077+399 hits for 62.02 MB = 394 visits
Googlebot, 2986+9 hits for 44.42 MB = 14,603 visits
Another one of my sites Slurp hit 70953+4405 times, consuming 1.23 GB of bandwidth to send me 307 visitors in May.
Is it really worth putting such a HUGE extra load on your server to allow Slurp in? And since I have 20 or so sites on that VPS, it’s working a lot harder than it should!
If you want to keep Slurp, then try their crawl delay recommendation.
But I’m with you Jack! Slurp’s goooone!
Stephen
BillyWarhol 06.09.07 at 2:28 am
Yeah Yahoo! is totally lame*
the graphics on their Daily News + Messenger is so butt ugly it’s really beyond belief* I can understand maybe 10 years ago people signing up cuz they only had a choice of Crap from AOL or MicroPOOP but they are a complete Joke now*
They did buy Flickr for $35 Million which was the steal of the Century but with their Censorship crap they will lose everybody there too*
Yahoo! is as boring & bland as it comes*
;))
Bill Urell 06.09.07 at 8:51 am
Hmmm…
After checking my logs, I found that Slurp uses 3 times the bandwith of Googlebot, but they still provide %27 of my search engine visitors. My I would be grateful if my site’s traffic for for a month was what this site does in a day. So while my blog is young and growing, I think I’ll take and welcome any visitor that comes along. What I have learned here is to take Yahoo for a grain of salt.
Bill Urell
Caribbean Guy 06.09.07 at 11:19 am
In my experience, Yahoo’s crawler uses a lot more bandwidth than Googlebot. Not sure why, but bandwidth is cheap, so it doesn’t bother me.
Our site isn’t a blog, and perhaps that’s why it’s generally done OK (not great) in Yahoo’s SERPs.
Another thing I’ve noticed is that when we have been in roughly the same position on G and Y for for the same a keyword, the volume of traffic we get from Yahoo seems less than it should be. My sense is that while Yahoo supposedly has 25% of all searches, the actual ratio of visitors we get is much lower than that.
Jack Humphrey 06.13.07 at 11:01 pm
What Chris said about apples and oranges is a big deal.
Yahoo is like AOL in user base for sure. Not tech savvy, not going to attract the readers I get from Google.
Well said and a good point.
So something to be wary of before you go chopping off the Slurp is analyze the quality of traffic you get from Yahoo, not just the quantity, in return for their abuse of your bandwidth, however cheap bandwidth may be.
You might find you, in your niche, need to keep going with Yahoo despite the glaring disparity in their slurp-to-results ratios.
Commissioner 08.06.07 at 3:08 am
Thanks for posting the Robots.txt code to disallow Slurp. it was eating my bandwidth too. after a Google search I found your site. that’s some sorry programming code they’ve got there.
Africa safari bookings, hotels, lodges 08.23.07 at 7:20 am
You know what guys, yahoo had first mover advantage. They can sleep all they want and people will still continue using them. They were there first. Before google there was yahoo. In fact the last time i checked, google was using yahoo links to gauge your sites popularity. I wouldn’t dismiss them that fast if i were you.
In fact for me google hasn’t even indexed my page yet. No google traffic for me so for me it’s still yahoo.
Chris Lang 10.05.07 at 12:25 am
How do you feel about banning slurp now that MyYahoo beta now offers a full blown feed reader? My latest article about MyYahoo feed reader….
Jack Humphrey 10.05.07 at 7:49 am
I’m not sure how the two are tied together really. I am thinking about lifting the ban to see what Slurp is doing these days, as the outcry was not just from this blog by any stretch.
I am hoping they’ve done some fixing over at Yahoo.
Still the essential point is that I rank at Yahoo for key phrases I rank with in Google and Google is getting the searchers for my stuff whereas Yahoo is not.
So I will leave the ban lifted if it is not too taxing, but I still don’t expect much traffic from Yahoo because all the geeks who search for my stuff are using Google.