Going from a PR7 to nil I should know what it feels like to be the coolest kid in class one day, and the geek back in the corner the next. I’m glad my income doesn’t rely on Google traffic, cause if it did I’d be in it deep.
Here’s the story: I’ve created a script, called phpODP that lets people put the content of dmoz (the open directory project) on their own pages. This is perfectly fine according to the dmoz licence, and my script is even listen on dmoz itself. A lot of people have downloaded this script, and are currently using it.
A little while ago I read Googleguy say something about Google wanting to do something about duplicate content. I think probably banning my script was one of the things he was talking about. They don’t want 100’s of dmoz out there, and frankly I can’t blame them (although I do think entire site banning is a bit to harsh). However, I can see a couple of positive things about including multiple dmoz sites in the index:
- The main site is often slow, so other sources may be preferable
- Sites can add content/functionality (e.g. user comments about the sites, rating, etc.)
Who is going to do that now? No one.
I see odp @ bie.no is still listed in Yahoo! – I don’t know if they’re heading in the same direction as Google, but I have a feeling they are.
The psychology of a Google Outlaw (me that is).. the first days after being banned Day 1: Something must be wrong. ‘I must’ve spelled “site:bie.no” incorrectly – is that how you search to tell if you’re included in the index? It’s always been, hasn’t it?’ etc.etc. Day 2: Realising one has been banned, searching for “google banned”, sending a complaint to Google Day 3: “I don’t need Google” – start using Yahoo! and actually learning it isn’t all that bad. It can be used as a replacement. Almost a full day. Day 4: Logging into Amazon Associates and checking the damage. Damn. Day 5 – next Google Dance: Apathy
Anyhow, today I put up a robots.txt that should help things:
User-agent: Googlebot Disallow: /products/phpodp/odp.php Disallow: /products/phpodp/go.php
I was delighted to find out that I could have User-agent in there, so I can make some special rules for Google. As you see, it shouldn’t access the odp script anymore.
Hopefully, this will get me unbanned – but it’s not for sure. I’ve got all those people using my script linking back to me from odp content as well. So Google might think “Duplicate content links to bie.no, bie.no must be bad”. I don’t know what I’m going to do with that if that’s a problem. But if it is a problem, then I’m not the only one in trouble (other scripts do the same, although maybe not with so much duplicate content).
Gaaah .. I’m tired after a long day of refurbising, so I’ll stop writing now :-)