Written by Bjørn Børresen
on 

The Google, The Bad and The Ugly

Going from a PR7 to nil I should know what it feels like to be the coolest kid in class one day, and the geek back in the corner the next. I’m glad my income doesn’t rely on Google traffic, cause if it did I’d be in it deep.

Here’s the story: I’ve created a script, called phpODP that lets people put the content of dmoz (the open directory project) on their own pages. This is perfectly fine according to the dmoz licence, and my script is even listen on dmoz itself. A lot of people have downloaded this script, and are currently using it.

A little while ago I read Googleguy say something about Google wanting to do something about duplicate content. I think probably banning my script was one of the things he was talking about. They don’t want 100’s of dmoz out there, and frankly I can’t blame them (although I do think entire site banning is a bit to harsh). However, I can see a couple of positive things about including multiple dmoz sites in the index:

Who is going to do that now? No one.

I see odp @ bie.no is still listed in Yahoo! – I don’t know if they’re heading in the same direction as Google, but I have a feeling they are.

The psychology of a Google Outlaw (me that is).. the first days after being banned Day 1: Something must be wrong. ‘I must’ve spelled “site:bie.no” incorrectly – is that how you search to tell if you’re included in the index? It’s always been, hasn’t it?’ etc.etc. Day 2: Realising one has been banned, searching for “google banned”, sending a complaint to Google Day 3: “I don’t need Google” – start using Yahoo! and actually learning it isn’t all that bad. It can be used as a replacement. Almost a full day. Day 4: Logging into Amazon Associates and checking the damage. Damn. Day 5 – next Google Dance: Apathy

Anyhow, today I put up a robots.txt that should help things:

User-agent: Googlebot Disallow: /products/phpodp/odp.php Disallow: /products/phpodp/go.php I was delighted to find out that I could have User-agent in there, so I can make some special rules for Google. As you see, it shouldn’t access the odp script anymore.

Hopefully, this will get me unbanned – but it’s not for sure. I’ve got all those people using my script linking back to me from odp content as well. So Google might think “Duplicate content links to bie.no, bie.no must be bad”. I don’t know what I’m going to do with that if that’s a problem. But if it is a problem, then I’m not the only one in trouble (other scripts do the same, although maybe not with so much duplicate content).

Gaaah .. I’m tired after a long day of refurbising, so I’ll stop writing now :-)

Comments

Comment by Oliver on 2005-11-01 10:57:50 +0000

Hi Bjorn, it might be correct that you have been banned because of using your own script without having a portal around (no “integrated service”). But I don`t think so! Have you heared about Google Bombing meanwhile? Unfortunataly there is a new Google Service that checks if your number of backlinks has grown natural or not. This means, if too many new odps are spidered too often you will be banned because Google is of the opinion that you bought lots of backlinks overnight. On the other hand it is a new sport in the US to buy lots of backlings for competitors to get them banned! Perhaps Google will think about it and bring us new updates. But who knows? My tip: Make all your sites spider friendly (including correct meta tags) and let Google spider your site again when you are ready. Can`t understand why you exclude Google although you are banned already? How should they check if your content gets better if you exclude their spiders? (German SEO – not leaving URL on banned sites ;-))

Comment by yellowriver on 2005-11-03 19:26:06 +0000

Hi bjorn, many spamming sites have abused of terms as “odp”, “opendirectoryproject”, and names of most popular sites using “phpODP” for to increase abusively their traffic rank. :( The age of democracy is a very utopy. :(:( yellowriver

Comment by Bjørn on 2005-11-15 21:48:20 +0000

Oliver, I have not banned Google’s spiders entirely – only from visiting the odp.php script. I think that’s the best option now, since that is probably what got me banned. – Bjørn

Comment by Oliver on 2006-04-18 18:11:19 +0000

Did you already try to ask Matt Cutts what to do? I think there is no reason why you shouldn`t. ;-) Have you had a look at the Google reinclusion request tool already? http://www.mattcutts.com/blog/reinclusion-request-howto/ Good luck! And keep us informed about new versions, please. I think we all need a possibility to add the odp content in order to make it much more unique and more up to date. It takes very very long to become listet in the odp and a bit longer to become listed in the Google Directory and others. I personally would like to include my own tips on top of the odp list without having to add a second catalog. Will there be a possibility to buy an add on like that from you later this year? Best regards Oliver

Comment by Bjørn on 2006-04-18 20:36:27 +0000

Oliver, I’ve sent several reinclusion requests as recommended by Matt Cutts. As for contacting him directly, I read his ‘comment guidelines’ where he states that he ‘does not have time to look at individual sites’. Sent one more request to Google today, maybe I should include it in my weekly routine ;) As for adding your own content, sure that could be done, just send me an e-mail (see ‘About’)