anyone familiar with Google bot?

Archived Admin Announcements
5 posts Page 1 of 1
Slayer
O6P Staff
Posts: 1568
Joined: Sat Oct 14, 2006 5:33 pm
Location: Hamilton
 
by Slayer » Wed May 14, 2008 12:44 pm
Hey, i was checking my web logs today, because i noticed we are creeping very close to our bandwidth limit, and it shows that googlebot is taking up double the bandwidth that the users and other site crawlers are using combined

googlebot took up almost a GIG of bandwidth this month alone (14 days only used this month so far)

at that rate, google can take up 2 gigs a month (our set amount on this free server)

i mean it is good that google crawls and ranks us, but come on, that is a waste of bandwidth

i dont want to put in a NOFOLLOW because that will only let them list our main page, not all the topics..

I think the problem right now is, when google comes here, they follow EVERYLINK on the site, meaning every message gets read and saved... what im looking for is a way to have google only crawl maybe once a week, or once every 2 weeks... it seems that they are crawling daily, and possably many times a day.
Every time i login to my admin panel, it looks liek googlebot is there

is there anyone out there that knows how to limit googlebots crawl frequency?


IN other news, i have a server setup and ready to roll, i will possably be transfering everything next tuesday/wednesday, as that seems to be the lowest used times on the site (mid week)

 

inflames625
6th Planet Addict
Posts: 154
Joined: Sun Apr 06, 2008 9:52 pm
Location: Lancaster PA
by inflames625 » Wed May 14, 2008 10:00 pm
honestly to conserve bandwidth ide find car club sites and try and get your site link on there. I hate googlebots personally. though if its what u want google itself should have some answers. ill go looking right now for ya.

 

inflames625
6th Planet Addict
Posts: 154
Joined: Sun Apr 06, 2008 9:52 pm
Location: Lancaster PA
by inflames625 » Wed May 14, 2008 10:05 pm
Someone else had the same problem. Here's the answer that was supplied...

"You don't have a robots.txt file to disallow such basic urls like
those for register, login, post reply, private messaging, any where
being logged in is mandatory, etc.

PHPBB forums (and most other forums) are robot traps unless you can
set them to not generate urls with session id's for non-authenticated
visitors. It is said the PHPBB version 3 has this feature available.
None of the lower versions does.

As such crawling a PHPBB forum will encounter an endless number of
different urls just because of different session id's.

Unrestricted by anyhting, not just GSC would run around in circles -
any other robots including Googlebot and other search engine robots.
Except some of the search engine robots would get tired of crawling so
would stop eventually meaning to return later to complete the run.

You'd have known in about 2 minutes after starting to run GSC, just by
watching the crawlers, that it's running around in circles, with lots
and lots of urls getting generated.

With your version of PHPBB you should disallow the forum altogether in
the robots.txt file, because it's just not possible to crawl it
finitely.

And this in your case would leave your homepage as the one and only
page for the site which has almost no content on it worthy of
indexing."

hope this helps!

 

sc2sick
Registered User
Posts: 10
Joined: Tue Dec 18, 2007 8:19 pm
 
by sc2sick » Fri May 16, 2008 1:25 am
shoulda e-mailed me, i didn't even notice. i upped the bandwidth to 20GB/mo

 

Slayer
O6P Staff
Posts: 1568
Joined: Sat Oct 14, 2006 5:33 pm
Location: Hamilton
 
by Slayer » Fri May 16, 2008 1:56 am
sc2sick wrote:shoulda e-mailed me, i didn't even notice. i upped the bandwidth to 20GB/mo


that helped cause i was told when i was at work that it was down, thanks for that... i actually signed up for WHB myself and shoul dhave hte site transfered over as soon as i get some free time, i dont want to do it over a weekend again, probably mi week next week

its still a piss off that googlebot uses that much bandwidth
5 posts Page 1 of 1
kuchnie na wymiar radomsko

Who is online

Users browsing this forum: No registered users and 2 guests