Hey, i was checking my web logs today, because i noticed we are creeping very close to our bandwidth limit, and it shows that googlebot is taking up double the bandwidth that the users and other site crawlers are using combined
googlebot took up almost a GIG of bandwidth this month alone (14 days only used this month so far)
at that rate, google can take up 2 gigs a month (our set amount on this free server)
i mean it is good that google crawls and ranks us, but come on, that is a waste of bandwidth
i dont want to put in a NOFOLLOW because that will only let them list our main page, not all the topics..
I think the problem right now is, when google comes here, they follow EVERYLINK on the site, meaning every message gets read and saved... what im looking for is a way to have google only crawl maybe once a week, or once every 2 weeks... it seems that they are crawling daily, and possably many times a day.
Every time i login to my admin panel, it looks liek googlebot is there
is there anyone out there that knows how to limit googlebots crawl frequency?
IN other news, i have a server setup and ready to roll, i will possably be transfering everything next tuesday/wednesday, as that seems to be the lowest used times on the site (mid week)