There’s a good post on Dark Reading by Ido Safruti about norms and etiquette for bot code. According to Imperva’s most recent bot traffic report, bots comprise the majority of Internet traffic. May bots are intentionally disruptive or misleading — for example, bots that create comment link spam on blogs. Others are useful — for example, they, allow a search engine to index web pages. Even useful bots can be disruptive, such as by using up site capacity, and the robots.txt standard has been developed so that site owners can limit or exclude bot traffic.
Safruti provides the following guidelines for ethical bot code:
1. Declare who you are;
2. Provide a method to accurately identify your bot;
3. Follow robots.txt;
4. Don’t be too aggressive.