Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Curl and libcurl doesn't. Wget doesn't, I think.


CURL isn't a bot, it's a tool. It can be part of a bot (which may or may not respect robots.txt) but it can also just act as a user agent directly for a human operator in which case it SHOULD just do what asked. Chrome doesn't follow robots.txt either for the same reason.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: