Search for a command to run...
Articles tagged with #web-scraping
Your fetch agent knows two endings to a request. 200: parse it. 403: back off, rotate, or skip. That branch has been the whole game for years. There's a third ending now, and it's the one your code fa
Turn 1 cost about 300 input tokens. Turn 20 cost 7,000. Same agent, same kind of page, 20 times more expensive for the last step than the first. Nothing was broken. The agent gave the right answer the
Yesterday I handed an agent a web_fetch tool. It fetched a page, got back a 200 and a screenful of text, and confidently built a plan on it. The text was a Cloudflare "Just a moment..." screen. The ag
Every MCP web-access tutorial I read this month pointed at a paid API. You don't need one. To let an AI agent read a public web page, sixty lines on the official MCP Python SDK give you a self-hosted
Your scheduled scraper re-downloaded the whole corpus last night. A few thousand records. About forty of them actually changed since the run before. It downloaded all of them anyway, because it had no
Your scraper ran clean. HTTP 200 on every request. The schema validated. Every price sat in a sane range, every date was in the past, every ISBN had thirteen digits. Zero errors, zero retries. You shi