tollbot
Tollbot is a certbot based tool that proxies a website automatically (similar to certbot) so that AI scrapers will have to pay a fee in order to scrape the content It starts with both a backwards compatible variation of robots.txt with payment information within the comments and a 402 redirect to a standard payment gateway. The user runs tollbot like they run certbot on any system. They connect their wallet, choose the host, and then price out the endpoints in USDC/unit amount If the Crawl-delay in the robots.txt or some other metric is violated, the host gets returned 402s for the restricted endpoints and then must fulfill the payment to continue upon which point they get a Proxy-Authorization (RFC 9110) header and can retry. proxy-authorization token then acts as a "balance" and is independent of any other header or ip src information.
Category tags:
For Innovators & Creators