ByteBriefBot

ByteBriefBot is the crawler behind ByteBrief, a tech news aggregator that links every story to its source. If you saw it in your server logs, this page explains what it is doing and how to control it.

What it fetches

It fetches public RSS and Atom feeds, and the public article pages those feeds point to so that a short summary can be written. Every ByteBrief story links back to the original article. It does not log in, submit forms or fetch anything behind a paywall or login.

How it identifies itself

The user agent is:

Mozilla/5.0 (compatible; ByteBriefBot/1.0; +https://bytebrief.ai/bot)

Requests are also signed with Web Bot Auth (HTTP Message Signatures, RFC 9421), so you can verify they really come from ByteBrief and not from someone copying the user agent. Our public keys are published at https://bytebrief.ai/.well-known/http-message-signatures-directory.

How it behaves

• It reads your robots.txt before fetching a page and obeys it, including Crawl-delay.
• It makes at most one page request to a host at a time, and waits out your Crawl-delay between requests.
• If your robots.txt cannot be fetched because of a server error, it does not fetch from that host.
• It does not retry blocked pages or try to get around bot protection.

How to block or slow it

Add this to your robots.txt and it stops fetching from your site:

User-agent: ByteBriefBot
Disallow: /

To slow it down instead, use Crawl-delay: 10 in the same group. Changes are picked up within about six hours.

Contact

Questions, problems or removal requests? Email [email protected].