My blog is full of bots and AI agents

Some time ago I wrote about how I developed my own analytics tool. I built it to respect personal data protection. So much so that it does not even need the visitor's consent. I also wrote about the problems I had to solve during development and how AI helped me along the way.

The first version already had built-in bot detection and flagging of suspicious behaviour (probable bots). It was not enough. Visits, especially direct visits, were strangely high and took an odd share of the total traffic. So I decided to dive back into the topic and improve my own analytics.

Now, after tuning it to a reasonable level, I read with interest posts where someone claims that 20% of the visits on their website are bots and 80% are humans. I have replied to a few of them, on Threads for example, saying that I used to think the same. Today I would say that on your website too, the opposite ratio is closer to the truth.

How did I arrive at this view?

My analytics tool. A switch to hide AI agents, and a note that the year-over-year comparison is understated for now, while older data still contains bots from before the filter.

What did the first version of my analytics identify as a bot?

The first version stood mainly on two things. The first was simple. I know the list of known robots and I look for their names in the browser header. Googlebot, GPTBot, ClaudeBot, Amazonbot and others. The second was a browser that does not send the page language. That tends to be a quiet sign that there is no human behind the visit.

This catches the obvious ones. But a new group of visits appeared that it does not catch. These are programs that pretend to be a regular browser. There is no bot name in the header. They have the parameters of a human. The original filter does not expose them.

Why scroll no longer proves a human

It occurred to me that I would recognise a human by activity on the page. Scrolling, reading, clicking. There was a time when a robot would not do this.

That no longer holds. These new programs actually run the page. They scroll through the article, click a button, behave almost like a human. Scroll tells me the visit showed some activity. It does not tell me whether a human was behind it.

Bot detection: several signals at once

So I stopped looking for one signal and started combining them. None is enough on its own. But when I put several together, a score emerges. It lets me mark a visit as a probable AI agent even when no single signal gives it away.