Article
My blog is full of bots and AI agents
Some time ago I wrote about how I developed my own analytics tool. I built it to respect personal data protection. So much so that it does not even need the visitor's consent. I also wrote about the problems I had to solve during development and how AI helped me along the way.
The first version already had built-in bot detection and flagging of suspicious behaviour (probable bots). It was not enough. Visits, especially direct visits, were strangely high and took an odd share of the total traffic. So I decided to dive back into the topic and improve my own analytics.
Now, after tuning it to a reasonable level, I read with interest posts where someone claims that 20% of the visits on their website are bots and 80% are humans. I have replied to a few of them, on Threads for example, saying that I used to think the same. Today I would say that on your website too, the opposite ratio is closer to the truth.
How did I arrive at this view?
What did the first version of my analytics identify as a bot?
The first version stood mainly on two things. The first was simple. I know the list of known robots and I look for their names in the browser header. Googlebot, GPTBot, ClaudeBot, Amazonbot and others. The second was a browser that does not send the page language. That tends to be a quiet sign that there is no human behind the visit.
This catches the obvious ones. But a new group of visits appeared that it does not catch. These are programs that pretend to be a regular browser. There is no bot name in the header. They have the parameters of a human. The original filter does not expose them.
Why scroll no longer proves a human
It occurred to me that I would recognise a human by activity on the page. Scrolling, reading, clicking. There was a time when a robot would not do this.
That no longer holds. These new programs actually run the page. They scroll through the article, click a button, behave almost like a human. Scroll tells me the visit showed some activity. It does not tell me whether a human was behind it.
Bot detection: several signals at once
So I stopped looking for one signal and started combining them. None is enough on its own. But when I put several together, a score emerges. It lets me mark a visit as a probable AI agent even when no single signal gives it away.
Join the Library
Full access to my findings, personal stories, and what I learn from the people I meet.
Join the Library · €29.99 per year How do you stop depending on social media? Can… How to Start With AI in a Company: Automating… Fable 5 vs Opus: I Ran the Same Audit… I Ran Object Detection on My Laptop, and Saw… Dependent on AI: Are We… Which Work Will AI Not Replace? Not Yet…Get the full article by email and feel free to reply if you want to discuss it further.
Summary
Common questions on this article's topic
What percentage of internet traffic is bots?
How can you tell a bot from a human visitor?
Why does scrolling not prove the visitor is human?
What are random URL parameters like ?subject= or ?utm_source=?
How accurate is Google Analytics about bots?
Are bots taking over web traffic?
If you have any thoughts, questions, or feedback, feel free to drop me a message at mail@richardgolian.com.
LinkedIn
Related articles
What AI can do, how I use it, what effect it will have on the world, how I try to adapt to the new world. I have written a lot about it. I need to arrange all those thoughts somehow, to simplify them. So that I have one compass that shows me the direction.
When you hear that artificial intelligence still does not deliver good enough output, that it is unusable for certain tasks, it is worth realising that this may no longer be true today. And even if it were, the most intelligent minds on this planet are working on making AI more useful. Most work activities will be automated. To understand where this is heading, some questions have to be asked.
One of my family members described a firm that has its data in several different tools, and its employees still join it by hand, in spreadsheets. That is the ordinary state of firms. When somebody tells such people not to do it by hand and to use AI, they will not understand. Most firms do not even have the basic connectors between the tools they use. These firms need help with AI transformation.
More articles
How and why I am gradually leaving social media and the big platforms. When I read that Meta is testing a new way to charge for organic posts that link outside its platform, it confirms how right I was to become steadily less dependent on the big platforms and on social media. It does not mean I have left them completely. It means that whether or not I stay in touch with my readers depends on them less and less, and today almost not at all.
The same task, two models. Fable 5 against Opus 4.8. On paper Fable is the better model, with a larger context and stronger specs. And still it lost. Opus handled the task with a single round of checking for 721,000 tokens, while Fable needed nine rounds and burnt through 2.78 million tokens. The difference was not in the model, but in how I set the task. And I know it, because I measured it.
A few weeks ago I installed a small local AI model on my laptop that watches a live camera feed. I turned the webcam on in the dark, and in near total darkness it recognised me and the objects in the room. That such things exist, I have known for a long time. What opened my eyes was the accessibility. I installed it in one prompt, free, and it runs entirely on my machine, sending data nowhere.
I have Heidegger and my notebook beside me. I am asking where all of this is heading, where artificial intelligence is taking us.
Seventy per cent. That is where the first AI output begins, even when you give it the full company context and the best examples from the past. We are talking about the kind of output that cannot be defined programmatically. It is more complex. Often it is creative work. On one repeated type of output I reached eighty per cent within a week. Every further percentage point is harder than the one before.
For a long time we treated the internet as the main road. The place where work and relationships happen. Yet most of what we see on it today is, or soon will be, AI-generated: text, images, profiles and comments. The internet is turning into an online game full of bots, where you cannot be sure that a human is on the other side of anything. So I ask: was the online world the main road, or only a temporary detour that part of us will return from, back offline?
A few days ago I interviewed a senior marketer. An experienced man, years of practice. I asked him about AI. He said he barely uses it. He had one bad experience with the output and decided he was too senior for it to add value when it is not perfect. I know the other side too: professionals who automate everything that can be automated.
Europe does not have the capacity to face a full-scale, mass drone war of the kind we see in Ukraine. Three dependencies weaken it: China supplies the physical material for defence systems, the United States supplies capabilities Europe does not have, and twenty-seven states cannot agree how fast, or who pays. Rearmament plans exist, but they are being carried out slowly.
AI produces the graphic, the newsletter and the product page faster than a person. What is left for the one who used to do it is the judgement, knowing whether the output is good. But most people have worse judgement than AI. And whoever cannot judge quality cannot delegate either. How do you tell whether yours is the judgement a company relies on, or the kind it can replace?
Four days in Catalonia. No computer, no AI, almost no social media. I bought this notebook so that I could write down what I would think about, and what I would come across and learn on the trip.