A colleague asked me a few months ago, “Listen, we have so many AI tools running, from cybersecurity to customer support. Are any of those tools agentic?”. I have been working with AI tools since 2018, and mind you, I had to stop right there, google a bit, and think as hard as I could just to answer that “it depends”. I simply didn’t know, and possibly don’t know, where the line is.
This chapter is less about my attempt to educate others, but rather to start a discussion about what makes a system agentic and whether we can define it cleanly, as we can define at least that a system is “distributed”.
Security Backlog
The agent was called “cross-service-reviewer,” and for several months nobody had much to say about it. Its job was the one some busy developers hate: every night the scanners ran across about 400 repositories and orchestrated an intricate set of static and AI tools to ensure our codebase is safe and/or has a good trajectory to become safe in the near future.
The toolbox is banal: Software Composition Analysis (SCA), Static Application Security Testing, a couple of Kubernetes-focused tools to reveal the configuration, resources, and connections between services, and a couple of sub-agents that check code bugs. As you may expect, a system like that may produce a ridiculous amount of noise, so the “cross-service-reviewer” made sure the bugs are real. SCA findings were reported as is, but taken into account by other agents. In the end, the system notifies owners of the repository and urges them to take action on the findings as soon as possible.
The “cross-service-reviewer” was a monstrosity: to do what it was supposed to do, it had read access to repositories, read access to the Kubernetes API (almost unrestricted), an API to create, update, and suppress previous findings, communicate via Slack, and the ability to call effectively any tool it deems it needs to prove or disprove findings. In its early version, it had been looking at the code from different angles, effectively simulating a source-sanitizer-sink model. The permissions of the agent were also largely unrestricted, so it could edit its own configuration when needed to give it more control over the scanning and verification process.
Of course, the system worked well until it stopped working. Two things, two problems of different kinds happened, and one of them is one of the key premises of the book: that one of the traits of agentic systems is that they are, in practice, distributed, or, to be precise, have the same pitfalls, dangers, and disadvantages as distributed systems once deployed for real use.
The scan scheduled at 02:00 against commits on main. My reasoning was as follows: nobody works that late. The system can take its time to tally up the day’s improvements without overloading the system or misreporting. Some teams, however automated security updates that come from Dependabot, and if all checks are green, a GitHub action merges them. Sometimes it takes time to run tests and builds, so the timing is inherently undeterministic. A repository scan also takes time, roughly 5 minutes per repository, depending on the codebase and diff size.
Of course, this night the stars aligned and the pull request got merged right when the “code-quality” agent was reviewing the codebase. Several bugs were introduced by third-party packages, and the “code-quality” agent justly left them be in its output, keeping the references to the CVEs they come from. The SCA was subsequently run by the “cross-service-reviewer” and found no such references in the code, as it pulled another commit, where Dependabot addressed the issues. Referencing nothing, the “cross-service-reviewer” simply decided to report nothing at all. That means that the security dashboard was left unchanged for many days until the line chart on the dashboard became suspiciously flat.
The agent didn’t have “to report nothing at all”: it had enough capabilities to compare commits or, for example, re-run the scan again, but it “decided” that doing nothing in this failure mode is the best way forward. Perhaps, it could even figure out how to communicate with engineers via Slack, but it didn’t. It recorded its last commit as a completed milestone and has been broken since then.
In a completely different context, for example, if we build a distributed system to handle transactions, this reads like struggles of a university first grader: everybody knows how to handle guarantees, idempotency, event sourcing, or locking. But here, I made a completely avoidable mistake simply by not thinking of the whole thing as distributed and not taking into account the fact that the system’s output, at several steps, was probabilistic.
To untangle the issue, an engineer had to look at what was going on. The logs the system produced were seemingly legit: with the combination of factors it was given, the old bugs never disappeared, and the new ones never added. Some of the bugs actually fixed were still reported as present, with a hallucinated but remotely believable justification why so. All steps were green, nothing failed, no error messages; the agent, deep inside its internal machinery, produced the outcome. However, a qualified engineer could change the “last scanned” commit record several weeks back, replay the processing, and everything worked. There’s a good chance the system would work without editing the commit property: the agent could decide differently this time, report fixed and newly introduced issues, or run the tools in a different order.
Is this system agentic?
Besides the obvious bugs we discussed, the system didn’t have enough guardrails or a failure mode that could force reconciliation. In essence, it was a long-running (many months) ChatGPT query that, at the end of the day, needed a human response (the commit editing) in order to make it work again. We are generally ok with the fact that a bash script cannot improvise and halts or misbehaves at any data input that wasn’t foreseen by its creators. Unlike with a bash script, a developer may rely on the ability of the LLM to take on the unknown, unexpected, or malformed input, map or restructure it, and execute the pipeline regardless. Less so when the system needs to make a judgment between two equally plausible inputs or states of its subsystems. In this specific example, the system didn’t improvise a new remediation strategy, despite visually, to a naked human eye, the output, especially plotted on the chart, was immediately recognized as faulty.