We started building Scout in November. The idea was simple enough: give it a question, let it go and find the evidence, let it work out what that evidence means, and let it decide whether it has actually answered the question. If it has not, it goes and finds more. If it discovers that it has been asking the wrong question, it changes the question. Eventually it either comes back with something defensible or says, in the most Australian way possible, "Yeah, nah. I've got no idea. But I'll find out for you."

That was the idea. Nine or ten months, an unreasonable number of Lambda functions, a lot of architecture diagrams and somewhere around thirty or forty 2am conversations later, we have built a great deal of useful technology. We just have not quite built the thing we originally described, and that is the part of the story worth telling.

We haven't wasted the year

It would be easy to look at where we are and conclude that twelve months of R&D have passed without an agent to show for it. In fact we have done quite a lot. Scout now has a proper question-decomposition capability: give it something complicated and it breaks the problem into questions that can actually be investigated, rather than handing the whole thing to an LLM and hoping it works out what "investigate" means. It has a discovery layer that works across real data sources rather than a toy collection of APIs - Salesforce, Azure DevOps, Jira, Confluence, Teams, Exchange, S3, CloudWatch, the web and the other sources we have accumulated along the way - and that has become useful in its own right. We are no longer building demonstrations; we are using the infrastructure.

We have built evidence ingestion and evidence management properly, so raw material is captured, normalised, processed and retained with provenance, and we can distinguish what we actually found from what a model subsequently made of it. That turned out to be enormously important. We have also built the knowledge layer, which is one of the things I am happiest with. We no longer treat everything that comes out of an LLM as knowledge: evidence becomes evidence, concepts become concepts, relationships need grounding, and conclusions have to be attributable to something. That sounds like basic hygiene, but it is not, because we spent months discovering the many creative ways you can accidentally turn a model's interpretation into an established fact. The knowledge-compiler work has proven useful beyond Scout as well: we can take evidence and compile it into something structured and reusable, rather than throwing documents into a vector database and hoping semantic similarity will save us. It will not. Semantic similarity is very clever right up until you need to know whether something is actually true.

We have built genuinely useful capability

This is the part I do not want to lose in the story: Scout has produced a lot of capability that we use. The discovery infrastructure, the evidence pipeline, the knowledge compilation, the learning components, the durable execution model and the connectors are all useful, as is the ability to run long-lived work without pretending that a Lambda invocation is an immortal consciousness. We have built a system that can take a reasonably complicated question, break it down, acquire information from multiple places, bring that information together, preserve where it came from, turn it into structured knowledge and then reason over it. That is not nothing; it is quite a lot.

The problem is that we built almost everything required to support an investigator without solving the final and rather important problem: letting the investigator decide what to do next when it does not know the answer. That is where it gets interesting.

Because it still wants to answer

The frustrating thing is that Scout is very good at finding information, very good at processing it, and increasingly good at understanding it. It can challenge an answer and reject a conclusion when the evidence does not meet the required standard, because we have deliberately built contention into the analysis so that the model proposing an answer is not simply allowed to declare itself correct. All of that works. And yet, at the end of the process, we still have a system that looks at the latest model response and essentially decides that it sounds pretty good.

That is the part that is hard to accept, because the system has all the machinery required to know better. It has the evidence, the provenance, the knowledge representation, the competing interpretation, the challenge, the adjudication and the learning. And still, when presented with a sufficiently plausible answer from a sufficiently capable model, the gravitational pull is toward answering the question. The model says something coherent, the evidence does not obviously contradict it, the answer sounds reasonable, and the machine concludes that it has solved the problem. It has not. It has produced an answer, and an answer is not the same thing as a resolution.

The thing we underestimated

This is the real lesson of the last twelve months. We underestimated how fundamentally different answering is from knowing whether you can answer. The former is something LLMs are astonishingly good at; the latter is the actual agent problem. Ask an LLM a question and it has a strong bias toward doing something useful with it, because that is what we have trained these systems to do. Ask what caused something and it wants to tell you the cause; ask whether something is compliant and it wants to give you a verdict; ask which of several explanations is correct and it wants to pick one. Even when the correct response is that there is not enough evidence to distinguish them, the model does not much like that answer, and neither do users, product managers or demos.

Reality, however, does not care. Sometimes there is not enough evidence, sometimes the evidence is contradictory, sometimes the question is badly formed, sometimes the obvious explanation is wrong, and sometimes the available data simply cannot answer the question. In those cases the correct behaviour is not to produce a more carefully worded answer; it is to go and find out.

We kept trying to solve it with more architecture

We can laugh at ourselves a little here, because we have had enough conversations about architecture over the last year to fill a substantial conference programme. Every time something did not quite work, we found a reason: perhaps the orchestration was not right, or the state boundary was wrong, or a subsystem needed to be separated, or the model needed another judge, or we needed a learning loop, or the learning loop needed to be durable, or the question needed another decomposition pass, or the evidence needed another normalisation pass, or the decision needed another adjudication boundary. Perhaps we needed another model, or a better model, or a cheaper model, or to ask the better model to critique the cheaper one. At 2am, all of these are extremely persuasive.

The problem is that there is always one more architecture you can build, and if you are not careful you can spend an entire R&D programme building increasingly sophisticated mechanisms around the thing you have not actually solved. We have done enough of that now to recognise it when we see it. At least I think we have. Ask me again at 2am.

The adversarial approach was always the point

The adversarial approach is not a new idea we bolted onto Scout once we finally accepted that LLMs can be wrong; it was there in the original design. Scout was meant to investigate, and investigation inherently means trying to disprove your current understanding. If I think I have found the answer, the next useful question is not whether we can make that answer sound better, but what would make it wrong, and then whether we can find evidence of that. That is a fundamentally different loop.

We now have real components that do this: the candidate answer can be challenged, the challenge can be adjudicated independently, evidence can be rejected, and a conclusion can be held rather than fabricated when the system has a capability gap. Those are genuine advances. What we are still learning is how to make the whole system behave that way without us telling it to do so for a particular case. That distinction matters: if we hard-code the behaviour, we have solved the demonstration; if the architecture lets the system discover the behaviour, we have solved something far more interesting.

The learning system taught us something else

The learning components have been particularly revealing. We originally thought of learning as memory - the agent remembers previous investigations, what worked and what did not. That is useful, but it is not really learning. The more interesting capability is when the system can recognise that it tried to answer a question from a particular perspective and the evidence was not capable of supporting it, and then change what happens next: not because it needs to remember the sentence, but because it needs to change its model of the problem.

That is what the question-evolution work is really about: reject the current frame and try another, rather than asking the same question with slightly different words. It is one of the pieces of Scout with genuine potential, because it moves us away from the idea that learning is a matter of putting more material into the context window. Learning should change behaviour.

The architectural insight we were missing

We have spent a year thinking about Scout as a collection of stages: questioning, discovery, evidence, knowledge, analysis, decision and learning. That is a perfectly reasonable way to build the machinery, but the agent is not the stages. The agent is the thing that decides which stage matters next, and that is the abstraction we were missing. Not another subsystem, another Lambda, another model or another prompt, but the ability to look at the current state of an investigation and decide that it is not done, or that new evidence has invalidated the starting assumption, or that there are two explanations with no way yet to distinguish them, or that an answer looks plausible but has not actually been established, and occasionally that the answer is now in hand. That decision has to emerge from the state of the investigation rather than from the fact that the pipeline has reached its final box. That is the difference between a pipeline and an agent.

Which brings us back to MCP and A2A

This is why the MCP and A2A direction has become interesting to us, and not because putting Scout's subsystems behind MCP magically solves the problem, because it does not. MCP gives us a clean way to expose capabilities, and A2A gives us a way to expose the investigation itself as something another agent can ask us to perform, which is a much more natural boundary. Another agent should not need to know how Scout searches Salesforce, how evidence is compiled, or whether the work takes thirty seconds or three hours. It should be able to ask us to investigate something, and Scout should be able to reply that it has started, then that it has found something, then that the evidence does not support the original hypothesis so it has changed direction, and eventually either that here is what we know, or that we cannot establish the answer from the available evidence. That is an agent - not because there is an LLM somewhere inside it, but because there is a durable process of investigation that can decide what to do next.

What we have actually achieved in twelve months

Quite a lot. We have built a serious investigation runtime, learned how to decompose questions into things that can actually be investigated, and built a discovery estate that reaches into real enterprise systems and the public web. We have built evidence handling that treats provenance as more important than a tidy citation at the bottom of a generated answer, and a knowledge-compilation layer that turns evidence into structured, reusable knowledge rather than stuffing documents into a RAG system. We have built adversarial analysis and independent decision adjudication, learning that can reject an approach and change how the next investigation is conducted, and durable execution so that an investigation does not die because a model call ended or a Lambda timed out. Perhaps most importantly, we have learned a rather large number of ways not to build an agent, which is difficult to put on a product brochure and is probably the most valuable R&D outcome we have.

The remaining problem is wonderfully simple

After all of that, the remaining problem fits in a single sentence: we need Scout to stop believing that producing the best available answer is the same as resolving the question. That is it. We do not need it to be better at writing, more eloquent, or equipped with another architecture diagram or another clever prompt telling it to think deeply. We need the system to understand that uncertainty is an actionable state. If it does not know, it needs to investigate; if it cannot investigate, it needs to explain what capability is missing; if the evidence contradicts itself, it needs to preserve the contradiction; if its hypothesis fails, it needs to change the hypothesis; and if it has genuinely established the answer, then, and only then, it can tell us. And if it reaches the end and the answer is still unknowable, that is fine. In fact it is one of the most useful answers an agent can give.

"Yeah, nah. I've got no idea. But I'll find out for you."

We just need to teach the machine that this is sometimes a far better answer than confidently making something up. After twelve months, thirty or forty 2am nights, several complete changes of architectural direction and an unreasonable number of conversations that began with "I've been thinking about this", I think that is the part we are finally in a position to build. And if we get it right, the interesting thing is that almost everything we have built so far stays. We did not spend twelve months building the wrong thing; we spent twelve months building almost everything the right thing needs. We just had not worked out what the right thing was until now.