Why Traceability Matters

In the last article, I argued that the value of a simulation system comes less from the AI model itself and more from the structure built around it. That leads us to a very simple, but core, question: How can we trust the output?

Even if a system is well-structured, every conclusion still needs to be understood and justified, otherwise you’re just replacing one black box with another. Once populations, context, and reasoning are treated as distinct layers, though, it becomes possible to ask not only what the system concluded, but how it reached that conclusion. Without this capability, Project Brainstorm would be no different from the many AI systems that are very good at producing answers yet offer little visibility into how those answers emerged. 

This may not matter if you're using AI to help manage an overflowing email inbox or draft a presentation. But if you’re using AI for research, strategy, investment decisions, market entry, or product development, understanding the reasoning process becomes critical.

As we uncovered in the earlier articles, the most useful question is usually not "What happened?" but "Why?". That’s what traceability is all about.

Most AI systems are optimised to produce conclusions, rather than to expose the process that produced them. People often refer to large language models as black boxes because researchers can never assign 100% certainty as to why any answer was given.

You'll often see outputs that look something like this:

     “Consumers are resistant to paying a premium price.”

Or:

     “This positioning is likely to create confusion.”

These statements by themselves may be useful to you. The problem for me, though, is that, on their own, they don’t tell you enough so you can figure out what to do next. I think the more useful questions are:

Without the answers to these questions, you're effectively accepting a conclusion on blind trust. But when you have them, you can begin to understand the mechanism behind it.

That's how traceability becomes valuable. The goal is never to simply generate outputs in response to queries. Instead, the goal is the whole journey from evidence to reasoning to conclusions visible enough that it can be examined, challenged, and improved.

But wait, you say: aren’t modern AI search engines such as Perplexity already solving this problem by attaching citations to their answers? 

To a degree, yes. An output with citations is certainly an improvement over an opaque output. Knowing where a piece of information came from is always better than not knowing. Duh.

But citations still only answer one question:

Another subtlety but the latter is so much more useful to understanding.

Imagine a report cites several articles discussing concerns about authenticity in a product category. The citations tell you these sources exist. What they don’t tell you is how those concerns flowed through the simulation, influenced different personas, shaped discussion, and ultimately contributed to a finding.

In other words, citations help you trace information, they can’t help you trace reasoning. Our goal isn’t simply to attach sources to an answer after the fact. It’s to make the path from sources, to context, to reasoning, and to conclusions, visible enough to be inspected. 

This is why, from very early on, I consciously designed Project Brainstorm to move beyond citations and towards something deeper.

In Project Brainstorm, traceability isn’t a single feature – it’s a series of layers where each layer answers a different question about where a conclusion came from.

I designed four major layers into the simulation system:

Taken together, these layers allow a finding to be traced backwards from a conclusion to the reasoning that produced it, the context that influenced that reasoning, and the evidence that informed the context itself.

Everything begins with the population.

Before a simulation can explore how people think, it first needs to establish who those people are. So, population grounding attempts to answer questions such as: 

This doesn’t tell us what people think. Rather, it creates the foundation from which thinking emerges by answering the most basic question: “Who exists inside the simulation?”

People do not make decisions in isolation. People consume information daily. They encounter narratives, they absorb news, local conversations, cultural references, and social signals.

That’s where context grounding comes in. Context grounding attempts to answer:

“What information exists within the world these people inhabit?”

This might include:

The goal here isn’t simply to collect information. The goal is to construct the information environment within which reasoning occurs.

That sounds straightforward in theory but presents challenges in implementation.

Traditional research often treats context as fixed because it must. Research takes time. Surveys, focus groups, analysis, and reporting can take weeks or months from start to completion. As a result, people’s concerns, priorities, and information environments are effectively frozen at a particular point in time.

Yet, even in the most mature markets, this is rarely the case. Markets rarely stand still. News changes. Political events change. Products enter and exist categories. Narratives evolve. Something that felt obvious six months ago may no longer be relevant today.

Project Brainstorm’s Dynamic Live Grounding system was created to address this. Rather than relying on a static body of information, the platform continuously retrieves and incorporates signals from the outside world when constructing its view of a market. Its objective is not to build a permanent model of reality but to build a current one. See the distinction?

This becomes especially important in a region like Asia-Pacific, where markets evolve quickly and information environments can vary dramatically from one city or country to the next. A consumer in Osaka, Jakarta, Bangkok, or even Shanghai versus Chongqing, is often operating within a very different information environment from someone in New York or London.

This raises another important question:

     “Whose version of reality should the system prioritise?”

Many AI systems implicitly default towards English-language content and globally visible narratives because that information is often the easiest to access. The problem is that global visibility and local relevance are not the same thing.

Local journalism, local narratives, local language sources, and local events frequently tell a very different story about what people care about, what concerns them, and how they interpret the world around them.

That's why Project Brainstorm attempts to prioritise local context wherever possible. The goal isn’t to ignore global perspectives but to avoid assuming that global perspectives are the same thing as local understanding. For understanding how people think, the information environment they live within usually matters far more than whatever happens to be most visible on the English-language internet.

Once populations and context have been established, the next challenge is making those inputs visible to the user. This is where study-level traceability comes in.

Today, every Project Brainstorm study already contains several traceability mechanisms. Users can see:

The purpose of these elements isn't simply documentation, they're there to help users understand where the simulation came from. A finding never emerges in isolation. It emerges from a chain of assumptions, information, context, and reasoning.

The more visible that chain becomes, the easier it is to understand what the system is saying. And the more trustworthy the output becomes.

As I noted earlier, citations are useful. Citations tell you where a piece of information came from. What they don’t tell you, though, is how that information influenced the conclusion.

A study may cite several articles discussing concerns about authenticity within a product category. The citation tells you those sources exist. They don’t explain is how those concerns flowed through the simulation, influenced different personas, shaped discussion, and ultimately contributed to a particular finding.

That's the gap between sourcing and reasoning which leads to the next layer.

Population grounding tells us who exists. Context grounding tells us what information exists. Study-level traceability shows us what sources informed the simulation. 

Claim-Level Evidence Grounding then asks: 

     “Which specific findings can be connected back to external evidence?”

This distinction is important even if the objective is not to prove the simulation correct. A simulation remains a simulation. A synthetic finding remains a synthetic finding. We’re not trying to transform simulated observations into facts. Rather, the goal is to make it easier to distinguish between simulated reasoning, external evidence, and the relationship between the two.

Suppose a report concludes:

     “Consumers are concerned about product authenticity.”

A first reaction is usually to agree with that conclusion and move on. After all, of course authenticity is preferable to something that’s an imitation. No? No. Not always. After all, Ralph Lauren has built an empire on imitation, with his prep school and urban cowboy themes. As with everything else, it depends on context.

So, Claim-Level Evidence Grounding does not attempt to declare that conclusion is true. Instead, it asks whether there are external signals that might help explain, support, contextualise, or even contradict the observation.

For example:

Over time, individual claims can be linked to supporting evidence, partially supporting evidence, contradictory evidence, or situations where no meaningful evidence can be found at all. The objective isn't to create a bibliography. The objective is to gauge the relationship between evidence and reasoning.

Put in another way, a citation explains where information came from. But Claim-Level Evidence Grounding helps explain how that information contributed to the reasoning process.

Imagine a study concludes:

     “Japanese consumers may not trust this premium tea brand.”

That's interesting. But what do you do with it? There's no obvious next step. The finding tells you what may be happening, but not why.

Now imagine the conclusion looking more like this:

     “Japanese consumers may not trust this premium tea brand because concerns about authenticity, provenance, and lack of sourcing information repeatedly emerged across multiple personas. Similar themes appeared in the broader market context and supporting external evidence suggested these concerns have been present elsewhere within the category.”

Suddenly, the conversation changes. You're no longer looking at a conclusion. You're looking at a chain of reasoning. You can see what emerged inside the simulation, what context influenced it, and what external signals may have contributed to the outcome.

And once you understand that chain, you can start making informed decisions.

Maybe the issue isn't price at all. Maybe it's packaging. Maybe it's provenance. Maybe it's trust. Maybe consumers simply need more information before they're comfortable paying a premium.

The finding itself isn't the most valuable part. Rather, understanding what produced the finding is.

That's where traceability becomes useful. It allows you to move beyond simply observing a result and start exploring what might need to change for the result to change as well. Ultimately, most decisions aren’t about understanding what happened but trying to understand what to do next.

At this point it's probably worth being clear about what Project Brainstorm is trying to do, and what it isn't.

The outputs are intended to be directional.

They're designed to help you understand how people reason, where assumptions begin to break down, what tensions exist between different perspectives, and what still needs to be explored or tested.

They're not predictions. They're not forecasts. It’s not to estimate market size. And they're certainly not a substitute for customers, experiments, or real-world validation. Those are important activities, but they answer a different set of questions.

As I continue to develop Project Brainstorm, we might arrive at more of those capabilities down the line. But for now, the system sits earlier in the process, to help navigate uncertainty before resources are committed, before strategies harden into plans, and before assumptions become expensive mistakes.

That’s why traceability matters so much. If the outputs are intended to inform decisions rather than provide definitive answers, then understanding how those outputs emerged becomes just as important as the outputs themselves.

That's the role it's designed to play. It’s to help you understand why a particular line of reasoning emerged, so you can decide for yourself what to do next.

The more I worked on Project Brainstorm, the more I came to believe that bad decisions are rarely caused by a lack of data. Much more often, they're caused by assumptions that were never challenged.

People move forward with a particular view of the market, a particular understanding of customers, or a particular belief about what matters, (like “authentic is always better than imitation!”) and those assumptions quietly shape everything that follows.

Sometimes they're right. Sometimes they're not. The difficulty is knowing the difference before resources have been committed.

To be sure, simulation doesn't eliminate that uncertainty. If anything, it often reveals just how much uncertainty exists. But what it can do is make the uncertainty more visible. And traceability extends that visibility one step further.

Instead of asking users to simply trust a conclusion, it helps them understand where the conclusion came from, what assumptions influenced it, what evidence informed it, and what context shaped it.

Of course, that doesn’t make the outputs infallible. But it does make them easier to inspect, challenge, and interpret. In practice, that means you’re no longer staring at a result and wondering whether to believe it.

You can examine the reasoning that produced it, you can identify the assumptions it depended on and then judge for yourself whether those assumptions make sense. Because the better you understand how a conclusion emerged, the easier it becomes to decide whether it deserves to influence a real-world decision.

As I keep saying, Project Brainstorm is still an imperfect system. It has limitations. It operates within constraints. But many of those constraints are deliberate, for now.

My goal isn't to replace research or to replace validation. At this stage, the goal isn't even to provide definitive answers.

The goal is to improve the stage that comes before all of that and make it more transparent. To make the reasoning easier to inspect. To make assumptions easier to challenge. To make the path from evidence to reasoning to conclusions easier to follow. And then gradually extending that traceability all the way from a market signal to the findings that emerge from it.

Ultimately, I think that’s the most useful kind of system.