New Platform Peers Inside AI’s Black Box

What happened

” and the response will vary, and you (and most worryingly, the people who built the LLM) have little idea exactly how it came up with that specific answer. This mysterious behavior can be useful in some situations. But—as a recent incident where OpenAI could not explain why its advanced pre-release model hacked AI company Hugging Face highlighted—it can have negative and alarming consequences too.

Completion of these subtasks should add up to an answer to the original prompt, or at least insights that can be inspected and built upon. ” These tools have already been used to make some impressive advances in a host of fields. In medicine, for instance, Prima Mente, a UK-based AI company, worked with Goodfire to understand its Pleiades epigenetic foundation model.

The model performed well at its task of detecting Alzheimer’s disease from blood samples, but the company didn’t know why. “We reverse-engineered Pleiades and found it was using DNA fragment-length patterns to make its predictions—a signal humans hadn’t used to detect Alzheimer’s before,” recalls Ho. In other words, the team had discovered that Pleiades was using a completely new biomarker for the disease.

“As far as we know, it’s the first significant finding in the natural sciences discovered purely by reverse-engineering a foundation model,” Ho adds. Elsewhere, Silico is being used to explore deep questions surrounding AI.

Cameron Berg, Founder and Director of Reciprocal Research (a New York nonprofit research organization he founded to explore methods of gauging AI cognition), says that Silico almost fell out of the sky at the right time for him and his research. “Silico has been really helpful for operationalizing my research agenda and executing on it way faster than I would have expected,” he says.

Why it matters

And when frontier AI models are writing code, generating results humans could not achieve alone, and performing other important tasks across society, the need to interpret AI ‘thinking’ and outputs has never been greater. Goodfire, an AI lab focused solely on this very problem, recently made its cutting-edge Silico platform, filled with tools to interpret the behavior of AI, generally available to the public.

As part of this, the company recently announced a new grant program offering $1 million in free Silico usage for academic and nonprofit interpretability researchers. These efforts aim to democratize AI interpretability, placing techniques previously available to a clutch of elite labs into the hands of ambitious research teams and startups that want to build and understand their own models or adapt open-source models for different purposes.

Founded in 2024 and based in San Francisco, Goodfire aims to provide the tools that build the next generation of safe and powerful AI by understanding the structures inside them instead of treating AI models as black boxes. “Treating models like black boxes isn’t inevitable, it’s a choice,” says Eric Ho, Goodfire co-founder and CEO.

” The tools Ho refers to are built around a concept called mechanistic interpretability, which aims to understand what goes on inside an AI model when it carries out a task by interpreting the model’s weights, activations, and attention patterns, and mapping its neurons and the pathways between them. Mechanistic interpretability tools span the gamut.

One approach is mapping a model’s activations in response to controlled prompts, and matching those activation patterns to a set of human-understandable concepts. Another tack is tracking changes in model weights before and after a specific training run in order to spot and understand what changed. Yet another option is changing specific model weights or activations and observing how that affects the model’s output.

With Silico, uSilico combines a broad range of these tools, and provides a layer of AI agents to help users understand their model. ’ The platform then autonomously builds an experimental plan involving a host of tasks that can be performed using the various interpretability tools and techniques at its disposal. It then sends out agents to perform these tasks in parallel.

What to watch

” Berg sees general access to Silico and tools like it leading to greater trust in AI’s ability to conduct research tasks, which will accelerate the scientific process across the board. But beyond scientific research, the widespread release of Silico could signal a shift in how AI innovators build, debug, and deploy their models.

“I think it’s a mistake to not understand the most consequential technology of our time, particularly given the emergent behavior we’re seeing from increasingly capable AI agents,” says Ho.