Startups Target Transformer Limits with New Attention

What happened

Nearly a decade after Google researchers published the seminal "Attention Is All You Need" paper, the underlying transformer architecture powering modern artificial intelligence is confronting fundamental scaling limits. Dense attention—the core mechanism in transformers that compares every token in a text sequence against every other token—requires quadratic computational overhead as sequence lengths increase.

This design creates extreme resource demands for extended context windows, advanced reasoning chains, and agentic workflows that must digest immense volumes of enterprise data, codebase files, or document libraries simultaneously.

To address these core technical bottlenecks, a fresh cohort of AI startups is actively engineering alternative architectural foundations. Miami-based startup Subquadratic has developed SubQ, a novel sparse attention framework that dynamically identifies and processes only the most critical word pairings during inference. Subquadratic claims its dynamic evaluation achieves performance parity with leading general-purpose frontier models on specialized tasks like code generation and search, while drastically scaling down required matrix multiplications.

Concurrently, San Francisco startup Manifest AI is pursuing a complete departure from traditional attention frameworks. Rather than retaining every token across a sliding context window, Manifest AI implemented a mechanism termed power retention. This system maintains a continuous rolling summary of state information, incrementally discarding lower-priority details as incoming data flows through the network. This structural change prevents context memory from expanding uncontrollably while keeping essential task parameters intact.

Why it matters

The AI industry's reliance on dense attention has driven compute infrastructure expenditures and energy consumption to unsustainable levels. Leading frontier developers face astronomical operational costs to run deep reasoning chains, while global energy grids face unprecedented strain from hyperscale data center expansion.

As model capabilities expand toward complex software development and multi-step agentic execution, traditional transformer mechanics force developers to build expensive algorithmic workarounds rather than addressing the core root cause.

Success in establishing viable post-transformer architectures could fundamentally shift the economics of artificial intelligence development. Lowering the quadratic compute burden directly reduces inference cost, lowers latency, and decreases energy demands across data centers worldwide. Furthermore, architectural innovations that handle ultra-long contexts efficiently will enable true continuous reasoning, seamless codebase integration, and persistent agent memory without requiring tens of billions of dollars in specialized hardware clusters.

What to watch

While alternative approaches like sparse attention and retention show technical promise in early benchmarks, mainstream adoption hinges on independent validation at enterprise scale. Industry observers should monitor whether these emerging architectures can maintain reasoning quality, structural accuracy, and broad versatility across diverse operational workloads without suffering benchmark drop-offs compared to full dense transformers.

Additionally, watch for established frontier developers and hyperscale cloud providers to potentially incorporate dynamic sparse attention or hybrid retention techniques into their next-generation base models. As compute costs escalate and context requirements expand, architectural efficiency will become a key competitive differentiator for enterprise deployment.