Are AI Models Working Harder Than They Need to?

What happened

Much of modern AI runs on multiplication. Neural networks behind everything from generated answers to photo organization and song recommendations perform millions or billions of operations that multiply inputs by learned weights. Lizy K. John thinks that’s more work than the job requires.

The industry took that and expanded it, millions and billions of neurons, to get something that works. But that doesn’t mean it’s needed. That’s my basic thinking. We may have a simpler way of coming up with the same answers. In the lookup case, we’re essentially looking up a zero or a one, saying go in this direction or don’t. There are no 30-bit or 16-bit weights involved.

So no multiplication, which is what takes all the energy, because multiplication is an expensive operation in hardware, and even for human minds. As children, we all struggled to learn the multiplication tables. It’s a hard operation! What have you been able to demonstrate so far?

John: In datasets like human-activity recognition or medical monitoring, ECG, EEG, blood pressure, weightless neural networks can do the job at 1,000 times less energy use. A lot of processing on smart sensors today just collects raw data and sends it to a server or phone, because the processing can’t happen on a battery-powered patch. Our network is small enough [that] it can sit right on the sensor.

The best other small AI model for one problem we looked at is 17 megabytes. Ours is 14 kilobytes, more than 1,000 times smaller. That means no transmitting raw data every millisecond, which is both an energy save and a privacy win, since your data never has to leave the device.

We’ve also shown gains on keyword spotting, the kind of listening a device does before it recognizes “Alexa” or a wake word. The current best industry model takes more than 5,000 nanojoules per inference. We do it in 42 to 79, depending on the variation. If weightless architecture took off tomorrow, what would change first? Chatbots, self-driving cars, phones?

Why it matters

John, a professor of electrical and computer engineering at the University of Texas at Austin, has spent the past five years working on a class of models called weightless neural networks. Instead of repeatedly multiplying inputs by weights, these networks pass binary inputs through interconnected lookup tables—closer to consulting a collection of stored answers than solving the same arithmetic problem repeatedly.

Depending on the task, she says, the networks can be 1,000 times as small or as fast as conventional alternatives while maintaining comparable accuracy. Her team’s work has so far focused on small, specific problems: medical sensors, activity tracking, keyword spotting. But she thinks the same approach could eventually reach much bigger targets, including the transformer models behind today’s chatbots.

What made you walk away from weights in the first place? Was there a specific moment that pushed you toward lookups instead? Lizy K. John: A friend casually invited me to a weekly meeting a few years ago to talk about using lookups instead of weights, a technique that wasn’t new. K. had built a commercial product around it in the ’80s for pattern recognition, and then it just disappeared.

A couple of professors at the Federal University of Rio [de Janeiro] kept working on it quietly for years, just one research group, so it wasn’t a lot of work. My friend knew I had hardware implementation experience, so he thought I could help make it real.

I had a student who I thought was perfectly positioned to take this on: He’d been working on spiking neural networks, another alternative to standard neural networks. The common thread for me was energy efficiency. I was upset by how much power was consumed by AI, and my ears were open for other ways of doing it with less technology, less power.

In less than six months, we had it running on an FPGA, basically a ready-made chip. We were able to create some very small neural networks that used the lookup methodology that could fit on tiny chips and didn’t need a GPU to run them. We could get them 1,000 times smaller than what everyone else was getting. That was encouraging.

Why is now the time to look into weightless neural networks? John: When you think about current AI models, it’s amazing what they do. I’m impressed by ChatGPT every time I use it, even though it gets things wrong sometimes. But when it’s coming up with the next word in a sentence, it’s doing millions or billions of multiplications to get there.

If you ask me a question, I’m not doing multiplications to answer you. I’m thinking, yes, no, I should say this, I shouldn’t say that. And the human brain only consumes about 20 watts of energy doing that. The model behind most popular networks today is based on a neuron model from a 1943 paper, the McCulloch-Pitts model.

What to watch

John: Our first target is medical monitoring, something as simple as a Band-Aid you put on someone for a week, so doctors can know what’s going on without an expensive connected device. Chemistry is another area we’re working in. Students run an experiment, then slowly send data out for offline processing, and there’s error introduced along the way.

We want to put the AI right at the point of the experiment, so the answer comes back immediately, like dipping litmus paper and seeing it turn blue. Can it help chatbots? We’re working on it. A transformer network alternates between an attention layer and a multilayer perceptron, over and over.