Extropic Looks To Revolutionize Modern Computing

For months, the AI community has waited with bated breath for the day they might see more of what Guillaume Verdon and his secretive start-up, Extropic, have been cooking up.

Finally, that day has arrived.

Guillaume Verdon, also known by his prolific pseudonymous Twitter persona @BasedBeffJezos, founded Extropic back in 2022. Formerly the quantum tech lead within the Physics & AI team at Alphabet’s X, he’s also a pioneer of the field of quantum deep learning after founding what would eventually become Google's TensorFlow Quantum project.

It was there, within Alphabet, that Verdon met Trevor McCourt, now CTO of Extropic. Together, along with a broad team of ‘cracked’ scientists and engineers, they embarked on a quest to do something unbelievable in the world of computing: they set out to transform it.

You see, there’s a crisis brewing. Amidst all the breakthrough innovations with large language models and generative AI, one of the most dominating trends in computing is finally slowing down; maybe, even, coming to a halt.

Moore’s Wall

What is Moore’s Law? Devised back in the 60s, it’s the observation that the number of transistors on an integrated circuit will double every two years with minimal rise in cost. This held true for decades. And it’s why, today, we have so much more computing power for a fraction of the cost. But now, it seems, Moore’s Law may be hitting “Moore’s Wall.”

Traditional computer chip design revolves around transistors getting smaller and smaller. But at the smallest scales, like atoms and electrons, all matter is moving about randomly. Transistors can only get so small before this “jiggling” starts to get in the way. Indeed, in the most advanced chips, the transistors are now infinitesimally-small—as small as just a few nanometers across.

At this scale, things get messy. The “noise” introduced into the system becomes a massive issue. And at the same time that traditional CMOS transistor technology is reaching the fundamental limits of physics, advances in AI are increasing the demand for compute at a breakneck pace.

Consider LLMs, which are just getting larger and larger. Meanwhile, in 2023, Nvidia sold over half a million H100 GPUs.

Humanity’s ravenous demand for compute will continue to grow astronomically. And with it, so will the need for energy to power it.

Nvidia’s next-gen GPUs will reportedly draw an enormous 1000W each, a 40% increase. Where does this lead? Nuclear reactors powering datacenters, human-level AIs that only nation-states can afford to host? Or is there another way?

This morning, Extropic released its very first “Litepaper,” detailing its novel attempt to solve the problem. From a “full-stack physics-based computing paradigm” to its cutting-edge new semiconductor design, the end result is a “complete AI hardware and software system from the ground up that thrives in an intrinsically noisy environment.”

If you’re like me, you were immediately overcome with excitement! –And then confused…very confused. But have no fear: this is the layman’s explanation that you’ve been looking for. I’ll walk you through:

• how Energy-Based Models (EBMs) can better unlock the full potential of generative AI,

• how Extropic is building transistor-less chips for a “noise-dominated yet well-behaved device” that embraces randomness instead of fighting it,

• and, finally, what all this could mean for the future of computing.

Let’s get started.

Energy-Based Models

If you read the Extropic release or listened to Yann LeCun on the Lex Fridman podcast a few days ago, you’re probably already wondering: what the hell are EBMs? Without getting too detailed, let’s explain the basics—and touch on why they’re important.

Here’s a simplified way of thinking about EBMs: you may imagine a model that, in effect, plays a very smart guessing game about how likely something is to happen, based on the "energy" or effort it would take. As opposed to LLMs (think: ChatGPT) which learn a probability distribution for the ‘next word’ and then pick the most likely option, an EBM would learn to predict how likely different situations are by associating them with an energy level. The lower the energy, the more probable the situation.

What about rare, tail events, ones that hardly show up in the data? Many other AI models struggle with such cases. But EBMs aren’t limited to simple patterns or relationships. They can capture a vast array of possibilities, making them extremely versatile. However, training such models is difficult in the current paradigm. As Extropic puts it so eloquently, “This process of hallucinating every possibility that is not included in a dataset and penalizing such occurrences energetically requires the usage of a lot of randomness, both at training and inference time.”

In effect, using such a model in digital hardware would be relying on a system that works effortlessly to eliminate all randomness, only then to pump the model full of randomness at massive cost. It’s a ridiculous idea. But what if your hardware isn’t fighting randomness, but embracing it? What then?

Extropic Accelerators

Therein lies the genius of Extropic’s hardware. In the era of AI, the deterministic nature of digital logic is outdated – machine learning is, instead, inherently probabilistic. And Extropics’ “thermodynamic computing” leverages this directly.

On an Extropic chip, what is happening can actually be thought of fairly simply. Imagine a large particle dropped into a fluid or a gas—it’s going to move around randomly based on the microscopic particles that continually run into it and exert force. But what if you anchored that particle to the wall of the space by some sort of spring or adjustable component? The particle would take on a much more limited range of motion, but one still impacted by the randomness of the system.

Now, if you adjusted the tension of that spring, you could change that distribution. Here’s where it gets interesting. The components on an Extropic chip are designed to be affected by inherent thermal fluctuations in the environment. This ingenious design means the system can produce new ‘random’ data simply by observing its current state.

With electrons serving as the ultimate random number generators, the randomness is truly random. And the shape of the distribution comes from the shape of a potential energy well in which electrons sit.

However, the behavior of these electrons, unpredictable by nature, can be shaped by carefully designed energy landscapes within the chips. These landscapes can be tweaked to accurately represent a wide range of complex patterns. All this is, you may have guessed, incredibly well-tailored towards the training and inference of EBMs.

So, briefly, how does Extropic engineer such a computing environment? The “Extropic accelerator” is a superconducting processor that operates at low temperatures and is meticulously nano-fabricated from aluminum. Its “neurons,” which form the most basic building block of the chip, can make use of the Josephson effect to access the complex probability distributions crucial for accurately modeling real-world data.

How does this work? The Josephson effect is when two superconducting materials are placed close to each other, with a thin layer of non-superconducting material (like an insulator or a normal metal) in between, forming a "Josephson junction." Even though the non-superconducting barrier would normally block the flow of electrons, the superconducting properties allow for the supercurrent to flow and, crucially, is the source of introducing nonlinearity to the system.

This nonlinearity is vital for the chips' ability to perform computations far beyond the scope of traditional digital processors, which rely on standard Gaussian sampling methods. As a whole, the chip comprises both linear and non-linear neurons, forming a system capable of handling diverse and high-dimensional data distributions. The neurons' properties can then be finely tuned, enabling a single chip to model a broad spectrum of probability distributions.

Designed to be entirely passive, these chips only consume energy during active measurement or manipulation, positioning them as possibly the most energy-efficient computational devices available. Additionally, to appeal to a wider market, Extropic is also developing room-temperature semiconductor devices, sacrificing a bit of the superconducting models' energy efficiency for the convenience of standard manufacturing and the ability to fit into consumer-friendly formats like GPU expansion cards.

Extropic’s new creation, the thermodynamic chip, would be an incredible leap forward in the era of computing for AI. So if they do pull it off, what does that mean for the future? What can we expect?

Looking Ahead to the Future

Energy efficiency, increasing scale, massive speed-ups in computation time: all of these benefits and more potentially arise downstream of Extropic’s new chip technology. Extropic's superconducting chips are designed to be the pinnacle of energy efficiency, operating passively except when actively being measured or manipulated. These systems will also be incredibly efficient at scale, specifically. Low-volume, high-value customers like governments, banks, and private clouds are exactly the targets Extropic has in mind with these systems.

Additionally, Extropic plans to build devices fit for an ever wider audience. These semiconductors would work at room temperature, utilizing transistors instead of Josephson junctions. While slightly less energy-efficient, these room-temperature devices can be built with standard manufacturing processes and supply chains, unlocking massive scale by integrating into consumer products like GPU cards.

The end goal? Making “thermodynamic AI acceleration” accessible to everyone.

To support the hardware, Extropic is also pioneering a new paradigm of software – all a part of their desire for a full stack, physics-based computing paradigm custom-built for artificial intelligence. This compilation layer will allow Extropic accelerators to “breakdown and run programs that are too big to fit on any given analog core.”

If successful, Extropic will revolutionize the world of AI, computing, and the future of these systems that are quickly coming to dominate every aspect of our world. It’s an ambitious goal. But today’s announcement demonstrates that they’re further along than maybe anyone would have guessed.

Ultimately, one thing is certain: they’re accelerating.