Here’s a mind-bending truth: artificial intelligence is advancing at breakneck speed, yet even the brightest minds in the field can’t fully explain how it works. And this is the part most people miss—the more powerful AI becomes, the less we understand its inner workings.
Last week, San Diego became the epicenter of this paradox as 26,000 academics, startup founders, and researchers from global tech giants gathered for NeurIPS (Neural Information Processing Systems), the premier AI conference. This year’s attendance shattered records, doubling since 2017, a testament to AI’s explosive growth. But amidst the buzz of AI-generated music and cutting-edge applications, one question loomed large: How do these systems actually function?
NeurIPS, now in its 39th year, has evolved from a niche academic gathering in a Colorado hotel to a sprawling event filling the San Diego Convention Center. Founded in 1987, it’s dedicated to exploring neural networks—computational models inspired by human and animal brains. Yet, despite decades of research, the field of interpretability—the quest to understand how AI models make decisions—remains in its infancy.
But here’s where it gets controversial: leading AI researchers and CEOs openly admit they don’t fully grasp how today’s AI systems operate. Shriyash Upadhyay, co-founder of Martian, a company focused on interpretability, compares the current state to the early days of physics: “We’re still asking, ‘What does it even mean to have an interpretable AI system?’” To accelerate progress, Martian launched a $1 million prize at NeurIPS, underscoring the urgency of the challenge.
The debate intensified as major players like Google and OpenAI unveiled diverging strategies. Google’s interpretability team announced a pragmatic shift, prioritizing real-world impact over ambitious reverse-engineering goals. Neel Nanda, a Google leader, admitted, “Grand goals like near-complete reverse-engineering feel far out of reach.” In contrast, OpenAI’s Leo Gao doubled down on a deeper, more ambitious approach to understanding neural networks. Is one strategy more viable than the other? Or are both necessary?
Skeptics like Adam Gleave, co-founder of FAR.AI, argue that fully reverse-engineering large-scale neural networks may be impossible. “Deep-learning models don’t have simple explanations,” he said. Yet, he remains hopeful that researchers can make meaningful progress in understanding AI behavior, paving the way for more reliable systems.
Adding to the complexity, current methods for evaluating AI capabilities fall short. “We lack tools to measure concepts like intelligence and reasoning,” said Sanmi Koyejo, a Stanford professor. This gap extends to domain-specific AI, such as biology, where evaluation methods are still in their infancy, according to Carnegie Mellon’s Ziv Bar-Joseph.
Despite these challenges, AI’s potential to revolutionize scientific research is undeniable. Upadhyay aptly noted, “People built bridges before Isaac Newton figured out physics.” Similarly, AI is already driving breakthroughs in fields like chemistry and biology, even if we don’t fully understand how it works.
At a NeurIPS offshoot event focused on AI for science, organizer Ada Fang highlighted the shared challenges across disciplines. “Our goal was to create a space where researchers could discuss not just breakthroughs, but the limits of AI,” she said. Jeff Clune, a pioneer in AI for science, marveled at the field’s momentum: “The interest level is through the roof. It’s heartwarming to see the world rallying to tackle humanity’s most pressing problems with AI.”
So, here’s the question for you: Can we responsibly deploy AI systems we don’t fully understand? Or is interpretability a non-negotiable prerequisite for progress? Share your thoughts in the comments—let’s spark a conversation that could shape the future of AI.
