AI Bubble: ‘LLMs are fundamentally flawed’ | Gary Marcus

Gary Marcus argues that the current AI boom rests on flawed large language models that are not reliable or good enough to justify the massive speculative investment they attract, with many projects failing to deliver returns. He welcomes the industry petition to slow AI as symbolically important but ultimately vague and unlikely to overcome heavy lobbying or global competition. Marcus warns that safety risks—such as jailbroken AI creating pathogens or spreading disinformation—are real, and that a temporary, internationally coordinated pause could force a beneficial rethink of fundamentally broken technology. He debunks the winner-take-all narrative, predicting a fragmented market resembling low-margin airlines rather than trillion-dollar monopolies, and notes that governments will not cross-buy AI, further undermining unlimited capital expenditure. Finally, he suggests LLMs are not indispensable—society would be fine without them—and that much current investment is driven by FOMO and fantasy, making investors vulnerable to a slowdown that could ultimately accelerate progress.

• ▶ 0:00 Current AI is "just not good enough" and LLMs are fundamentally flawed, yet vast investment continues while public trust collapses over contradictory job-loss messaging.
• ▶ 1:40 Gary Marcus welcomes the industry petition to slow AI as symbolically important, but notes it offers no concrete government guidance and calls for global, not just national, cooperation.
• ▶ 3:13 Marcus sees mixed motives behind the slowdown calls: genuine concern after the Hugging Face incident, marketing, and a possible "distraction move" from OpenAI amid cost pressures and billion-dollar IPO valuations.
• ▶ 5:22 Marcus is skeptical the pacing letter will change anything, calling it “almost like a cry for help”—it’s vague, the causal path is “rocky,” and heavy lobbying would likely water down any resulting action.
• ▶ 6:19 He still argues for slowing down or pausing to focus on safety because current AI “cannot be aligned with human interest very well,” seen in failures like not repeating copyrighted material, not hallucinating, and not always refusing to create pathogens.
• ▶ 7:02 There is a realistic fear that someone could jailbreak AI defenses to create a deadly pathogen; guardrails are “permeable,” and there are no good ways to fight abuse, so the field should put far more energy into safety.
• [9:07-9:15] A slowdown would "cost a lot of people a lot of money," and affected investors would likely resist it because their entire thesis is a bet on rapid AI improvement.
• [9:17-9:50] The fundamental issue is that current AI isn't good enough, especially on reliability—many companies are disappointed with results, and studies show 70% to 90% of experimental AI projects don't return on investment.
• [9:51-10:20] The market relies on a "fantasy" that AI will soon solve its problems and generate massive profits, which has justified speculative trillion-dollar IPO talk; a slowdown would threaten this entire financial calculus.
• [10:22-10:31] Massive AI infrastructure spending, especially on data centers, is highly vulnerable to disruption if an AI slowdown occurs; investors should be mindful of this risk.
• [10:41-10:47] Marcus argues counterintuitively that a temporary slowdown focused on safety could actually accelerate long-term AI progress.
• [11:13-11:20] A temporary slowdown could lead to a better, faster path forward by forcing a rethink of fundamentally flawed LLM technology rather than continuing current investment patterns.
• ▶ 11:23 The interviewer suggests the "China threat" may be a post-hoc justification for unrealistic AI capital expenditure rather than a real reason to keep investing at full speed.
• ▶ 11:53 Marcus says he doesn't think U.S. companies have lost the race, but rejects the winner-take-all narrative — the likely outcome is "more like a tie, like Coke and Pepsi."
• ▶ 12:11 Governments will not cross-buy AI or robotics: the U.S. won't buy China's superior humanoid robots, and China won't buy U.S. AI services, creating a bifurcated market that weakens the case for unlimited capex.
• ▶ 12:46 Gary Marcus argues the LLM market is not winner-take-all, unlike Facebook or Google Search.
• ▶ 12:59 He expects the AI landscape to be more fragmented and competitive rather than dominated by one model.
• ▶ 13:02 He begins addressing the common “China competition” rebuttal, pushing back on that framing of AI investment debates.
• ▶ 13:06 Gary Marcus rejects the idea that LLMs will create trillion-dollar companies or generate trillions in profits.
• ▶ 13:23 LLMs are a commodity with very low or even negative margins—a point he has made since August 2023.
• ▶ 13:36 The LLM business model resembles the airline industry: a rough, low-margin business with little chance of huge profits.
• ▶ 13:56 The interviewer frames a hypothetical: assume LLMs, at least in their current form, become as fundamental as airline travel, separating the question of necessity from the soundness of the current race.
• ▶ 14:05 Airline travel is characterized as a “necessity,” implying that even indispensable infrastructure can have complicated or problematic economic dynamics.
• ▶ 14:07 The interviewer begins to note that this necessity is “one that everybody’s having a race to,” hinting that a fundamental good can still trigger overinvestment, bubbles, or instability.
• ▶ 14:15 LLMs will certainly be replaced by better technology eventually; for now they are the best we have for certain tasks.
• ▶ 14:20 LLMs are not actually indispensable — they matter for coders and some ad copy, but many imagined business transformations are not materializing.
• ▶ 14:51 Many businesses are not really getting a return on their LLM/AI investments.
• ▶ 14:57 Marcus argues current AI investment is driven by FOMO and competitive anxiety—fear that rivals will figure out AI first—rather than proven business results.
• ▶ 15:14 He says he struggles to name any companies that have genuinely leaped ahead because of AI, and many supposed success stories fall apart under scrutiny.
• ▶ 15:16 A NYT story about a billion-dollar, one-person company looked like an AI win, but turned out to involve drugs and fake accounts—highlighting that AI hype often masks dubious or unrelated behavior; real uses, like in mathematics, remain more modest ▶ 15:35.
• ▶ 15:42 Marcus questions whether LLMs are truly "necessities," arguing the framing is overstated.
• ▶ 15:47 He proposes a thought experiment: if LLMs disappeared and society returned to 2020, "that would be fine" — the pre-LLM world lacked nothing essential.
• ▶ 15:56 He emphasizes LLMs have "so many downsides," harming education and adding to disinformation, and concludes society wouldn't be "significantly worse off" without them.

• ▶ 16:18 International agreement on AI pacing is not impossible but far from guaranteed; the key possibility rests on the fact that everyone has legitimate worries about AI.

• ▶ 16:35 Even powerful nations have reason to cooperate — e.g., China cannot afford to ignore AI risks, while "reckless" or desperate countries are far less likely to accept limits.

• ▶ 17:55 There is precedent for cooperation in cybersecurity, but verification and economic pressure remain major obstacles to any enforceable AI pacing deal.

• ▶ 18:44 The interviewer questions whether industry-led pacing would just mean AI companies marking their own homework, and asks what would make it genuinely effective.
• ▶ 19:16 Marcus insists that in a rational world companies should not oversee themselves; independent scientists must be at the table, or companies will create oversight that exists "in name" and simply move the bar to what they can get away with.
• ▶ 19:48 Marcus argues that AI companies face a huge public trust crisis, so they will prioritize looking like the "good guys" over being genuinely independent or effective.
• ▶ 20:22 Companies will likely offer "window dressing" rather than substantive safety measures because corporate incentives favor looking good without high costs.
• ▶ 20:40 The only way to get meaningful oversight is to include independent scientists with no vested interest, like Marcus and Yoshua Bengio; otherwise oversight is "meaningless."
• ▶ 20:55 Governments are not a clean solution either, since companies heavily influence them through campaign financing and similar channels.
• ▶ 21:08 Marcus admits he doesn't have a full answer, but knows where to start methodologically.
• ▶ 21:11 The essential first step is a genuine US–China agreement: "Hey, we actually want to do something here."
• ▶ 21:21 Until that real bilateral agreement is reached, "there's no point in talking to the rest" of the world.
• ▶ 21:28 Pacing agreement must start with a core deal between the US and China before involving the rest of the world.
• ▶ 21:35 A first feasible mechanism is mandating an off switch for AI software; the main question then becomes when it should be invoked.
• ▶ 21:47 Additional mechanisms include information sharing and incident reporting between countries; later, governments could require much more transparency through blue-ribbon panels.
• ▶ 22:42 Marcus proposes a "dimmer switch" model for AI control—gradually scaling back development (e.g., limiting data center investment) rather than relying on a simple binary off switch.
• ▶ 23:06 A key obstacle is verification: without knowing how many data centers others are building, no one should take assurances on faith, and historical disarmament agreements show such approaches are problematic.
• ▶ 23:21 Verification will get harder over time because AI systems may become far more efficient (humans run on 20 watts, not 20 megawatts), making development less visible and agreements "not trivial at all."

• ▶ 24:19 Marcus agrees that big tech firms constantly seek to keep competitors out and are not above using regulatory capture, so an outside panel of independent experts is needed to flag when rules hurt startups.

• ▶ 24:49 He stresses that AI regulation involves trade-offs, such as balancing the benefit of preventing pathogen spread against the value of allowing smaller companies to compete.

• ▶ 25:00 Marcus worries that very few government officials discuss AI with technical nuance—they rarely distinguish LLMs from AI in general or address distribution shift—which is why independent scientists must be involved.

• ▶ 0:00 AI's Problems and the Safety Petition (0:00 - 5:05) - Opens with AI’s shortcomings, introduces Gary Marcus, and examines the Anthropic/OpenAI-backed petition to slow AI development.
• ▶ 5:05 The Case for Slowing AI Down (5:05 - 8:43) - Covers skepticism about the pacing letter, alignment risks, current AI abuse, and the need to pause until safety is solved.
• ▶ 8:43 Slowdown Fallout, China, and LLM Economics (8:43 - 16:18) - Discusses investor and industry fallout from a slowdown, the China competition argument, and LLMs as commodities with weak ROI.
• ▶ 16:18 What Pacing Could Actually Look Like (16:18 - 23:49) - Addresses international pacing agreements, independent oversight, company trust, and the need for transparency and verification.
• ▶ 23:49 Regulation, Trade-offs, and Outro (23:49 - 25:56) - Covers regulatory capture, outside panels, government nuance, and closes with thanks and a subscription reminder.