Thomas Graham, right, co-founder of chip startup Lightmatter, told Mandeep Singh of Bloomberg Intelligence that data centers equivalent to eight times the power draw of New York City will be under construction come 2026 to serve deployment of AI.
Nvidia and its partners and customers have steadily built larger and larger computer facilities around the world to handle the compute-intensive needs of training giant artificial intelligence (AI) programs such as GPT-4. That effort will gain continued importance as more AI models are put into production, says one startup serving the tech giants.
"People will want more compute, not necessarily because of scaling laws, but because you're deploying these things now," said Thomas Graham, co-founder of optical computing startup Lightmatter, during an interview last week in New York with Mandeep Singh, a senior technology analyst with Bloomberg Intelligence.
Also: I test AI tools for a living, and these are the only two worth paying for
Singh asked Graham if large language models (LLMs) such as GPT-4 will continue to "scale," meaning grow in size as OpenAI and others try to achieve more ambitious models.
Graham turned the question around, suggesting that the next stage of AI's compute appetite is putting trained neural nets into production.
"If you view training as R&D, inferencing is really deployment, and as you're deploying that, you're going to need large computers to run your models," said Graham. The discussion was part of a daylong conference hosted by Bloomberg Intelligence called "Gen AI: Can it deliver on the productivity promise?"
Also: AI isn't hitting a wall, it's just getting too smart for benchmarks, says Anthropic
Graham's view echoes that of Nvidia CEO Jensen Huang, who has told Wall Street in recent months that "scaling up" the "agentic" forms of AI will require "both more sophisticated training [of AI models], but also increasingly more sophisticated inference," and that, as a result, "inference compute scales exponentially."
"If you view training as R&D, inferencing is really deployment, and as you're deploying that, you're going to need large computers to run your models," said Graham.
Lightmatter, founded in 2018, is developing a chip technology that can join multiple processors together on a single semiconductor die using optical connections — which can replace conventional network links between dozens, hundreds, or even thousands of chips needed to build AI data centers. Optical interconnects, as they're called, can move data faster than copper wires at a fraction of the energy draw.
The technology can be used between computers in a data center rack and between racks to simplify the computer network, making the entire data center more economical, Graham told Singh.
Also: AWS says its AI data centers just got even more efficient – here's how
"So, really taking away the copper traces that you have in data centers, both in the server on the printer circuit board and in the cabling between racks, or replace that all with fiber, all with optics, that really dramatically increases the bandwidth you get," said Graham.
Lightmatter is working with numerous tech companies on plans for new data centers, Graham said. "Data centers are being built from scratch," he said. Lightmatter has already announced a partnership with contract semiconductor manufacturer Global Foundries, which has facilities in upstate New York and serves numerous chip makers, including Advanced Micro Devices.
Outside of that collaboration, Graham declined to name partners and customers. The implication of his talk was that his company partners with silicon providers such as Broadcom or Marvell to fashion custom integrated parts for tech giants that design their own processors for their data centers, such as Google, Amazon, and Microsoft.
Also: Today's AI ecosystem is unsustainable for most everyone but Nvidia, warns top scholar
For a sense of the scale of the deployment, Graham pointed out that there are at least a dozen new AI data centers planned or in construction now that require a gigawatt of power to run.
"Just for context, New York City pulls five gigawatts of power on an average day. So, multiple NYCs." By 2026, he said, it's expected the world's AI processing will require 40 gigawatts of power "specifically for AI data centers, so eight NYCs."
Lightmatter recently received a venture capital infusion of $400 million, and the company is valued at $4.4 billion. Lightmatter intends to enter production "over the next few years," said Graham.
Also: 8 ways Intel can still pull itself out of free fall after its CEO's forced exit
When Singh asked him what could up-end the company's plans, Graham expressed confidence in the continued need to expand AI computing infrastructure.
"If in the next few years researchers come up with a new algorithm to do AI that requires way less compute, that is way more performant than what we have today, that achieves AGI [artificial general intelligence] way quicker, that would throw a monkey wrench into everybody's assumptions on wanting to keep investing in exponential compute," he said.