‘Real Transformation of AI Has Not Been on Engineering, But Product & Design Teams’

AI coding may be saving time in generating the code but its gains are lost in debugging and security audits.

Amjad Masad, CEO of AI coding platform Replit, recently shared on X his conversation with the CEO of a publicly traded company, who mentioned that while AI coding has had little effect on his engineering teams, the product and design teams using Replit observed “real change”.

“Whatever time is saved in generating the code is lost in debugging, reverting bugs, and security audits. So if you measure time to ship, PRs merged, or whatever high-level metric, you don’t see any impact,” the CEO told Masad.

These claims align with a recent study that states engineering teams and developers are indeed spending more time debugging and clearing code written with AI, which rather has a counterproductive effect.

A METR study found that developers who use AI tools believe they are reducing their time by 20%, but actually take 19% longer. Several developers have also observed that they frequently spend time fixing AI-written code, effectively spending the time saved by AI on performing corrections.

However, the executive mentioned that non-technical teams, such as product and design, now possess a ‘superpower’, thanks to AI and rapid prototyping tools like Replit. These tools have significantly shortened the time required to develop prototypes before passing them on to engineering teams.

“I was surprised to hear the part about engineering teams, and I’m sure every company will be different, but it made sense, the profound impact coding agents are having on non-technical folks,” said Masad.

Product Managers’ Critical Role

Industry experts aligned with Masad’s anecdote. Jon Sakoda, a venture capitalist and founding member of Decibel, replied to Masad, saying, “Many people have product intuition–they just have struggled to share it with others. Your platform [Replit] now makes it possible for all.”

Some also believe that product teams using AI will free up additional time for the engineering team to focus on more valuable work.

“Giving PMs [product managers] the ability to prototype has second-order benefits to engineers who no longer have to waste time building prototypes,” said Anish Acharya on X, an investor at Andreessen-Horowitz (a16z).

For context, a product manager, or a PM, is one who defines a product’s vision, strategy, and roadmap, aligning business goals, customer needs, and technical constraints to guide cross-functional teams from concept to delivery and ensure market success.

In several previous interactions with AIM, product designers and managers resonated a similar sentiment. Suhas Motwani, founder of The Product Folks, stated that the role of product managers will only become more critical in the future, as it is more important than ever to understand what to build with AI.

“Tools like Cursor, v0, are helping you take a wireframe and build a prototype. Imagine I can get to that stage in 1/10th of the time, and take that to my development,” he said.

Akshay Verma, a product innovation specialist and founding partner at Prophecy, said that he views the rise of AI in product design and development as a net positive.
Instead of worrying about displacement, he highlighted how AI lowers long-standing barriers in the prototyping process.
“We’ve long had two choices, mostly: ‘a’–create quick, but non-functional experience prototypes, or ‘b’–spend time and resources waiting on our developer counterparts to create functional prototypes. AI makes it much easier for anyone with a little coding experience to bring their ideas to life,” he said, noting that this shift allows product teams to validate ideas faster and involve non-technical stakeholders earlier in the process.

Need More PMs Than Ever

Furthermore, Motwani also added that we will see a stronger relationship between PMs and LLMs, and these product roles will only become more substantial, earning more demand in various specialised forms.

True to his statement, the industry has seen an increasing number of product manager roles in the AI space. Several firms — from big tech companies and startups alike are hiring for AI product managers.

Recently, a hiring post from Apple seeking an AI product manager was doing the rounds on social media. Similarly, Moody’s Corporation, an American business and financial services company, is hiring a product manager to propose AI solutions that enhance efficiency and productivity within tasks.

Similarly, ServiceNow, a US-based service and consulting firm, is hiring a product manager to guide the roadmap and development of the company’s AI agent platform, aligning it with both user and business needs. The role also involves collaborating with cross-functional teams to ‘develop and launch innovative PoCs (proof of concepts).’

Aaron Levie, CEO of the cloud storage platform Box, took to X to speak about such PM roles for AI agents, and called it the “wildest form of product management in history.”

“As a PM (or engineer), you basically spend your time trying to reverse engineer ‘what would a human need as context to perform this task,’ and then figure out how to design systems to get the agent that data in the right sequence, with the right tools, and instructions,” he said.

He added that people with deep domain expertise — or those who can acquire it quickly are well-equipped to build such agents. “The ability to anticipate the context that the agent would need to be successful is a huge determining factor in how effective the agent will be.”

“This partly explains why coding agents have worked so well out of the gate; because its builders deeply understand the domain that they’re working to automate,” said Levie, and suggested that similar results would be observed across sectors such as legal, healthcare, and finance, as a new generation of product managers takes over.

Aman Khan, director of product at Arize AI, talked about the ongoing evolution in the roles of product managers during a recent podcast interview.

“At the end of the day, you really want to be in love with the problem,” Khan said, adding that AI PMs don’t add value only while building AI-centric products, but by understanding how to solve the customer’s needs.

The post ‘Real Transformation of AI Has Not Been on Engineering, But Product & Design Teams’ appeared first on Analytics India Magazine.