Creative tech conglomerate Adobe has recently migrated 2100 servers and two petabytes of data from its data lake to Microsoft Azure and AWS. This migration, one of the largest in the industry, positions Adobe well for bolstering its generative AI initiatives.
Bin Mu, the VP of enterprise data and analytics, Adobe, confirmed this to AIM in an exclusive interview at Databricks’ Data + AI Summit in San Francisco, earlier this month.
“Now, we are 100% in the cloud. It provides all the flexibility and capability that enable us to actually get to the next level of our generative AI for enterprise internal data analytics. We also want to focus on enterprise applications with GenAI, either as an engine or as part of the workflow,” he said.
In March this year, the company announced a strategic partnership with Databricks to improve the former’s consumer experiences by integrating their platforms. This collaboration allows data merging between the Databricks Data Intelligence Platform and the Adobe Experience Platform.
The transition to serverless computing aims to establish Databricks as Adobe’s unified data platform for all data sources, enabling analytics, decision-making, and data intelligence. All data sources are unified through Databricks, ensuring certification, security, governance, and curation.
However, migrating from legacy open-source systems like Apache Hadoop to Databricks presents unique challenges. The primary difficulties lay on both technological and human sides.
How did Adobe Make this Happen?
“The main challenge is the technology itself – that’s the most difficult part,” said Mu.
For users accustomed to the traditional on-premise system, transitioning to the cloud marks a huge change. To address these challenges, the company focused on three key areas. First, it established strong partnerships to ensure that the new technology was superior to the on-premise solutions. This involved building systems that not only matched but exceeded the functionalities of the previous method.
Second, it assembled a highly capable migration team skilled in executing the transition efficiently. The team, consisting of around 50 engineers, product managers, and change managers, focused on automation and removing unnecessary processes.
Third, they emphasised change management by collaborating closely with partners and showcasing a successful pilot project. This project served as a model for others, supported by a detailed playbook.
This transition also includes implementing various training and engagement initiatives. Tools like automatic converters ease the migration process by converting code from one language to another and managing metadata.
Mu, who loves to use an analogy to make things easy, explained that the shift is akin to moving into a new house – having the right tools, a clear plan, and organising belongings beforehand. And he is right!
Adobe’s Secret Sauce for GenAI
Adobe’s strategy for integrating generative AI is rooted in a philosophy of flexibility and responsible AI implementation. According to Mu, since the release of ChatGPT in September 2022, the company has been exploring various enterprise applications for generative AI.
However, privacy, legal compliance, and data security are critical considerations in this journey. Adobe’s approach avoids dependency on any single language model, ensuring that its generative AI platform is model-agnostic.
“Our approach includes making the generative AI architecture responsible, scalable, flexible, and deployable anywhere,” the spokesperson explained.
AIM spoke with Adobe India MD Prativa Mohapatra in May, and she echoed similar sentiments.
“We want to use AI responsibly and prioritise the protection of intellectual property. We employ measures like content credentials, which have become the industry standard for digital content provenance,” said Mohapatra.
What Next?
Looking ahead, Adobe sees generative AI as a catalyst for creating new opportunities and job roles. While some traditional engineering jobs may disappear, they will be replaced by roles that are more aligned with the capabilities of generative AI.
“Certain engineering jobs may disappear, replaced by AI capabilities, but this creates new opportunities. Just as switchboard operators became obsolete, new, more exciting roles will emerge,” Mu explained with another interesting analogy. Continuous learning and staying ahead of change is crucial for success in this evolving landscape.
Mu said that even internally, the company is using generative AI tools to revamp the engineering landscape at Adobe, leading to less coding and more configuration so that engineers can focus on understanding business needs and designing solutions. The modular approach simplifies the engineering process, with each function handled by specific modules.
“It’s much more like we’re experiencing what feels like an industrial revolution. We see less coding from an engineering perspective, and more configuration and a deeper understanding of business needs,” concluded Mu.
The post Adobe Rewrites History with Databricks in World’s Largest Data Migration appeared first on Analytics India Magazine.