Data and AI giant Databricks announced a host of new generative AI capabilities and a major push to its open-source strategy at the annual Data + AI Summit. The new offerings, such as Mosaic AI Model Training, Mosaic AI for RAG, and Mosaic AI Gateway, in addition to open-sourcing their Unity Catalog, aim to help enterprises build high-quality, domain-specific AI applications.
“We want to help people get the best quality possible in their domain for their GenAI application,” said CTO and co-founder Matei Zaharia in an exclusive interview with AIM. “And to do that, we see a lot of companies are building what we call compound AI systems.”
These compound AI systems involve multiple components, such as calls to different models, retrieval of relevant data, use of external APIs and databases, and breaking problems into smaller steps. At the same time, Databricks is also focusing on open-source models.
Why is Databricks Betting Big on Open-Source?
While acknowledging the rapid advancements in closed-source models, Zaharia noted that Databricks is definitely betting big on an open-source strategy. He also believes that the performance gap between closed-source and open-source models is rapidly narrowing.
This is evidenced by recent open-source models like DBRX, Mistral 8×22 billion, and Llama 3 approaching the quality of the best closed-source models. “They’re all quite good, and they’re all in that space, getting really close to the best closed models. Meanwhile, the best closed models haven’t gotten that much better.”
Acknowledging the possibility that significantly higher investments could lead to superior closed models, Zaharia believes that open-source development will continue to thrive as companies seek to share development costs.
While consumer AI applications may stagnate, Zaharia predicts that the most exciting advances in generative AI will come from open, customisable models in the B2B world, applied to complex industry use cases.
“I actually think the most exciting sort of advances in GenAI will next be in the B2B world with custom AI for challenging mission-critical domains,” said Zaharia.
“That’s another reason that we’re betting on open models,” he added.
He drew parallels to how open-source big data technologies initially powered consumer applications but later had a transformative impact on the enterprise.
He elaborated, “Let’s say you build a model for chemistry. That’s really good. Even if it’s not as good at chatting about random topics as GPT-4, it’s still extremely valuable.”
Mosaic AI for Training Cost-Effective Models & Quality Monitoring
The new offerings in Mosaic AI are designed to address major hurdles like quality, cost, governance and security that organisations face in building and deploying generative AI applications.
“If you don’t get the right kind of quality for your application, then you’re stuck,” Zaharia emphasised.
One key offering is the RAG framework in Mosaic AI, which provides a quick way to deploy and manage an entire generative AI application, including the vector database, data pipeline, and serving layer.
Databricks is also introducing quality monitoring capabilities for compound AI applications. This includes the ability to see detailed traces, review results, and even use LLMs as automated judges to score outputs.
“So, of course, you can do prompt engineering, you can try to tell the model to do different things, but at some point, if you have examples of data that you can label and give it, you’ll do a lot better,” explained Zaharia.
“And we actually packaged up all the stuff that we used to train DBRX, which our research team had just developed. It’s now behind a very simple serverless API,” he added.
Additionally, Mosaic AI Training enables organisations to fine-tune models using their own labelled data, resulting in higher-quality outputs.
Fine-Tuning and Cost Reduction
Zaharia underscored the significance of fine-tuning foundation models on organisations’ own data with Mosaic AI Model Training.
He cited the example of FactSet, a financial data vendor that initially built an application using GPT-4, which was, at best, 55% accurate and took 10 seconds per user query. By switching to a multi-step AI system with Databricks, FactSet achieved 87% accuracy, reduced query time to three seconds, and lowered costs by approximately five times compared to GPT-4 calls.
Mosaic AI Training is an optimised software stack that makes training LLMs cost-effective. Through system-level optimisations, tuned parallelism strategies, and model training science, it can reduce training costs by up to 10x.
Zaharia emphasised that another factor to consider for cost-efficiency benefits is using custom and open-source models. “You might have something that works, but it’s very expensive, very slow.
“This is where custom models and open-source models provide a huge benefit because you can often take something that works well with a very expensive model, collect a bunch of examples of it and then fine-tune a small, low-cost model to do it well,” he explained.
Databricks’ DBRX model, for example, surpasses GPT-3.5 in quality while being faster and more cost-effective to serve, with costs similar to a 13 billion parameter model.
By incorporating Databricks Mosaic AI into their data strategy, organisations can experience reduced training time and costs, improved model performance, increased developer productivity, enhanced scalability, and democratised AI.