NVIDIA has announced a new text-to-audio model called Fugatto, which can create ‘any combination of music, videos and sounds’. Fugatto, which stands for ‘Foundational Generative Audio Transformer Opus,’ can generate audio that is a mixture of music, voices, and sounds, as described by the prompt.
“For example, it can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice — even let people produce sounds never heard before,” said NVIDIA, in the announcement.
NVIDIA also states that it is the first generative AI model to show ‘emergent properties,’ meaning its capabilities are derived from the interaction within the training properties.
“During inference, the model uses a technique called ComposableART to combine instructions that were only seen separately during training. For example, a combination of prompts could ask for text spoken with a sad feeling in a French accent,” said NVIDIA in the announcement.
They’re positioning this tool for audio producers, video game developers, language learning tools, and ad agencies.
Fugatto is trained on 2.5 billion parameters and built on top of NVIDIA DGX systems, with 32 NVIDIA H100 GPUs. While NVIDIA hasn’t released the model for use, it has released a demo on GitHub and a paper.
The GitHub demo dives deeper into the tool’s capabilities. It showcases examples of sound mixtures like ‘violin melody and baby laugh’, a synthesis of ‘a cello shouting with anger and a cello screaming’, and other amusing examples, like making a ‘trumpet bark’ or a ‘saxophone meow’.
The paper released by NVIDIA also lists all the open-source vocal and non-vocal datasets, including a few from BBC, and Epidemic Sounds.
“We work towards a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale. Our proposed framework ComposableART Fugatto establishes our first step towards this direction,” mentioned the authors in the paper.
When Fugatto is eventually released, it will compete with ElevenLabs and SunoAI. That said, none of the existing tools can generate sounds outside their training database.
The post NVIDIA Strikes a Chord with Fugatto appeared first on Analytics India Magazine.
