Рубрика: AI

One fun thing about CES (aka, the Consumer Electronics Show) is that it's not all consumer or electronics.

In the case of industrial enterprise vendor Siemens, CES 2024 is an opportunity to showcase a wide range of technology innovations.

Also: CES 2024: What's next in tech

The company was kind enough to share its announcement plans ahead of time with ZDNET, and we chose the following items to spotlight. This will give you a good overview of the scope of what Siemens is doing, particularly in the area of industrial AI and mixed reality.

Mixed reality headset partnership with Sony

Siemens and Sony today announced that they're working together to produce a next-generation mixed-reality headset designed to utilize Siemens design tools like NX Immersive Designer. What caught my attention was that Sony used Siemens tools to design the headset, and liked them so much, they reached out to form this exclusive partnership.

Also: Sony and Siemens unleash the power of immersive engineering

Red Bull Racing partnership

Another company making active use of the Siemens Xcelerator portfolio is Red Bull Racing. While the details are minimal, the companies report that they are showing "how this new solution empowers engineers to free them from traditional constraints to bring together the virtual and physical worlds by immersing them in the industrial metaverse."

Siemens Industrial Copilot

In another partnership, Siemens is working with Microsoft to bring Microsoft's generative AI assistant Copilot to industrial projects and processes. It's expected to help improve human-machine collaboration and boost manufacturing productivity.

Also: What are Microsoft's different Copilots?

The companies plan to add additional copilots for sectors including infrastructure, transportation, and health. Already the technology is in early adoption use by Schaeffler AG, a maker of rolling element bearings for automotive, aerospace, and industrial mechanisms. These are used in a wide range of applications to reduce radial friction and support radial and axial loads.

Increasing Amazon partnership

Mendix, an enterprise low-code vendor, was founded in 2005 in Rotterdam. In 2018, Siemens scooped up the company after using Mendix to build more than 450 specialized applications. Mendix is now part of the Xcelerator portfolio offered by Siemens.

At CES, Siemens and Amazon's AWS are announcing that they are "strengthening their partnership" to make it easier for any sized business to build and scale generative AI applications. The partnership involves the integration of Amazon Bedrock and Mendix. Think of Bedrock as generative AI as-a-service, a tool that lets developers use a variety of foundational models to build and customize specialty applications.

Also: Generative AI is a developer's delight. Now, let's find some other use cases

By combining the AI toolkit resources of Bedrock with the low-code approach of Mendix, it will be possible for developers to rapidly prototype low-code AI solutions, possibly cutting months or years off of the development cycle.

Affordable prosthetic arm

I describe the digital twin concept in some detail in my coverage of Sony's new mixed-reality headset. The idea is that it's a virtual replica of something that exists in the physical world. By using digital twins, engineers and scientists can simulate real-world conditions and then analyze the results, enabling them to optimize the design without the overall cost of producing numerous prototypes.

Siemens is showcasing how their Siemens Xcelerator portfolio and tools like the NX designer have helped scientists like Easton LaChappelle build products. In LaChapelle's case, he's the founder of a company called Unlimited Tomorrow, which makes affordable, lightweight, high-quality prostheses for children and adults.

The process involves a remote scan, 3D printing, custom engineering, and one-off design production, all of which result in a more comfortable limb while also reducing costs.

Intelligent habitat solutions

Siemens is presenting its smart home energy management solutions under the umbrella name Inhab. Three key components include a load manager that installs into your current electrical panel, smart circuit breakers, and an energy monitor app for managing and controlling it all.

Also: The best home EV chargers you can buy, according to experts

The company's focus is not just on controlling energy, but also on allowing homeowners greater visibility into patterns of energy consumption, which can lead to better decision-making over time. Siemens is laying the groundwork for homeowners who purchase electric vehicles and want to charge them in their garages or carports.

In addition to usage patterns, the system can also be programmed to provide a series of alerts based on a variety of designated conditions. This can keep the home safe, and also help reduce overall energy costs.

Tackling food insecurity

Blendhub is a fascinating story all on its own. The company uses shipping containers as food factories, pioneering the idea of food-as-a-service. The containers can be deployed worldwide to provide manufacturing services, creating customized ingredients for local food industries, customized food products for local needs, and high-nutrient value food products for populations in need of better nutrition.

At CES, Siemens is showcasing how it's equipping Blendhub with technology solutions that are key for automating and managing its portable food factories efficiently. Siemens further contributes to Blendhub's food production model by providing automation components like its TIA Process Control System, controllers, and motors; implementing Opcenter RD&L for lab management; and helping to implement a transition to the company's Teamcenter X product lifecycle management solution for global recipe management.

Intelligent predictive maintenance

Modern manufacturing is characterized by lots of machines and relatively few humans. The machines need to stay functional, be efficient, and produce quality work. The problem is that machines have moving parts, and over time those machines begin to break down or wear out.

Also: This company says AI can help design sustainable smart home appliances

Siemens is showcasing its Senseye technology, which uses AI to predict maintenance and upkeep requirements across factories. It combines the use of existing and new-generation sensors with AI insights, human insights, behavior models, and asset management to give managers a dynamic and even "see the future" view of factory performance.

More here on ZDNET

Be sure to poke around ZDNET for more coverage on CES 2024. My colleagues and I have spent quite a bit of time putting together a comprehensive series of articles to help you understand the product introductions and innovative tech coming out of Las Vegas, especially in light of the very hot topics of AI and mixed reality.

You can follow my day-to-day project updates on social media. Be sure to subscribe to my weekly update newsletter on Substack, and follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.

CES 2024

Samsung brings back Ballie, its home robot, at CES 2024 — with a few upgrades Kyle Wiggers 13 hours

Remember Ballie, Samsung’s spherical home robot from CES 2020? I sure didn’t — until Samsung brought it back at this year’s keynote with a few on-trend AI upgrades.

The new and improved Ballie, which Samsung previewed during its press conference at CES 2024 in Las Vegas today, is around the size of a bowling ball, packing a battery that’s designed to last two to three hours. Ballie sports a spatial lidar sensor to help it navigate rooms and obstacles, as well as a 1080p projector with two lenses that allows the robot to project movies and video calls and even act as a second PC monitor.

“Use [Ballie] to project images and stream content on walls, and it can automatically adjust the picture based on the wall distance and lighting conditions,” Samsung writes in press release. “It [can] automatically detect people’s posture and facial angle and adjust the optimal projection angle for you.”

Ballie can project content onto a wall or other surface, adjusting the angle of the picture as needed. Image Credits: Samsung

Ballie can be controlled with voice commands or, intriguingly, requests sent via text message (e.g. “play a movie on the nearest wall”). In the latter case, Ballie will respond with the aid of a chatbot to confirm requests before taking action.

Like other home robots in its class, Ballie can automatically turn on smart lights and, thanks to a built-in infrared transmitter, “non-smart” devices like air conditioners and older TVs. And the robot can map a floor plan, identifying where smart devices might be located inside a home.

Samsung’s promising a lot beyond these basics, like automatic reminders to water plants around the house, access to remote medical services (for older household members) and personalization depending on who the robot senses nearby. “With its built-in front [and] rear camera, [Ballie] can detect and analyze its surroundings and learn recurring user patterns,” Samsung continues in the press release.

But the details of these — as with Ballie’s availability and pricing — have yet to be firmed up.

The question is, will any of these features compel homeowners to buy Ballie when — or if, rather — it reaches market? Home robots have never been a slam-dunk, as recently demonstrated by Amazon’s attempt. Another promising attempt within the last few years, Mayfield Robotics, which hoped to sell a home robot in partnership with Bosch, ceased operations before shipping a single unit to early customers.

Perhaps Samsung will fare better. We’ll have to wait and see.

At CES you encounter so many new products and innovative technologies that it's not often that something stops you dead in your tracks, but this technology merited that reaction from me. Holograms are no longer limited to sci-fi movies and may be in your future sooner than you think with the Holobox.

Also: CES 2024: What's Next in Tech

At CES Unveiled, a press-only pre-show event, Holoconnects showcased its AI-powered holographic solution, Holobox. The box, similar to a phone booth, can display a full-sized holographic rendering of the person you are talking to with nearly no latency, making it feel like a normal conversation.

I had the chance to talk to Andre Smith, CEO of Holoconnects, via the Holobox at the event. Even though he was in Amsterdam, and I was 5,000 plus miles away in Las Vegas, it felt like I was talking to him in person, being able to carry out a regular conversation with the typical pauses. You can view a snippet of my experience below.

For me to see a hologram of him, he had to be well-lit with a white background. Since I was standing in the showroom without the proper background and lighting, he couldn't see a hologram of me but rather a camera shot, similar to a Zoom call.

Although it seems like super advanced technology, the only things the Holobox needs to function are electricity and the internet, making it a "plug and play" system.

Also: The best robots and AI innovations we've seen at CES 2024 so far

Some other features of the box are anti-glare glass, two HiFi built-in speakers, an 86" transparent LCD screen, and an advanced touch system.

According to Holoconnects, many companies are already harnessing the technology, including UNICEF, the United Nations, Nike, Vodafone, BMW, Deloitte, and T-Mobile.

The company also offers a desktop version of Holobox Mini technology. The website does not reveal the pricing for its solutions, but interested customers can request a pricing list.

Artificial Intelligence

CES 2024: Everything revealed so far, from Nvidia and AI to Samsung’s Ballie robot Christine Hall 11 hours

CES 2024 is here! The TechCrunch team is in Las Vegas this week to take in all of the action and decipher what it means to you. You already know what we’re expecting, so sit back, relax and stay tuned throughout the week as we bring you the products, announcements and startup news that you need to know.

Kicking off the first day are some bigger announcements from companies, including Nvidia, LG and Samsung. Here are all the ways you can watch live.

And here’s how you can follow along with our team’s coverage.

Kia’s new modular EV van lineup

Kia PBV Concept Lineup

Kia’s new EV vans come with a modular twist. In addition to using a modular powertrain, the vehicles will also have modular tops that allow for many different cabin options. But they remained vague on pricing, specs and expected launch dates for this new fleet of commercial EVs. Read more.

Samsung brings back Ballie; renews green initiative

Image Credits: Samsung

Meet the new and improved Ballie, Samsung Electronic’s home robot, which it previewed today. It’s around the size of a bowling ball with a battery designed to last two to three hours. Ballie sports a spatial lidar sensor to help it navigate rooms and obstacles, as well as 1080p projector with two lenses that allows the robot to project movies and video calls even act as a second PC monitor. Learn more.

Expanding beyond cute, rolling robots, Samsung showcased its wider initiatives for connected homes. Aside from expected UI and feature updates for its existing SmartThings home automation platform, Samsung showed off a “map view” for users, that creates an interactive home map that even includes animated avatars of residents and pets. Learn more.

Also, Samsung devoted some of its keynote speech to its commitment of sustainability. “We start by incorporating recycled materials into some of our most loved products, such as recycled fishing nets in our Galaxy,” said Inhee Chung, VP of corporate sustainability at Samsung. “Smartphones, recycled plastic in our TVs, and recycled aluminum in our bespoke refrigerators. Recycled plastic accounted for 14% of the total plastic used in our products in 2022. And we’re working towards increasing this amount.” Read more.

X1 Interpreter Hub: A new real-time translator

Image Credits: Timekettle

Timekettle announced the X1 Interpreter Hub, a more robust solution, designed for meetings. Timekettle calls it “the world’s first multi-language simultaneous interpretation system.” The system works out of the box, without having to download a separate app. For in-person meetings, two devices are touched together to initiate conversation translation. The handheld devices house earbuds, similar to past Timekettle products. All told, the X1 is capable of supporting up to 20 people at once in five different languages. Read more.

LG’s transparent television

LG Signature OLED T. Image Credits: LG

Televisions aren’t naturally pretty or used as a design feature, but LG Electronics is out to change that perception. Today, consumer technology giant unveiled what it touts is “the world’s first” wireless transparent OLED TV. The LG Signature OLED T combines a transparent 4K OLED screen with LG’s wireless video and audio transmission technology.

Breathe easy with bio-engineered houseplants

Neoplants at CES 2024. Image Credits: Haje Kamps / TechCrunch

French startup Neoplants is showing off its progress with its houseplants that work as air purifiers designed for the home. The bio-engineered plants can, according to the company, replace 20 “regular” houseplants, as measured by how many pollutants the plants can remove from the air.

More from Samsung: bigger, foldier, more rollable displays

Image Credits: Samsung

Ahead of Samsung Electronics’ press conference later today, we look at some of its product plans that include a “new generation of products that can be folded inward and outward,” along with “monitor-sized” folding and sliding OLEDs. Samsung also unveiled a “Transparent MICRO LED” display for the first time.

Nvidia gets its game on

GeForce RTX 4080 SUPER. Image Credits: Nvidia

Today, Nvidia gets into artificial intelligence in a big way with the unveiling of its GeForce RTX, including the GeForce RTX 40 Super series of desktop graphics cards. Much of these are meant for gaming, and Nvidia said 14 titles will get the RTX upgrade treatment, including Horizon Forbidden West, Pax Dei, and Diablo IV. The RTX 4080 Super starts at $999.

More chip updates from AMD

Speaking of chips, AMD debuted its new Ryzen 8000G processors for the desktop, with a big focus on their AI capabilities.

Bosch’s in-car eye-tracking

Image Credits: Bosch

Bosch is showing off two technologies this week in eye-tracking while driving: One will see that you have tired eyes and ask if you need an espresso when you arrive home. If yes, its connect technology will tell your fancy machine to have one ready. The other is a bit more complicated in that it’s developed to track what you’re looking at as you drive.

Smart cooking

Image Credits: Haje Kamps (opens in a new window) / TechCrunch (opens in a new window)

We have a collection of small home appliances for the kitchen, from grills to smart microwaves and everything in between, that is sure to get you cooking this year, if you aren’t already.

ChatGPT in Volkswagen

Image Credits: Volkswagen

The German automaker plans to add an AI-powered chatbot into all Volkswagen models equipped with its IDA voice assistant. For now, it’s not available in the U.S.

Apple Vision Pro to go on sale February 2

Image Credits: Apple

And Apple, in a surprise announcement preempting CES, stole some of the show’s thunder by announcing the Vision Pro will be available in the U.S. The consumer electronics giant confirmed that the Vision Pro will be available in the U.S. starting February 2. Pre-orders for the $3,500 spatial computing device open Friday, January 19.

Some companies made announcements ahead of the big event. Check out what’s already made headlines:

Withings’ new multiscope device checks vitals for telehealth visits

Invoxia has a new smart collar suitable for both cats and dogs

This app lets restaurants and coffee shops charge to use the bathroom

Aurora and Continental pass first major hurdle in commercial self-driving trucks deal

This startup is bringing a ‘voice frequency absorber’ to CES 2024

For just $139, this startup turns your iPhone into a BlackBerry-era relic

Qualcomm next-gen XR chip promises up to 4.3K resolution per eye

Urbanista integrates Powerfoyle tech with solar-powered headphones

Moonwalker robotic shoes get lighter and smarter

Today at CES 2024, HP announced new laptops for professional, consumer and gaming use. The HP Spectre x360 line of 2-in-1 laptops for professional or consumer use now come with AI enhancements, and HP has announced new peripherals. The AI features in the Spectre x360 line are mostly cosmetic, such as transcription and video polishing for meetings. HP also released its Series 5 monitors in 24-, 27- and 32- inch sizes.

In addition, HP announced the OMEN Transcend 14 Gaming Laptop PC, with the assertion that it is the “coolest and lightest” gaming laptop of its size available in the world.

HP introduces Spectre x360 laptops

HP’s newest offerings applicable to business use are the HP Spectre x360 2-in-1 Laptop PC in 14-inch (Figure A) and 16-inch models. Both laptops can use up to a 2.8K OLED screen, include 9 MP cameras for calls and have an NPU for processing AI workloads.

Figure A

The HP Spectre x360 2-in-1 Laptop PC line has Intel Core Ultra5 processors and can include an optional NVIDIA GeForce RTX 4050 Laptop GPU. Calls and video are adjusted automatically by Windows Studio for enhancements, such as automatic framing and background blur, which are handled by the NPU.

SEE: Dell’s new XPS laptops show off one of the trends at CES 2024: Dedicated buttons for Microsoft’s generative AI assistant, Copilot for Windows. (TechRepublic)

“We believe that the best innovations are also the most personal ones,” wrote Samuel Chang, senior vice president and division president of personal systems consumer solutions at HP Inc., in a press release. “New technologies from HP deliver solutions that allow us to be more personalized than ever, taking advantage of game-changing innovations like AI that will alter the way that technology moves us forward.”

The Spectre x360 line is available at HP.com or BestBuy.com now. The 14-inch model starts at $1,499.99, while the 16-inch model starts at $1,599.99.

New peripherals work with the HP Spectre x360 line

The following peripherals were revealed at CES alongside the HP Spectre x360 2-in-1 Laptop PC line:

• Poly Voyager Free 20 wireless, noise-filtering earbuds. Coming in May 2024 for $149.
• HP 960 ergonomic, split wireless keyboard with programmable keys. This keyboard is expected to be available in April on HP.com for $119.
• HP 690 rechargeable wireless mouse with Qi charging and programmable buttons. Available now on HP.com for $59.99.
• HP 430 programmable wireless keypad. Available now on HP.com for $49.99.
• HP USB-C Travel Hub G3, which adds one USB-C port, two USB-A ports and an HDMI port. Coming late February on HP.com for $69.99.
• HP 400 backlit wired keyboard. Available now on HP.com for $49.99.

HP Series 5 monitors offer more screen space

HP announced its Series 5 monitors at CES 2024, offering 24-, 27- and 32- inch screens for people looking for larger displays. The HP Series 5 monitors connect to other devices through HDMI ports; plus, these monitors offer a 1500:1 contrast ratio and 100Hz refresh rates.

The HP Series 5 monitors are expected to be available in spring 2024. HP has not yet released pricing.

HP debuts ultra-light OMEN Transcend 14-inch gaming laptop PC

The 14-inch OMEN Transcend gaming laptop PC (Figure B) weighs just 3.5 pounds. Its IMAX Enhanced Certified 2.8K 120Hz VRR OLED display is suitable for high-performance gaming or for creative work such as digital painting and graphic design. The most high-performance version runs on an Intel Core Ultra 9 185H processor and NVIDIA GeForce RTX 4070 Laptop GPU. Cooling is achieved through a pressurized zone using a vapor chamber for direct heat dissipation through rear vents.

Figure B

The Intel and NVIDIA processors enable onboard AI to offer tools for work, such as:

• Live transcript and real-time captions during meetings.
• A record function for transcribing audio.
• AI-generated notes.

The OMEN Transcend Gaming Laptop PC 14-inch model is available for preorder now, starting at $1,499.99. The 16-inch model with up to the newest Intel processors and up to OLED displays will be available January 10, starting at $1,899.99.

Competitors to HP’s Spectre x360 and OMEN Transcend 14 laptops

The HP Spectre x360’s directly competes against the Dell XPS 13, the MacBook Pro, the Lenovo Flex 14 and the ASUS VivoBook Flip.

The OMEN Transcend 14 laptops’ weight make them stand out. They go head-to-head with other gaming laptops like the ASUS ROG Zephyrus G14.

Note: TechRepublic is covering CES 2024 remotely.

Samsung’s new smart home features include household maps with ‘AI characters’ Kyle Wiggers 8 hours

Samsung wants to make the smart home smarter — if your home’s a Samsung home, that is.

During its CES 2024 keynote in Las Vegas tonight, the company announced a range of additions to — and capabilities for — its SmartThings home automation platform. A new dashboard screen, Now Plus, is headed to select Samsung TVs, programmed to turn on as you approach to display info about smart home devices and stats like the current indoor temperature. Accompanying it is a new “quick panel” with access to shortcuts for controlling connected devices, as well as functions like finding misplaced smartphones and other mobile gadgets.

Image Credits: Samsung

Elsewhere, Samsung launched a new “map view” for SmartThings similar to Amazon’s recently launched Map View. Samsung’s take shows an interactive map of your home complete with the location of any smart home devices (e.g. washing machines, refrigerators and so on) within. Maps can be created manually or automatically with the help of a photo of an existing floor plan or with a lidar-enabled Samsung device, like the company’s forthcoming Ballie robot or new JetBot robot vacuum.

In a cute (or creepy, depending on your point of view) touch, the new SmartThings maps show “AI characters” that stand in for family members and pets inside the home. The animated avatars “respond” to real-time conditions, for example appearing to sweat if the house gets too warm.

Image Credits: Samsung

Maps have to be generated using the SmartThings app on a smartphone or tablet. But once that’s done, they’ll display on supported Samsung TVs, the screen of the Samsung Family Hub smart fridge and Samsung’s M8 monitors.

As we experience the world, our senses (vision, sounds, smells) provide a diverse array of information, and we express ourselves using different communication methods, such as facial expressions and gestures. These senses and communication methods are collectively called modalities, representing the different ways we perceive and communicate. Drawing inspiration from this human capability, large multimodal model (LMM), a combination of generative and multimodal AI, are being developed to understand and create content using different types like text, images, and audio. In this article, we delve into this newly emerging field, exploring what LMMs (Large Multimodal Models) are, how they're constructed, existing examples, the challenges they face, and potential applications.

Evolution of Generative AI in 2024: From Large Language Models to Large Multimodal Models

In its latest report, McKinsey designated 2023 as a breakout year for generative AI, leading to many advancements in the field. We have witnessed a notable rise in the prevalence of large language models (LLMs) adept at understanding and generating human-like language. Furthermore, image generation models are significantly evolved, demonstrating their ability to create visuals from textual prompts. However, despite significant progress in individual modalities like text, images, or audio, generative AI has encountered challenges in seamlessly combining these modalities in the generation process. As the world is inherently multimodal in nature, it is crucial for AI to grapple with multimodal information. This is essential for meaningful engagement with humans and successful operation in real-world scenarios.

Consequently, many AI researchers anticipate the rise of LMMs as the next frontier in AI research and development in 2024. This evolving frontier focuses on enhancing the capacity of generative AI to process and produce diverse outputs, spanning text, images, audio, video, and other modalities. It is essential to emphasize that not all multimodal systems qualify as LMMs. Models like Midjourney and Stable Diffusion, despite being multimodal, do not fit into the LMM category mainly because they lack the presence of LLMs, which are a fundamental component of LMMs. In other words, we can describe LMMs as an extension of LLMs, providing them with the capability to proficiently handle various modalities.

How do LMMs Work?

While researchers have explored various approaches to constructing LMMs, they typically involve three essential components and operations. First, encoders are employed for each data modality to generate data representations (referred to as embeddings) specific to that modality. Second, different mechanisms are used for aligning embeddings from different modalities into a unified multimodal embedding space. Third, for generative models, an LLM is employed to generate text responses. As inputs may consist of text, images, videos and audios, researchers are working on new ways to make language models consider different modalities when giving responses.

Development of LMMs in 2023

Below, I have briefly outlined some of the notable LMMs developed in 2023.

• LLaVA is an open-source LMM, jointly developed by the University of Wisconsin-Madison, Microsoft Research, and Columbia University. The model aims to offer an open-source version of multimodal GPT4. Leveraging Meta's Llama LLM, it incorporates the CLIP visual encoder for robust visual comprehension. The healthcare-focused variant of LLaVa, termed as LLaVA-Med, can answer inquiries related to biomedical images.
• ImageBind is an open-source model crafted by Meta, emulating the ability of human perception to relate multimodal data. The model integrates six modalities—text, images/videos, audio, 3D measurements, temperature data, and motion data—learning a unified representation across these diverse data types. ImageBind can connect objects in photos with attributes like sound, 3D shapes, temperature, and motion. The model can be used, for instance, to generate scene from text or sounds.
• SeamlessM4T is a multimodal model designed by Meta to foster communication among multilingual communities. SeamlessM4T excels in translation and transcription tasks, supporting speech-to-speech, speech-to-text, text-to-speech, and text-to-text translations. The model employs non-autoregressive text-to-unit decoder to perform these translations. The enhanced version, SeamlessM4T v2, forms the basis for models like SeamlessExpressive and SeamlessStreaming, emphasizing the preservation of expression across languages and delivering translations with minimal latency.
• GPT4, launched by OpenAI, is an advancement of its predecessor, GPT3.5. Although detailed architectural specifics are not fully disclosed, GPT4 is well-regarded for its smooth integration of text-only, vision-only, and audio-only models. The model can generate text from both written and graphical inputs. It excels in various tasks, including humor description in images, summarization of text from screenshots, and responding adeptly to exam questions featuring diagrams. GPT4 is also recognized for its adaptability in effectively processing a wide range of input data formats.
• Gemini, created by Google DeepMind, distinguishes itself by being inherently multimodal, allowing seamless interaction across various tasks without relying on stitching together single-modality components. This model effortlessly manages both text and diverse audio-visual inputs, showcasing its capability to generate outputs in both text and image formats.

Challenges of Large Multimodal Models

• Incorporating More Data Modalities: Most of existing LMMs operate with text and images. However, LMMs need to evolve beyond text and images, accommodating modalities like videos, music, and 3D.
• Diverse Dataset Availability: One of the key challenges in developing and training multimodal generative AI models is the need for large and diverse datasets that include multiple modalities. For example, to train a model to generate text and images together, the dataset needs to include both text and image inputs that are related to each other.
• Generating Multimodal Outputs: While LMMs can handle multimodal inputs, generating diverse outputs, such as combining text with graphics or animations, remains a challenge.
• Following Instructions: LMMs face the challenge of mastering dialogue and instruction-following tasks, moving beyond mere completion.
• Multimodal Reasoning: While current LMMs excel at transforming one modality into another, the seamless integration of multimodal data for complex reasoning tasks, like solving written word problems based on auditory instructions, remains a challenging endeavor.
• Compressing LMMs: The resource-intensive nature of LMMs poses a significant obstacle, rendering them impractical for edge devices with limited computational resources. Compressing LMMs to enhance efficiency and make them suitable for deployment on resource-constrained devices is a crucial area of ongoing research.

Potential Use Cases

• Education: LMMs have the potential to transform education by generating diverse and engaging learning materials that combine text, images, and audio. LMMs provide comprehensive feedback on assignments, promote collaborative learning platforms, and enhance skill development through interactive simulations and real-world examples.
• Healthcare: In contrast to traditional AI diagnostic systems that target a single modality, LMMs improve medical diagnostics by integrating multiple modalities. They also support communication across language barriers among healthcare providers and patients, acting as a centralized repository for various AI applications within hospitals.
• Art and Music Generation: LMMs could excel in art and music creation by combining different modalities for unique and expressive outputs. For example, an art LMM can blend visual and auditory elements, providing an immersive experience. Likewise, a music LMM can integrate instrumental and vocal elements, resulting in dynamic and expressive compositions.
• Personalized Recommendations: LMMs can analyze user preferences across various modalities to provide personalized recommendations for content consumption, such as movies, music, articles, or products.

• Weather Prediction and Environmental Monitoring: LMMs can analyze various modalities of data, such as satellite images, atmospheric conditions, and historical patterns, to improve accuracy in weather prediction and environmental monitoring.

The Bottom Line

The landscape of Large Multimodal Models (LMMs) marks a significant breakthrough in generative AI, promising advancements in various fields. As these models seamlessly integrate different modalities, such as text, images, and audio, their development opens doors to transformative applications in healthcare, education, art, and personalized recommendations. However, challenges, including accommodating more data modalities and compressing resource-intensive models, underscore the ongoing research efforts needed for the full realization of LMMs' potential.