Рубрика: AI

Anthropic in Talks with Menlo Ventures to Raise $750M December 22, 2023 by Drew Jolly

Shutterstock/203818450

Another influx of cash might be on the horizon for Anthropic. The OpenAI competitor is hoping to raise $750 million in a new round of funding, according to Bloomberg and originally reported by The Information. The AI startup is in advanced talks to secure this investment, led by Silicon Valley's Menlo Ventures, and the potential round could raise the company's valuation to an estimated $18.4 billion. However, this deal is not yet finalized and remains confidential, as sources close to the matter reveal.

Founded in 2021 by former members of OpenAI, Anthropic is the creator of Claude, an advanced conversational AI chatbot with a strong focus on safety and ethical considerations.

This is only the most recent chapter in Anthropic’s funding saga. The company commenced with a $124 million Series A in May 2021, followed by a $580 million Series B in April 2022. March saw a strategic investment of $300 million from Google, valuing Anthropic at $4.1 billion. The upward trajectory continued with a $450 million Series C in May 2023, led by Spark Capital. Sapphire Ventures LLC pledged over $1 billion in June, and in August, South Korean telecommunications giant SK Telecom invested $100 million.

The year culminated with significant commitments from major tech players. Amazon announced in September that it will invest up to $4 billion and have a minority ownership position in the company. Google quickly followed suit in October with its own pledge of up to $2 billion.

Prior to this development, Anthropic was last in the news in November with the release of Claude 2.1. The latest iteration of the chatbot features a massive 200K token context window, significant reductions in rates of model hallucination, and a new tool use beta feature.

Earlier in the year, Anthropic finalized a major agreement with Google for cloud computing which, among other aspects, involves Anthropic utilizing Google’s numerous cloud services over an extended timeline. Furthermore, Anthropic has a similar arrangement with Amazon tied to the investment announcement, wherein the AI startup will tap into Amazon AWS’s cloud services and technology.

Editor's note: This article originally appeared on Datanami.

Related

Cybersecurity company ESET released its H2 2023 threat report, and we’re highlighting three particularly interesting topics in it: the abuse of the ChatGPT name by cybercriminals, the rise of the Lumma Stealer malware and the Android SpinOk SDK spyware.

Jump to:

• ChatGPT name is being abused by cybercriminals
• Lumma Stealer malware-as-a-service is going strong
• Android SpinOk SDK case is a spyware standout

ChatGPT name is being abused by cybercriminals

In the second half of 2023, ESET has blocked 650,000 attempts to access malicious domains whose names include “chatgpt” or similar string in an apparent reference to the ChatGPT chatbot.

One of the frauds resides in the OpenAI API for ChatGPT. The API needs a private API key that must be carefully protected and never exposed by users, yet some apps ask users to provide their API keys so the applications can use ChatGPT. As written by ESET researchers, “if the app sends your key to the developer’s server, there may be little to no guarantee that your key will not be leaked or misused, even if the call to the OpenAI API is also made.”

A “ChatGPT Next Web” web application taken as an example by ESET has been installed on 7,000 servers. It is unknown if this app was created as an effort in a ChatGPT API keys phishing campaign or exposed on the internet for another reason.

The use of the API key is billed by OpenAI. So once in possession of someone’s private API key and depending on the users or company’s subscription, an attacker might use it for their own needs without paying; the attacker might also resell it to other cybercriminals.

In addition, the second half of 2023 saw a lot of ChatGPT-inspired domain names all leading to malicious Google Chrome browser extensions detected as “JS/Chromex.Agent.BZ”. One example is gptforchrome(.)com, leading to such a malicious extension (Figure A).

Figure A

Recommendations related to these ChatGPT security threats

Users should be educated to detect such threats and avoid browsing suspicious websites related to ChatGPT. They must secure their private ChatGPT API key and never share it.

Lumma Stealer malware-as-a-service is going strong

In H2 2023, malicious cryptominers declined by 21% in the cryptocurrencies malware threat landscape, according to ESET; however, cryptostealers are on the rise by more than 68% for the same period, wrote the researchers.

This strong augmentation was caused by a single specific threat: Lumma Stealer, which is also known as LummaC2 Stealer. This malware-as-a-service threat targets multiple cryptocurrency wallets as well as users’ credentials and two-factor authentication browser extensions. It also has exfiltration capabilities, rendering it a tool that might be used for financial fraud as well as for cyberespionage purposes.

According to ESET, the deployment of Lumma Stealer tripled between H1 and H2 2023. Multiple tiers are offered for the malware with prices ranging from $250 USD to $20,000 USD. The highest option allows the buyer to get access to the full C source code for the malware. The buyer is also allowed to resell the malware independently of its developer.

The Lumma Stealer malware shares a common code base with the infamous Mars, Arkei, and Vidar information stealers and is very likely to be developed by the same author, according to cybersecurity company Sekoia.

Various distribution vectors are used for spreading Lumma Stealer; ESET observed these methods in the wild: cracked installations of software, YouTube, fake browser update campaigns, content delivery network of Discord and installation via third-party malware loader Win/TrojanDownloader.Rugmi.

Tips for protecting against such malware threats

It is highly recommended to always keep operating systems and their software up to date and patched to avoid being compromised by any common vulnerability that could lead to malware infection. And, users should never be allowed to download and install software without proper analysis from the organization’s IT team.

Android SpinOk SDK is a spyware standout

A mobile marketing software development kit identified as the SpinOk spyware by ESET climbed to being the seventh most detected Android threat for H2 2023 and the most prevalent type of spyware for the period.

The SpinOk SDK offered developers a gaming platform intended to monetize application traffic. Multiple developers incorporated the SDK in their apps, including apps already available on official Android marketplaces. Once running, the application starts to act as spyware and connects to a command & control server before starting to extract data from the Android device, including potentially sensitive clipboard content, according to ESET.

The malicious code has features to try to stay undetected. It uses the device’s gyroscope and magnetometer to determine if it is running in a virtual or lab environment; if so, it changes its behavior in an attempt to avoid being detected by researchers.

The SDK has been incorporated into various legitimate Android applications. In fact, 101 Android apps have used the malicious SDK, with more than 421 million cumulated downloads, as reported in May 2023 by cybersecurity company Doctor Web, who contacted Google; then, Google removed all those applications from the Google Play Store. The company responsible for SpinOk contacted Doctor Web and updated its module to version 2.4.2, which removed all the spyware features.

A company called Roaster Earn explained how they ended up installing the SDK in their own application. Basically, they have been approached by the OkSpin company responsible for the SpinOk SDK with a “revenue growth program,” which they accepted, before Google notified them of their app removal because it contained spyware. This case once is once again a reminder of the complex problem of incorporating third-party code in software that is increasingly abused by cybercriminals.

How to mitigate the risk of using third-party code in software

• Analyze the third-party code for any anomalies, when possible. This might help to avoid falling for code containing malicious content or functionalities.
• Use static analysis tools to detect potential vulnerabilities or behavior.
• Monitor network traffic for any suspicious or unexpected traffic.
• Scrutinize the reputation of the code provider and feedback about the organization, as well as security certifications or audits the provider might share.

Disclosure: I work for Trend Micro, but the views expressed in this article are mine.

Proposed new rule: All AI output must rhyme Haje Jan Kamps 9 hours

Welcome to Startups Weekly. Sign up here to get it in your inbox every Friday.

I know I usually lead Startups Weekly with my own column (and I did write one this week, so if you are somehow a fan of everything I write, go ham), but Devin’s piece taking a stand against the pseudanthropy of AI is a must-read. He proposes a new set of rules for AIs to follow to preserve our humanity:

1. AI must rhyme​​.
2. AI may not present a face or identity​​.
3. AI cannot “feel” or “think.”​​
4. AI-derived figures, decisions and answers must be marked ‘⸫’ ​​.
5. AI must not make life or death decisions​​.
6. AI imagery must have a corner clipped​​.

Yes, these suggestions can’t realistically be implemented, but read the article anyway; it goes deep about some of the interesting challenges we are facing as AIs get more mature and ubiquitous.

Okay, with today’s philosophy lesson out of the way, let’s dive in and see what else is up in the Right Honorable Royal Kingdom of Startups.

Flying high, dippin’ deep

Image Credits: Bird

The turbulent journey of startups is continuing.

Bird, the once high-flying electric scooter company, has crash-landed into bankruptcy. After a wild ride from a $2 billion valuation to a financial face-plant, this former micromobility poster child is now restructuring its finances faster than one of its scooters racing downhill with a tailwind. Now they’re banking on Chapter 11 to keep their wheels spinning, but only after laying off some feathers and hoping someone finds enough value in their assets to buy them out. The irony? Their Canadian and European operations are still scooting along as if nothing happened.

I’m not gonna say “I told you so,” but there was definitely no coincidence that I picked Bird as the example for the ‘Undertanding the levers in your business’ post I wrote back in 2018 . . .

Anyway. Here’s a few more stories that’ve kept y’all clickin’:

Back to startups: Eric Wu, the co-founder of Opendoor, is ditching his executive chair for a bean bag in the startup world again. After a decade of playing real estate Tetris, Wu’s ready to go back to building things from scratch, amid the toughest real estate market in more than 40 years.

Feeling safer yet?: In a move that’s less surprising than finding out your password is still “password123,” Okta has snapped up security firm Spera for a cool $100 million-ish. The latter is like a cybersecurity Sherlock Holmes, sniffing out digital weaknesses before they become full-blown disasters.

I bet this newsletter triggers their algorithms: Meltwater, the media monitoring maestro that’s been dancing around print and digital news like a tech-savvy ballerina, just got a $65 million pat on the back from Verdane, valuing the company at a cool $592 million.

When artificial intelligence is more prevalent than real intelligence

Image Credits: Getty Images

Devin breaks open the crystal ball for AI in 2024 and predicts a roller-coaster ride from hype to a reality check. He suggests that OpenAI, post-leadership shuffle, might morph into an Apple-esque “ship it” product powerhouse with its own AI app store. Meanwhile, niche AI applications, like agent-based models and generative multimedia, could graduate from “meh” to “hmm, interesting,” especially in monotonous tasks like insurance claims. In the political arena, AI might become a tool for misinformation and manipulation in the 2024 elections, with bot accounts and fake news ramping up the chaos.

I can’t say that I disagree. When media literacy hits rock bottom and AI is on the upswing, we have a perfect storm.

Cool cool cool. What else has been cookin’ in the AI kitchen?

Composers, composers, composers: Microsoft Copilot, the AI-powered chatbot, is now dipping its digital toes into the world of music composition through an integration with the GenAI music app Suno. Users can prompt Copilot to create complete songs, including lyrics and instrumentals, with requests like “Create a pop song about adventures with your family.”

Hey, Spotify, make me a playlist where every song starts with the letters W, T, and F: Spotify is testing an “AI playlists” feature that lets users create playlists using AI prompts. Users can type prompts into an AI chatbot-style box or choose from suggested prompts like “Get focused at work with instrumental electronica,” or “Songs most likely to drive my parents up the wall.”

Sorry, Charles Ponzi, you can’t shop here: Rite Aid has been banned from using facial recognition software for five years, after it was found that its “reckless use of facial surveillance systems” resulted in customer humiliation and jeopardized sensitive information.

There’s an app for that

Image Credits: Bryce Durbin / TechCrunch

Apple has been ordered to cough up $25 million to settle a lawsuit over its Family Sharing feature. The Cupertino-based software giant was promoting a “share-all-the-things” feature for apps that were . . . not sharable. Despite Apple’s “Who, us?” stance, they decided to throw money at the problem rather than endure the endless courtroom drama. Now, some lucky Family Sharing users from the good ol’ days (2015–2019) might get a whopping $30 payout. That’s three months of a post-price-hike Apple TV+ subscription. Yay.

Apple got off lightly compared to Google’s recent day in court. In a “My bad, here’s some cash” move, Google’s digging into its sofa cushions for a spare $700 million to settle a lawsuit over its Play Store monopoly antics. Of this, $630 million goes to U.S. consumers and a neat $70 million to U.S. states. The search giant, once known for its “Do No Evil” motto, apparently didn’t extend that to app distribution on Android. As part of the deal, Google’s also revamping its user choice billing program in the U.S., allowing developers more freedom in billing methods. They’re even making sideloading (i.e., installing apps without Google’s blessing) less of a digital obstacle course. But let’s not get too excited — as Epic Games’ VP of public policy points out, consumers are still likely to overpay for digital goods thanks to Google’s hefty fees. So, while Google’s wallet gets lighter, our wallets might not feel much different.

Court shenanigans aside . . .

Sharing is caring: Claim, the new social media kid on the block, is trying to make sharing rewards with friends the next big thing. They’ve scooped up $4 million from Sequoia Capital to turn buying stuff into a multiplayer game.

Oh, hi, didn’t see you there: Jagat, a location-based social network that’s all about real-life connections, has zoomed past 10 million users. Launched in March, this app, which is like a social map for friends and activities, is aiming to make social networking, well, social again.

Link in bio: Linktree, the Australian link-in-bio platform, has scooped up its competitor Koji from GoMeta. In this game of digital Monopoly, Linktree’s not only expanding its empire but also sending Koji’s product into retirement by January 2024.

Top reads on TechCrunch this week

I dipped into our analytics software to see what else might be worth highlighting from the past week or so. Here are a few additional reads:

Taking the oxygen out of Apple’s Christmas sails: Apple has paused sales of its Apple Watch Series 9 and Ultra 2 due to a patent dispute with Masimo, a medical tech firm. The dispute involves the blood oxygen sensor in Apple’s smartwatches.

To Xfinity and beyond: Comcast’s Xfinity service fell victim to a cyberattack, affecting almost 36 million customers. The breach potentially exposed customer usernames, hashed passwords, contact information, dates of birth, parts of Social Security numbers, and secret questions and answers.

Where were you, tho?: Google, in a move that could make Big Brother a little less nosy, announced plans to store users’ location data on their devices instead of on its servers. This change aims to put an end to the use of “geofence warrants,” where police demand Google hand over data from devices in a specific area at a certain time. These warrants have been criticized as overly broad and possibly unconstitutional.

As I completed this blog series, the European Union (EU) announced its AI Regulation Law. The European Union’s AI Regulation Act seeks to ensure AI’s ethical and safe deployment in the EU. Coming on the heels of the White House’s “Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence,” we should expect a steady stream of legislation designed to ensure that AI is being used to deliver meaningful, relevant, responsible, and ethical outcomes.

The mandate to deploy AI in a manner that creates a more fair, just, and prosperous world for everyone is upon us. What are the key concepts and actions we must undertake to achieve the goal of the just and fair application of AI? Well, here is one view of how to achieve that goal:

(1) Officials Negotiate Laws and Regulations. Elected legislators would continue to define our society’s laws and regulations. Unfortunately, the biased and opaque enforcement of today’s laws and regulations leads to manipulation, distrust, anger, and citizen disenfranchisement.

(2) AI Models Enforce Laws and Regulations. To address the issues of manipulation, distrust, anger, and citizen disenfranchisement, we would create AI models to enforce society’s laws and regulations in a more unbiased, consistent, and transparent manner. For these laws and regulations, legislators would need to collaborate across diverse groups of constituents to identify the desired outcomes, their importance (weights), and the measures against which laws and regulations compliance would be enforced.

(3) AI Utility Function Engineered for Compliance. The AI Utility Function, a mathematical function that evaluates the different actions that the AI system can take, would include the desired outcomes of the laws and regulations, their importance (weights), and their supporting measures. The AI Utility Function would ensure the enforcement of the laws and regulations in an unbiased, consistent, and transparent manner.

(4) AI Utility Function Becomes More Human. To address the concern of AI model authoritarian decision-making, we can expand the variables and metrics in the AI Utility Function to provide enforcement “leniency” for those “who have earned the benefit” of leniency consideration. Then, when leniency is given in enforcement, the measures that drove the leniency decision would be transparent to everyone.

(5) Leverage Legal Evidence Admissibility Tests. We would leverage time-tested legal evidence admissibility tests – like the Frye and Daubert standards – to help select the measures that comprise a healthy, unbiased AI Utility Function. This would help safeguard the protected class of constituents as defined by our legal systems.

(6) AI Model Audit Trail Tracks AI Governance. To monitor the ongoing effectiveness and fairness of the AI models, we would mandate an AI Model audit trail that captures the AI models’ desired outcomes, their respective “weights,” and the metrics and variable weights at the time of any decision or action.

(7) AI Certification Programs Assess AI Compliance. Government, independent agencies, and third-party auditors would develop AI Certification models (e.g., GDPR AI Enforcement Model, CCPA AI Enforcement App, ACLU Enforcement App, EY Enforcement App) to analyze, assess, benchmark, and grade the AI models’ audit trail to officially certify responsible and ethical performance.

(8) New AI Governance Jobs. New, high-paying jobs would emerge to ensure AI’s relevant, meaningful, responsible, and ethical deployment, root out nefarious AI models, and prosecute organizations and their responsible senior executives on compliance violations in an unbiased, consistent, and transparent manner.

Eight steps to ensure the unbiased, transparent design and deployment of AI to deliver more relevant, meaningful, responsible, and ethical outcomes. A mandate in which everyone’s role, responsibilities, and rights are carefully and thoroughly articulated.

This does seem like the brave new world in which everyone would want to live (well, the nefarious evil-doers might not like this world).

=====

Note: You can find more details about these eight steps in these supporting blogs:

In part 1 of the series “A Different AI Scenario: AI and Justice in a Brave New World,” I outlined some requirements for the role that I would play in enforcing our laws and regulations in a more just and fair manner and what our human legislators must do to ensure that outcome (Figure 1).

Figure 1: Dimensions of Economic Value Creation

In part 2, I discussed a real example of how AI is being used to enforce our laws and regulations more unbiased, consistent, and transparently – robot umpires – and what we can do to humanize those robot umpires (and AI) just a little. I also tapped some time-tested standards that courts use to guide the admissibility of evidence and how those lessons can help us build more fair and just AI models in this brave new world of AI (Figure 2).

Figure 2: Humanizing AI Models for “Earned Leniency”

In part 3, I discussed the challenges of operationalizing the AI governance necessary and introduced the concept of the AI Model Audit Trail concept and how third parties could create AI certification models that ensure that the AI models follow the laws and regulations and respect the rights and freedoms of the data subjects and natural persons. Finally, we outlined the challenges to the AI Model Audit Trail concept and the steps we can take to ensure its viability.

Figure 3: AI Model Audit Trail Concept

In the realm of big data, the ability to efficiently manage, interpret, and leverage vast amounts of diverse information is crucial. This is where the concept of a semantic layer comes into play, serving as a vital component in the architecture of big data systems. The semantic layer is a technology that sits between the user and the complex databases, providing a user-friendly interface that simplifies data retrieval and analysis. It's important to understand the intricacies of semantic layers in big data, exploring their significance, functionality, and impact on data management and analysis.

The Essence of Semantic Layers

At its core, a semantic layer is an abstraction layer that provides a consistent, business-oriented framework for accessing data. It translates complex technical data structures into familiar business terms, making it easier for non-technical users to interact with the data without needing to understand the underlying database languages or structures. Essentially, the semantic layer acts as a translator, converting the technical language of data into a language that is understandable and usable for business analysts and decision-makers.

Functionality and Components

The functionality of a semantic layer is multi-faceted. It includes the capability to define relationships between different data elements, apply business logic, and standardize metrics across various data sources. Key components of a semantic layer typically include:

• Metadata Repository: This stores information about data sources, relationships, hierarchies, calculations, and business rules.
• Query Engine: Responsible for converting user queries into commands that the underlying databases understand.
• Data Modeling Tools: These tools allow users to define, manipulate, and manage semantic models that represent business concepts and data relationships.

Benefits in Big Data Environments

In big data contexts, semantic layers offer several significant advantages:

• Improved Data Accessibility: By presenting data in familiar terms, semantic layers democratize data access, enabling a broader range of users to extract insights.
• Enhanced Data Consistency: They ensure that everyone in the organization uses the same definitions and business rules, leading to consistent and reliable analytics.
• Increased Efficiency: Semantic layers streamline the process of data analysis, reducing the time and technical know-how required to derive insights.
• Scalability and Flexibility: They can efficiently handle the volume, variety, and velocity of big data, providing flexibility in accommodating new data sources and evolving business needs.

Challenges and Considerations

While semantic layers bring numerous benefits, they also pose certain challenges:

• Complexity in Design: Creating an effective semantic layer requires a deep understanding of both the technical and business aspects of an organization’s data.
• Performance Issues: Poorly designed semantic layers can lead to performance bottlenecks, especially when dealing with large volumes of data.
• Maintenance: As business needs and data sources evolve, maintaining and updating the semantic layer can be resource-intensive.

Practical examples

To illustrate how semantic layers are implemented and utilized in various industries:

Retail Industry: Unified View of Customer Data

In a retail company, data is collected from multiple sources like sales transactions, online shopping behavior, and customer feedback surveys. A semantic layer integrates this diverse data, translating it into a unified view. For example, it can correlate customer demographic information with purchasing patterns and preferences, presented in understandable terms like “Average Spend per Visit” or “Frequent Purchase Categories.” This unified view helps in personalized marketing and inventory management.

Healthcare: Patient Data Analysis

Hospitals and healthcare providers deal with vast amounts of patient data, including medical records, lab results, and treatment histories. A semantic layer in a healthcare data system can standardize medical terms and patient data across various departments. It enables healthcare professionals to query complex medical records using simple, familiar terms, such as “Patient Recovery Rate” or “Medication Effectiveness,” facilitating quicker and more accurate medical analyses and decisions.

Finance: Risk Management and Compliance

In the financial sector, institutions manage extensive data related to transactions, market trends, and customer profiles. A semantic layer helps in simplifying and interpreting this data into business-oriented insights. For instance, it can transform complex financial metrics into comprehensible risk scores or compliance ratings, aiding in making strategic decisions in risk management and regulatory adherence.

Manufacturing: Supply Chain Optimization

Manufacturing companies often operate with complex supply chains. A semantic layer can integrate data from various stages of the supply chain, from raw material procurement to product distribution. By translating this data into more accessible terms like “Inventory Turnover Ratio” or “Supplier Reliability Score,” companies can optimize their supply chain processes more efficiently.

E-Commerce: Personalized Shopping Experience

E-commerce platforms gather data on customer interactions, preferences, and buying habits. A semantic layer helps in translating this data into insights that can be used to personalize the shopping experience. For example, it can identify patterns and preferences, allowing the platform to recommend products as “Frequently Bought Together” or highlight “Trending Items in Your Area.”

Education: Learning Management Systems

Educational institutions use learning management systems (LMS) that generate vast amounts of data from student activities, assessments, and course interactions. A semantic layer can help educators and administrators analyze this data in a more straightforward way, such as understanding “Student Performance Trends” or “Course Engagement Levels,” to improve teaching methods and curriculum development.

Conclusion

In each of these examples, the semantic layer plays a crucial role in transforming raw, complex data into actionable insights, tailored to the specific needs and language of each industry. This demonstrates the layer's versatility and importance in various big data applications.

The semantic layer is a pivotal element in the big data ecosystem, bridging the gap between complex data infrastructures and business users. It enhances the accessibility, consistency, and efficiency of data analysis, empowering organizations to harness the full potential of their data assets. As businesses continue to navigate the vast seas of big data, the semantic layer stands out as a beacon, guiding them towards more informed and impactful decision-making. However, it is essential to approach its implementation and maintenance with thorough planning and consideration, ensuring that it serves its purpose effectively and continues to evolve with the changing data landscape.

Top robotics names discuss humanoids, generative AI and more

Leading thinkers from CMU, UC Berkeley, Meta, NVIDIA, Boston Dynamics and the Toyota Research Institute share their predictions

Brian Heater @bheater / 10 hours

Last month, I took an extended break. In a bid to keep my robotics newsletter Actuator (subscribe here) up and running, however, I reached out to some of the biggest names in the industry. I asked people from CMU, UC Berkeley, Meta, NVIDIA, Boston Dynamics and the Toyota Research Institute the same six questions, covering topics like generative AI, the humanoid form factor, home robots and more. You’ll find all of the answers broken up by question below. You would be hard pressed to find a more comprehensive breakdown of robotics in 2023 and the path its blazing for future technologies.

What role(s) will generative AI play in the future of robotics?

Image Credits: Getty Images

Matthew Johnson-Roberson, CMU: Generative AI, through its ability to generate novel data and solutions, will significantly bolster the capabilities of robots. It could enable them to better generalize across a wide range of tasks, enhance their adaptability to new environments and improve their ability to autonomously learn and evolve.

Dhruv Batra, Meta: I see generative AI playing two distinct roles in embodied AI and robotics research:

• Data/experience generators
  Generating 2D images, video, 3D scenes, or 4D (3D + time) simulated experiences (particularly action/language conditioned experiences) for training robots because real-world experience is so scarce in robotics. Basically, think of these as “learned simulators.” And I believe robotics research simply cannot scale without training and testing in simulation.
• Architectures for self-supervised learning
  Generating sensory observations that an agent will observe in the future, to be compared against actual observations, and used as an annotation-free signal for learning. See Yann’s paper on AMI for more details.

Aaron Saunders, Boston Dynamics: The current rate of change makes it hard to predict very far into the future. Foundation models represent a major shift in how the best machine learning models are created, and we are already seeing some impressive near-term accelerations in natural language interfaces. They offer opportunities to create conversational interfaces to our robots, improve the quality of existing computer vision functions and potentially enable new customer-facing capabilities such as visual question answering. Ultimately we feel these more scalable architectures and training strategies are likely to extend past language and vision into robotic planning and control. Being able to interpret the world around a robot will lead to a much richer understanding on how to interact with it. It’s a really exciting time to be a roboticist!

Russ Tedrake, TRI: Generative AI has the potential to bring revolutionary new capabilities to robotics. Not only are we able to communicate with robots in natural language, but connecting to internet-scale language and image data is giving robots a much more robust understanding and reasoning about the world. But we are still in the early days; more work is needed to understand how to ground image and language knowledge in the types of physical intelligence required to make robots truly useful.

Ken Goldberg, UC Berkeley: Although the rumblings started a bit earlier, 2023 will be remembered as the year when generative AI transformed Robotics. Large language models like ChatGPT can allow robots and humans to communicate in natural language. Words evolved over time to represent useful concepts from “chair” to “chocolate” to “charisma.” Roboticists also discovered that large Vision-Language-Action models can be trained to facilitate robot perception and to control the motions of robot arms and legs. Training requires vast amounts of data so labs around the world are now collaborating to share data. Results are pouring in and although there are still open questions about generalization, the impact will be profound.

Another exciting topic is “Multi-Modal models” in two senses of multi-modal:

• Multi-Modal in combining different input modes, e.g. Vision and Language. This is now being extended to include Tactile and Depth sensing, and Robot Actions.

• Multi-Modal in terms of allowing different actions in response to the same input state. This is surprisingly common in robotics; for example there are many ways to grasp an object. Standard deep models will “average” these grasp actions which can produce very poor grasps. One very exciting way to preserve multi-modal actions is Diffusion Policies, developed by Shuran Song, now at Stanford.

Deepu Talla, NVIDIA: We’re already seeing productivity improvements with generative AI across industries. Clearly, GenAI’s impact will be transformative across robotics from simulation to design and more.

• Simulation: Models will be able to accelerate simulation development, bridging the gaps between 3D technical artists and developers, by building scenes, constructing environments and generating assets. These GenAI assets will see increased use for synthetic data generation, robot skills training and software testing.
• Multimodal AI: Transformer-based models will improve the ability of robots to better understand the world around them, allowing them to work in more environments and complete complex tasks.
• Robot (re)programming: Greater ability to define tasks and functions in simple language to make robots more general/multipurpose.
• Design: Novel mechanical designs for better efficiency — for example, end effectors.

What are your thoughts on the humanoid form factor?

Image Credits: NanoStockk (opens in a new window) / Getty Images

Ken Goldberg, UC Berkeley: I’ve always been skeptical about humanoids and legged robots, as they can be overly sensational and inefficient, but I’m reconsidering after seeing the latest humanoids and quadrupeds from Boston Dynamics, Agility and Unitree. Tesla has the engineering skills to develop low-cost motors and gearing systems at scale. Legged robots have many advantages over wheels in homes and factories to traverse steps, debris and rugs. Bimanual (two-armed) robots are essential for many tasks, but I still believe that simple grippers will continue to be more reliable and cost-effective than five-fingered robot hands.

Deepu Talla, NVIDIA: Designing autonomous robots is hard. Humanoids are even harder. Unlike most AMRs that mainly understand floor-level obstacles, humanoids are mobile manipulators that will need multimodal AI to understand more of the environment around them. An incredible amount of sensor processing, advanced control and skills execution is required.

Breakthroughs in generative AI capabilities to build foundational models are making the robot skills needed for humanoids more generalizable. In parallel, we’re seeing advances in simulations that can train the AI-based control systems as well as the perception systems.

Matthew Johnson-Roberson, CMU: The humanoid form factor is a really complex engineering and design challenge. The desire to mimic human movement and interaction creates a high bar for actuators and control systems. It also presents unique challenges in terms of balance and coordination. Despite these challenges, the humanoid form has the potential to be extremely versatile and intuitively usable in a variety of social and practical contexts, mirroring the natural human interface and interaction. But we probably will see other platforms succeed before these.

Max Bajracharya, TRI: Places where robots might assist people tend to be designed for people, so these robots will likely need to fit and work in those environments. However, that does not mean they need to take a humanoid (two arms, five-fingered hands, two legs and a head) form factor; simply, they need to be compact, safe and capable of human-like tasks.

Dhruv Batra, Meta: I’m bullish on it. Fundamentally, human environments are designed for the humanoid form factor. If we really want general-purpose robots operating in environments designed for humans, the form factor will have to be at least somewhat humanoid (the robot will likely have more sensors than humans and may have more appendages, as well).

Aaron Saunders, Boston Dynamics: Humanoids aren’t necessarily the best form factor for all tasks. Take Stretch, for example — we originally generated interest in a box-moving robot from a video we shared of Atlas moving boxes. Just because humans can move boxes doesn’t mean we’re the best form factor to complete that task, and we ultimately designed a custom robot in Stretch that can move boxes more efficiently and effectively than a human. With that said, we see great potential in the long-term pursuit of general-purpose robotics, and the humanoid form factor is the most obvious match to a world built around our form. We have always been excited about the potential of humanoids and are working hard to close the technology gap.

Following manufacturing and warehouses, what is the next major category for robotics?

Image Credits: Getty Images

Max Bajracharya, TRI: I see a lot of potential and needs in agriculture, but the outdoor and unstructured nature of many of the tasks is very challenging. Toyota Ventures has invested in a couple of companies like Burro and Agtonomy, which are making good progress in bringing autonomy to some initial agricultural applications.

Matthew Johnson-Roberson, CMU: Beyond manufacturing and warehousing, the agricultural sector presents a huge opportunity for robotics to tackle challenges of labor shortage, efficiency and sustainability. Transportation and last-mile delivery are other arenas where robotics can drive efficiency, reduce costs and improve service levels. These domains will likely see accelerated adoption of robotic solutions as the technologies mature and as regulatory frameworks evolve to support wider deployment.

Aaron Saunders, Boston Dynamics: Those two industries still stand out when you look at matching up customer needs with the state of art in technology. As we fan out, I think we will move slowly from environments that have determinism to those with higher levels of uncertainty. Once we see broad adoption in automation-friendly industries like manufacturing and logistics, the next wave probably happens in areas like construction and healthcare. Sectors like these are compelling opportunities because they have large workforces and high demand for skilled labor, but the supply is not meeting the need. Combine that with the work environments, which sit between the highly structured industrial setting and the totally unstructured consumer market, and it could represent a natural next step along the path to general purpose.

Deepu Talla, NVIDIA: Markets where businesses are feeling the effects of labor shortages and demographic shifts will continue to align with corresponding robotics opportunities. This spans robotics companies working across diverse industries, from agriculture to last-mile delivery to retail and more.

A key challenge in building autonomous robots for different categories is to build the 3D virtual worlds required to simulate and test the stacks. Again, generative AI will help by allowing developers to more quickly build realistic simulation environments. The integration of AI into robotics will allow increased automation in more active and less “robot-friendly” environments.

Ken Goldberg, UC Berkeley: After the recent union wage settlements, I think we’ll see many more robots in manufacturing and warehouses than we have today. Recent progress in self-driving taxis has been impressive, especially in San Francisco where driving conditions are more complex than Phoenix. But I’m not convinced that they can be cost-effective. For robot-assisted surgery, researchers are exploring “Augmented Dexterity” — where robots can enhance surgical skills by performing low-level subtasks such as suturing.

How far out are true general-purpose robots?

Image Credits: Yuichiro Chino / Getty Images

Dhruv Batra, Meta: Thirty years. So effectively outside the window where any meaningful forecasting is possible. In fact, I believe we should be deeply skeptical and suspicious of people making “AGI is around the corner” claims.

Deepu Talla, NVIDIA: We continue to see robots becoming more intelligent and capable of performing multiple tasks in a given environment. We expect to see continued focus on mission-specific problems while making them more generalizable. True general-purpose embodied autonomy is further out.

Matthew Johnson-Roberson, CMU: The advent of true general-purpose robots, capable of performing a wide range of tasks across different environments, may still be a distant reality. It requires breakthroughs in multiple fields, including AI, machine learning, materials science and control systems. The journey toward achieving such versatility is a step-by-step process where robots will gradually evolve from being task-specific to being more multi-functional and eventually general purpose.

Russ Tedrake, TRI: I am optimistic that the field can make steady progress from the relatively niche robots we have today towards more general-purpose robots. It’s not clear how long it will take, but flexible automation, high-mix manufacturing, agricultural robots, point-of-service robots and likely new industries we haven’t imagined yet will benefit from increasing levels of autonomy and more and more general capabilities.

Ken Goldberg, UC Berkeley: I don’t expect to see true AGI and general-purpose robots in the near future. Not a single roboticist I know worries about robots stealing jobs or becoming our overlords.

Aaron Saunders, Boston Dynamics: There are many hard problems standing between today and truly general-purpose robots. Purpose-built robots have become a commodity in the industrial automation world, but we are just now seeing the emergence of multi-purpose robots. To be truly general purpose, robots will need to navigate unstructured environments and tackle problems they have not encountered. They will need to do this in a way that builds trust and delights the user. And they will have to deliver this value at a competitive price point. The good news is that we are seeing an exciting increase in critical mass and interest in the field. Our children are exposed to robotics early, and recent graduates are helping us drive a massive acceleration of technology. Today’s challenge of delivering value to industrial customers is paving the way toward tomorrow’s consumer opportunity and the general purpose future we all dream of.

Will home robots (beyond vacuums) take off in the next decade?

Image Credits: Lego

Matthew Johnson-Roberson, CMU: The advent of true general-purpose robots, capable of performing a wide range of tasks across different environments, may still be a distant reality. It requires breakthroughs in multiple fields, including AI, machine learning, materials science and control systems. The journey toward achieving such versatility is a step-by-step process where robots will gradually evolve from being task-specific to being more multi-functional and eventually general purpose.

Deepu Talla, NVIDIA: We’ll have useful personal assistants, lawn mowers and robots to assist the elderly in common use.

The trade-off that’s been hindering home robots, to date, is the axis of how much someone is willing to pay for their robot and whether the robot delivers that value. Robot vacuums have long delivered the value for their price point, hence their popularity.

Also, as robots become smarter, having intuitive user interfaces will be key for increased adoption. Robots that can map their own environment and receive instructions via speech will be easier to use by home consumers than robots that require some programming.

The next category to take off would likely first be focused outdoors — for example, autonomous lawn care. Other home robots like personal/healthcare assistants show promise but need to address some of the indoor challenges encountered within dynamic, unstructured home environments.

Max Bajracharya, TRI: Homes remain a difficult challenge for robots because they are so diverse and unstructured, and consumers are price-sensitive. The future is difficult to predict, but the field of robotics is advancing very quickly.

Aaron Saunders, Boston Dynamics: We may see additional introduction of robots into the home in the next decade, but for very limited and specific tasks (like Roomba, we will find other clear value cases in our daily lives). We’re still more than a decade away from multifunctional in-home robots that deliver value to the broad consumer market. When would you pay as much for a robot as you would a car? When it achieves the same level of dependability and value you have come to take for granted in the amazing machines we use to transport us around the world.

Ken Goldberg, UC Berkeley: I predict that within the next decade we will have affordable home robots that can declutter — pick up things like clothes, toys and trash from the floor and place them into appropriate bins. Like today’s vacuum cleaners, these robots will occasionally make mistakes, but the benefits for parents and senior citizens will outweigh the risks.

Dhruv Batra, Meta: No, I don’t believe the core technology is ready.

What important robotics story/trend isn’t getting enough coverage?

Image Credits: Yurii Karvatskyi / Getty Images

Aaron Saunders, Boston Dynamics: There is a lot of enthusiasm around AI and its potential to change all industries, including robotics. Although it has a clear role and may unlock domains that have been relatively static for decades, there is a lot more to a good robotic product than 1’s and 0’s. For AI to achieve the physical embodiment we need to interact with the world around us, we need to track progress in key technologies like computers, perception sensors, power sources and all the other bits that make up a full robotic system. The recent pivot in automotive towards electrification and Advanced Driver Assistance Systems (ADAS) is quickly transforming a massive supply chain. Progress in graphics cards, computers and increasingly sophisticated AI-enabled consumer electronics continues to drive value into adjacent supply chains. This massive snowball of technology, rarely in the spotlight, is one of the most exciting trends in robotics because it enables small innovative companies to stand on the backs of giants to create new and exciting products.

Ken Goldberg, UC Berkeley: Robot motion planning. This is one of the oldest subjects in robotics — how to control the motor joints to move the robot tool and avoid obstacles. Many think this problem has been solved, but it hasn’t. Robot “singularities” are a fundamental problem for all robot arms; they are very different from Kurzweil’s hypothetical point in time when AI surpasses humans. Robot singularities are points in space where a robot stops unexpectedly and must be manually reset by a human operator. Singularities arise from the math needed to convert desired straight-line motion of the gripper into the corresponding motions for each of the six robot joint motors. At certain points in space, this conversion becomes unstable (similar to a divide-by-zero error), and the robot needs to be reset.

For repetitive robot motions, singularities can be avoided by tedious manual fine-tuning of repetitive robot motions to adjust them such that they never encounter singularities. Once such motions are determined, they are repeated over and over again. But for the growing generation of applications where robot motions are not repetitive, including palletizing, bin-picking, order fulfillment and package sorting, singularities are common. They are a well-known and fundamental problem as they disrupt robot operations at unpredictable times (often several times per hour). I co-founded a new startup, Jacobi Robotics, that implements efficient algorithms that are *guaranteed* to avoid singularities. This can significantly increase reliability and productivity for all robots.

Russ Tedrake, TRI: We hear a lot these days about generative AI and about the incredible progress and investments in hardware. Many of these successes, though, were made possible by the quiet revolution that we’ve seen in simulation. Only a few years ago, most roboticists would have said it was impossible to train or test a computer vision system in simulation; now it’s standard practice. Some researchers are still skeptical that we could develop a control system for, say, a dexterous hand completely in simulation and have it work in reality, but the trend is increasingly moving this way. Big investments from companies like Nvidia, Google DeepMind and TRI are helping to make this happen.

Dhruv Batra, Meta: That we can now test navigation robots in real homes and things actually work! Notice, unlike self-driving cars, these in-home navigation robots do not have the luxury of building precise maps over millions of miles of driving. We take a robot to a new house and ask it to find objects.

Deepu Talla, NVIDIA: The need for a platform approach. Many robotics startups are unable to scale as they are building robots that work well for a specific task or environment. For commercial viability at scale, it’s important to develop robots that are more generalizable — that is, they can add new skills rapidly or bring the existing skills to new environments.

Roboticists need platforms with the tools and libraries to train and test AI for robotics. The platform should provide simulation capabilities to train models, generate synthetic data and exercise the entire robotics software stack, with the ability to run the latest and emerging generative AI models right on the robot.

Tomorrow’s successful startups and robotics companies should focus on developing new robot skills and automation tasks and leverage the full extent of available end-to-end development platforms.

Matthew Johnson-Roberson, CMU: Despite significant advancements in certain niche areas and successful robotic implementations in specific industries, these stories often get overshadowed by the allure of more futuristic or general-purpose robotic narratives. The incremental but impactful successes in sectors like agriculture, healthcare or specialized industrial applications deserve more spotlight as they represent the real, tangible progress in the field of robotics.