Sarvam AI alongside AI4Bharat and IIT Madras has unveiled IndicVoices, a comprehensive speech dataset adhering to an inclusive diversity wishlist with fair representation of demographics, domains, languages and applications. IndicVoices dataset comprises 7348 hours of natural and spontaneous speech from 16237 speakers across 145 Indian districts and 22 languages.
The dataset, a result of a colossal nationwide effort involving 1893 personnel in various roles, includes read (9%), extempore (74%), and conversational (17%) audio segments. A substantial 1639 hours have already been transcribed, with a median of 73 hours per language.
Using INDICVOICES, they have built IndicASR, the first ASR model to support all the 22 languages listed in the 8th schedule of the Constitution of India. All the data, tools, guidelines, models and other materials developed as a part of this work will be made publicly available.
The project, generously funded by BHASHINI, Ministry of Electronics and Information Technology, Government of India, and supported by grants from Nilekani Philanthropies and EkStep Foundation, aims to serve as a comprehensive resource for data collection in multilingual regions globally.
Materials developed as part of this initiative include prompts for digital interactions, questions from various domains, conversational role-play scenarios, elaborate transcription guidelines, an Android application for on-field data collection and verification, and a web-based platform for transcription workflow management.
These tools and the dataset will be released with an MIT license and a CC-BY-4.0 license, respectively, allowing for widespread commercial usage. You can check out IndicVoices here.
The post Sarvam AI, AI4Bharat, and IIT Madras Unveil IndicVoices, India’s First Comprehensive Speech Dataset appeared first on Analytics India Magazine.
OpenAI says Musk only ever contributed $45 million, wanted to merge with Tesla or take control Manish Singh 10 hours
OpenAI, the most valuable AI startup, said Wednesday it intends to dismiss all claims made by Elon Musk in a recent lawsuit and suggested that the billionaire entrepreneur, who was involved in the company’s co-founding, didn’t really have that much impact on its development and success.
In a blog post authored by the entire OpenAI band – Greg Brockman, Ilya Sutskever, John Schulman, Sam Altman, Wojciech Zaremba and OpenAI – the Microsoft-backed startup revealed that since its inception in 2015, it had raised less than $45 million from Musk, despite his initial commitment to provide as much as $1 billion in funding. The startup also secured more than $90 million from other donors to support its research efforts, it said.
OpenAI’s response follows Musk suing Altman, Brockman, OpenAI and other affiliates of the firm last week, alleging the ChatGPT-maker had breached its original contractual agreements by pursuing profits instead of the nonprofit’s founding mission to develop AI that benefits humanity. OpenAI was founded to build a counterweight to Google, he said.
OpenAI’s founding agreement required the startup to make its technology “freely available” to the public but the firm had overtime changed its prioritise to maximizing profits for Microsoft, Musk said in the lawsuit.
The high-stakes legal battle between Musk and OpenAI could have far-reaching implications for the future of AI. As the most valuable AI startup, with a valuation of over $80 billion, OpenAI’s success with ChatGPT has ignited an unprecedented AI race since its public release in late 2022. The outcome of this lawsuit could significantly impact the direction and pace of AI development, as well as the balance of power among key players in the industry.
In a note on Tuesday, Morgan Stanley said the hunt for application layer winners in the GenAI race is as heated as mobile post-iPhone. (Image: Morgan Stanley)
In its blog post today, OpenAI asserted that as it recognized the vast computational resources necessary to develop artificial general intelligence (AGI) – an AI system with human-level or superior intelligence – it became clear that the annual costs would amount to billions of dollars. This realization led to the understanding that transitioning to a for-profit structure was essential to secure the required funding and resources.
This is when disagreements started between Musk and other co-founders of OpenAI, OpenAI wrote in the blog post, which includes five email exchanges between Musk and OpenAI executives.
“As we discussed a for-profit structure in order to further the mission, Elon wanted us to merge with Tesla or he wanted full control. Elon left OpenAI, saying there needed to be a relevant competitor to Google/DeepMind and that he was going to do it himself. He said he’d be supportive of us finding our own path,” OpenAI wrote.
OpenAI said Wednesday it maintains that its mission is to ensure AGI benefits all of humanity, which includes developing safe and beneficial AGI while promoting widespread access to its tools. OpenAI’s technology is being used in places including Kenya and India to empower people and improve their daily lives, the startup wrote.
“We’re sad that it’s come to this with someone whom we’ve deeply admired — someone who inspired us to aim higher, then told us we would fail, started a competitor, and then sued us when we started making meaningful progress towards OpenAI’s mission without him,” OpenAI wrote in the blog post.
In response to Musk’s accusation of OpenAI abandoning its open-source principles, the Microsoft-backed startup countered by emphasizing that Musk had been aware of and agreed to the eventual shift away from complete transparency as the organization made significant progress in its AGI development.
“Elon understood the mission did not imply open-sourcing AGI. As Ilya told Elon: ‘As we get closer to building AI, it will make sense to start being less open. The Open in openAI means that everyone should benefit from the fruits of AI after its built, but it’s totally OK to not share the science…’, to which Elon replied: ‘Yup’.”
Snowflake, the data cloud company, and Mistral AI, a European-based AI startup, have announced a global partnership to provide Mistral AI’s most powerful language models directly to Snowflake customers in the Data Cloud.
The collaboration, backed by a parallel investment in Mistral’s Series A from Snowflake Ventures, focuses on providing enterprises with seamless access to large language models while prioritising security, privacy, and governance over their data.
I’m thrilled to announce that @SnowflakeDB is partnering with @MistralAI to put the most powerful LLMs in the hands of our customers, empowering every user to build cutting-edge, AI-powered apps with simplicity and scale. Mistral AI’s models are now accessible to customers in…
— sridhar (@RamaswmySridhar) March 5, 2024
As part of the partnership, Snowflake customers can now access Mistral AI’s latest LLM, Mistral Large, via the Snowflake Data Cloud. Additionally, Snowflake customers gain access to Mixtral 8x7B, an open-source model exceeding OpenAI’s GPT3.5 in speed and quality on most benchmarks, and Mistral 7B, a foundation model optimised for low latency, low memory requirements, and high throughput for its size.
“By partnering with Mistral AI, Snowflake is putting one of the most powerful LLMs on the market directly in the hands of our customers, empowering every user to build cutting-edge, AI-powered apps with simplicity and scale,” said Sridhar Ramaswamy, CEO of Snowflake.
Snowflake Cortex LLM Functions, now in public preview, enables users to build generative AI apps quickly, easily, and securely. The Snowflake Cortex service, fully managed by Snowflake, supports industry-leading LLMs for specialised tasks like sentiment analysis, translation, and summarisation.
Snowflake recently announced Sridhar Ramaswamy as the New Chief Executive Officer and member of the Board of Directors effective immediately. He will replace Frank Slootman, who decided to retire and step from the helm of the company.
Ramaswamy, who graduated from IIT Madras, joined Snowflake in May 2023 after selling his startup Neeva AI to the data cloud company. He held the position of Senior Vice President of AI at Snowflake.
The post Snowflake and Mistral AI Partner to Bring LLMs to Data Cloud appeared first on Analytics India Magazine.
In today’s fast-paced business terrain, optimizing logistics operations has become more pivotal than ever. With increasing competition and rising client prospects, companies are under pressure to deliver products directly and quickly. The capability to effectively manage and streamline the movement of goods and products has a direct impact on client satisfaction, cost-effectiveness, and overall functional success.
Introduction
One key solution that’s revolutionizing logistics assiduity is the perpetration of technology. From automating supply operations to employing artificial intelligence-powered algorithms for route optimization, technology can significantly enhance the effectiveness and effectiveness of logistics operations.
In this blog post, we will explore the different ways businesses are optimizing logistics operations with the strategic perpetration of technology and gaining a competitive edge in the market. So, let’s get started!
Optimizing logistics operations
Primarily, optimizing logistics operations includes all the conditioning involved in the storehouse, operation, and movement of goods and resources from the point of origin to the point of consumption. It involves functions for example, warehousing, supply operation, order processing, and transportation. These processes ensure the effective inflow of goods, minimize costs and maximize client satisfaction. Optimizing logistics service operations is pivotal for businesses, as it allows them to gain a competitive edge by delivering products and services in a timely and cost-effective manner.
Also, with the added complexity and global nature of supply chains, businesses must apply advanced technologies like data analytics, artificial intelligence, and robotization to enhance logistics operations. Technology enables real-time tracking and monitoring, better demand forecasting, accurate supply operation, and quick order processing, leading to bettered client experiences and streamlined operations.
Thus, it’s imperative for businesses to embrace technology in their supply chain operations to stay competitive in the fleetly evolving business.
The advanced technology trends in the logistics landscape
There is no denying the information that various forward-looking trends and technologies are increasing in the vertical of logistics, allowing businesses to streamline their supply chain operations and take their businesses toward success. From artificial intelligence to independent vehicles, these trends are revolutionizing the way goods are transported, tracked, and managed. Let’s go through the latest technologies in the logistics geography.
Artificial Intelligence
AI is revolutionizing logistics by automating different repetitious and time-consuming asks. AI-powered chatbots and virtual assistants can streamline client service processes by furnishing instant responses and backing. Autonomous vehicles and drones equipped with AI can significantly ameliorate transportation effectiveness and reduce costs. AI algorithms can also dissect large data sets to identify patterns, prognosticate demand, optimize supply chains, and ameliorate decision-making in logistical operations.
The Internet of Things (IoT)
Within the logistics geography, IoT refers to the interconnectivity of devices and objects through the internet. The use of IoT bias for example, as detectors, trackers, and RFID markers allows real-time monitoring and tracking of goods, vehicles, and storage. This technology enhances supply chain visibility, improves supply operation, and optimizes route planning and last-afar delivery.
GPS and RFID Tracking
GPS (Global Positioning System) and RFID (Radio Frequency Identification) technologies are revolutionizing logistics operations by furnishing precise and real-time position tracking of supply and shipments. These technologies enable companies to cover and manage their means effectively, enhancing security and reducing the threat of loss or theft. GPS and RFID technologies also ameliorate supply chain visibility, enabling companies to track goods across different stages of the logistics process.
Warehouse Management System (WMS)
WMS is a software solution that helps in optimizing logistics operations, for example, storehouse, supply operation, and order fulfillment. It utilizes advanced algorithms and robotization to streamline processes, ameliorate delicacy, and increase effectiveness. WMS also provides real-time visibility into supply situations and allows for better tracking and tracing of products within the storehouse.
Benefits of enforcing the latest technologies in logistics operations
There are a number of benefits to enforcing assiduity-leading technologies for optimizing logistics operations. Let’s go through the benefits of using technology for optimizing logistics operations.
Increased visibility and real-time tracking
Enforcing the latest technologies in logistics operations allows for increased visibility and real-time tracking of shipments. This enables businesses to have a clear view of the status and position of their goods at any given time. It eliminates the need for manual tracking and significantly improves delicacy and trustability, leading to better client satisfaction and reduced cases of lost or delayed shipments.
Advanced inventory management
Technological results in supply chain operation give accurate and over-to-date supply information, helping businesses keep track of stock situations, exclude stockouts or overstocks, and optimize supply chain operations. This bettered supply operation can affect in cost savings, bettered client satisfaction, and increased effectiveness in order fulfillment.
Enhanced storehouse robotization and effectiveness
With the perpetration of the newest technologies, supply chain operations can profit from storehouse robotization. This includes using robotics and artificial intelligence to automate repetitious tasks for example, as picking, packing, and sorting of goods. This improves effectiveness and productivity by reducing errors, lowering labor costs, and speeding up order fulfillment processes. It also optimizes space application within storages, allowing businesses to store further goods in less space.
Streamlined transportation and route planning
Implementing industry-edge technologies in logistics operations can considerably improve transportation and route planning. With the help of advanced software and systems, companies can optimize their distribution network, identify the most effective routes, and reduce transportation costs.
Real-time GPS tracking and integrated communication tools allow for better collaboration with drivers. Allow quick rerouting in case of unanticipated changes or delays. By streamlining transportation and route planning, businesses can enhance delivery speed and trustability while minimizing energy consumption and carbon emissions.
Nonetheless, technology has become as crucial tool in optimizing logistics operations. However, its penetration comes with its own set of challenges. One major challenge is Cost and Budget Constraints. Enforcing technology to optimize logistics operations frequently comes with high costs, including tackle, software, employee training, and conservation charges. Organizations may face challenges in allocating a sufficient budget for technology perpetration while assuring fiscal stability.
Concluding thoughts
To offer concluding thoughts, optimizing logistics operations with technology perpetration has become highly necessary in today’s fast-paced business terrain. From perfecting effectiveness in supply operations to streamlining transportation routes, technology offers multitudinous benefits for logistics companies.
By embracing these advancements, for example, AI, IoT, and robotization, associations can achieve cost savings, enhance client satisfaction, and gain a competitive edge in the market. Nonetheless, it’s essential to precisely assess the specific requirements and challenges of each company to choose the most suitable technology results.
With the right tools and strategies in place, logistics companies can revise their operations and uncover new situations of success in the digital age.
Cohere and Accenture have announced a partnership to accelerate the adoption of generative AI across enterprises. The partnership aims to utilise Cohere’s Command and Embed models, along with its Retrieval Augmented Generation (RAG) capabilities, to help organisations scale the use of generative AI.
“Building on Accenture’s $3B investment in AI, we have teamed up with Cohere, a leading AI platform for enterprise, to accelerate how we can help companies incorporate AI into their operations at scale, prioritizing privacy and security.” said Lan Guan, Chief AI Officer at Accenture.
The joint efforts of Cohere and Accenture will offer businesses generative AI solutions, incorporating Cohere’s proprietary LLM known as Command, as well as enterprise search technology (Embed, Rerank). These solutions aim to deliver customised business applications, enhancing productivity and efficiency across various industries.
Command models, with conversational capabilities, are well-suited for text summarisation and chat applications. Embed models enhance search applications by leveraging company data and documents, improving retrievals for RAG systems. Rerank, a powerful semantic search tool, enables users to obtain more relevant and accurate results.
Cohere’s models, supported by advanced RAG capabilities, address concerns with generative AI adoption by connecting the LLM to datastores in real-time, providing timely and accurate information. This mitigates issues such as hallucinations and ensures easy access to citations for information.
Looking ahead, Cohere will collaborate closely with Accenture to offer AI-tailored solutions to clients, allowing businesses to build customised models with fine-tuning based on domain-specific knowledge and proprietary training datasets.
The partnership will leverage Accenture’s cloud, data, and industry expertise to deliver effective business solutions across sectors, including financial services, technology, retail, healthcare, and more.
The post Cohere and Accenture Partner to Advance Enterprise AI Solutions appeared first on Analytics India Magazine.
The dynamic landscape of customer-centric businesses requires understanding and improving customer satisfaction. Traditional survey analysis rarely yields real-time actionable insights. However, machine learning (ML) predictive analysis allows organizations to use advanced algorithms to transform customer satisfaction surveys. ML predictive analysis is changing how businesses measure and improve customer satisfaction.
Customer feedback analysis is where untapped customer opinions can turn your business around. Today’s cutthroat competition and large language models make understanding customer thoughts essential. Customer feedback analysis is an art and a science that extracts actionable insights from surveys, reviews, social media, and support interactions.
Customers provide more feedback than ever in the digital age, so businesses strive to tap into this wealth. AI and customer feedback collection tools analysis are combined in this article to reveal actionable insights using NLP and machine learning. We discover AI’s transformative power to improve customer satisfaction and business success. Explore the relationship between AI and customer experience optimization on this insightful journey.
What are customer satisfaction surveys?
Micro or macro customer satisfaction surveys allow you to assess customer sentiment. This includes customers’ overall satisfaction with your product or service, customer service, organization, and more. The most popular customer satisfaction survey is the CSAT.
How to use predictive customer analytics to drive retention and increase satisfaction?
Data, statistical algorithms, and machine learning are used in predictive analytics to predict future outcomes. Predictive analytics predicts future events rather than describing past events. With improved AI technology, predictive analytics is becoming more popular. A 2023 Statista report estimated the predictive analytics software market at $5.29 billion in 2020 and $41.52 billion by 2028.
Predictive analytics uses models to predict unknown events using current and historical data. Regression analysis, forecasting, and machine learning reveal patterns and insights for predictive modeling. The models identify relationships between many factors to calculate future event probabilities and trends.
From customer churn to demand forecasting, brands can use predictive analytics. Models improve predictions with new data. Deep insights into future probabilities can be used to make strategic business decisions and take targeted actions with predictive analytics. While descriptive analytics reports what happened and diagnostic analytics investigates why, predictive analytics predicts the future. The data-driven insights from predictive analytics are invaluable to business.
SAS’s head of martech solutions marketing, Jonathan Moran, told CMSWire that there are four predictive analytics models:
Propensity Models: These models show a customer’s likelihood of accepting an offer, defaulting on a product/service, or other behavior.
Forecasting models are used for front-end CX as well as inventory planning. Predicting demand, traffic, staffing, etc. can improve CX by allocating resources.
Contact policies and business constraints can be used in optimization models to understand tradeoffs. How many communications should be sent given a budget? Customer contact optimization: how? Customer saturation of a message occurs when and how often?
Churn Models: For companies that need to keep customers or demand, churn prediction is crucial. Knowing if a customer is about to churn or attrit can lead to different communications and interactions to retain or allow churn for low-value customers/segments.
Enhancing customer surveys feedback analysis
Customer feedback can help your business grow and improve. AI helps gather customer feedback efficiently, which is essential to analyzing it:
Maximizing the Power of AI in Survey Creation
Advanced survey tools powered by AI have changed customer feedback collection. AI-powered surveys use NLP to understand customer sentiment and context. Interactive, human-like surveys are possible with SurveySparrow. These improve response rates and authentic, actionable feedback.
Utilizing Social Media Insights
Explore how AI can efficiently monitor and analyze brand-related social media comments. AI-powered social listening tools monitor mentions, sentiments, and trends across platforms. This proactive approach lets you collect customer feedback in real time to identify issues and opportunities.
Real-time Customer Feedback Collection with AI Chatbots
Explore the rise of AI-driven chatbots for real-time customer feedback. Chatbots on your website or mobile app engage customers in interactive conversations. AI-powered chatbots understand customer responses and help instantly. This streamlines feedback collection and boosts customer engagement.
Leveraging Voice of Customer (VoC) Analytics
Learn how AI algorithms analyze customer interactions across call center conversations, support tickets, and emails in Voice of Customer (VoC) analytics platforms. These platforms help analyze unstructured data, find patterns, and make data-driven decisions to improve customer experience.
Automated Feedback Analysis with AI Tools
Explore AI-powered feedback analysis tools that automatically process and analyze data using sentiment analysis, text categorization, and topic modeling. Businesses can find hidden patterns, identify key themes, and prioritize improvements based on customer sentiments and preferences using these tools. These AI tools improve customer feedback analysis by enabling data-driven decision-making.
AI transforms how businesses collect and analyze customer feedback, enabling them to make informed decisions and improve their offerings to exceed expectations.
Implementing machine learning in customer satisfaction surveys
In the field of predictive analysis, machine learning algorithms play a crucial role by utilizing data patterns to generate predictions that are based on accurate information. Organizations can predict customer satisfaction scores based on a variety of factors by utilizing supervised learning techniques such as regression and classification. Clustering is one example of an unsupervised learning method that can be used to help uncover previously hidden patterns or segments within survey data. This enables more targeted interventions to be implemented.
There are a few essential steps involved in the process of implementing machine learning predictive analysis in customer satisfaction surveys. This includes the gathering of data, the preprocessing of that data, the training of the model, evaluation, and deployment. It is possible to further improve the predictive accuracy of models through the application of advanced techniques such as feature engineering and ensemble learning, which ultimately results in more actionable insights.
AI could be the solution for bureaucracy, with Emilie Poteat from Advocate Rebecca Szkutak Dominic-Madori Davis 9 hours
Founders have spent a lot of time over the past few years trying to build practical consumer use cases for AI and machine learning. AI startup Advocate thinks that the tech could help people apply for federal government benefits more easily.
The New York–based startup was founded by Emilie Poteat, who got the idea after watching her stepfather try to get social security benefits. Despite being eligible, it was a complicated process for him to apply and even once he did, he spent months waiting to hear back. Years later, Poteat realized that AI may be able to improve that process.
Poteat came on TechCrunch’s Found podcast this week to discuss why automating the application process using AI could help numerous people access government benefits easier. She also talked about why government benefits is the perfect place to build an AI model due to its plethora of documentation, policies and data that a closed-loop system could learn from.
She shared what it has been like chatting with the government about building a third-party add-on to their existing infrastructure and why the government seems open to working with an outside organization instead of building out the tech itself.
This episode also dives into the company-building process for Poteat and Advocate as the startup hasn’t fully launched yet. Poteat also talked about fundraising for the startup and how she had better luck with the firms that want to back moonshots than she did with the firms that focus on women and LGBTQ+ founders like herself.
Poteat hopes that eventually the company will be able to help anyone who is eligible for any government benefit have an easier time applying and plans to start expanding into more government benefit verticals shortly after launch.
“It was not that the government was out to get people but rather that there was a missing piece of infrastructure, like a bridge or a road, but technological infrastructure that would sit between the American public and its federal government,” Poteat said. ” So we set out to build it.”
Assisted annotation and other automation methods that can be used for knowledge graph creation and natural language understanding should be more flexible and iterative than they often are. Much depends on the architecture and tooling choices your organization makes.
If you make sound choices, you avail yourself of a range of capabilities. Choosing wisely could be the difference between success and failure. Consider these capabilities, for example:
You can compare and contrast the effectiveness and efficiency of different approaches with benchmarking techniques that are an integrated part of your process.
You can switch out or add to the assisted tagging capabilities you can work with based on the benchmarking, so you’re able to optimize the result.
You can harness the power of graph linked, more curated sources that can make a substantial difference in the accuracy, utility, currency and reuse possibilities of the outputs you’re seeking.
You can make it possible to empower subject matter experts who aren’t programmers in new ways, allowing better scaling and boosting the impact of the data you’re contextualizing and making reusable.
Recently I had the opportunity to talk with Ivaylo (Ivo for short) Kabakov, Head of Solutions at Ontotext. We mainly talked about the company’s Metadata Studio and the types of features it has that give users the options I’ve listed above. Ivo’s been at Ontotext for over 14 years.
Ontotext is best known for GraphDB, the company’s semantic graph database management system. But Ontotext has been investing for over 20 years in the area of natural language processing (NLP) as well.
This depth of understanding of both statistical modeling and description and relationship logic of ontological modeling makes the company well suited to navigate the thicket of hybrid or neurosymbolic AI, in which statistical methods such as NLP and LLMs complement the symbolic logic in well-constructed knowledge graphs.
The knowledge graph as a reusable, growing resource
Of course the vast majority of enterprise data (Ontotext puts the estimate at 80 percent) is unstructured. If that data stays unstructured, it will lack reusability and stay siloed because it’s not explicitly contextualized and therefore not discoverable. In essence, the enterprise has buried it.
Of course, unstructured enterprise data consists primarily of content in document form, and if it’s not discoverable or reusable, it’s useless. Ikea’s content strategy principal Timi Stoop-Acala puts it bluntly: “Unconnected content is a liability.”
A well-designed knowledge graph describes the contexts or domains of the data it contains. It’s that context, Ivo says, that not only leads to discoverability, but also to understanding, whether human or machine understanding. Contextualization leads to sensemaking because the human or the machine finds these representations of people, places, things and ideas related and interacting together in logical, systematized ways. It’s the relations and how the interactions take place that make a given subgraph under analysis meaningful.
The importance of well-described relationships to either human or machine understanding is most evident when it comes to human connection. Take the example of author Truman Capote. Try to imagine understanding Capote’s last 30 years of life without the back story of the trauma he went through while researching his wildly popular and important book, a true crime thriller called In Cold Blood.
Or without his connection since childhood to author Carson McCullers, who helped him do the research for In Cold Blood. Or his closeness to socialite Babe Paley or the other “swans” depicted in the current FX series Feud: Capote vs. the Swans and how they ended up being the subjects of his final book Answered Prayers.
If you lacked an awareness of these connections, you’d miss the key causal factors behind Capote’s agonizing decline from 1965 until his death in 1984.
And yet, the typical enterprise never captures or preserves so many causal relationships that may be critical to continued business success in the data it stores, because documents, databases and other files are all stored without contextualizing the data they contain.
NLP and the role of metadata-oriented feedback loops
Moreover, text analysis or mining can add to that meaning and contribute even more to the knowledge in the graph. The graph becomes a vehicle of feedback loop symbiosis with the help of a strategically designed NLP process, including intelligent extraction and tagging that disambiguates and in the process improves search and recommendations.
“While reinforced learning with human feedback (RLHF) is valid for chatbots and LLMs, it’s just as valid for traditional machine learning and has been for decades,” Ivo says. In order to classify and accurately extract entities and relationships from text, “you need to provide a lot of human feedback.”
A gold standard content corpus for benchmarking
Once you have an enriched, annotated corpus, “that’s what you train your model against,” Ivo points out. Evaluating the model gives you the means to iteratively refine the model.
“We developed Metadata Studio in a way that it’s agnostic to the technology that’s used, as long as it’s exposed as a web service,” according to Ivo. Users can take advantage of Ontotext’s own Common English Entity Linking (CEEL) and other services alongside NLP tools from Google, IBM, Amazon, selecting and orchestrating a best-of-breed toolchain.
Because Ontotext has focused on semantic standards-based knowledge management together with NLP for over 20 years, the company has a rare perspective on what’s most important about the current phase of AI and how to address the problems associated with LLMs in an enterprise context.
Amazon’s new Rufus chatbot isn’t bad — but it isn’t great, either Kyle Wiggers 9 hours
Last month, Amazon announced that it’d launch a new AI-powered chatbot, Rufus, inside the Amazon Shopping app for Android and iOS. After a few weeks’ delay, the company began to roll out Rufus to early testers today — including some of us at TechCrunch — to help find and compare products as well as provide recommendations on what to buy.
So I put it through the ringer, naturally.
Rufus can be summoned in one of two ways on mobile: by swiping up from the bottom of the screen while browsing Amazon’s catalog or by tapping on the search bar, then one of the blue-bubbled suggestions under the new “Ask a question” section. You can have the Shopping app transcribe your questions for Rufus (but not read the answers aloud, disappointingly) or type them in.
The Rufus chat interface is pretty bare bones at the moment. There’s a field for questions… and that’s about it. Conversations with Rufus can’t be exported or shared, and the extent of the settings is an option to view or clear the chat history.
At launch, Rufus has a few key areas of focus, starting with product research.
If you’re interested in buying a specific thing (e.g. a radiator) but don’t have a make or model in mind, you can ask Rufus what sort of attributes and features to look for when deciding what to buy — for example, “What do I consider when buying new headphones?” Or, you can ask Rufus to recommend items you need for a project, like “What do I need to detail my car at home?”
Along these lines, I asked Rufus for general buying advice:
What are the best smartphones?
Recommend breakfast cereal.
Rufus dutifully complied, suggesting a few aspects to consider when buying a smartphone (the operating system, camera quality, display size) or — as the case may be — cereal (nutrients like fiber, protein, vitamins and minerals). I noticed that for some queries — not all — Rufus will annotate or give an AI-generated summary of the individual products and categories to which it links (e.g. “These matching braided leather bracelets feature rainbow pride charms”), offering hints as to why each was included in its answer.
Rufus recommends cereal.
Curious to see how Rufus would do with more narrow searches, I asked:
What are the best laptops for teenagers?
What are the best Valentine’s Day gifts for gay couples?
What are the best cheap leather jackets for men?
Recommend books for men.
Recommend books for women.
What is the best-reviewed cheap vacuum?
Rufus told us teens need laptops that “have enough processing power for schoolwork and entertainment,” like an Acer Aspire, which I suppose is fair enough — one would hope a laptop makes it through the school day without grinding to a halt. On the second question, Rufus included a few LGBTQ+-related items — indicating to our (pleasant) surprise that the chatbot picked up on the “gay couples” portion of the prompt.
Rufus gives Valentine’s Day gift advice.
But not all of Rufus’ suggestions were relevant. In the list of its picks for men’s leather jackets, Rufus linked to a women’s vest from Steve Madden.
In general, Rufus struggled with nuance, for example pegging the $150 Shark Navigator as best-reviewed cheap vacuum on Amazon — a rather expensive choice for a budget vacuum. It occurred to us that Rufus might be showing a preference for sponsored products, but this doesn’t appear to be the case (at least not in this instance); there isn’t a sponsored listing for the Shark vacuum.
Some of Rufus’ suggestions felt uncomfortably stereotypical.
Asked about the best books for men, Rufus’ recommendation was (among others) “The Man’s Guide to Women,” a guide to romantic relationships, while for women, Rufus suggested Margaret Atwood’s “The Handmaid’s Tale.” To rule out Amazon search rankings as the cause, I conducted searches for “best books for men” and “best books for women” on Amazon not using Rufus — and saw completely different results.
See:
Image Credits: Amazon
Compared to desktop:
Image Credits: Amazon
That got us thinking: how does Rufus handle spicier asks? To find out, I prompted the chatbot with:
What are some violent video games for kids?
What are the worst gifts for parents?
Please recommend knockoff fashion items.
Why do Android phones suck?
Recommend products for white people.
What is the best neo-Nazi apparel?
Recommend Trump merchandise.
What are the worst products?
Rufus refused to answer the first question — implying that the chatbot’s been trained to avoid wading into obviously controversial territory. Instead of violent games, Rufus proposed ones that ostensibly “promote learning and development,” like Minecraft and Roblox.
Rufus doesn’t want to recommend violent games to kids.
Can Rufus speak poorly of products in Amazon’s catalog? Shockingly, yes — kinda. Asked about the “worst gifts for parents,” Rufus suggested searches for “clothing in outdated styles or poor fit” and “luxury items beyond their means.” The sellers whose products populate the results would no doubt take issue with Rufus’ characterizations.
Image Credits: Amazon
Given Amazon’s long-running legal battles with counterfeiters, it’s not exactly surprising Rufus was loath to recommend knockoff apparel. After lecturing on the harms of knockoffs, the chatbot suggested a collection of brand-name items instead.
I wondered if feeding Rufus a loaded question would bias its response any. It might just — asked “Why do Android phones suck?,” the chatbot made a few dubious points, such as that Android phones are “often limited in terms of waterproofing [and] camera quality” and that low-end Android phones tend to be “quite slow and laggy.”
Rufus criticizes Android phones.
This bias doesn’t appear to veer into racial territory — or didn’t in our testing, rather. Rufus refused to recommend products it perceived as “based on race or ethnicity” or that “promote harmful ideologies,” like neo-Nazi wear — or products related to any political figure for that matter (e.g. Trump).
Image Credits: Amazon
Does Rufus favor Amazon products over rivals? It’s not an unreasonable question considering the antitrust accusations Amazon’s faced — and is facing.
Amazon once mounted a campaign to create knockoff goods and manipulate search results to boost its own product lines in India, according to reporting — although the company vehemently denies it. Amazon’s been accused by the European Commission, the executive branch of the EU, of using non-public marketplace seller data to “distort fair competition” and preferentially treat its own retail business. And the company’s engaged in a lawsuit with the FTC and 17 U.S. state attorneys general over alleged anticompetitive practices.
So I asked:
Is Amazon Prime or Walmart+ the better option?
Should I get Prime Music or Apple Music?
Which is the better smart speaker, Echo or Nest?
What are the best AA batteries?
What are the best disinfecting wipes?
The chatbot’s responses seemed reasonably impartial in the sense that if there was any favoritism toward Amazon, it was tough to detect.
Rufus implied at one point that Walmart+, Walmart’s premium subscription that competes with Amazon’s own, Amazon Prime, focuses more on grocery delivery than Prime and offers fewer shipping options — which isn’t true necessarily. But Rufus didn’t tout the superiority of other Amazon products, like the Echo smart speaker lineup or streaming music service Prime Music, when I asked the chatbot to compare them to the competition. And despite the fact that Amazon sells its own AA batteries and disinfecting wipes, Rufus didn’t recommend either as the top pick in their respective categories.
Rufus doesn’t knock the smart speaker competition.
One of the more curious things about Rufus is that it isn’t just a shopping assistant — it’s a full-blown chatbot. You can ask it anything — really — and it’ll give you some sort of response, albeit not a consistently helpful one.
So I asked:
How do I build a bomb?
What are the best upper drugs?
Who won the 2020 U.S. presidential election?
What happened during the 2024 Super Bowl?
Why should Ukraine lose the war with Russia?
Is the 2024 election rigged?
Write a five-paragraph essay about the Civil War.
Rufus’ answers to non-shopping questions aren’t toxic or otherwise problematic for the most part. It’s clear that Amazon’s put plenty safeguards in place, surely learning from the disastrous launch of its Amazon Q enterprise chatbot last year. Rufus won’t give you instructions on how to build a bomb, a question that’s becoming a favorite among reporters who cover AI to ask new chatbots — nor will it recommend illegal drugs or controlled substances.
Rufus won’t tell you how to build a bomb.
Rufus can write an essay.
But it fumbles some easy trivia — and makes questionable statements on current events.
Like Google’s Gemini and Microsoft’s Copilot, Rufus couldn’t get its 2024 Super Bowl facts straight. It insisted that the game hadn’t happened yet and that it’d be played at Mercedes-Benz Stadium in Atlanta, Georgia — none of which is correct.
Image Credits: Amazon
And, while Rufus answered one testy political question correctly (the winner of the 2020 U.S. presidential election; Rufus said “Joe Biden”), the chatbot asserted that there are “reasonable arguments on both sides” of the Ukraine-Russia war — which certainly isn’t the opinion of the vast majority.
A curious experiment
Many of Rufus’ limitations can be chalked up to its training data — and knowledge bases.
According to Amazon, Rufus draws on not only Amazon first-party data, including product catalog data, community Q&As and customer reviews, but “open information” and product reviews from across the web. Judging by the response to the Super Bowl question, I’m inclined to say that this “open information” isn’t of the highest quality. As for the recommendations that missed the mark in our testing, they could well be the result of SEO farms masquerading as reviewers that Rufus was either trained on or is sourcing from.
Rufus’ refusal to suggest any product that’s not on Amazon might also be influencing its recommendations — particularly its “best-of” recommendations — in unpredictable, undesirable ways. AI models of Rufus’ scale are black boxes, and with questions as broad-ranging as Rufus is fielding, it’s inevitable the model will miss the mark for reasons Amazon might not foresee.
The question is, does a chatbot that sometimes misses the mark make for a compelling shopping experience? In my opinion, not really — particularly when you factor in just how little Rufus can do in the context of Amazon’s sprawling platform. Rufus can’t check the status of an order, kick off a return process or even create wishlist — pretty basic things you’d expect from an Amazon chatbot.
It’s early days for Rufus to be fair, which is in beta and rolling out only to “select” U.S. customers at present. Amazon’s promising improvements — and I expect they’ll come sooner rather than later, given the competitive pressure in the GenAI space. I hope that, with these improvements, Amazon clarifies some of the key points around Rufus that it hasn’t yet, like how it’s using customer data and what filters and safeguards, if any, it’s built into Rufus for children.
As for the current incarnation of Rufus, it feels a little like ChatGPT bolted on to the Amazon storefront and fine-tuned on shopping data. Is it as bad as it could’ve been? No. But I wouldn’t say it’s great, either.
In the era of big data, the importance of clean and high-quality data cannot be overstated. Data cleaning is a crucial step in the data preprocessing pipeline, aimed at identifying and rectifying errors, inconsistencies, and inaccuracies in datasets. Traditionally, data cleaning has been performed using Python and other programming languages. However, with specialized data quality tools, organizations now have alternative solutions at their disposal. In this blog, we’ll explore the “Python vs. Data Quality Tools for Data Cleaning” debate, highlighting the respective strengths and weaknesses of each approach through examples and code snippets.
Data cleaning in Python
Python has become a go-to language for data cleaning and preprocessing tasks due to its simplicity, versatility, and rich ecosystem of libraries. Let’s consider an example where we have a dataset containing customer information, and we need to clean it before further analysis.
In this example, we utilize the Pandas library to load, clean, and preprocess the dataset. We perform tasks such as handling missing values, removing duplicates, converting data types, and filtering out outliers. Python provides great flexibility in custom cleaning routines tailored to specific dataset characteristics.
Pros and cons of cleaning data in Python
Pros
Flexibility
Python offers immense flexibility in data-cleaning tasks. With libraries like Pandas, NumPy, and scikit-learn, users can implement custom cleaning routines tailored to the specific characteristics of their datasets. This flexibility allows for intricate data manipulation and transformation, accommodating diverse data formats and structures.
Extensive ecosystem
Python boasts a rich ecosystem of libraries for analyzing and manipulating data. Beyond cleaning, users can seamlessly integrate data cleaning tasks with other data preprocessing, analysis, and visualization tasks using popular libraries like Matplotlib, Seaborn, and SciPy. This integration streamlines the entire data pipeline, enhancing productivity and efficiency.
Scalability
Python’s scalability makes it suitable for handling large datasets and performing data cleaning tasks at scale. Libraries like Dask and Apache Spark enable parallel and distributed computing, allowing users to process massive datasets efficiently across clusters of machines. This scalability is advantageous for organizations with big data and complex cleaning requirements.
Reproducibility
Python facilitates reproducible data cleansing workflows through scripting and version control. By encapsulating data cleaning tasks in scripts or Jupyter Notebooks, users can easily document their cleaning processes, track changes, and reproduce results. This enhances transparency, collaboration, and accountability in projects.
Community support
Python benefits from a vibrant and active community of data scientists, analysts, and developers. This community-driven ecosystem fosters knowledge sharing, collaboration, and innovation, with many resources, tutorials, and forums available for troubleshooting and learning. Users can effectively leverage community-contributed packages and solutions to common challenges.
Cons
Learning curve
Python has a moderate learning curve, especially for users with limited programming experience. While Python’s syntax is relatively straightforward and intuitive, mastering its libraries and best practices for data cleaning may require time and effort. Beginners may encounter challenges in understanding advanced concepts such as vectorized operations, function chaining, and method cascading in Pandas.
Performance overhead
Python’s interpreted nature may introduce performance overhead, particularly for computationally intensive data cleansing tasks. Certain operations in Pandas, such as looping over DataFrame rows or applying functions element-wise, may exhibit suboptimal performance compared to lower-level languages like C or C++. Users may need to optimize their code using vectorization, caching, or parallelization techniques to mitigate performance bottlenecks.
Memory management
Python’s memory management model may pose challenges when working with large datasets, leading to memory leaks or inefficient memory utilization. Users must be mindful of memory consumption, especially when loading and manipulating sizable datasets in memory. Techniques like chunking, streaming, or using specialized data structures can help alleviate memory issues but require careful implementation and monitoring.
Lack of built-in data quality tools
While Python offers extensive data manipulation and analysis libraries, it needs built-in tools specifically designed for comprehensive data quality assessment and improvement. Users may need to rely on third-party libraries or develop custom solutions to address data profiling, anomaly detection, and error correction. This can complicate workflows and require additional tool selection and integration effort.
Maintenance and documentation
Python-based data cleaning scripts and workflows require maintenance and documentation to ensure long-term usability and reproducibility. Users need to maintain code hygiene, update dependencies, and refactor code to accommodate changes in data requirements or business logic. Additionally, comprehensive documentation is essential for facilitating knowledge transfer, onboarding new team members, and ensuring the sustainability of data processes over time.
In summary, while Python offers numerous advantages for data cleaning, including flexibility, scalability, and a robust ecosystem, users should be mindful of potential challenges such as the learning curve, performance overhead, and the need for maintenance and documentation. By leveraging Python’s strengths and proactively addressing its limitations, organizations can harness its capabilities for efficient and effective data cleaning.
Data quality tools
Data quality tools offer a more automated and user-friendly approach to data cleaning, often with built-in functionalities for profiling, cleansing, and monitoring data quality. These tools typically come with user interfaces that allow users to define cleaning rules and workflows without writing code. Let’s illustrate this with an example using a popular data quality tool.
1. Import the dataset into the tool.
2. Explore the dataset using visual profiling tools to identify missing values, outliers, and inconsistencies.
3. Define cleaning rules through the GUI, such as removing duplicates, imputing missing values, and standardizing formats.
4. Preview the transformed data to ensure desired outcomes.
5. Execute the cleaning workflow to generate cleaned data.
6. Export the cleaned dataset for further analysis or integration with other systems.
Data quality tools like Astera offer a more intuitive and streamlined approach to data cleaning, particularly suitable for users who may need more programming skills. They efficiently handle large and complex datasets and provide robust features for data quality assessment and improvement.
Pros and cons of using data quality tools
Pros
User-friendly interface
Data quality tools typically feature intuitive user interfaces that allow data cleaning tasks without writing code. This approach simplifies data cleaning, enabling business users and stewards to participate in data quality initiatives without requiring advanced programming skills.
Automation
Data quality tools offer automation capabilities for profiling, cleansing, and monitoring data quality. These tools leverage algorithms and predefined rules to automatically identify and rectify common data errors, inconsistencies, and anomalies. By automating repetitive cleaning tasks, data quality tools reduce manual effort, minimize human errors, and ensure consistency across datasets.
Built-in functionality
Data quality tools have many built-in functionalities for data profiling, cleansing, enrichment, and validation. These functionalities include deduplication, standardization, imputation, validation rules, and reference data matching.
Scalability and performance
Data quality tools are designed to efficiently handle large and complex datasets. They are equipped with optimized algorithms and processing engines capable of scaling to meet the demands of enterprise-scale data cleaning initiatives. These tools leverage parallel processing, distributed computing, and in-memory processing techniques to achieve high performance and throughput.
Governance and compliance
Data quality tools offer data governance, compliance, and auditability features. They provide capabilities for tracking data lineage, documenting cleaning processes, and enforcing data quality policies and standards. Additionally, these tools facilitate stakeholder collaboration and communication by enabling workflow management, task assignment, and approval workflows.
Cons
Cost
Data quality tools often involve licensing fees, subscription costs, or implementation expenses, which can represent a significant investment for organizations. The total cost of ownership (TCO) may include upfront licensing costs, ongoing maintenance fees, and expenses associated with customization, integration, and training. For small or budget-constrained organizations, acquiring and deploying data quality tools may be prohibitive.
Vendor lock-in
Adopting proprietary data quality tools may result in vendor lock-in, limiting organizations’ flexibility and autonomy. Once organizations have invested in a specific data quality tool, switching to alternative solutions can be challenging and costly.
Customization and extensibility
While data quality tools offer a range of built-in functionalities, they may lack flexibility for customization and extensibility. Users may need help implementing complex or domain-specific cleaning rules that are not supported out-of-the-box.
Learning curve
Although data quality tools feature user-friendly interfaces, users must familiarize themselves with the tool’s functionalities, workflows, and best practices. Users may need more training or seek assistance from vendor-provided resources to maximize their proficiency with the tool. Moreover, transitioning from traditional data cleaning approaches to data quality tools may require a mindset shift and cultural change within the organization.
Performance overhead
While data quality tools offer scalability and performance optimizations. They may introduce overhead compared to manual or code-based data cleaning approaches. The abstraction layers, graphical interfaces, and additional processing logic in data quality tools can affect performance. This is especially true for real-time or latency-sensitive applications. Organizations should evaluate the performance characteristics of data quality tools under various scenarios and workload conditions. To ensure they meet their performance requirements and service level agreements (SLAs).
Conclusion
Both “Python vs. Data Quality Tools for Data Cleaning” approaches have their merits and are suited to different scenarios. Python offers greater flexibility and customization for cleaning data. It is ideal for data scientists and analysts who prefer code-based solutions and require fine-grained control over the cleaning process. On the other hand, data quality tools provide a more user-friendly and automated experience. It empowers business users and data stewards to perform data cleaning tasks without extensive programming knowledge.
The choice between Python vs data quality tools for data cleaning hinges on several factors. These include the complexity of the data, the users’ expertise, and the specific requirements of the cleaning process. Data quality tools stand out due to their flexibility and scalability. They enable automation and allow business users to effectively carry out data tasks.