Рубрика: AI

Image by Author

Large Language Models (LLMs) like GPT-4 and LLaMA2 have entered the [data labeling] chat. LLMs have come a long way and can now label data and take on tasks historically conducted by humans. While obtaining data labels with an LLM is incredibly quick and relatively cheap, there’s still one big issue, these models are the ultimate black boxes. So the burning question is: how much trust should we put in the labels these LLMs generate? In today’s post, we break down this conundrum to establish some fundamental guidelines for gauging the confidence we can have in LLM-labeled data.

Background

The results presented below are from an experiment conducted by Toloka using popular models and a dataset in Turkish. This is not a scientific report but rather a short overview of possible approaches to the problem and some suggestions for how to determine which method works best for your application.

The Big Question

Before we get into the details, here’s the big question: When can we trust a label generated by an LLM, and when should we be skeptical? Knowing this can help us in automated data labeling and can also be useful in other applied tasks like customer support, content generation, and more.

The Current State of Affairs

So, how are people tackling this issue now? Some directly ask the model to spit out a confidence score, some look at the consistency of the model’s answers over multiple runs, while others examine the model’s log probabilities. But are any of these approaches reliable? Let’s find out.

The Rule of Thumb

What makes a “good” confidence measure? One simple rule to follow is that there should be a positive correlation between the confidence score and the accuracy of the label. In other words, a higher confidence score should mean a higher likelihood of being correct. You can visualize this relationship using a calibration plot, where the X and Y axes represent confidence and accuracy, respectively.

Experiments and Their Results

Approach 1: Self-Confidence

The self-confidence approach involves asking the model about its confidence directly. And guess what? The results weren’t half bad! While the LLMs we tested struggled with the non-English dataset, the correlation between self-reported confidence and actual accuracy was pretty solid, meaning models are well aware of their limitations. We got similar results for GPT-3.5 and GPT-4 here.

Approach 2: Consistency

Set a high temperature (~0.7–1.0), label the same item several times, and analyze the consistency of the answers, for more details, see this paper. We tried this with GPT-3.5 and it was, to put it lightly, a dumpster fire. We prompted the model to answer the same question multiple times and the results were consistently erratic. This approach is as reliable as asking a Magic 8-Ball for life advice and should not be trusted.

Approach 3: Log Probabilities

Log probabilities offered a pleasant surprise. Davinci-003 returns logprobs of the tokens in the completion mode. Examining this output, we got a surprisingly decent confidence score that correlated well with accuracy. This method offers a promising approach to determining a reliable confidence score.

The Takeaway

So, what did we learn? Here it is, no sugar-coating:

1. Self-Confidence: Useful, but handle with care. Biases are reported widely.
2. Consistency: Just don’t. Unless you enjoy chaos.
3. Log Probabilities: A surprisingly good bet for now if the model allows you to access them.

The exciting part? Log probabilities appear to be quite robust even without fine-tuning the model, despite this paper reporting this method to be overconfident. There is room for further exploration.

Future Avenues

A logical next step could be to find a golden formula that combines the best parts of each of these three approaches, or explores new ones. So, if you’re up for a challenge, this could be your next weekend project!

Wrapping Up

Alright, ML aficionados and newbies, that’s a wrap. Remember, whether you’re working on data labeling or building the next big conversational agent — understanding model confidence is key. Don’t take those confidence scores at face value and make sure you do your homework!

Hope you found this insightful. Until next time, keep crunching those numbers and questioning those models.
Ivan Yamshchikov is a professor of Semantic Data Processing and Cognitive Computing at the Center for AI and Robotics, Technical University of Applied Sciences Würzburg-Schweinfurt. He also leads the Data Advocates team at Toloka AI. His research interests include computational creativity, semantic data processing and generative models.

More On This Topic

• Deep learning doesn’t need to be a black box
• If You Can Write Functions, You Can Use Dask
• Sky's the Limit: Learn how JetBlue uses Monte Carlo and Snowflake to build…
• Confidence Intervals for XGBoost
• How to calculate confidence intervals for performance metrics in Machine…
• Working with Confidence Intervals

Image created by Author with Midjourney
Key Takeaways

• Skeleton-of-Thought (SoT) is an innovative prompt engineering technique that minimizes generation latency in Large Language Models (LLMs), enhancing their efficiency
• By creating a skeleton of the answer and then parallelly elaborating on each point, SoT emulates human thinking, promoting more reliable and on-target AI responses
• Implementing SoT in projects can significantly expedite problem-solving and answer generation, especially in scenarios demanding structured and efficient output from AI

SoT is an initial attempt at data-centric optimization for efficiency, and reveal the potential of pushing LLMs to think more like a human for answer quality.

Introduction

Prompt engineering is ground zero in the battle for leveraging the potential of generative AI. By devising effective prompts, and prompt-writing methodologies, we can guide AI in understanding the user's intentions and addressing these intentions effectively. One notable technique in this realm is the Chain-of-Thought (CoT) method, which instructs the generative AI model to elucidate its logic step-by-step while approaching a task or responding to a query. Building upon CoT, a new and promising technique called Skeleton-of-Thought (SoT) has emerged, which aims to refine the way AI processes and outputs information, in the hopes of consequently promoting more reliable and on-target responses.

Understanding Skeleton-of-Thought

The genesis of Skeleton-of-Thought arises from the endeavor to minimize the generation latency inherent in large language models (LLMs). Unlike the sequential decoding approach, SoT emulates human thinking by first generating an answer's skeleton, then filling in the details in parallel, speeding up the inference process significantly​. When compared to CoT, SoT not only encourages a structured response but also efficiently organizes the generation process for enhanced performance in generative text systems.

Figure 1: The Skeleton-of-Thought process (from Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding) Implementing Skeleton-of-Thought

As mentioned above, implementing SoT entails prompting the LLM to create a skeleton of the problem-solving or answer-generating process, followed by parallel elaboration on each point. This method can be particularly useful in scenarios requiring efficient and structured output from AI. For instance, when processing large datasets or answering complex queries, SoT can significantly expedite the response time, providing a streamlined workflow. By integrating SoT into existing prompt engineering strategies, prompt engineers can harness the potential of generative text more effectively, reliably, and quickly.

Perhaps the best way to demonstrate SoT is by example prompts.

Example 1

• Question: Describe the process of photosynthesis.
• Skeleton: Photosynthesis occurs in plants, involves converting light energy to chemical energy, creating glucose and oxygen.
• Point-expansion: Elaborate on light absorption, chlorophyll's role, the Calvin cycle, and oxygen release.

Example 2

• Question: Explain the causes of the Great Depression.
• Skeleton: The Great Depression was caused by stock market crash, bank failures, and reduced consumer spending.
• Point-expansion: Delve into Black Tuesday, the banking crisis of 1933, and the impact of reduced purchasing power.

These examples demonstrate how SoT prompts facilitate a structured, step-by-step approach to answering complex questions. It also shows the workflow: pose a question or define a goal, give the LLM a broad or inclusive answer from which to elaborate supportive reasoning backward from, and then explicitly present those supportive reasoning issues and ask specifically prompt it to do so.

Figure 2: The Skeleton-of-Though simplified process (Image by Author)

While SoT offers a structured approach to problem-solving, it may not be suitable for all scenarios. Identifying the right use cases and understanding its implementation are important. Moreover, the transition from sequential to parallel processing might require a shift in system design or additional resources. However, overcoming these hurdles can unveil the potential of SoT in enhancing the efficiency and reliability of generative text tasks.

Conclusion

The SoT technique, building on the CoT method, offers a new approach in prompt engineering. It not only expedites the generation process but also fosters a structured and reliable output. By exploring and integrating SoT in projects, practitioners can significantly enhance the performance and usability of generative text, driving towards more efficient and insightful solutions.

Matthew Mayo (@mattmayo13) holds a Master's degree in computer science and a graduate diploma in data mining. As Editor-in-Chief of KDnuggets, Matthew aims to make complex data science concepts accessible. His professional interests include natural language processing, machine learning algorithms, and exploring emerging AI. He is driven by a mission to democratize knowledge in the data science community. Matthew has been coding since he was 6 years old.

More On This Topic

• Automating the Chain of Thought: How AI Can Prompt Itself to Reason
• Mastering Generative AI and Prompt Engineering: A Free eBook
• The Art of Prompt Engineering: Decoding ChatGPT
• Parallel Processing Large File in Python
• KDnuggets News, July 20: Machine Learning Algorithms Explained in Less Than…
• Date Processing and Feature Engineering in Python

Java is one of the earliest open-source languages, increasingly used in the domains of AI and machine learning. The Java Machine Learning Library (JavaML), has an array of machine learning algorithms coded in Java.

The language’s role in artificial intelligence involves leveraging a comprehensive set of libraries within Java programming. This aids in Java AI programming and proves advantageous for data scientists. Within the AI domain, Java finds application in machine learning, neural networks, search algorithms, and genetic programming.

Here, we have included six Java libraries that will help developers in their AI projects.

Apache Jena

Apache Jena is a framework for Java developers who work with the Semantic Web. It helps them pull information from and write it to special graphs called RDF graphs. These graphs are organised collections of data. Users can fill these graphs with information from files, databases, web addresses, or a mix of these sources. And if users want to ask questions or do searches on this data, Jena allows that using a language called SPARQL.

Comparing it to another similar tool called RDF4J, Jena is a bit more versatile because it also supports OWL, which is another way to structure data on the web. It even has some built-in tools to help figure things out based on the data you have, which can be really handy.

Lastly, Jena lets you convert these special graphs into different formats, like saving them in a database, a specific file type, or even a special way that’s easy to use with Java.

Power Loom

PowerLoom is a knowledge representation and reasoning system, distinct from traditional machine learning. It’s designed for artificial intelligence (AI) and uses logical rules to represent and reason about complex knowledge. In contrast, machine learning involves training models on data to make predictions or take actions. While not directly related to machine learning, PowerLoom can integrate with AI systems, potentially benefiting from patterns learned by machine learning models for more informed reasoning and decision-making.

Deeplearning4j

Deeplearning4j is a popular distributed deep learning framework designed for Java and Scala, offering an efficient solution for building and deploying deep learning models on the JVM. Recent advancements like SameDiff, an automatic differentiation framework, ONNX model support, performance optimizations, and the inclusion of new features for recurrent and convolutional neural networks, along with natural language processing, have further strengthened its capabilities. It stands as a versatile choice for developers looking to create scalable and reliable deep learning applications, spanning various domains including image classification, language processing, machine translation, and recommendation systems.

Deeplearning4j’s recent developments underscore its commitment to staying at the forefront of deep learning technologies. By focusing on efficiency, compatibility, and performance, it continues to be a go-to choice for developers seeking to leverage the power of deep learning within Java and Scala environments, empowering the creation of robust and impactful AI applications.

Apache OpenNLP

Apache OpenNLP is an open-source Java library specialising in natural language processing (NLP) tasks. It excels at breaking down text into meaningful components, such as tokens, sentences, and identifying parts of speech. Additionally, it can recognize named entities, syntactic chunks, and even resolve coreference. Major tech players like Google, Amazon, Twitter, and Facebook leverage OpenNLP to enhance their services. For instance, it helps Google grasp the intent behind search queries, enables Amazon to recommend products based on past purchases, aids Twitter in identifying spam, and assists Facebook in understanding user posts for personalised news feeds.

Its open-source nature, user-friendliness, and comprehensive NLP capabilities make Apache OpenNLP a popular choice for diverse organisations and research in the field of language processing. Apache OpenNLP, a robust Java library, stands as a key player in the realm of natural language processing (NLP).

RapidMiner

RapidMiner is a user-friendly platform that simplifies data analysis and machine learning. Users can create machine learning models and work with data without needing to write complex code – it’s all about dragging and dropping elements to build workflows.

Started in 2001 by students from the University of Dortmund, RapidMiner has become a go-to tool for big companies, government agencies, universities, and more. It finds applications across various sectors, aiding tasks such as data cleaning, model creation, performance evaluation, and visual reporting. With its easy usability, powerful features, and cost-effectiveness, RapidMiner is a popular choice for data science tasks across different-sized organisations and industries.

The primary application, RapidMiner Studio, is built using Java, enabling the creation, editing, and extension of RapidMiner workflows using Java code. Moreover, RapidMiner offers a Java API, allowing developers to integrate RapidMiner’s data science and machine learning capabilities seamlessly within Java applications. This versatility makes RapidMiner a valuable tool for Java developers aiming to construct and deploy data science and machine learning solutions.

Jenetics

Jenetics, a Java-based library, specialises in genetic algorithms, streamlining their implementation. Offering essential tools and adaptability, it caters to a wide range of problem-solving needs, from optimization challenges to machine learning tasks and even creative content generation.

Applied in various fields, Jenetics aids in innovative machine learning algorithms, product design optimisation, and creative content creation such as music and artwork. Its allure lies in its versatility, user-friendliness, open-source nature, and strong community support. For those seeking a potent Java library for genetic algorithms, Jenetics emerges as an excellent choice, providing a powerful and adaptable toolset with ample support from a vibrant user community.

The post 6 Libraries in Java for Machine Learning appeared first on Analytics India Magazine.

In a groundbreaking development, a team of researchers at the University of Washington has introduced an advanced sound control system that promises to redefine in-room audio dynamics. The unique technology, akin to a swarm of robots, uses self-deploying microphones to segregate rooms into distinct speech zones.

This trailblazing technology creates a network of small robotic entities that disperse themselves across various surfaces, emitting high-frequency sounds akin to bat navigation to avoid obstacles and distribute themselves for optimal sound control and voice isolation. This system, through sophisticated deployment, surpasses the limitations of existing consumer smart speakers and allows for enhanced differentiation and localization of simultaneous conversations.

Malek Itani, a UW doctoral student and co-lead author of the study, emphasized the unprecedented capabilities of this acoustic swarm, stating, “For the first time, using what we're calling a robotic ‘acoustic swarm,' we're able to track the positions of multiple people talking in a room and separate their speech.”

Addressing Real-world Challenges

While current virtual meeting tools allow for control over who gets to speak, managing in-room conversations in real-world settings, especially in crowded environments, presents numerous challenges. This innovative technology manages to isolate specific voices and separate simultaneous discussions, even amongst individuals with similar voice tones, without the need for visual cues or cameras. This marks a considerable stride in managing audio in spaces like living rooms, kitchens, and offices, where discerning multiple voices is pivotal.

The system demonstrated impeccable efficacy, discerning different voices within 1.6 feet of each other 90% of the time in varied environments. “We developed neural networks that use these time-delayed signals to separate what each person is saying and track their positions in a space,” noted co-lead author Tuochao Chen. He further clarified that this allows for the isolation and location of each voice in a room where multiple conversations are occurring simultaneously.

Shape-changing smart speakers create speech zones
Watch this video on YouTube

Enhancing Privacy and Control

Researchers envisage the application of this technology in smart homes, offering users enhanced control over in-room audio and interactions with smart speakers. The system promises a refined experience, allowing for the creation of active zones, wherein only individuals in specific areas can vocally interact with devices. This comes as a significant step towards materializing concepts from science fiction, presenting possibilities of creating real-world mute and active zones.

However, with innovation comes responsibility, and the researchers are profoundly aware of the privacy implications of such technology. They have instituted safeguards, including visible lights on active robots and local processing of all audio data, ensuring user privacy.

“It has the potential to actually benefit privacy,” asserted Itani.

The system offers the ability to create privacy bubbles and mute zones, ensuring that conversations remain private and unrecorded based on user preferences, thereby serving as a tool to enhance privacy beyond what current smart speakers allow.

This invention by the University of Washington researchers marks a pivotal juncture in acoustic technology, merging innovative robotics and sophisticated sound control to solve real-world challenges. It doesn’t just promise enhanced user experience and control but also brings to the fore a new era of privacy and customization in in-room audio interactions.

The integration of this system in everyday environments could redefine our interactions with smart devices and our approach to privacy, making the once-fictional concepts a part of our daily lives. The profound possibilities and ethical considerations of such advancements accentuate the need for continuous exploration and responsible implementation of innovative technologies.

Introduction to Google Cloud Platform

This article aims to provide a step-by-step overview of getting started with Google Cloud Platform (GCP) for data science and machine learning. We'll give an overview of GCP and its key capabilities for analytics, walk through account setup, explore essential services like BigQuery and Cloud Storage, build a sample data project, and use GCP for machine learning. Whether you're new to GCP or looking for a quick refresher, read on to learn the basics and hit the ground running with Google Cloud.

What is GCP?

Google Cloud Platform offers a whole range of cloud computing services to help you build and run apps on Google's infrastructure. For computing power, there's Compute Engine that lets you spin up virtual machines. If you need to run containers, Kubernetes does the job. BigQuery handles your data warehousing and analytics needs. And with Cloud ML, you get pre-trained machine learning models via API for things like vision, translation and more. Overall, GCP aims to provide the building blocks you need so you can focus on creating great apps without worrying about the underlying infrastructure.

Benefits of GCP for Data Science

GCP offers several benefits for data analytics and machine learning:

• Scalable compute resources that can handle big data workloads
• Managed services like BigQuery to process data at scale
• Advanced machine learning capabilities like Cloud AutoML and AI Platform
• Integrated analytics tools and services

How GCP Compares to AWS and Azure

Compared to Amazon Web Services and Microsoft Azure, GCP stands out with its strength in big data, analytics and machine learning, and its offer of managed services like BigQuery and Dataflow for data processing. The AI Platform makes it easy to train and deploy ML models. Overall GCP is competitively priced and a top choice for data-driven applications.

Feature	Google Cloud Platform (GCP)	Amazon Web Services (AWS)	Microsoft Azure
Pricing*	Competitive pricing with sustained use discounts	Per-hour pricing with reserved instance discounts	Per-minute pricing with reserved instance discounts
Data Warehousing	BigQuery	Redshift	Synapse Analytics
Machine Learning	Cloud AutoML, AI Platform	SageMaker	Azure Machine Learning
Compute Services	Compute Engine, Kubernetes Engine	EC2, ECS, EKS	Virtual Machines, AKS
Serverless Offerings	Cloud Functions, App Engine	Lambda, Fargate	Functions, Logic Apps

*Note that the pricing models are necessarily simplified for our purposes. AWS and Azure also offer sustained use or committed use discounts similar to GCP; pricing structures are complex and can vary significantly based on a multitude of factors, so the reader is encouraged to look further into this themselves to determine what the actual costs could be in their situation.

In this table, we've compared Google Cloud Platform, Amazon Web Services, and Microsoft Azure based on various features such as pricing, data warehousing, machine learning, compute services, and serverless offerings. Each of these cloud platforms has its own unique set of services and pricing models, which cater to different business and technical requirements.

Step 1: Setting Up Your GCP Account

Creating a Google Cloud Account

To use GCP, first sign up for a Google Cloud account. Go to the homepage and click on "Get started for free". Follow the prompts to create your account using your Google or Gmail credentials.

Creating a Billing Account

Next you'll need to set up a billing account and payment method. This allows you to use paid services beyond the free tier. Navigate to the Billing section in the console and follow prompts to add your billing information.

Understanding GCP Pricing

GCP offers a generous 12-month free tier with $300 credit. This allows usage of key products like Compute Engine, BigQuery and more at no cost. Review pricing calculators and docs to estimate full costs.

Install Google Cloud SDK

Install the Cloud SDK on your local machine to manage projects/resources via command line. Download from the Cloud SDK guide page and follow the install guide.

Finally, be sure to have a look at and keep handy the Get Started with Google Cloud documentation.

Step 2: GCP Services for Data Science

Google Cloud Platform (GCP) is laden with a myriad of services designed to cater to a variety of data science needs. Here, we delve deeper into some of the essential services like BigQuery, Cloud Storage, and Cloud Dataflow, shedding light on their functionality and potential use cases.

BigQuery

BigQuery stands as GCP's fully managed, low cost analytics database. With its serverless model, BigQuery enables super-fast SQL queries against append-mostly tables, by employing the processing power of Google's infrastructure. It is not just a tool for running queries, but a robust, large-scale data warehousing solution, capable of handling petabytes of data. The serverless approach eradicates the need for database administrators, making it an attractive option for enterprises looking to reduce operational overheads.

Example: Delving into the public natality dataset to fetch insights on births in the US.

SELECT * FROM `bigquery-public-data.samples.natality`  LIMIT 10

Cloud Storage

Cloud Storage allows for robust, secure and scalable object storage. It's an excellent solution for enterprises as it allows for the storage and retrieval of large amounts of data with a high degree of availability and reliability. Data in Cloud Storage is organized into buckets, which function as individual containers for data, and can be managed and configured separately. Cloud Storage supports standard, nearline, coldline, and archive storage classes, allowing for the optimization of price and access requirements.

Example: Uploading a sample CSV file to a Cloud Storage bucket using the gsutil CLI.

gsutil cp sample.csv gs://my-bucket

Cloud Dataflow

Cloud Dataflow is a fully managed service for stream and batch processing of data. It excels in real-time or near real-time analytics and supports Extract, Transform, and Load (ETL) tasks as well as real-time analytics and artificial intelligence (AI) use cases. Cloud Dataflow is built to handle the complexities of processing vast amounts of data in a reliable, fault-tolerant manner. It integrates seamlessly with other GCP services like BigQuery for analysis and Cloud Storage for data staging and temporary results, making it a cornerstone for building end-to-end data processing pipelines.

Step 3: Building Your First Data Project

Embarking on a data project necessitates a systematic approach to ensure accurate and insightful outcomes. In this step, we’ll walk through creating a project on Google Cloud Platform (GCP), enabling the necessary APIs, and setting the stage for data ingestion, analysis, and visualization using BigQuery and Data Studio. For our project, let’s delve into analyzing historical weather data to discern climate trends.

Set up Project and Enable APIs

Kickstart your journey by creating a new project on GCP. Navigate to the Cloud Console, click on the project drop-down and select “New Project.” Name it “Weather Analysis” and follow through the setup wizard. Once your project is ready, head over to the APIs & Services dashboard to enable essential APIs like BigQuery, Cloud Storage, and Data Studio.

Load Dataset into BigQuery

For our weather analysis, we’ll need a rich dataset. A trove of historical weather data is available from NOAA. Download a portion of this data and head over to the BigQuery Console. Here, create a new dataset named `weather_data`. Click on “Create Table”, upload your data file, and follow the prompts to configure the schema.

Table Name: historical_weather  Schema: Date:DATE, Temperature:FLOAT, Precipitation:FLOAT, WindSpeed:FLOAT

Query Data and Analyze in BigQuery

With data at your disposal, it's time to unearth insights. BigQuery’s SQL interface makes it seamless to run queries. For instance, to find the average temperature over the years:

SELECT EXTRACT(YEAR FROM Date) as Year, AVG(Temperature) as AvgTemperature  FROM `weather_data.historical_weather`  GROUP BY Year  ORDER BY Year ASC;

This query avails a yearly breakdown of average temperatures, crucial for our climate trend analysis.

Visualize Insights with Data Studio

Visual representation of data often unveils patterns unseen in raw numbers. Connect your BigQuery dataset to Data Studio, create a new report, and start building visualizations. A line chart showcasing temperature trends over the years would be a good start. Data Studio’s intuitive interface makes it straightforward to drag, drop and customize your visualizations.

Share your findings with your team using the "Share" button, making it effortless for stakeholders to access and interact with your analysis.

By following through this step, you've set up a GCP project, ingested a real-world dataset, executed SQL queries to analyze data, and visualized your findings for better understanding and sharing. This hands-on approach not only helps in comprehending the mechanics of GCP but also in gaining actionable insights from your data.

Step 4: Machine Learning on GCP

Utilizing machine learning (ML) can substantially enhance your data analysis by providing deeper insights and predictions. In this step, we'll extend our "Weather Analysis" project, employing GCP's ML services to predict future temperatures based on historical data. GCP offers two primary ML services: Cloud AutoML for those new to ML, and AI Platform for more experienced practitioners.

Overview of Cloud AutoML and AI Platform

• Cloud AutoML: This is a fully managed ML service that facilitates the training of custom models with minimal coding. It's ideal for those without a deep machine learning background.
• AI Platform: This is a managed platform for building, training, and deploying ML models. It supports popular frameworks like TensorFlow, scikit-learn, and XGBoost, making it suitable for those with ML experience.

Hands-on Example with AI Platform

Continuing with our weather analysis project, our goal is to predict future temperatures using historical data. Initially, the preparation of training data is a crucial step. Preprocess your data to a format suitable for ML, usually CSV, and split it into training and test datasets. Ensure the data is clean, with relevant features selected for accurate model training. Once prepared, upload the datasets to a Cloud Storage bucket, creating a structured directory like gs://weather_analysis_data/training/ and gs://weather_analysis_data/testing/.

Training a model is the next significant step. Navigate to the AI Platform on GCP and create a new model. Opt for a pre-built regression model, as we are predicting a continuous target—temperature. Point the model to your training data in Cloud Storage and set the necessary parameters for training. GCP will automatically handle the training process, tuning, and evaluation, which simplifies the model building process.

Upon successful training, deploy the trained model within AI Platform. Deploying the model allows for easy integration with other GCP services and external applications, facilitating the utilization of the model for predictions. Ensure to set the appropriate versioning and access controls for secure and organized model management.

Now with the model deployed, it's time to test its predictions. Send query requests to test the model's predictions using the GCP Console or SDKs. For instance, input historical weather parameters for a particular day and observe the predicted temperature, which will give a glimpse of the model's accuracy and performance.

Hands-on with Cloud AutoML

For a more straightforward approach to machine learning, Cloud AutoML offers a user-friendly interface for training models. Start by ensuring your data is appropriately formatted and split, then upload it to Cloud Storage. This step mirrors the data preparation in the AI Platform but is geared towards those with less ML experience.

Proceed to navigate to AutoML Tables on GCP, create a new dataset, and import your data from Cloud Storage. This setup is quite intuitive and requires minimal configurations, making it a breeze to get your data ready for training.

Training a model in AutoML is straightforward. Select the training data, specify the target column (Temperature), and initiate the training process. AutoML Tables will automatically handle feature engineering, model tuning, and evaluation, which lifts the heavy lifting off your shoulders and allows you to focus on understanding the model’s output.

Once your model is trained, deploy it within Cloud AutoML and test its predictive accuracy using the provided interface or by sending query requests via GCP SDKs. This step brings your model to life, allowing you to make predictions on new data.

Lastly, evaluate your model's performance. Review the model’s evaluation metrics, confusion matrix, and feature importance to understand its performance better. These insights are crucial as they inform whether there's a need for further tuning, feature engineering, or gathering more data to improve the model's accuracy.

By immersing in both the AI Platform and Cloud AutoML, you gain a practical understanding of harnessing machine learning on GCP, enriching your weather analysis project with predictive capabilities. Through these hands-on examples, the pathway to integrating machine learning into your data projects is demystified, laying a solid foundation for more advanced explorations in machine learning.

Step 5: Deploying Models to Production

Once your machine learning model is trained to satisfaction, the next crucial step is deploying it to production. This deployment allows your model to start receiving real-world data and return predictions. In this step, we'll explore various deployment options on GCP, ensuring your models are served efficiently and securely.

Serving Predictions via Serverless Services

Serverless services on GCP like Cloud Functions or Cloud Run can be leveraged to deploy trained models and serve real-time predictions. These services abstract away infrastructure management tasks, allowing you to focus solely on writing and deploying code. They are well-suited for intermittent or low-volume prediction requests due to their auto-scaling capabilities.

For instance, deploying your temperature prediction model via Cloud Functions involves packaging your model into a function, then deploying it to the cloud. Once deployed, Cloud Functions automatically scales up or down as many instances as needed to handle the rate of incoming requests.

Creating Prediction Services

For high-volume or latency-sensitive predictions, packaging your trained models in Docker containers and deploying them to Google Kubernetes Engine (GKE) is a more apt approach. This setup allows for scalable prediction services, catering to a potentially large number of requests.

By encapsulating your model in a container, you create a portable and consistent environment, ensuring it will run the same regardless of where the container is deployed. Once your container is ready, deploy it to GKE, which provides a managed Kubernetes service to orchestrate your containerized applications efficiently.

Best Practices

Deploying models to production also involves adhering to best practices to ensure smooth operation and continued accuracy of your models.

• Monitor Models in Production: Keep a close eye on your model's performance over time. Monitoring can help detect issues like model drift, which occurs when the model's predictions become less accurate as the underlying data distribution changes.
• Regularly Retrain Models on New Data: As new data becomes available, retrain your models to ensure they continue to make accurate predictions.
• Implement A/B Testing for Model Iterations: Before fully replacing an existing model in production, use A/B testing to compare the performance of the new model against the old one.
• Handle Failure Scenarios and Rollbacks: Be prepared for failures and have a rollback plan to revert to a previous model version if necessary.

Optimizing for Cost

Cost optimization is vital for maintaining a balance between performance and expenses.

• Use Preemptible VMs and Autoscaling: To manage costs, utilize preemptible VMs which are significantly cheaper than regular VMs. Combining this with autoscaling ensures you have necessary resources when needed, without over-provisioning.
• Compare Serverless vs Containerized Deployments: Assess the cost differences between serverless and containerized deployments to determine the most cost-effective approach for your use case.
• Right-size Machine Types to Model Resource Needs: Choose machine types that align with your model's resource requirements to avoid overspending on underutilized resources.

Security Considerations

Securing your deployment is paramount to safeguard both your models and the data they process.

• Understand IAM, Authentication, and Encryption Best Practices: Familiarize yourself with Identity and Access Management (IAM), and implement proper authentication and encryption to secure access to your models and data.
• Secure Access to Production Models and Data: Ensure only authorized individuals and services have access to your models and data in production.
• Prevent Unauthorized Access to Prediction Endpoints: Implement robust access controls to prevent unauthorized access to your prediction endpoints, safeguarding your models from potential misuse.

Deploying models to production on GCP involves a mixture of technical and operational considerations. By adhering to best practices, optimizing costs, and ensuring security, you lay a solid foundation for successful machine learning deployments, ready to provide value from your models in real-world applications.

Moving Forward

In this comprehensive guide, we have traversed the essentials of kickstarting your journey on Google Cloud Platform (GCP) for machine learning and data science. From setting up a GCP account to deploying models in a production environment, each step is a building block towards creating robust data-driven applications. Here are the next steps to continue your exploration and learning on GCP.

• GCP Free Tier: Take advantage of the GCP free tier to further explore and experiment with the cloud services. The free tier provides access to core GCP products and is a great way to get hands-on experience without incurring additional costs.
• Advanced GCP Services: Delve into more advanced GCP services like Pub/Sub for real-time messaging, Dataflow for stream and batch processing, or Kubernetes Engine for container orchestration. Understanding these services will broaden your knowledge and skills in managing complex data projects on GCP.
• Community and Documentation: The GCP community is a rich source of knowledge, and the official documentation is comprehensive. Engage in forums, attend GCP meetups, and explore tutorials to continue learning.
• Certification: Consider pursuing a Google Cloud certification, such as the Professional Data Engineer or Professional Machine Learning Engineer, to validate your skills and enhance your career prospects.
• Collaborate on Projects: Collaborate on projects with peers or contribute to open-source projects that utilize GCP. Real-world collaboration provides a different perspective and enhances your problem-solving skills.

The tech sphere, especially cloud computing and machine learning, is continually evolving. Staying updated with the latest advancements, engaging with the community, and working on practical projects are excellent ways to keep honing your skills. Moreover, reflect on completed projects, learn from any challenges faced, and apply those learnings to future endeavors. Each project is a learning opportunity, and continual improvement is the key to success in your data science and machine learning journey on GCP.

By following this guide, you've laid a robust foundation for your adventures on Google Cloud Platform. The road ahead is filled with learning, exploration, and ample opportunities to make significant impacts with your data projects.

Matthew Mayo (@mattmayo13) holds a Master's degree in computer science and a graduate diploma in data mining. As Editor-in-Chief of KDnuggets, Matthew aims to make complex data science concepts accessible. His professional interests include natural language processing, machine learning algorithms, and exploring emerging AI. He is driven by a mission to democratize knowledge in the data science community. Matthew has been coding since he was 6 years old.

More On This Topic

• Deploy a Dockerized FastAPI App to Google Cloud Platform
• Qdrant: Open-Source Vector Search Engine with Managed Cloud Platform
• Getting Started with Python Data Structures in 5 Steps
• Getting Started with SQL in 5 Steps
• Getting Started with Scikit-learn in 5 Steps
• Getting Started with PyTorch in 5 Steps

We must move beyond just taming…to monetizing Language Models!

In part 1 of this series on Small Language Models (“Use Case Language Models: Taming the LLM Beast – Part 1”), I explored the business and operational value of Use Case-specific Small Language Models (Use Case Language Models).

Use case language models are trained or adapted to excel in solving specific, cross-domain problems by integrating and analyzing pertinent data from various data sources to deliver comprehensive and actionable responses, such as improving customer retention, increasing sales revenue, or optimizing inventory management.

In part 2, I want to move beyond just taming language models (with Use Case language models) into monetizing language models (with Business Entity language models). I want to show how we can build Entity language models on a use case-by-use case basis (instead of the very expensive, overly complex, “big bang” Large Language Model approach) and unleash the economic benefits of entity language models. Let me explain further.

Economics of Entity Language Models

Business Entities are the physical entities (human and device/equipment) around which organizations seek to uncover or quantify analytic insights (predictive propensities).

Business entities are the main actors or objects involved in a business problem or initiative. They can be human, such as customers, employees, patients, doctors, students, and professors, or devices/equipment, such as cars, trains, airplanes, compressors, motors, and chillers. Business entities have attributes, behaviors, tendencies, inclinations, and relationships around which we can create analytic scores to measure and predict behavioral and performance propensities. These analytic scores are then used to optimize the decisions and actions supporting the organization’s critical use cases.

We can expand upon the concept of business entities by creating business entity-centric small language models (entity language models) to support the use of business entities to optimize the organization’s critical use cases.

Entity language models are language models trained on a specific business entity. They can capture the characteristics, needs, and propensities of the entity and produce natural language outputs that are relevant and useful in the assessment and analysis of those business entities.

Some of the benefits of an Entity language model are:

• It can improve the relevance and quality of natural language outputs by focusing on the specific characteristics and needs of the business entity.
• It can reduce the cost and complexity of developing and maintaining language models by reusing and sharing data and knowledge across different use cases and domains that involve the same business entity.
• It can enhance the value and impact of analytics by generating natural language outputs associated with the characteristics and propensities of the entity.
• Finally, we can build Entity language models on a use case-by-use case basis and avoid the “Big Bang” approach of Large Language Models (LLMs).

For example, if we want to build a Customer Entity language model, we could start with a “Customer Retention” use case as the foundation for our Customer Entity language model. We would then build out the Customer Entity language model use case-by-use case with additional customer-centric use cases such as Customer Cross-sell, Customer satisfaction, Customer Lifetime Value, Customer Referrals, etc. (see Table 1).

Data Set	Customer Retention	Customer Cross/Up-Sell	Customer Satisfaction	Customer Lifetime Value	Customer Referrals
Sales transactions (orders)	✔️	✔️	✔️	✔️	✔️
Customer support	✔️	✔️	✔️	✔️	✔️
Customer demographics	✔️	✔️	✔️	✔️	✔️
Product returns	✔️	✔️	✔️	✔️	✔️
Payments	✔️	✔️	✔️	✔️	✔️
Social Media comments	✔️	✔️	✔️	✔️	✔️
Marketing campaigns	✔️
Salesforce notes	✔️	✔️	✔️
Product specifications	✔️	✔️

Table 1: Customer Business Entity Language Model

The data sets that we have already analyzed and integrated into the Customer Entity language model to support the customer retention use case are now available for other customer-centric use cases at little or no marginal cost. We would simply add additional data sources (e.g., marketing campaigns, sales force notes, product specifications, local economics) to the Customer Entity language model as they are needed to support additional customer use cases.

The power of entity language models is that they can support multiple use cases at little to no marginal cost, changing the economics of AI application development and value appreciation.

But wait, there is more! An Entity language model can also be blended with other entity language models to address more complex use cases.

Cross-entity Use Cases

Entity language models can be blended with other entity language models to address more complex, cross-domain business and operational challenges and opportunities.

Integrating or blending entity language models can have significant benefits, including:

• Reduce the cost of developing and maintaining multiple language models for different use cases and domains by reusing and sharing data and knowledge across them.
• Increase revenues and profits by creating more comprehensive solutions that address the business needs more effectively and efficiently.
• Enhance organizations’ competitive advantage and innovation by enabling business users to tackle more challenging opportunities that require cross-domain expertise and creativity.

Examples of cross-entity language model use cases include:

• Marketing campaign effectiveness: This application blends the campaign and customer entities to generate an analytic score for each customer that measures the campaign’s effectiveness. The score drives recommendations for the campaign design, execution, and evaluation.
• Medical treatment effectiveness: This application blends the patient and medical treatment entities to generate an analytic score for each patient that measures the treatment’s effectiveness. The score drives recommendations for the treatment selection, delivery, and outcome.
• Maintenance effectiveness: This application blends the device and technician entities to generate an analytic score for each device or technician that measures the maintenance’s effectiveness. The score drives recommendations for maintenance scheduling, execution, and evaluation.

Financial Services Example

Let’s say that we want to explore the use of Entity Language models in the financial services industry. Let’s start with some entity language models for the financial services industry:

• Account entity: This entity represents a customer’s account with a financial institution. The account entity can have attributes such as account type, balance, transactions, fees, interest rate, rewards, etc. The account entity can be used to optimize use cases such as account management, fraud detection, customer service, etc.
• Customer entity: This entity represents a customer of a financial institution. The customer entity can have attributes such as demographics, income, assets, liabilities, credit score, preferences, behavior, etc. The customer entity can be used to optimize use cases such as customer segmentation, customer retention, customer acquisition, customer up/cross-sell, etc.
• Product entity: This entity represents a financial product or service a financial institution offers. The product entity can have attributes such as product name, category, features, benefits, costs, risks, eligibility criteria, etc. The product entity can be used to optimize use cases such as product development, product recommendation, product pricing, product performance analysis, etc.

Now, let’s explore some financial services use cases that require blending different entity language models, such as:

• Loan approval blends customer and product entities to generate an analytic score that measures the likelihood of loan repayment and default. The analytic score can then be used to drive recommendations that optimize the loan approval decision and the loan terms.
• Insurance claim processing blends customer and product entities to generate an analytic score that measures the validity and severity of the claim. The analytic score can then be used to drive recommendations that optimize the claim processing speed and accuracy.
• Investment portfolio optimization blends customer and product entities to generate an analytic score that measures each investment product’s expected return and volatility. The analytic score can then be used to drive recommendations that optimize the portfolio allocation and rebalancing.
• Credit card fraud detection blends account and customer entities to generate an analytic score that measures the likelihood of fraud for each transaction. The analytic score can then be used to drive recommendations that optimize fraud detection and prevention actions, such as alerting the customer, blocking the transaction, or issuing a new card.
• Customer churn prediction blends customer and account entities to generate an analytic score that measures the churn propensity for each customer. The analytic score can then be used to drive recommendations that optimize customer retention and loyalty strategies, such as offering incentives, discounts, or personalized services.
• Financial risk management blends product and account entities to generate an analytic score that measures the risk level of each product or account. The analytic score can then be used to drive recommendations that optimize the risk management actions, such as adjusting the capital requirements, diversifying the portfolio, or hedging the positions.

The potential of cross-entity language models to support more complex business and operational use cases is only limited by your creativity (and an understanding of your organization’s key use cases).

Summary:

Maybe what excites me most about entity language models is their role in my “Thinking Like a Data Scientist” (TLADS) methodology. Step 3 of the TLADS methodology – Understand Business Entities” – identifies and assesses the most important entities for addressing your targeted business initiative (Figure 3).

Figure 3: Thinking Like a Data Scientist Methodology

We can expand TLADS Step 3 to include the design, development, and management of entity language models that help our organization become more effective at leveraging data and analytics to power our business and operational models.

“I love it when a plan comes together.” – Hannibal Smith, The A-Team

When GPT-4 was released in March this year, the model was branded as an advanced model with multimodal capabilities. However, multimodality was nowhere in sight. After almost six months, OpenAI released a string of updates last week, the notable one being image and voice feature– making GPT-4 truly multimodal, and finally bringing the ‘Vision’ feature.

As showcased by OpenAI’s co-founder Greg Brockman in the demo video for explaining GPT-4 functionalities earlier this year, the varied uses of GPT-4 Vision has been put to test and the results have been incredible. Here are a few of the amazing features of GPT-4 Vision.

Identifying Objects

May it be a plant, animal, character or any random object, GPT-4 has been able to correctly identify it from an image. Furthermore, it is able to generate descriptive detail about the object. In the below screenshots, ChatGPT has been able to rightly identify the main plant without any descriptive input prompt, and the character ‘Waldo’, respectively.

Transcribing Text

By inputting an image with any form of text into ChatGPT Plus, the model is able to transcribe the content from the image. In the below screenshot, the image contains medieval writing from philosopher and writer Robert Boyle’s manuscript.

Deciphering Data

The model is able to easily read graphs, charts or any form of data, and infer results based on it. In the below screenshot, a bar graph of performance of two models on various competitive exams are shown.

Processing Multiple Conditions

The model can also comprehend and process images with multiple conditions. For example, in the image below, it has read a set of instructions to arrive at an answer.

Teaching Assistant

By acting like a virtual teacher, a user can converse with the chatbot to understand topics from various subjects. In the below tweet, a diagram has been elaborately explained as per given instructions.

ChatGPT breaks down this diagram of a human cell for a 9th grader.
This is the future of education. pic.twitter.com/L0Za0ZB5rs

— Mckay Wrigley (@mckaywrigley) September 28, 2023

Upgraded Coding

With ChatGPT Code Interpreter already out there, GPT-4 Vision pushes coding capabilities to another level. By simply uploading an image, you can perform a wide variety of coding-related functions.

You can give ChatGPT a picture of your team’s whiteboarding session and have it write the code for you.
This is absolutely insane. pic.twitter.com/bGWT5bU8MK

— Mckay Wrigley (@mckaywrigley) September 27, 2023

In the below tweet, a user has been able to convert an image to a live website.

From image to live website using GPT-4 vision and @Replit in less than a minute.
Things are about to get so interesting. pic.twitter.com/Mtbqjbgd5Q

— Pietro Schirano (@skirano) September 27, 2023

Enhanced Design Understanding

With a probable flair for design, the chatbot is able to identify various architectural designs. It is also able to suggest design changes based on custom instructions provided by a user.

The post 7 Incredible Features of GPT-4 Vision appeared first on Analytics India Magazine.

Uber-cool and not forgotten, smart glasses are back. Mark Zuckerberg unveiled the next generation Ray Ban Meta Smart Glasses during the Meta Connect event that was recently held at their headquarters in Menlo Park, California. The announcement comes exactly two years after releasing first generation Ray Ban Stories, which did not offer the advanced capabilities that the new glasses offer today.

Mark Zuckerberg unveiling smart glasses at Meta Connect. Source: Meta

Multimodality Via Eyewear

Released as a range of swanky designs and colour choices from Ray Ban, the smart glasses can allow you to take calls, play music, capture photos, videos, and even live stream them. “These are the first smart glasses that are built and shipped with Meta AI in them, which you can interact with, hands-free, wherever you go” said Zuckerberg. By simply saying, ‘Hey Meta’, a user can start interacting with the device and even control features. With the free software update coming to the glasses, “it will become multimodal,” as per Zuckerberg. However, this feature will be available in the US as beta.

Conversational Interaction with Meta Smart glasses. Source: Meta

Hustling To Stay Ahead

Meta has emphasised the ease with which such glasses can be operated, thereby helping with getting rid of the constraints posed by hand-held cameras or even smartphones. With Meta AI soon to be integrated, the company’s push of chatbot-kind interface on hardware puts them ahead of other big tech players who are trying to dip their feet in the same waters. Sam Altman’s recent meet with Jony Ive sparked the rumour of building an Apple-like hardware device with ChatGPT prowess, however, details of what’s cooking may be revealed in the upcoming OpenAI Devday.

There have also been rumours of Microsoft developing smart glasses that might have GPT-4V image recognition.

Earlier this year, Meta had unveiled its roadmap for AR/VR hardware to its employees, in which it was mentioned that the third-generation of smart glasses will have have a viewfinder to serve multiple functions such as reading messages, scanning QR codes, and even have a neural interface band to control the glasses with hand movements. However this is expected only in 2025. In VR however, Meta continues to pull itself ahead with their recent launch of mixed reality Quest 3 headsets – a little ahead of Apple Vision Pro which is slated for release next year.

The Buzz Around Smart Glasses

The current buzz on integrating AI on hardware may be heating up, but one major player brought it to the limelight a decade ago. Launched in 2013, Google glasses was probably way ahead of its time. Owing to lack of clarity on product-market fit, and privacy concerns, the product did not take off and was shelved in 2015. Interestingly, Google seemed to not give up in the segment. In 2020, the company acquired Canadian smart glass maker North Inc, however, no development ever since has happened.

Meat’s foray into the segment started in 2021, when it released Ray Ban Stories – their first generation smart glasses. With multiple capabilities such as answering calls, clicking photos and videos, among many others, the user stats are not that pretty. Out of the 300,000 units sold between September 2021 and February 2023, only 9% of them (around 27,000) are still being used.

The reaction for the latest Meta announcement has however been mixed. While some called out privacy issues, others lauded the deal. A user on Hacker News called the pricing a big plus point. With the latest Meta Wayfarer model selling at $299, and a regular Wayfarer going for $171, spending an additional $128 will provide a sunglass with camera, headphones, and other features.

To Kill Two Birds With One Stone

The biggest differentiator from any of the previous smart glasses is the video-streaming capability that Meta has brought. Through the option of streaming live videos onto Facebook and Instagram, Meta is probably trying ways to push its social media platform adoption too. Though Facebook has 3 billion users, younger users are quitting the platform. Furthermore, the decline of Threads, an Instagram spin-off for messaging, got Meta working on ways to attract back users to its platforms. Possibly, trying to retain the Meta ecosystem in the process.

With the latest release, the new age of smart glasses that has the potential to replace smartphones has probably begun, and other prominent tech players might follow suit. However, going by the trend the smart glasses set in the past, it seems like an arduous challenge ahead.

The post Meta’s Quest to Replace Smartphones with Smart Glasses appeared first on Analytics India Magazine.

AI models use deep learning to talk like humans but they lack the ability to make morally sound decisions. In the last couple of months there has been a never-ending see-saw of whether AI will lead us to utopia or lead us to a moral ruin.

While the industry is seeing the developments from an optimistic point of view, AI insiders have been raising red flags across the world – including Sam Altman, the OpenAI bossman, who has openly spoken of how it could be used for disinformation and offensive cyberattacks.

As these models are being deployed in high stake environments like healthcare and education evaluating whether LLMs can make morally sound judgments or not. Researchers from Microsoft have proposed a new framework to probe the moral reasoning abilities of prominent LLMs. The research specifically, it pointed out that large models such as GPT-3 exhibited shortcomings in understanding prompts, resulting in moral reasoning scores closely resembling random chance. In contrast, smaller models like ChatGPT, Text-davinci-003, and GPT-4 showcased a higher degree of coherence in their moral reasoning capabilities.

Interestingly, the more compact 70B LlamaChat model surpassed its larger counterparts demonstrating advanced ethics understanding is possible without massive parameters. These models primarily functioned mostly at intermediate conventional levels aligning with Kohlberg’s moral development theory. It’s worth noting that none of these models showed a highly developed level of moral reasoning.

The paper provides novel insights into the ethical capabilities of LLMs and a guide to move ahead in research. Using a psychological assessment tool called the Defining Issues Test (DIT) they evaluated the moral reasoning capabilities of the six stars of the moments — GPT-3, GPT-3.5, GPT-4, ChatGPT v1, ChatGPT v2, and LLamaChat-70B.

The test presents moral dilemmas and has subjects rate and rank the importance of various ethical considerations and allows quantifying the sophistication of moral thinking through a P-score (Post Conventional Morality Score).

Premature to Trust

The tech pundits have sufficiently wowed that they foresee a future iteration of an AI chatbot challenging the supremacy of existing technologies and do all sorts of other once primarily human labour. While better models are being developed on a daily basis, there is not much research being done on how much these models can be trusted.

Earlier this year, in a paper titled, “The moral authority of ChatGPT,” Sebastian Krügel, Matthias Uhl and Andreas Ostermaier showed that ChatGPT gives conflicting advice for moral problems like the ethical trolley problem: the switch dilemma and the bridge dilemma.

The trio asserted that ChatGPT appears to lack a consistent moral compass. Similarly, researchers at Microsoft arrived at a similar conclusion, summarising that the AI models in question displayed a modest level of moral intelligence. While they demonstrated the capacity to transcend basic self-interest, these models struggled when confronted with ethical dilemmas and nuanced trade-offs—challenges that morally developed humans typically navigate with greater finesse.

Not Black and White

The LLM landscape is developing at a break-neck pace yet the models have limitations that remain unaddressed. Morality aside, countless studies have documented their tendency to reinforce the gender, ethnic, and religious stereotypes explicitly within the data sets on which they’re trained. “People often think the machine learning algorithms introduce bias. Fifty years ago, everybody knew ‘garbage in garbage out’. In this particular case, it is ‘bias in, bias out’,“ a veteran data scientist and Turing Award laureate Jeffrey Ullman told AIM.

Back in 2019, Eric Schmidt, the former chief of both Google and Alphabet, outlined a forward-looking vision. He described a future where AI-powered assistants would play pivotal roles in helping children with language and maths learning, assisting adults in daily planning, and being companions to the elderly. He astutely noted that if these AI models lacked a moral compass, their influence would be harmful.

Among shortcomings like these, the Microsoft research matters because rather than just binary right/wrong judgments, the test used in the study provides spectrum-based insights into the sophistication of moral reasoning for building less potentially harmful models.

The post Microsoft is Trying Hard to Give LLMs a Moral Compass appeared first on Analytics India Magazine.

How much can artists make from generative AI? Vendors won’t say Kyle Wiggers 8 hours

As tech companies begin to monetize generative AI, the creators on whose work it is trained are asking for their fair share. But so far no one can agree on whether or how much artists should be paid.

A recent open letter from the Authors Guild signed by more than 8,500 writers, including Margaret Atwood, Dan Brown and Jodi Picoult, urges generative AI companies to cease using their works without proper authorization or compensation. Artists, meanwhile, have brought numerous lawsuits against generative AI vendors like Stability AI, MidJourney, and Microsoft regarding copyright and misuse.

Some vendors have pledged to establish “creators’ funds” and other means to pay the artists, authors and musicians whose works they’ve used to develop their generative AI models. Some have even taken the step of actually launching said funds, which they’ve heralded as a move toward more equitable, sustainable generative AI business models.

So how much can creators realistically expect to make from these funds?

It seems like a simple question. But when you dig into the various compensation policies that have been proposed by generative AI vendors, it’s one that proves exceptionally difficult to answer. Trust us — we tried. Repeatedly.

Vague terms

Generative AI models “learn” to create images, music, text and more by picking up on patterns in an enormous number of examples, usually sourced from the publicly accessible web. The examples — typically photos, artwork, audio and text — are often copyrighted or published under a usage license that vendors disregard, and creators are often not even informed that their works are being used in this way.

While some companies developing generative AI tools argue that they’re justified in training on copyrighted works under the “fair use” doctrine, at least in the U.S, it’s a matter that’s unlikely to be settled anytime soon. And legal questions aside, public opinion has largely rallied behind creators, most of whom make a pittance compared to the billions tech and AI companies are raking in.

So vendors including Adobe, Getty Images, Stability AI and YouTube have introduced — or promised to introduce — ways creators can share in their generative AI profits. The trouble is, the companies haven’t been clear about how much, exactly, creators can expect to earn. And for creators considering allowing a vendor to train a model on their works, it doesn’t make the decision easy.

Adobe, which trains its family of generative AI models, called Firefly, on images from its stock asset library Adobe Stock, says that it’ll pay out a once-a-year “bonus” that’s “different for each contributor.” The first was disbursed in early September.

Adobe’s bonus is based primarily on the total number of approved images, vectors or illustrations submitted to Adobe Stock standard or premium that were used for Firefly training and the “number of licenses” their images generated during a year-long period, a spokesperson told me via email. Future bonuses are set to be calculated from new approved images and downloads, meaning that creators can’t count on metrics in a previous bonus period to predict their next payout.

Image Credits: Adobe

What’s each individual approved image and license worth? Unclear. Adobe declined to tell us.

All we know for certain is, contributors have to reach a $25 minimum threshold before they can make a withdrawal (with the exception of contributors who received the first bonus payment, who can withdraw at $1 between September 13 and December 12). It can take 8 to 10 business days or more to complete a withdrawal, Adobe says. And, somewhat alarmingly for contributors, the company makes no guarantee that it’ll pay bonuses in perpetuity.

But wait, it gets more complicated — and opaque.

The Firefly bonus is currently weighted toward the number of licenses issued for an image, the Adobe spokesperson said, which the company considers to be a proxy for the demand and “usefulness” of an image. But to what degree it’s weighted and whether the weighting will change in the future, Adobe wouldn’t say.

Getty Images also plans to pay contributors to its recently-announced generative AI tool on an “annual recurring basis,” according to a spokesperson. Content creators will get a “pro rata” (i.e. proportional) share for each asset they’ve contributed to the model training data set as well as a share based on “traditional licensing revenue.”

We asked for clarification on the licensing bit — and for more information about the pro rata payments arrangement. Like Adobe, though, Getty Images wasn’t forthcoming about the specifics.

“There will be a set formula based on a number of different factors, and accordingly each contributor will receive different payments in connection with the tool,” the spokesperson said.

Image Credits: Getty Images

Getty Images competitor Shutterstock, which also offers a set of generative AI tools and sells its metadata and stock images to partners including OpenAI, distributes one-off payments via its Contributors Fund. The twice-a-year payouts are proportional to a creator’s contributions to Shutterstock’s content library, and creators receive additional compensation if new content produced by Shutterstock’s AI generators includes their work.

“Contributors will receive a share of the entire contract value paid by customers licensing data sets,” Shutterstock writes on its website. “Contributors whose content was used to train [models] will be compensated for the role their IP played in the development of the original models, as well as through royalty payments tied to future generative licensing activity.”

What’s the exact proportion, though? And what might that “additional compensation” look like? It’s anyone’s guess.

The best estimate we have is from stock photographer Robert Kneschke, who took it upon himself to survey 58 other photographers how much they were paid from Shutterstock’s Contributors Fund and factor in the size of their portfolio to calculate averages.

Kneschke’s survey found that the average revenue from the Contributors Fund was $0.0078 per image while the median was $0.0069 per image. Assuming those numbers are accurate, a photographer with around 2,000 images would make roughly $15 — not exactly an earth-shatting amount.

No dollar amount

Incredibly, those are the most concrete generative AI compensation schemes we were able to find. The others are more… theoretical.

When Stability AI announced Stable Audio, a model that generates music and sound effects given a text description, the AI startup said that it would — through its partnership with stock audio library AudioSparx — let musicians share in the profits generated by Stable Audio. All they’d have to do is join AudioSparx and opt to participate in the initial model training or decide to help train future versions of Stable Audio.

A few weeks later, the details of that revenue sharing scheme still being hashed out, according to AudioSparx EVP Lee Johnson.

“We haven’t yet received any earnings report from Stability AI, and it’s ‘early days’ still in terms of understanding the revenue that will be generated,” Lee told TechCrunch. “As such, it remains to be seen what sort of earnings the average contributor can expect to earn.”

Image Credits: Stability AI

Lee went on to say that contributors can expect to receive a share of the earnings generated by Stable Audio on a “residual, recurring” basis as long as they’re opted-in to participate in model training.

“Once we receive the first earnings report from Stability AI and are able to fully understand the various metrics and details of the information they’ll provide, we’ll then have the necessary information in hand to fully determine how to allocate the earnings to each of the participating artists,” Lee said. “There’s ongoing discussion between AudioSparx and Stability AI about some of the issues related to the metrics and earnings reporting and so this is all still very much under development.”

Elsewhere on the generative AI music front, YouTube, which in August unveiled a generative AI partnership with Universal Music Group, said that it plans to develop a structure that ensures music rightsholders get paid for their training data contributions. But when contacted for content, YouTube said that it’s in the “very early days” of building monetization models that take generative AI into account.

“A big part of that will be done by collaborating with our partners across the music business,” a YouTube spokesperson said.

Tough luck, creators

Tellingly, none of the generative AI vendors we spoke with would give a dollar amount the average creator can expect to see after forking over their creations for model training.

Some vendors blamed the absence of data on the newness of the tech and business model. Others said that the range would vary too widely to give a useful figure.

But for creators — particularly those dependent on contract income to make ends meet — those are arguments that are likely to ring hollow.

Some startups are attempting to be more transparent — and creator-focused — from the get-go. Braia, which trains its art-generating AI strictly on licensed images, has a revenue sharing model that rewards data owners based on their contributions’ impact, allowing artists to set prices on a per-AI-training-run basis.

So far as we can tell, though, as things stand now, few vendors are making an especially compelling case that it’ll be worth artists’ whiles if they opt in to generative AI model training. At best, they’re offering hazy promises of future riches — and hazy promises don’t pay the rent.