Рубрика: AI

OpenAI recently said that it would solve AI Alignment within four years — an idea that seems farfetched. And people have reacted. Meta founder Mark Zuckerberg commented that at a time when we are not even able to solve cyber crime issues, solving super alignment is no less than climbing a mountain peak.

Meta AI chief Yann LeCun said that one cannot solve the AI alignment problem in four years. “One doesn’t just ‘solve’ the safety problem for turbojets, cars, rockets, or human societies, either,” he said, adding that engineering-for-reliability is always a process of continuous and iterative refinement.

But, is the AI threat even real?

A few days back ago, ‘The Terminator’ Arnold Schwarzenegger expressed his concerns about AI taking over humanity. He claimed ‘The Terminator’ franchise, which launched his career in 1984, predicted the future of artificial intelligence and how the movie predicted about the machines becoming self-aware and taking over. As AI continues to advance, discussions around its responsible development and potential risks are at the forefront. The concerns surrounding AI turning against humans are not unfounded.

In a blog post, Altman, the founder of Open AI, emphasised the significance of ensuring alignment in the development of super-intelligent AGI. He acknowledged that a misaligned AGI could pose significant harm to the world, and even an autocratic regime with superior AI capabilities could potentially cause similar damage. These remarks underscore the importance of cautious and ethical approaches in shaping the future of AI. In another blog post, he went a step ahead warning about dangers of AI saying “it could lead to the disempowerment of humanity or even human extinction”.

Earlier, in an interview with Fridman, when asked if AI would kill humans, Altman said there was a chance that it would, and it’s important to acknowledge it. “If we don’t treat it as potentially real, we won’t put enough effort into solving it. And I think we have to discover new techniques to be able to solve it.”

It looks like Altman is taking his concerns seriously as OpenAI launched ‘Superalignment’. OpenAI plans to invest a significant investment and resources to create a new research team to ensure its artificial intelligence remains safe for humans. This new team will be co-led by Ilya Sutskever and Jan Leik and the company will dedicate 20% of the compute they’ve secured to date to this effort.

Should we be worried?

Currently, there is no definitive answer to this question, and various experts and founders hold differing viewpoints on the matter. Altman, for instance, emphasises the importance of acknowledging the issue and actively addressing it. In contrast, Meta’s founder, Zuckerberg, takes a different stance, suggesting there is no immediate need for excessive concern. These divergent perspectives reflect the ongoing discourse surrounding the appropriate approach to handling this complex issue.

In an interview with Lex Fridman, Zuckerberg said that we are quite a few steps away from super intelligence so the existential threat is much further out. There are more near term risks of people using AI to do harmful things like fraud or scams etc that need to be tackled now. These are going to be a pretty big set of challenges to tackle. So he is worried that people are focused on the tail risk of existential threat instead of doing a good job with the risks that are more certain near term.

Along the same lines, LeCun said, “I think that the magnitude of the AI alignment problem has been ridiculously overblown and our ability to solve it is widely underestimated.” LeCun thinks that for machines to be in control, they should “want to take control” and our instant assumption that they will obviously dominate humans is purely drawn from science fiction dreams.

Meanwhile, the inventor of Markov Logic Network Pedro Domingos believes that OpenAI is going to waste billions of dollars on an unworkable solution to a fake problem.

In conclusion

Apart from OpenAI, no one else has taken the lead to address the issue of Alignment. Many authors have expressed concerns in their columns regarding alignment but most of them take their inspiration from fiction. Open AI said that currently it doesn’t have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue.

Their present techniques for aligning AI, such as reinforcement learning from human feedback, rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us. They need new scientific and technical breakthroughs.

The post OpenAI’s Pursuit of AI Alignment is Farfetched appeared first on Analytics India Magazine.

Chat With Your Data: Mixpanel Integrates Generative AI to Simplify Analytics July 7, 2023 by Jaime Hampton

Mixpanel, a platform for event analytics, announced it has integrated generative AI capabilities into its services to allow companies to “chat with their data.”

Using a new feature called Spark, Mixpanel users can now conduct a natural language chat with their data to receive immediate insights into customer experience and the impact of their product and marketing decisions.

“Generative AI is the next interface to computing, and it’s unlocking huge productivity gains,” said Amir Movafaghi, CEO of Mixpanel. “In our world, this means it’s much easier for anyone to query their data in plain English by asking the AI a question. Making analytics accessible, so literally everyone can participate, will significantly improve decision making across companies.”

Mixpanel’s goal is to make analytics less technical and more accessible to all users. The Mixpanel platform is based on event analytics where every action a user performs within a digital product like an eCommerce site or rideshare app is captured and used for analysis. The company says this granular view helps companies understand how different groups of users behave at various points during their experience. Traditional analytics and BI tools often require writing complex SQL queries which can leave non-technical users at the mercy of their (often very busy) data scientist colleagues when gaining insights.

“Mixpanel changed this with its event-based analytics system, which non-technical employees use to ask questions of their data with drop-down menus. The introduction of generative AI reimagines the data analytics process again, so anyone can use Mixpanel to support better decision making by easily asking questions of their data,” the company said in a release.

Mixpanel users can use text prompts to query data without writing a complex SQL query. (Source: Mixpanel)

Spark leverages OpenAI’s GPT-3.5 Turbo model, a smaller, more refined version of GPT-3. Users can ask business questions in plain English and the model constructs the necessary query, executes it in Mixpanel, and delivers a dashboard of the relevant data.

As an example of how this new feature can be used, a non-technical employee working for a rideshare platform might ask, “Which group of users most frequently convert when we apply surge pricing across our key markets?” Using this prompt, Spark can build the necessary query, execute it in Mixpanel, and return a relevant chart displaying conversion trends for different cohorts across different markets.

Inaccuracy via hallucination is a concern with large language models like GPT-3.5 Turbo which is said to have a hallucination rate between 15-20%. There are also privacy and security concerns when using LLMs with proprietary data.

Mixpanel is addressing these concerns with built-in features that allow users to check the source of the information given in a generated report. The company says it prioritizes privacy and that company data is not ingested by the LLM. The AI only builds queries and Mixpanel analyzes the underlying data, the company asserts.

“When Spark builds a report, it’ll be viewable and editable like any other report, meaning you can go into its query builder view and see details like what events are being used. From there, you can even add your own edits to the report to make modifications or improvements,” Movafaghi wrote in a blog post.

The company is also making its generative AI feature optional. Even though Spark will eventually be available to all users, customers can choose to keep using the existing Mixpanel interface. Spark will soon be available as part of a closed Beta program to select customers, but the company says it will be rolling it out as an optional interface to all Mixpanel users in the coming weeks.

“Generative AI is a bit like electricity, you can build it into other products to make things faster and easier. We’re using it to speed up workflows and simplify how people ask questions of their data. But this is just the start, and we expect LLMs will enhance analytics for years to come,” said Movafaghi.

This article first appeared on sister site Datanami.

Related

YouTube tests AI-generated quizzes on educational videos Lauren Forristal 8 hours

YouTube is experimenting with AI-generated quizzes on its mobile app for iOS and Android devices, which are designed to help viewers learn more about a subject featured in an educational video. The feature will also help the video-sharing platform get a better understanding of how well each video covers a certain topic.

The AI-generated quizzes, which YouTube noted on its experiments page yesterday, are rolling out globally to a small percentage of users that watch “a few” educational videos, the company wrote. The quiz feature is only available for a select portion of English-language content, which will appear on the home feed as links under recently watched videos.

Not all of YouTube’s experiments make it to the platform, so it will be interesting to see if this one sticks around. We’re not sure how many people – especially if they’re no longer in school – would want to take a quiz while they scroll through videos.

However, YouTube has long established itself as a destination for users to learn new things, whether it’s how to change a car tire or even perform a backflip. “Edutainment” accounts like TED-Ed and HowToBasic are among the more popular educational YouTube channels, with 18.8 million and 17.3 million subscribers, respectively.

Additionally, many teachers go on YouTube to create educational content or discover and share videos with their students. So, for those that genuinely want to learn more about a topic, the new quiz feature could be an effective way to gain a deeper understanding of the material.

Earlier this year, YouTube partnered with Crash Course and Arizona State University to launch its “Study Hall” initiative, which gives college students free access to four courses covering various subjects, including college math, U.S. history, English composition and more. Crash Course, a channel run by John and Hank Green, has 14.8 million subscribers and approximately 1.6 billion views.

Separately, YouTube has also been testing features such as a three-strikes ad-blocking policy and a new lock screen feature for Premium subscribers.

YouTube is experimenting with a new lock screen feature for Premium users

AI music generators use algorithms to create music by combining patterns, loops, chords, and melodies. Users choose a genre, lyrics, tune and other specifics and the AI algorithms can generate a unique soundscape. The generators can adjust the song length and structure and match uploaded videos with suitable music. These are versatile tools for content creators, providing copyright-free sound and music for various projects. Here is a list of the few excellent ones.

Amper Music

Amper Music, which has been acquired by Shutterstock, is user-friendly, making it an ideal choice for those interested in exploring AI-generated music. And being a cloud-based platform, it’s a suitable option for content creators, as well as individuals working on soundtracks and audio for games, movies, or podcasts. The premium edition offers additional features that complement the artist’s creative process.

One doesn’t even require extensive knowledge of music theory or composition since it generates musical tracks using pre-recorded samples. These samples are then transformed into real audio, allowing users to modify aspects such as music keys, tempo, and individual instruments. For instance, you can adjust the instrument to match the desired mood or vibe.

Musenet and Jukebox

Jukebox, developed by OpenAI, is an AI music generator that can produce audio in various genres and artist styles. By inputting information such as genre, artist, and lyrics, Jukebox can generate unique music that differs from the original training data. It empowers AI to autonomously create original compositions.

Similarly, MuseNet, another AI music generator from OpenAI, is capable of crafting 4-minute musical compositions using 10 different instruments. It possesses a remarkable ability to blend styles from different eras and genres, encompassing country, Mozart, the Beatles, and more. Unlike rule-based programming in other AI music generators, MuseNet is built on a general-purpose unsupervised technology like GPT-2.

AIVA

AIVA is another famous AI music generator, which was initially developed in 2016. The site undergoes continuous enhancements to compose soundtracks for various applications such as ads, video games, and movies. They debuted with the publication of Opus 1 for Piano Solo and has since released an album, along with composing music for a video game. This tool allows users to create original music from scratch and even generate variations of existing songs, eliminating concerns about music licensing processes.

With AIVA, users can effortlessly generate music across multiple genres and styles by selecting preset options. Additionally, AIVA offers the flexibility to make edits to tailor the soundtracks to specific requirements.

Soundraw

Another excellent choice for an AI music generator is Soundraw. It offers a range of features, including the ability to customise a song using AI-generated phrases and more. The tool leverages a combination of AI technology and manual tools to make music generation and customisation a seamless experience.

One notable aspect of the platform is its customisation feature, which allows you to improvise and fine-tune individual pieces of music. While free users can create music using the music generator, unlimited downloads are available through a subscription plan. They also have compatible plug-ins with Google Chrome and Premiere Pro.

Mubert

Mubert AI is an AI-powered music generation tool that specializes in creating personalized music streams based on individual preferences. Much like Soundraw, Mubert AI operates in a similar space, offering comparable services and approaching AI music generation from a similar standpoint. This similarity in their services has led to comparisons, where Soundraw and Mubert are often referred to as the “Microsoft and Apple of AI Music Generation”.

While Soundraw places a stronger emphasis on customization through its editing features and Pro mode, Mubert distinguishes itself by providing additional offerings. For instance, Mubert Studio enables users to collaborate with its AI system, allowing them to earn money with their tracks and samples. On the other hand, Mubert Play offers carefully curated AI-generated music tailored to individual musical preferences using Mubert’s intelligent algorithms.

Moreover, Mubert stands out by offering a mobile app, enabling users to experiment with and enjoy AI-generated music on the go.

Beatoven

It is an AI music generator that simplifies the music creation process. What sets Beatoven apart is its transparent and ethical approach, as it involves contributions from real artists in its training. With Beatoven, users have the freedom to choose from eight genres and 16 moods, allowing them to match the music to the tone and theme of their content. The platform also offers the flexibility to make cuts and adjustments, providing a familiar experience similar to popular music editors like Audacity and Ableton.

Whether you’re creating a travel vlog, a meditation podcast, or a promotional video, Beatoven.ai provides features that can fulfill your musical needs without any promotional hype.

Ecrett

Ecrett Music is another AI-powered music generator that empowers users to create unique, royalty-free music, soundscapes, and sound effects. Similar to Soundraw and Mubert, Ecrett Music offers an intuitive interface and an extensive library of sounds, instruments, and effects. Its user-friendly editor caters to musicians, sound designers, and individuals of all skill levels.

Users can effortlessly mix and match various elements to craft their own music, searching for sounds and instruments based on genre, mood, or key. It stands out as a cost-effective solution in the market. Users can customise the music length, instruments, and structure. Its library features over 500,000 unique monthly generated music patterns, all available royalty-free for personal and commercial projects. Ecrett Music simplifies the process of music composition for videos and games and their tool’s straightforward interface and extensive selection of scenes, emotions, and genres.

Melobytes

Melobytes offers more than just an AI music generator—it is an expansive Artificial Intelligence playground where users can explore various AI-powered applications. The website hosts a wide array of apps, ranging from random story generators to crossword solvers, all powered by AI. Moreover, Melobytes provides an extensive selection of music composition tools, such as ‘Drawing to Music’, ‘Movie Music Maker’, and ‘Text to Rap Song’.

The site features a dedicated MIDI section for file conversions and an algorithmic music and lyrics creation section. The platform boasts numerous features that would require an extensive article to cover comprehensively, making it an exciting tool for users to discover and experiment with personally.

The post 9 AI Music Generators You Need to Know About appeared first on Analytics India Magazine.

OpenAI is seeking researchers to work on containing super-smart artificial intelligence with other AI. The end goal is to mitigate a threat of human-like machine intelligence that may or may not be science fiction.

“We need scientific and technical breakthroughs to steer and control AI systems much smarter than us,” wrote OpenAI Head of Alignment Jan Leike and co-founder and Chief Scientist Ilya Sutskever in a blog post.

Jump to:

• OpenAI’s Superalignment team is now recruiting
• AI trainer may keep other AI models in line
• Superintelligent AI: Real or science fiction?

OpenAI’s Superalignment team is now recruiting

The Superalignment team will devote 20% of OpenAI’s total compute power to training what they call a human-level automated alignment researcher to keep future AI products in line. Toward that end, OpenAI’s new Superalignment group is hiring a research engineer, research scientist and research manager.

OpenAI says the key to controlling an AI is alignment, or making sure the AI performs the job a human intended it to do.

The company has also stated that one of its objectives is the control of “superintelligence,” or AI with greater-than-human capabilities. It’s important that these science-fiction-sounding hyperintelligent AI “follow human intent,” Leike and Sutskever wrote. They anticipate the development of superintelligent AI within the last decade and want to have a way to control it within the next four years.

SEE: How to build an ethics policy for the use of artificial intelligence in your organization (TechRepublic Premium)

AI trainer may keep other AI models in line

Today, AI training requires a lot of human input. Leike and Sutskever propose that a future challenge for developing AI might be adversarial — namely, “our models’ inability to successfully detect and undermine supervision during training.”

Therefore, they say, it will take a specialized AI to train an AI that can outthink the people who made it. The AI researcher that trains other AI models will help OpenAI stress test and reassess the company’s entire alignment pipeline.

Changing the way OpenAI handles alignment involves three major goals:

• Creating AI that assists in evaluating other AI and understanding how those models interpret the kind of oversight a human would usually perform.
• Automating the search for problematic behavior or internal data within an AI.
• Stress-testing this alignment pipeline by intentionally creating “misaligned” AI to ensure that the alignment AI can detect them.

Personnel from OpenAI’s previous alignment team and other teams will work on Superalignment along with the new hires. The creation of the new team reflects Sutskever’s interest in superintelligent AI. He plans to make Superalignment his primary research focus.

Superintelligent AI: Real or science fiction?

Whether “superintelligence” will ever exist is a matter of debate.

OpenAI proposes superintelligence as a tier higher than generalized intelligence, a human-like class of AI that some researchers say won’t ever exist. However, some Microsoft researchers think GPT-4 scoring high on standardized tests makes it approach the threshold of generalized intelligence.

Others doubt that intelligence can really be measured by standardized tests, or wonder whether the very idea of generalized AI approaches a philosophical rather than a technical challenge. Large language models can’t interpret language “in context” and therefore don’t approach anything like human-like thought, a 2022 study from Cohere for AI pointed out. (Neither of these studies is peer-reviewed.)

SEE: Some high-risk uses of AI could be covered under the laws being developed in the European Parliament. (TechRepublic)

OpenAI aims to get ahead of the speed of AI development

OpenAI frames the threat of superintelligence as possible but not imminent.

“We have a lot of uncertainty over the speed of development of the technology over the next few years, so we choose to aim for the more difficult target to align a much more capable system,” Leike and Sutskever wrote.

They also point out that improving safety in existing AI products like ChatGPT is a priority, and that discussion of AI safety should also include “risks from AI such as misuse, economic disruption, disinformation, bias and discrimination, addiction and overreliance, and others” and “related sociotechnical problems.”

“Superintelligence alignment is fundamentally a machine learning problem, and we think great machine learning experts — even if they’re not already working on alignment — will be critical to solving it,” Leike and Sutskever said in the blog post.

Subscribe to the Innovation Insider Newsletter

Catch up on the latest tech innovations that are changing the world, including IoT, 5G, the latest about phones, security, smart cities, AI, robotics, and more.

Delivered Tuesdays and Fridays Sign up today

OpenAI announced that GPT-4, the latest version of the company's language model, will be generally available to paying API customers. OpenAI says GPT-4 is the company's most capable language model. Since GPT-4's release in March, it has powered apps and services that offer visual accessibility, wealth management, language preservation, and language learning.

Also: GPT-3.5 vs GPT-4: Is ChatGPT Plus worth its subscription fee?

In a blog post, OpenAI details the general availability of GPT-4 and the depreciation of several APIs that will retire by January of next year.

Who can access GPT-4?

According to OpenAI, existing API developers with an OpenAI API payment history can access GPT-4 with 8K context. Context windows dictate how much text GPT-4 uses to generate an output. The 8K context option amounts to about 13 pages of text.

OpenAI plans to raise rate limits and open access to GPT-4 to new developers by the end of July based on compute availability. Additionally, the company plans to make GPT-3.5 Turbo, DALL-E, and Whisper APIs generally available to developers.

Depreciation of GPT-3

The company also announced that beginning Jan. 4, 2024, GPT-3 and its variations using the Completions API will retire to free up computing capacity. Although the API will continue to be accessible, it will receive a legacy label and will be replaced by a new model.

Also: GPT-4: A new capacity for offering illicit advice and displaying 'risky emergent behaviors'

Applications that use stable names for base GPT-3 models, such as ada, babbage, curie, and davinci, will automatically be replaced by their successors, ada-002, babbage-002, curie-002, and davinci-002.

Depreciation of embedding API

By Jan. 4, 2024, developers who use older embedding models will need to migrate to the new embedding model that OpenAI released in December 2022. The company says the new model is "more capable and cost-effective than previous models."

Also: 7 advanced ChatGPT prompt-writing tips you need to know

Although the new model is responsible for 99.9% of all embedding API use, according to OpenAI, the company is aware that making the switch can be costly and inconvenient for some developers. As a result, the company says it will cover any costs associated with re-embedding content.

Artificial Intelligence

Meta in a blogpost on Wednesday announced that its new offering “Threads” will soon be built on ActivityPub, a protocol that is used to post on other decentralised networks. This is unique as it would make Threads interoperable with other platforms like WordPress, Mastodon and Bluesky.

Meta’s plan is to convert Threads into a fediverse platform, a new type of network that allows people to follow and interact with each other on different decentralised social media platforms, such as Mastodon and Bluesky. This would also enable an individual with a public profile on Threads to have an audience across similar decentralised applications. Alternatively, with a private profile, you can approve followers and interact with them, similar to instagram.

“An individual from other fediverse platforms can find people on Threads using full usernames, such as ‘@mosseri@threads.net’,” said Instagram head Adam Mosseri.

These additions would give Threads an edge over twitter allowing new types of connections. It would be one of the few platforms of its kinds which would also allow you to carry your audience over as well.

Threads Reaping The Fruit of Mastodon’s Labours

Launched in 2016, Mastodon was the first decentralised microblogging platform built on the ActivityPub protocol. It gained popularity because it allowed users to exercise more control over their data and flexibility in comparison to a centralised platform like twitter. It was a noble idea, but couldn’t make much impact until Elon Musk announced to take over Twitter.

Social media users looked out for alternatives and Mastodon was there. In early December 2022, it hit an all time high of 2.2 million active users. But the numbers didn’t hold for long. In January 2023, the user base fell down 30% to 1.8 million. The current user base has shrunk to 1.4 million.

On the other hand, Bluesky launched by ex-Twitter CEO Jack Dorsey is attracting former Twitter users and people who haven’t quite adapted to the more formal and organised settings of Mastodon.

It was recently reported that its systems were experiencing “some degraded performance” as a result of record-high traffic. Its community has topped 50,000 users because it is an invite only platform. However, the Twitter alternative has over 375,000 worldwide instals on iOS as of April.

Though the early decentralised players haven’t been able to make a bigger impact, Meta is going all in on the decentralised social media front. It claimed in its blogpost that it believes that “decentralised approach similar to protocols governing email and the web itself will play an important role in the future of online platforms”.

It seems like Meta’s bet paid off. In just over 24 hours since the app was rolled out, it has got 50 million users.

The company could reap all the benefits of the work put in by Mastodon and Bluesky to create an appeal and user base of a decentralised platform. It could benefit from their learning, and strike a balance between a formal and organised setting of Mastodon and an informal site filled with memes and shitposting like Bluesky.

Threads could also strike a chord because of the already existing monthly user base of more than 2 billion on Instagram—because an individual can sign up to threads using an already existing Instagram account.

Besides giving an open experience in terms of followers and interaction across platforms, developers can enhance user experiences and introduce features that can be seamlessly incorporated into different open social networks. Meta has also signalled that threads is one of its many offerings which would be compatible with an open social networking protocol.

Meta’s Threads move can also steer the entire decentralised ecosystem. Each decentralised platform could also establish its own community standards and content moderation policies, granting users the freedom to choose spaces that align with their beliefs and values.

The post Why is Everyone Obsessed with Twitter? Threads Poses Real Threat to Mastodon, Bluesky appeared first on Analytics India Magazine.

What Is Reinforcement Learning?

Reinforcement learning is a branch of machine learning that deals with an agent learning—through experience—how to interact with a complex environment.

From AI agents that play and surpass human performance in complex board games such as chess and Go to autonomous navigation, reinforcement learning has a suite of interesting and diverse applications.

Remarkable breakthroughs in the field of reinforcement learning include DeepMind’s agent AlphaGo Zero that can defeat even human champions in the game of Go and AlphaFold that can predict complex 3D protein structure.

This guide will introduce you to the reinforcement learning paradigm. We’ll take a simple yet motivating real-world example to understand the reinforcement learning framework.

The Reinforcement Learning Framework

Let's start by defining the components of a reinforcement learning framework.

Reinforcement Learning Framework | Image by Author

In a typical reinforcement learning framework:

• There is an agent learning to interact with the environment.
• The agent can measure its state, take actions, and occasionally gets a reward.

Practical examples of this setting: the agent can play against an opponent (say, a game of chess) or try to navigate a complex environment.

As a super simplified example, consider a mouse in a maze. Here, the agent is not playing against an opponent but rather trying to figure out a path to the exit. If there are more than one paths leading to the exit, we may prefer the shortest path out of the maze.

Mouse in a Maze | Image by Author

In this example, the mouse is the agent trying to navigate the environment which is the maze. The action here is the movement of the mouse within the maze. When it successfully navigates the maze to the exit—it gets a piece of cheese as a reward.

Example | Image by Author

The sequence of actions happens in discrete time steps (say, t = 1, 2, 3,…). At any time step t, the mouse can only measure its current state in the maze. It doesn’t know the whole maze yet.

So the agent (the mouse) measures its state s_t in the environment at time step t, takes a valid action a_t and moves to state s_(t + 1).

State | Image by Author

How Is Reinforcement Learning Different?

Notice how the mouse (the agent) has to figure its way out of the maze through trial and error. Now if the mouse hits one of the walls of the maze, it has to try to find its way back and etch a different route to the exit.

If this were a supervised learning setting, after every move, the agent would get to know whether or not that action—was correct—and would lead to a reward. Supervised learning is like learning from a teacher.

While a teacher tells you ahead of time, a critic always tells you—after the performance is over— how good or bad your performance was. For this reason, reinforcement learning is also called learning in the presence of a critic.

Terminal State and Episode

When the mouse has reached the exit, it reaches the terminal state. Meaning it cannot explore any further.

And the sequence of actions—from the initial state to the terminal state—is called an episode. For any learning problem, we need multiple episodes for the agent to learn to navigate. Here, for our agent (the mouse) to learn the sequence of actions that would lead it to the exit, and subsequently, receive the piece of cheese, we’d need many episodes.

Dense and Sparse Rewards

Whenever the agent takes a correct action or a sequence of actions that is correct, it gets a reward. In this case, the mouse receives a piece of cheese as a reward for etching a valid route—through the maze(the environment)—to the exit.

In this example, the mouse receives a piece of cheese only at the very end—when it reaches the exit.This is an example of a sparse and delayed reward.

If the rewards are more frequent, then we will have a dense reward system.

Looking back we need to figure out (it’s not trivial) which action or sequence of actions caused the agent to get the reward; this is commonly called the credit assignment problem.

Policy, Value Function, and the Optimization Problem

The environment is often not deterministic but probabilistic and so is the policy. Given a state s_t, the agent takes an action and goes to another state s_(t+1) with a certain probability.

The policy helps define a mapping from the set of possible states to the actions. It helps answer questions like:

• What actions to take to maximize the expected reward?

• Or better yet: Given a state, what is the best possible action that the agent can take so as to maximize expected reward?

So you can think of the agent as enacting a policy π:

Another related and helpful concept is the value function. The value function is given by:

This signifies the value of being in a state given a policy π. The quantity denotes the expected reward in the future if the agent starts at state and enacts the policy π thereafter.

To sum up: the goal of reinforcement learning is to optimize the policy so as to maximize the expected future rewards. Therefore, we can think of it as an optimization problem to solve for π.

Discount Factor

Notice that we have a new quantity ɣ. What does it stand for? ɣ is called the discount factor, a quantity between 0 and 1. Meaning future rewards are discounted (read: because now is greater than much later).

Exploration vs. Exploitation Tradeoff

Going back to the food loop example of mouse in a maze: If the mouse is able to figure out a route to exit A with a small piece of cheese, it can keep repeating it and collecting the cheese piece. But what if the maze also had another exit B with a bigger cheese piece (greater reward)?

So long as the mouse keeps exploiting this current strategy without exploring new strategies, it is not going to get the much greater reward of a bigger cheese piece at exit B.

Exploration vs. Exploitation | Image by Author

But the uncertainty associated with exploring new strategies and future rewards is greater. So how do we exploit and explore? This tradeoff between exploiting the current strategy and exploring new ones with potentially better rewards is called the exploration vs exploitation tradeoff.

One possible approach is the ε-greedy search. Given a set of all possible actions , the ε-greedy search explores one of the possible actions with the probability ε while exploiting the current strategy with the probability 1 — ε.

Wrap-up and Next Steps

Let's summarize what we’ve covered so far. We learned about the components of the reinforcement learning framework:

• The agent interacts with the environment, gets to measure its current state, takes actions, and receives rewards as positive reinforcement. The framework is probabilistic.
• We then went over value functions and policy, and how the optimization problem often boils down to finding the optimal policies that maximize the expected future awards.

You’ve now learned just enough to navigate the reinforcement learning landscape. Where to go from here? We did not talk about reinforcement learning algorithms in this guide, so you can explore some basic algorithms:

• If we know everything about the environment (and can model it completely), we can use model-based algorithms like policy iteration and value iteration.

• However, in most cases, we may not be able to model the environment completely. In this case, you can look at model-free algorithms such as Q-learning which optimizes state-action pairs.

If you’re looking to further your understanding of reinforcement learning, David Silver’s reinforcement learning lectures on YouTube and Hugging Face’s Deep Reinforcement Learning Course are some good resources to look at.
Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she's working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.

More On This Topic

• How to Make Sense of the Reinforcement Learning Agents?
• eBook: How to use third-party data to make smarter decisions
• eBook: 101 Ways to Use Third-Party Data to Make Smarter Decisions
• AWS Webinar: How are data-driven companies using ESG and sustainability…
• Getting Started with Reinforcement Learning
• Reinforcement Learning for Newbies

SpeedyBrand uses generative AI to create SEO-optimized content Kyle Wiggers 9 hours

While working at Vetan, a startup helping small- and medium-sized (SMBs) businesses manage employee payroll, Jatin Mehta realized that SMBs often lack the tools to thrive online with organic sales. The cost to hire an agency is beyond their budget, and generating content is costly — both in terms of time and money.

“Having a great online presence is critical for e-commerce stores like Shopify and Woo, as online traffic is the bread and butter of their business,” Mehta told TechCrunch in an email interview. “But existing content marketing solutions are not complete and require search engine optimization (SEO) expertise. Businesses need multiple SEO tools and to hire content strategists, writers and agencies to outsource their content marketing work.”

So along with Ranti Dev Sharma and Ayush Jasuja, Mehta co-founded SpeedyBrand, which aims to bring “high-quality,” affordable SEO content to SMBs using generative AI. SpeedyBrand today announced that it raised $2.5 million in a funding round led by GV (Google’s venture arm) and Y Combinator that values the company at $15 million post-money.

SpeedyBrand’s platform, powered by generative AI, can create custom SEO-optimized content — including websites and social media posts — for brands. Brands first choose a topic. Then they have the platform generate text and suggest images that might be appropriate for the type of content they’re generating.

From SpeedyBrand’s dashboard, generated content can be edited and further customized before being published to various channels. An analytics component allows brands to track the performance of the content once it’s live.

Image Credits: SpeedyBrand

“The economic slowdown requires cost-effective marketing solutions,” Mehta said. “Speedy is well-positioned to help businesses with an affordable solution.”

But there’s reason to be wary of the tech.

For one, generative AI, no matter how good, can — and does — run amok. Thanks to a phenomenon known as “hallucination,” AI models sometimes confidently make up facts. And, as a result of biases and other imbalances in their training data, text-generating AI can spew toxic, wildly offensive remarks.

In another potential problem for brands, generative AI has been shown to plagiarize copyrighted work. One study found that an indirect predecessor to ChatGPT, GPT-2, can be prompted to “copy and paste” entire paragraphs from its training data.

Then there’s the matter of generative AI spamming up the internet. As The Verge’s James Vincent wrote in a recent piece, generative AI models are changing the economy of the web — making it cheaper and easier to generate lower-quality content. Newsguard, a company that provides tools for vetting news sources, has exposed hundreds of ad-supported sites with generic-sounding names featuring misinformation created with generative AI.

Mehta asserts that SpeedyBrand isn’t a content mill — and that it takes steps to mitigate any toxic content that the platform’s AI might generate. SpeedyBrand’s AI can be personalized to brand tone and generates provably “plagiarism-free” content, he claims, incorporating feedback from content edits to improve future output.

To what extent is all this true? It’s tough to say without a third-party audit. But brands, no doubt eager to jump on the generative AI train, appear to be embracing SpeedyBrand.

The company, which has a six-person team, has around 50 paying customers and over 1,000 users. Annual recurring revenue stands at $100,000, and Mehta anticipates that it’ll reach $1 million in the next year.

That’s impressive considering the competition. SpeedyBrand faces off against Typeface, which recently emerged from stealth with $65 million in venture capital. Startups like Movio, Copysmith, Copy.ai, Sellscale, Jasper, Omneky and Regie.ai, too, are using generative AI to create (ostensibly) better marketing copy, imagery and even video for ads, websites and emails.

It’s a large and growing market. Statista reports that 87% of current AI adopters are already using, or considering using, AI for improving their email marketing. Another report projects that the market for generative AI will be worth more than $110 billion by 2030.

Given that almost half of SMB owners handle content marketing themselves, there’s even stronger incentive within that cohort to adopt tools that could — at least on the surface — save time, money and massive headache.

“Speedy saves a company’s marketing workforce hours of marketing hustle — from strategy to content generation and then posting,” Mehta said. “Speedy gives them and their team hours back every day so they can focus on the core of their business.”

With the proceeds from the funding round, SpeedyBrand plans to roll out additional tools for text and image generation.