Рубрика: AI

Voice changing software is a type of AI application that allows users to modify their voice in real-time or alter pre-recorded audio. These software solutions provide different effects, such as changing the pitch or speed of the voice, or transforming the user's voice to sound like someone or something else, such as a famous celebrity, cartoon character, robot, or different genders and ages.

These tools are widely used in different industries and applications, such as video gaming (where players may wish to mask their identity or role-play different characters), multimedia production, telecommunication, podcasts, and many other areas where voice manipulation can enhance the user experience or support creative processes.

There are many voice changing software available on the market, each offering a unique set of features and capabilities.

1. Hitpaw Voice Changer

This easy to use AI tool is considered to be one of the best applications for Gamers, Streamers, YouTubers, and Meetings. Gamers enjoy it because it enables you to easily sound like a favorite character in a video game, entrepreneurs enjoy it because it can make them sound authoritative.

Unleash your creativity, change voices with endless possibilities. Be a robot, demon, chipmunk, woman, man, ghostface, or anime actor, HitPaw Voice Changer offers a huge number of voice-changing effects to meet your needs and give you more options to act like the character you want.

• Change your voice with various voice-changing effect in real-time
• Integrate perfectly with all popular games and programs
• Perfect voice modifier for gameplay, content creator, Vtuber, or live streamer
• Remove noise and echo while changing voices
• Change voice effortlessly with high quality

2. Murf

One of the most popular and impressive AI voice changers on the market is Murf, which enables anyone to convert text to speech, voice-overs, and dictation. It is especially useful for product developers, podcasters, educators, and those in the business world.

Murf creates natural voices in a very short amount of time and with minimal effort needed. They can then be used in nearly any sector. With a library consisting of over 110 voices in 15 different languages, Murf has a wide range of uses.

Here are some of the main features of Murf:

• Large library of voices and languages
• Expressive emotional speaking styles
• Pitch and fine-tune voice tones
• Audio and text input support

3. Synthesys

Synthesis is one of the most popular and powerful AI voice changers and generators, it enables anyone to produce a professional AI voiceover or AI video in a few clicks.

This platform is on the leading edge of developing algorithms for text to voiceover and videos for commercial use. Imagine being able to enhance your website explainer videos or product tutorials in a matter of minutes with the aid of a natural human voice. Synthesys Text-to-Speech (TTS) and Synthesys Text-to-Video (TTV) technology transform your script into vibrant and dynamic media presentations.

A myriad of features is offered including:

• Choose from a large library of professional voices: 34 Female, 35 Male
• Create and sell unlimited voiceovers for any purpose
• Extremely lifelike voices unlike competing platforms
• The choice of emphasizing specific words to be able to express a range of emotions like happiness, excitement, sadness, etc.
• Add pauses when the user wants to give the voiceovers an even more human feel.
• Preview mode to see results quickly and apply changes without losing time rendering.
• Use for sales videos, letters, animations, explainers, social media, TV commercials, podcasts, and more.

4. Voice Over by Speechify

Speechify can turn text in any format into natural-sounding speech. Based on the web, the platform can take PDFs, emails, docs, or articles and turn it into audio that can be listened to instead of read. The tool also enables you to adjust the reading speed, and it has over 200 natural-sounding voices to select from.

The software is intelligent and can identify more than 15 different languages when processing text, and it can seamlessly convert scanned printed text into clearly audible audio.

Here are some of the top features of Speechify:

• Web-based with Chrome and Safari extensions
• Over 200+ high-quality voices voices to select from
• 20+ languages & accents
• Granular controls on the pitch, tone and speed
• Commercial usage rights
• Custom soundtracks

30% discount code: SPEECHIFYPARTNER30

5. Altered

Altered Studio is a next-generation audio editor that integrates multiple voice AI technologies into a single user friendly application. It runs online as well as locally on Windows and Mac using local computing resources.

The Voice AI tools can help you with your dubbing workflow. Transcribe, voice over, text-to-speech and Translations.
‍
Altered Studio provides a unique speech-to-speech, performance-to-performance Speech Synthesis technology that pushes the boundaries of what can be done.

One option of the unique technology allows you to modify your voice to a custom voice. You can also transcribe, add voice-over with text-to-Speech and translate audio files.

Main features include:

• Create a specific voice. It might be the voice of a famous actor, a captivating voice-talent, a friend or a grandparent.
• Use life-like Text-To-Speech to add Voice-Over to your content in 70+ languages.
• From personal audio notes to long meetings conversations, quick and accurate transcription is just one click away.
  
• Google Drive integration, easily work from anywhere and easily share files.
• Voice Editor can record directly from the browser through the microphone or any other recording device.
• Import and export your files in many different formats, lossless and raw.
• Spectrogram and spectrum visualisation are one click away, for detailed frequency analysis.

6. Lovo.ai

Lovo.ai is an award-winning AI-based voice generator and text-to-speech platform that also be used as a voice changer. It is one of the most robust and easiest platform to use that produces voices that resemble the real human voice.

Lovo.ai has provided a wide range of voices, servicing several industries, including entertainment, banking, education, gaming, documentary, news, etc., by continuously refining its voice synthesis models. Because of this, Lovo.ai has garnered a lot of interest from esteemed organizations on a global scale, making them stand out as innovators in the voice synthesis sector.

LOVO has recently launched Genny, a next-gen AI voice generator equipped with text-to-speech and video editing capabilities. It can produce human-like voices with stunning quality and content creators can simultaneously edit their video.

Genny lets you choose from over 500 AI voices in 20+ emotions and 150+ languages. Voices are professional grade voices that sound human-like and realistic. You can use the pronunciation editor, emphasis, speed and pitch control to perfect your speech and customize how you want it to sound.

Features:

• World's largest library of voices of over 500+ AI voices
• Granular control for professional producers using pronunciation editor, emphasis, and pitch control.
• Video editing capabilities that allow you to edit videos simultaneously while generating voiceovers.
• Resource database of non-verbal interjections, sound effects, royalty free music, stock photos and videos

With 150+ languages available, content can be localized with the click of a button.

7. Listnr

Another top listing for voice changers, is Listnr, an AI text-to-speech voice generator tool that converts text-to-speech in various formats, such as genre selection, pauses, accent selection, and more. One of the best features of Listnr is that it enables you to get your own customizable audio player embed, which can be embedded into your blog as an audio version.

Listnr is personalized to each individual listener's routine and preferences. It is also a great tool for creating, managing, and publishing podcasts. Whether you are a commercial or freelance podcaster, Listnr can help you monetize your content through advertising. You can use the AI voice generator tool to distribute and convert audio with commercial broadcasting rights on the world's biggest platforms like Spotify, Apple, and Google Podcasts. When it comes to podcasting, Listnr supports more than 17 languages, and the AI technology can convert blog posts into several different languages and dialects.

Listnr also helps you improve conversation rates by enabling the option to read-listen and watch-listen for users.

Here are some of the main features of Listnr:

• Embed customizable audio player
• Personalized to each listener
• Improves conversion rates
• AI voice-overs for YouTube, blog posts, and audiobooks
• Audio analytics

8. Play.ht

A powerful AI text-to-speech generator, Play.ht relies on AI to generate audio and voices from IBM, Microsoft, Amazon, and Google. The tool is especially useful for converting text into natural voices, and it allows you to download the voice-over as MP3 and WAV files.

With Play.ht, you can choose a voice type and either import and type text, which the tool will instantly convert into a natural human voice. The audio can then be enhanced with SSML tags, speech styles, and pronunciations.

Play.ht is used by major brands like Verizon and Comcast.

Here are some of the main features of Play.ht:

• Convert blog posts to audio
• Integrate real-time voice synthesis
• Over 570 accents and voices
• Realistic voice-overs for podcasts, videos, e-learning, and more

9. Deepbrain AI

The Deepbrain AI tool offers the ability to easily create AI-generated videos using basic text instantly quickly and easily. Simply prepare your script and use the Text-to-Speech feature to receive your first AI video in 5 minutes or less.

There are 3 quick steps to get started they are as following:

1. First, create a new project. You can start with your own PPT template or choose one of the starter templates.
2. You can manually type in or copy and paste your script. Contents of your uploaded PPT will be entered in automatically.
3. Once you select the appropriate language and AI model and finish editing, you can export the synthesized video.

This tool offers the following benefits:

• Easy find a custom-made AI avatar that best fits your brand.
• The Intuitive tool is designed to be super easy to use for beginners.
• Offers significant time savings in video preparation, filming, and editing.
• Cost-saving in the entire video production process.

10. Sonantic

Sonantic has risen in popularity since it was used to help actor Val Kilmer reclaim his voice with a synthetic voice replica. The easy-to-use AI tool is popular in the entertainment industry since it enables lively voice expressions.

The tool allows you to change the tone of the speech generated, with tones like happy, sad, or angry. You can also customize the level of emotion through adjustments, and it works by simply copying and pasting a written text into the editor before waiting for it to be converted to audio.

These reasons are why Sonantic has been used for animations, films, and games.

Here are some of the top features of Sonantic:

• Human-like voice generator
• Emotion adjustments
• Voice parameters
• Voice projects like Shouts or Fear

This Week in AI: Musk Launches xAI, Anthropic Takes Claude 2 Public, Did OpenAI Nerf GPT-4? July 13, 2023 by Jaime Hampton

The artificial intelligence space has been characteristically busy this week. Elon Musk is officially throwing his hat into the AI ring, Anthropic has released a more powerful version of its Claude chatbot, and OpenAI is under an FTC investigation for data privacy concerns while rumors abound that the company has redesigned GPT-4, lowering its performance.

Elon Musk Officially Launches an AI Startup

Tesla and SpaceX CEO Elon Musk debuted his new AI company dubbed xAI. The company’s goal is to “understand the true nature of the universe,” its website says. There will presumably be more information about the new AI firm during a live event on Twitter Spaces scheduled for Friday.

Elon Musk at VivaTech in Paris last month. (Source: Frederic Legrand-COMEO/Shutterstock)

The xAI site also has team members listed and says they are alumni of DeepMind, OpenAI, Google Research, Microsoft Research, Twitter, and Tesla and have worked on projects including DeepMind’s AlphaCode and OpenAI’s GPT-3.5 and GPT-4 models.

A news report from April suggested Musk was launching an AI startup after he incorporated xAI in Nevada on March 9 and had secured “thousands” of Nvidia GPUs. Musk previously shared details of plans for an AI tool called TruthGPT during an interview on Fox News in April where he claimed companies like OpenAI are creating politically correct systems, positioning his AI as an anti-woke alternative.

Executive Director of the Center for AI Safety Dan Hendrycks has been tapped as an advisor for Musk’s new startup. The Center for AI Safety authored a May letter urging prompt global action in mitigating the risk of extinction from AI that many AI ethicists saw as a distraction from current problems algorithmic bias is causing for marginalized groups.

Anthropic Takes Claude 2 to the Masses

There’s a new chatbot available to test out: Anthropic’s Claude 2. The company released a blog post Tuesday announcing this new model, boasting its improved performance and longer responses, as well as API access and a new public-facing beta website, claude.ai.

Anthropic has also increased the length of Claude’s input and output. Users can now input up to 100,000 tokens in each prompt, which adds up to hundreds of pages, the company claims. Claude can now write longer responses up to a few thousand tokens, as well.

The company said it has made improvements from previous modes on coding, math, and reasoning. Claude 2 scored 71.2% on the CodexHumanEval python coding test, compared to the first Claude’s score of 56%. On GSM8k, a large set of grade-school math problems, Claude 2 scored 88.0% up from 85.2%.

Safety improvements were also a consideration when training Claude 2, as the company noted it has been iterating the model to be more harmless and more difficult to prompt for offensive or dangerous output. “We have an internal red-teaming evaluation that scores our models on a large representative set of harmful prompts, using an automated test while we also regularly check the results manually. In this evaluation, Claude 2 was 2x better at giving harmless responses compared to Claude 1.3,” the company wrote.

Anthropic said it will be rolling out a roadmap of capability improvements for Claude 2 to be deployed over the coming months.

Did OpenAI Nerf GPT-4? Rumors Swirl Amid FTC Investigation

Rumors have been swirling around the internet that OpenAI has nerfed the performance of GPT-4, its largest and most capable model available to the public. Users on Twitter and the OpenAI developer forum were calling the model “lazier” and “dumber” after it appeared to be giving faster but less accurate answers compared to the slower but more precise responses it initially gave.

An Insider report says the industry insiders are questioning whether OpenAI has redesigned its GPT-4 model. Some have said the company could be creating a group of smaller GPT-4 models that could act as one model and be less expensive to run. This approach is called a Mixture of Experts, or MOE, where smaller expert models are trained on specific tasks and subject areas. When asked a question, GPT-4 would know which model to query and might send a query to more than one of these expert models and mash up the results. OpenAI did not respond to Insider’s request for comment on this matter.

Whether or not GPT-4 is actually “dumber,” OpenAI is also in the news this week due to a new investigation opened by the Federal Trade Commission.

The FTC is looking into whether ChatGPT has harmed consumers through its collection of data and publication of false information on individuals. The agency sent a 20-page letter to OpenAI this week with dozens of questions about how the startup trains its models and how it governs personal data.

The letter detailed how the FTC is examining whether OpenAI “engaged in unfair or deceptive privacy or data security practices or engaged in unfair or deceptive practices relating to risks of harm to consumers.”

“It is very disappointing to see the FTC's request start with a leak and does not help build trust,” wrote OpenAI CEO Sam Altman in a tweet. “That said, it’s super important to us that our technology is safe and pro-consumer, and we are confident we follow the law. Of course we will work with the FTC.”

Altman went on to say the company built GPT-4 on top of years of safety research and spent six months aligning it before release. “We protect user privacy and design our systems to learn about the world, not private individuals,” he said.

Related

Generative AI has so many different applications that can help people advance their careers including coding, writing, resume assistance, and more.

However, to take advantage of this help, you need to know how to use it first, which is an obstacle for many.

Also: Generative AI is coming for your job. Here are 4 reasons to be excited

Salesforce surveyed 4,000 desk workers regarding their feelings toward AI, and the results were overwhelmingly positive.

Out of the respondents in the survey, 54% of workers believe generative AI will advance their careers. 65% believe that generative AI will allow them to focus on more strategic work.

Despite the optimistic look, the workers felt like they don't have the right skill set to properly use generative AI or take full advantage of the new technology.

Also: How to use ChatGPT: Everything you need to know

A whopping 62% of respondents said they don't have the skills to effectively and safely use generative AI. Over half of the respondents (53%) also said they didn't know how to get the most value out of generative AI.

Particularly, 43% of the respondents felt they didn't know how to use generative AI while keeping first-party data secure.

This is a common concern with using generative AI in the workplace because generative AI models typically use user data and input to further train their models, which puts the security of the data you are inputting in question.

The feeling of lacking the proper skillset is also shared by a majority of business leaders (70%) who believe that their teams don't have the skills to safely use generative AI. However, the business leaders could be to blame for the lack of team members' preparedness.

Also: '5 ways to explore the use of generative AI at work

Two-thirds of the desk workers surveyed said they expect their employers to provide learning opportunities, but their employers haven't offered training on the technology.

With proper understanding and usage, workers and businesses can increase their productivity and get assistance in meeting their end goals. However, this requires training and education on the topic.

Artificial Intelligence

In the ever-evolving world of artificial intelligence, Anthropic, a start-up co-created by ex-OpenAI leaders, has taken another step towards industry dominance. They recently announced the debut of their AI chatbot, Claude 2, marking a significant milestone in the firm's journey to establish itself alongside AI titans like OpenAI and Google.

The birth of Anthropic in 2021 served as a precursor for the current rapid advancements in AI chatbots. Their latest progeny, Claude 2, is a testament to their dedicated focus on the evolution of this technology. It's a successor to Claude 1.3, Anthropic's initial commercial model, and was launched in beta in the U.S. and U.K. The pricing remains untouched, still around $0.0465 for 1,000 words, and has attracted various businesses like Jasper and Sourcegraph to start piloting Claude 2.

Anthropic is the brainchild of former OpenAI research executives and has enjoyed the backing of significant corporations like Google, Salesforce, and Zoom. A host of businesses like Slack, Notion, and Quora have become testing grounds for its AI models over the past two months. The start-up has successfully garnered interest from over 350,000 individuals, eagerly waiting to gain access to Claude's application programming interface and its consumer offering.

Anthropic's co-founders, Daniela and Dario Amodei, have stressed the importance of robust safety in Claude's development. According to them, Claude 2 is the safest iteration yet, and they are thrilled about its potential impact on both the business and consumer world. Currently limited to users in the U.S. and U.K., Claude 2's availability is set to expand in the near future.

Claude 2 – AI Evolution in Practice

Much like its predecessor, Claude 2 demonstrates an impressive ability to search across documents, summarize, write, code, and answer topic-specific questions. However, Anthropic asserts that Claude 2 surpasses its predecessor in several key areas. For example, Claude 2 outperforms Claude 1.3 on the multiple-choice section of the bar exam and the U.S. Medical Licensing Exam. Its programming ability has also improved, demonstrated by its superior score on the Codex Human Level Python coding test.

Claude 2 exhibits improved capability in mathematics, scoring higher on the GSM8K collection of grade-school-level problems. Anthropic has focused on enhancing Claude 2's reasoning and self-awareness, making it more competent at processing multi-step instructions and recognizing its limitations.

The introduction of more recent data for Claude 2's training, including a mix of web content, licensed datasets from third parties, and voluntarily-supplied user data, has likely contributed to these performance enhancements. Despite the vast improvements, the underlying architecture of Claude 1.3 and Claude 2 remains similar. The latter is viewed as a refined version of its predecessor, rather than a completely new invention.

A notable attribute of Claude 2 is its large context window of 100,000 tokens, matching Claude 1.3's capacity. This enables Claude 2 to generate and ingest a significantly larger volume of text, allowing it to analyze approximately 75,000 words and produce around 3,125 words.

However, Claude 2 is not without its limitations. It still grapples with the problem of hallucination, where responses can be irrelevant, nonsensical, or factually incorrect. It can also generate toxic text, which reflects biases in its training data. Despite these limitations, Claude 2 is said to be twice as likely to give harmless responses compared to Claude 1.3, based on an internal evaluation.

Anthropic suggests refraining from using Claude 2 in scenarios involving physical or mental health and well-being or high-stakes situations where a wrong answer could cause harm. Nevertheless, they are hopeful about the chatbot's potential and are committed to further improving its performance and safety.

Implications and Future Prospects

The introduction of Claude 2 signifies more than just the birth of a new AI chatbot. It stands as an emblem of Anthropic's ambitious pursuit of a self-teaching AI algorithm. This ambition, if realized, could ignite a revolution in various sectors, from virtual assistance to content generation, posing significant implications for the AI industry.

The AI industry is closely observing Anthropic's progress, with competitors such as OpenAI, Cohere, and AI21 Labs all developing their AI systems. Claude 2's introduction underscores a larger industry trend towards more sophisticated and user-friendly AI models. It is poised to drive a new wave of innovation and improvements in AI technology as it competes with other AI chatbots in the market.

A New Era of AI: Charting the Course of Future Innovations

The introduction of Claude 2 by Anthropic is a defining moment that is not just significant to the company, but is emblematic of a broader shift within the field of AI. This new model ushers in a fresh era of AI advancement, where the line between human and artificial intelligence continues to blur. Claude 2’s improved capabilities exemplify significant strides taken in AI technology, offering a peek into the future of how artificial and human interactions might evolve.

The launch of Claude 2 also sheds light on the growing complexity of ethical issues related to AI. As AI models become more sophisticated, ethical considerations around their development and usage become increasingly critical. These range from privacy concerns and data security to the biases embedded in AI and how it might influence our society. It is now more vital than ever for AI developers to work alongside ethicists, policymakers, and society at large to ensure these considerations are thoroughly addressed.

In the competitive landscape of AI chatbots, Claude 2, along with its counterparts, is likely to be a significant catalyst for innovation and technological progress. The competition between AI chatbots could be likened to an intellectual arms race, pushing the boundaries of AI and leading to the development of more sophisticated, user-friendly, and reliable models. This competition is not just about who has the most advanced AI, but who can utilize it effectively and responsibly in real-world applications.

The development of Claude 2 and other similar models promises to have wide-ranging implications for a multitude of sectors. This goes beyond the realm of virtual assistance and content generation, extending to industries such as education, healthcare, and even entertainment. These AI chatbots could potentially revolutionize the way we learn, communicate, and interact with technology, paving the way for a new phase of digital evolution.

Looking at Anthropic's strategy for Claude 2 and their larger objective of creating a “next-gen algorithm for AI self-teaching” offers a glimpse into the company’s ambitious vision. The successful achievement of these goals could indeed instigate a seismic shift in the AI industry, bringing us closer to a future where AI is a seamless part of our daily lives.

However, such grand ambitions don't come without their fair share of challenges. From technical hurdles and data privacy issues to societal acceptance and regulatory landscapes, there are multiple factors that could impact the realization of these plans. It will indeed be intriguing to follow Anthropic’s journey, to see how they maneuver around these challenges, and how their vision shapes the future of Claude 2 and the broader AI industry.

The unveiling of Claude 2 is more than just another product launch; it represents the promise of what AI can achieve, the responsibility that comes with such advancements, and the start of an exciting new chapter in the story of AI. As we stand on the precipice of this new era, it's an opportune time to not only celebrate the technological marvel that AI represents but also to engage in a thoughtful conversation about its implications for our society.

NVIDIA has cashed in on the AI rush, leaving Intel scrambling for the second place. However, when compared to its closest competitor AMD, Intel finds itself in a difficult position. After shuttering its server business earlier this year, team blue recently shut its mini PC business as well.

According to reports, Intel has ceased investment in its NUC (Next Unit of Computing) line of mini PC products. These small form factor NUC PCs saw use all over the enterprise sector, but Intel has now handed them over to its manufacturing partners. This move is in line with Intel’s bid to concentrate their resources on finding a foothold into the exploding AI compute market.

Since its acquisition of AI chipmaker Habana Labs in 2019 for $2 billion, Intel has tried hard to break into the AI compute market. As NVIDIA’s software moat grows stronger, Intel is left with an ever-shrinking market share. Will redoubling its efforts into AI hardware give it the boost it needs?

Going up against a giant

While backing out of the mini PC market is mainly driven by a reluctance to compete with its OEMs, it also comes at a time when the company is cutting costs across the board. Intel has also targeted $10 billion in cost cutting for 2023 as a part of the IDM 2.0 strategy which it adopted in 2021. Reportedly, even the server business was sold in a bid to focus on its new strategy.

This strategy doesn’t seem to be working for the company, as Intel’s stock has dipped heavily, dropping 69% of its value from a high in 2021. While its competitors AMD and NVIDIA saw huge growth in the same time period due to the boom in AI applications, Intel’s AI offerings have fallen to the wayside. Even comments from Intel’s management echo this sentiment. Eitan Medina, Habana’s chief business officer, said in 2020, “We have to realise that we’re starting from zero and NVIDIA is 100%. The uphill battle or the process of taking market share has to go through convincing end developers to try it out.”

Even as the company is fighting an uphill battle against the green giant, its hardware seems to be doing well against NVIDIA’s lineup. When testing the Habana Gaudi 2 accelerator against NVIDIA’s A100 chips, researchers at Hugging Face found that Intel’s offerings beat out NVIDIA’s chips in common AI workloads. In a gauntlet that tested the chips’ BERT pre-training, text-to-image on Stable Diffusion, and fine-tuning T5-3B, Habana was upto 2.8x faster than NVIDIA A100 in these workloads.

Even with this performance bump, Habana chips are facing the heat against NVIDIA’s Grace Hopper chips. This new line thoroughly beats Intel’s AI accelerators while ironically using Intel Xeon CPUs to do so. However, Habana is still capable enough to compete in the industry, hardware-wise. Regardless, Intel falls short in an important facet of AI compute: software. Intel’s One API offering works well for basic AI workloads, but NVIDIA’s software stack services nine different verticals, from enterprise AI to Omniverse.

Strategic realignment needed

As part of the cost-cutting measures, Intel has also moved towards a disaggregated CPU architecture. First seen with the much-delayed Meteor Lake line of chips, Intel has moved to a chiplet design to not only cut down on manufacturing costs, but to better compete with its peers.

Using the AI accelerator know-how from Habana, Intel has been slowly eating up the on-device AI compute market. While Gaudi is competing with NVIDIA’s heaviest cards like Grace Hopper and the H100, the new Meteor Lake chips will compete with NVIDIA GPUs in laptops. Intel’s integrated offerings will offer a low-power alternative to NVIDIA’s more power-hungry GPUs in everyday laptops, a strategy that has already seen moderate success in Apple’s M-series chips.

At a glance, this market seems to be a better field to approach NVIDIA from, especially in terms of building the software stack. Let’s take CUDA for example, which was built over decades through constant feedback by developers who used NVIDIA’s chips as accelerators. While Intel doesn’t have as big a timeframe, shipping AI chips on laptops can vastly expedite the creation of a robust software stack.

The third generation of Gaudi chips is on the horizon, and will take on competing offerings from other specialised chipmakers like Graphcore, SambaNova, Tenstorrent, and Cerebras. While NVIDIA continues to enjoy its market dominance thanks to its strong software stack, it seems that Intel will slowly build up momentum as AI accelerators begin to become cheaper to manufacture for Team Blue.

Combined with its strategy of ‘innovate and integrate’, Intel could soon start offering end-to-end AI solutions which include CPUs, GPUs, and AI accelerators. When combined with a software stack, this might finally make Intel relevant as a third player in the market alongside NVIDIA and AMD.

The post Intel is Giving AI Hardware Everything It’s Got appeared first on Analytics India Magazine.

Hello, data enthusiast! In this article let’s discuss “Data Modelling” right from the traditional and classical ways and aligning to today’s digital way, especially for analytics and advanced analytics. Yes! Of course, last 40+ years we all worked for OLTP, and followed by we started focusing on OLAP.

After cloud ear come into the picture the Data become very crazy level and every industry started zooming them and looking at different levels and perspectives. So, Big Data, Data Platform, Data Analytics, Data Science, and many more buzzwords are popping out of the window

“This is the technique is used to characterize the data and help us to know how it is stored in the available tables and alongside with other table and association between them”

image designed by author shanthababu

Before getting into Data Modelling, let’s understand the few terminologies which is the ground for DATA architecting and modeling, which are nothing but OLTP and OLAP.

What is OLTP?

OLTP is nothing but Online Transaction Processing, and we can call this database workload used for transactional systems, which we use to play around with DDL, DML, and DCL.

OLAP is Online Analytical Processing, database workloads are used for modern data warehousing systems, in which we use to play around SELECT queries with simple or complex queries by filtering, grouping, aggregating, and portioning a large data set quickly for reporting/visualization for Data Analyst and Dataset for Data Scientists for specific reasons.

OLTP	OLAP
Focus	Day-To-Day Operations	Analysis and Analytics
DB Design	Application-Specific	Business Driven
Nature Of the Data	Current [RDBMS]	Historical and Dimensional
DB Size	In GB	In TB

What is Data Modelling

• Data modelling is the well-defined process of creating a data model to store the data in a database or Modren Data warehouse (DWH) system depending on the requirements and focused on OLAP on the cloud system.
• Always this is a conceptual interpretation of Data objects for the Applications or Products.
• This is specifically associated with the different data objects, and the business rules derived to achieve the goals
• It helps in the visual description of data and requires business rules, governing compliances, and government policies on the data like GDPR, PII and etc.,
• It ensures stability in naming conventions, default values, semantics, and security while ensuring the quality of the data.

Data Model

This defines the abstract model that organizes the Description, Semantics, and Consistency constraints of data.

What is really the Data Model underlines on

• What data need for DWH?
• How it should be organized in the DWH system,

DWH Data Model is like an architect’s building plan, which helps to build conceptual models and set a relationship between data-item, let’s say Dimension and Fact, and how they are linked together.

How we could implement DWH Data Modelling Techniques

• Entity-Relationship (E-R) Model
• UML (Unified-Modelling Language)

Consideration factors for Data Modelling

While deriving the data model, there are several factors that need to be considered, these factors vary based on the different stages of the Data Lifecycle.

• Scope of the Business: There are several departments and diverse business functions around.
• ACID property of the data during transformation and storage.
• Feasibility of the data granularity levels of filtering, aggregation, slicing, and dicing
• Key features of Modern Data Warehouse
• Starts with logical modelling across multi-platforms and an extensive-architecture approach, its enhanced performance, and scalability.
• Serving data for all types and different categories of consumers
  • [Data Scientist, Data Analysts, Downstream applications, API-based system, Data Sharing systems]
• Highly flexible deployment and decoupling approach for cost-effectiveness.
• Well-defined Data Governance model to support quality, visibility, availability security
• Streamlined Master Data Management and Data Catalog and Curation to support functionally and technically.
• Perfect monitoring and tracking of the Data Linage from Source into Serving layer
• Ability to facilitate Batch, Real-Time analysis, and Lambda process of high-velocity, verity, and veracity data.
• Supports Analytics and Advanced Analytics components.
• Agile Delivery approach from Data modelling and delivering aspects to satisfy, their business model.
• Excellent-Hybrid Integration with multiple cloud service providers and maximize the benefits for the customer

Why the modern DWH be necessary for us?

Yes! The Modern Data Warehouse systems solve many problems in business challenges

• Data Availability
  Data sources divided across organizations – Certainly, the Modern DWH system allows us to bring the data faster into our table in the form of different ranges and helps to analyze across the organizations, divisions, and behavior. It keeps getting the agility model and stimulates more and more.
• Data Storage
  Data Lakes – In the modern cloud the storage and computation are very flexible and extendable ways, instead of storing in hierarchical files and folders as we used in a traditional data warehouse, a data lake is an extensive repository that holds a massive amount of raw data, and you can store in its native format until required for processing layer.
• Data Maintainability
  As you know that we can’t maintain the historical data in a normal database like RDBMS, there were lots of challenges with respect to querying or fetching the data is a tedious process. So we have to build the DWH with Facts and Dimensions, and we could use the data for data perspective very easily and quickly.
• IoT/ Streaming Data
  Since we’re in the internet world the data flowing across the different applications and Internet of Things data has transformed and based on the business scenarios, needs, etc.

So far, we have discussed the concepts around the Modern DWH system, Let’s move on to data modelling components and techniques.

Data Model evaluation

Generally, before building the model, each table would undergo the below stages, conceptual, logical, and physical, so exactly in the last stage only we would realize the model as accepted by the business.

Source: image designed by author shanthababu

Multi-dimensional Data Modelling components

The main components are Fact and Dimension tables are the main two tables that are used when designing a data warehouse. The fact table contains the measures of columns and a special key called surrogate, that link to the dimensions tables.

Facts: To define FACTS in one word that is nothing but Measures

It can be measured attributes of the fields, it can be Quantitatively Measured, and in Numerical Quantities. Generally, it would be a number of orders received and products sold.

Dimensions: It has the attributes and basically “Category Values” or “Descriptive Definition” would be the Product Name, Description, Category, and so on.

Source: image designed by author shanthababu

Modeling techniques

For most of the scenarios, while developing the data modelling for DWH, we use to follow the Star Schema or Snowflake Schema, or Kimball’s Dimensional Data Modelling.

Source: image designed by author shanthababu

Star Schema: This is the most common technique and basic modelling type and is easy to understand. In which Fact table is connected with other all Dimension tables and considerably accepted architectural model and used to develop DWH and Data marts. Each dimension table in the star schema has a Primary-Key and which is related to a Foreign-Key. In the Fact table. joining the tables and querying a little complex and performance a bit slow.

The representation of this model seems like a star with the Fact table at the center and dimensions-tables connecting from all other sides of it, constructing a STAR-like model

Source: image designed by author shanthababu

Snowflake Schema: This is an extension of the Star Schema with little modification and reduced load and improved performance. here the dimensions tables are normalized into multiple related tables as sub-dimension. So, it minimizes data redundancy. Apparently, it has multiple levels of joins which leads to less query complexity and ultimately improves query performance.

Tables are arranged logically and a many-to-one relationship hierarchy structure and it is resembling a SNOWFLAKE-like pattern. It has more joins between dimension tables, so performance issues might be in place, which leads to the slow query processing times for data retravel.

Source: image designed by author shanthababu

Let’s do a quick comparison of Star & Snowflake Schema

Star Schema	Snowflake Schema
Simplified design and easy to understand	Complex design and a little difficult to understand
Top-Down model	Bottom-Up model
Required more space	Less Space
The fact table is surrounded by Dimension tables	The fact table is connected with dimension tables and dimension tables are connected with sub-dimension tables in normalized
Low query complexity	Complex query complexity
Not normalized, so there is a lesser number of relationships and foreign keys.	Normalized, so required number of foreign keys and the well-defined relationship between tables
Since not normalized, a High volume of data redundancy	Since normalized, Low volume data redundancy.
Fast query execution time	Low query execution time due to more joins
One Dimensional	Multidimensional

Everything is fine with the star schema, as we understood that this is Flexible, Extensible, and many more. But not answered business process and questions from DWH.

Kimball’s answer to below dimensional data modelling.

• The business process to a model – Keeping customer model, product model
• ATOMIC model – Depth of data level stored in the fact table in the concrete ATOMIC model so, we can’t split further for any analysis and not required too
• Building fact tables – designing the fact tables with a strong set of dimensions with all possible categories.
• Numeric facts – Identifying the most important numeric measures use to store at the fact table layer
• The part of the Data Analytics environment where structured data is broken down into low-level components and integrated with other components in preparation for exposure to data consumers

Then why do we need Kimball’s Approach? Obviously, we need them to Expedite the business value and Performance enhancement.

Expedite the business value: When you want to speed to business value, the data needs to be denormalized, so that BI teams can deliver to the business quickly and reliably and improve analytical workloads and performance.

• Bottom-up approach. the DWH is provisioned from the collection of DataMart.
• The Datamart is cooked from OLTP systems that are usually RDBMS and well-tuned with 3NF
• Here the DWH is central to the core model and de-normalized star schema.

Source: image designed by author shanthababu

Let’s quickly go through Inmon DWH Modelling, it follows a top-down approach. In this model, OLTP systems are a data source for DWH and play as a central repository of data in 3NF. Followed by this Datamart is plugged in and in 3NF. Comparatively with Kimball’s model, this Inmon is not that great option while dealing with BI and AI and data provisioning.

Kimball	Inmon
De-normalized data model.	Normalized data model.
Bottom-Up Approach	Top-Down Approach
Data Integration mainly focuses on Individual business-area(s).	Data Integration focuses on Enterprise specific
Data source systems are highly stable since the Datamart stage will take care of the challenges	Data source systems have a high rate of change Since DWH is plugged with the Data source directly.
Building time-lime takes less time.	Little complex and required more time.
Involves an iterative mode and is very cost-effective.	Building the blocks might consume a high cost.
Functional and Business knowledge is enough to build the model.	Understanding of Database, Table, Columns and key relationship knowledge is required to build the model.
Challenge in maintenance	Comparatively easy to maintenance
Less DB space is enough	Comparatively more DB space is required

So far, we have discussed various data modelling techniques and their benefits around them.

Data Vault Model (DVM): What had discussed models earlier are predominantly focused on Classical or Modern Data Warehousing and Reporting systems. All we know now is we’re in the digital world delivering a Data Analytics Service to support enterprise-level systems like rich BI, Modern DWH, and Advanced Analytics like Data Science, Machine Learning, and extensive AI. This methodology is an agile way of designing and building modern, efficient and effective DWHs.

DVM is composed of multiple components like Model, Methodology, and Architecture, this is quite different from other DWH modelling techniques in current use. Another way around this is simply we can say that this is NOT a framework, product, and any service, instead, we can say this is Very Consistency, Scalability, highly Flexibility, easily Auditability, and specifically AGILITY. Yes! It is a modern agile way of designing DWH for various systems as mentioned earlier. Along with we can incorporate and implement the standards, policies, and best practices with the help of a well-defined process.

This model consists of three elements Hub, Link, and Satellite.

Hubs: This is one of the core building blocks in DVM. Which is to record a unique list of all the business keys for a single entity. Let’s say, for example, an It may contain a list of all Customer IDs, Employee IDs, Product IDs, and Order IDs in the business.

Links: Is fundamental component in a DVM is Links, which form the core of the raw vault along with other elements Hubs, and Satellites. Generally speaking, this is an association or link, between two business keys in the model. A typical example is Orders and the Customers in the respective table which is associated with customers and orders. And one more I can say store and employee working in store under various department so the link would be link_employee_store

Satellites: In DVM, Satellites connect to other elements in DVM (Hubs or Links). Satellite tables hold attributes related to a link or hub and update them as they change. For example, SAT_EMPLOYEE may feature attributes such as the employee’s Name, Role, Dob, Salary, or Doj. Simply say “The Point in Time Record in the table”. In simple language, we can say Satellites contain data about their parent Hub or Link and Metadata along with when the data has been loaded, from where, and effective business date details. Where the actual data resides for our business entities in the other elements discussed earlier (Hubs and Links).

In DVM architecture each Hub and Link record may have one or more child Satellite records, all the changes to that Hubs or Link.

Source: image designed by author shanthababu

Pros and cons

Pros

• This model tracks historical records
• Agile way of building the model as incrementally
• DVM use to provide the facilities of audibility
• Adaptable to changes without re-engineering
• The high degree of parallelism with respect to loads of data
• Supports the fault-tolerant ETL pipelines

Cons

• At a certain point, the models became more complex
• Implementation and understanding of Data Vault are a few challenges
• Since storing historical data capacity storage needed is high
• The model building takes time, so the value to the business is slower than another model

Conclusion

So far, we discussed data and modelling concepts in the below items in detail,

• What are OLTP and OLAP and their major difference?
• What is Data Modelling and what factors influence Data modelling?
• Discussed why the modern DWH is important for us? And various data availability, storage, maintainability, and IoT/ streaming data
• Data Model Evaluation and Data Modelling Components in depth
• Discussed various modelling techniques -Star Schema, Snowflake Schema, Kimball, Inmon, and Data Vault Model, and their components

Google’s Bard, one of today’s high-profile generative AI applications, is used with a grain of salt within the company. In June 2023, Google asked its staff not to feed confidential materials into Bard, Reuters found through leaked internal documents. It was reported that engineers were instructed not to use code written by the chatbot.

Companies including Samsung and Amazon have banned the use of public generative AI chatbots over similar concerns about confidential information slipping into private data.

Find out how Google Cloud approaches AI data, what privacy measures your business should keep in mind when it comes to generative AI and how to make a machine learning application “unlearn” someone’s data. While the Google Cloud and Bard teams don’t always have their hands on the same projects, the same advice applies to using Bard, its competitors such as ChatGPT or a private service by which your company could build its own conversational chatbot.

Jump to:

• How Google Cloud approaches using personal data in AI products
• What businesses should consider about using public AI chatbots
• Cracking machine unlearning

How Google Cloud approaches using personal data in AI products

Google Cloud approaches using personal data in AI products by covering such data under the existing Google Cloud Platform Agreement. (Bard and Cloud AI are both covered under the agreement.) Google is transparent that data fed into Bard will be collected and used to “provide, improve, and develop Google products and services and machine learning technologies,” including both the public-facing Bard chat interface and Google Cloud’s enterprise products.

“We approach AI both boldly and responsibly, recognizing that all customers have the right to complete control over how their data is used,” Google Cloud’s Vice President of Engineering Behshad Behzadi told TechRepublic in an email.

Google Cloud makes three generative AI products: the contact center tool CCAI Platform, the Generative AI App Builder and the Vertex AI portfolio, which is a suite of tools for deploying and building machine learning models.

Behzadi pointed out that Google Cloud works to make sure its AI products’ “responses are grounded in factuality and aligned to company brand, and that generative AI is tightly integrated into existing business logic, data management and entitlements regimes.”

SEE: Building private generative AI models can solve some privacy problems but tends to be expensive. (TechRepublic)

Google Cloud’s Vertex AI gives companies the option to tune foundation models with their own data. “When a company tunes a foundation model in Vertex AI, private data is kept private, and never used in the foundation model training corpus,” Behzadi said.

What businesses should consider about using public AI chatbots

Businesses using public AI chatbots “must be mindful of keeping customers as the top priority, and ensuring that their AI strategy, including chatbots, is built on top of and integrated with a well-defined data governance strategy,” Behzadi said.

SEE: How data governance benefits organizations (TechRepublic)

Business leaders should “integrate public AI chatbots with a set of business logic and rules that ensure that the responses are brand-appropriate,” he said. Those rules might include making sure the source of the data the chatbot is citing is clear and company-approved. Public internet search should be only a “fallback,” Behzadi said.

Naturally, companies should also use AI models that have been tuned to reduce hallucinations or falsehoods, Behzadi recommended.

For example, OpenAI is researching ways to make ChatGPT more trustworthy through a process known as process supervision. This process involves rewarding the AI model for following the desired line of reasoning instead of for providing the correct final answer. However, this is a work in progress, and process supervision is not currently incorporated into ChatGPT.

Employees using generative AI or chatbots for work should still double-check the answers.

“It is important for businesses to address the people aspect,” he said, “ensuring there are proper guidelines and processes for educating employees on best practices for the use of public AI chatbots.”

SEE: How to use generative AI to brainstorm creative ideas at work (TechRepublic)

Cracking machine unlearning

Another way to protect sensitive data that could be fed into artificial intelligence applications would be to erase that data completely once the conversation is over. But doing so is difficult.

In late June 2023, Google announced a competition for something a bit different: machine unlearning, or making sure sensitive data can be removed from AI training sets to comply with global data regulation standards such as the GDPR. This can be challenging because it involves tracing whether a certain person’s data was used to train a machine learning model.

“Aside from simply deleting it from databases where it’s stored, it also requires erasing the influence of that data on other artifacts such as trained machine learning models,” Google wrote in a blog post.

The competition runs from June 28 to mid-September 2023.

Subscribe to the Google Weekly Newsletter

Learn how to get the most out of Google Docs, Google Cloud Platform, Google Apps, Chrome OS, and all the other Google products used in business environments.

Delivered Fridays Sign up today

Indian IT giant HCL has recently acquired a 100% stake in German autonomous vehicle tech company ASAP Group, according to media reports.

This deal is poised to enhance the IT giant’s portfolio and strengthen its presence in the automotive industry. Furthermore, it will facilitate the company’s expansion into significant automotive markets across Europe, the Americas, and Japan.

The deal is expected to be around USD 279.72 million and is expected to be completed through HCL’s UK-based subsidiary.

“Core engineering is at the heart of HCLTech’s DNA and truly differentiates our services portfolio. ASAP has developed some exciting capabilities in automotive engineering, and we share their vision for the future of mobility.

“This agreement will enable us to scale these capabilities and innovations across our global network,” Hari Sadarahalli, Corporate Vice President, Engineering and R&D Services, HCL Tech said.

It’s noteworthy that HCL’s interest in the autonomous driving space is not new. In 2018, HCL revealed that the company was working on autonomous driving technology.

In fact, other IT giants too have been indulging in this space for quite some time. In 2020, Infosys showcased their innovation by retrofitting an existing buggy on their campus with a patented Drive-by-Wire technology, transforming it into an autonomous buggy.

The post HCL Acquires German Autonomous Vehicle Tech Firm ASAP Group appeared first on Analytics India Magazine.

Image by freepik.com

Data scientists are now considered to be among the most popular STEM (Science, Technology, Engineering, and Mathematics) professions. As information technology and AI becomes more important in our lives, the field of data science will continue to grow. The US Bureau of Labor Statistics projects a 36% increase in the demand for data scientists from 2021 — 2031. The demand for data scientists will continue to grow. For data science aspirants interested in the field, one important question to ask is: how much do data scientists make?

The salary for data scientists vary based on several factors such as geographic location, experience, education, industry, job title, etc.

In this article, we perform elementary statistical analysis to estimate the 2023 average annual salary for data scientists in the United States.

Data Collection

To estimate the average annual salary for data scientists in the United States, data was collected from indeed.com. Using the keyword search box and keyword “data scientists job”, we found 26,661 open data scientists positions in the United States as of March 8, 2023. The open positions vary from entry level to senior data scientists positions. The salary ranges for the available positions are displayed on the Table below.

Data Source: Indeed.com

Annual salary range and number of available data scientists positions in the United States. Data retrieved from source URL, using the search criteria “data science jobs” on March 8, 2023. Image by Author.

The figure below shows a bar plot of the salary ranges for the open data scientists positions.

Data Scientists open positions and annual salary ranges. Image by Author. Average Salary for Data Scientists

Using the data obtained from indeed.com, we performed elementary statistical calculations to determine the mean annual salary and standard deviation. The average salary for data scientists based on the information retrieved from indeed.com was estimated to be $124,000 per year. The standard deviation was estimated to be $21,000. The 95% confidence interval for the salary data is [$83,000 — $166,000] per year.

The distribution of the salary range could be estimated using a normal distribution, as shown in the Figure below.

Annual salary distribution for data scientists | Image by Author. Comparison with Data from US Bureau of Labor Statistics

According to the US Bureau of Labor Statistics, the 2021 median salary for data scientists was $100,000 per year. This value is slightly less than our estimated average annual salary of %124,000. The discrepancy could be due to increasing data scientists salary for 2023 to offset high inflation rates and the increased demand for data scientists associated with labor shortage.

Conclusion

In summary, we’ve performed elementary statistics to estimate the average salary for data scientists in the United States using salary data for open data scientists positions retrieved from indeed.com. Our analysis shows that the nationwide average salary for data scientists in the United States is estimated to be $124,000 per year, with a 95% confidence interval of [$83,000 — $166,000] per year. Data scientists should expect to make between $83,000 and $166,000 per year in 2023. The actual salary for any given role depends on factors such as geographic location, experience, education, industry, job title, etc.
Benjamin O. Tayo is a Physicist, Data Science Educator, and Writer, as well as the Owner of DataScienceHub. Previously, Benjamin was teaching Engineering and Physics at U. of Central Oklahoma, Grand Canyon U., and Pittsburgh State U.

More On This Topic

• Data Scientist Job Salaries Analysis
• Top Jobs and Salaries in Data Science in 2022
• Top Data Python Packages to Know in 2023
• Top 19 Skills You Need to Know in 2023 to Be a Data Scientist
• Data Analytics Tools You Need To Know in 2023
• A List of 7 Best Data Modeling Tools for 2023

Image Source: Author

Introduction

Data Engineers and Data Scientists need data for their Day-to-Day job. Of course, It could be for Data Analytics, Data Prediction, Data Mining, Building Machine Learning Models Etc., All these are taken care of by the respective team members and they need to work towards identifying relevant data sources, and associated with the business problems.

Data Sources

Data Sources can be identified in two different ways.

• Functional aspects
• Technical aspects

1 Functional aspect

With respect to functional aspects, it can be sub-divided into Primary and Secondary sources. Let’s quickly discuss this.

• Primary Sources: Data in the form of documents, a person details (First Name/Last Name/Address/Date of Birth/Phone Number/Passport Number/Driver’s License/Aadar card/SSN/National ID Number and etc.,)
• Secondary Sources: Derived from Primary.

2 Technical aspects

Both above said is nothing but in the form of non-digital form. When we convert them into meaningful ways. then it got the feel of technical rhythm. Then it would be given the way to below divisions

• Relational ( Relational Data Model)
• Multidimensional (OLAP Data Model)

Image Source: Author

What is Data Handling?

It refers to the set of processes, Let’s will walk through them one by one in detail along with effective python libraries

• Data-collection
• Data cleaning/cleansing
• Data preparation
• Data Wrangling

Data-Collection (DC)

General statements about “DATA COLLECTION” is a highly time-consuming and manual intervention, but in this digital world, it would be from an application source, mobile application, IoT devices etc., using automated tools and technologies.

• Conducting a campaign
• Quantitative research
• Interviews
• Observation and research
• Online Sales/Marketing analysis
• Social Media
• IoT and IIOT

Collecting data from Clients/Customers/End-users is a key process and business strategy to reach your perfect target audience to improve your presence in the leading market and support. So, in recent years industries are funding to collect data and draft big game plans for their business advancements.

Source https://www.fotolog.com/steps-in-data-science-process/

Why is so important?

From the data collection,

• We could analyze the root level information and identify your existing and potential customers in the market.
• You can build customer relationships strong and plan for your future marketing space
• Data in digital format would remove potential bias

Data-Collection is the first and major step in the Machine Learning(ML) life cycle. specifically for training, testing and building the right ML model to address the problem statement. The data which we’re collecting will define the outcome of the ML systems after lots of iterations and the process, So this process is very important for Data Science (or) ML team. Obviously, there are multiple challenges during this period, let’s review a few of them here.

• The collected data should be related to the problem statement.
• Inaccurate, missing data, null values in columns, and irrelative/missing images from the source would lead to errored prediction.
• Imbalance, an anomaly, and outliers are deviating from our focus and take us to the under-represented stage of model building.

Strategies to fix the challenges and issues with DC

• Pre-cleaned, freely available datasets. If the problem statement aligns with a cleaned, properly drafted dataset, then take advantage of existing, open-source expertise.
• Web crawling and scraping methods to collect the data using bots and automated tools.
• Private data. ML engineers can create their own data when the volume of data is required to train the model is very small and does not align with the problem statement.
• Custom data, Organizations can create the data.

Data cleaning/cleansing

In the ML lifecycle, 60% or more of that timeline will be demanded in Data Preparation, Loading, Cleaning/Cleansing, Transforming and reshaping/rearranging.

When we start looking at the Cleaning (or) Cleansing process, The below list of options is provided by Python.

• Missing Data handling techniques
• Transformation of Data
• Manipulation Methods

Missing Data handling techniques: Missing data analysis is a very common technique in the ML world. Data missing impacts the analysis and model. Certainly, the model couldn’t train properly and misguide the prediction or forecasting at a later point.

In Python’s pandas, we use to adapt NA (Not Available or Not exist)

(i) Finding Null values

I will show a few sample codes here

(a) Output – if Null item in the list, it should be NaN

import pandas as pd import numpy as np string_collection=pd.Series(['Apple','Ball','Cat',np.nan,'Dog'])

string_collection

0 Apple
1 Ball
2 Cat
3 NaN
4 Dog
dtype: object

(b) Output – if Null item in the list, it should be True

string_collection.isnull()

0 False

1 False

2 False

3 True

4 False

dtype: bool

(c) Dropping NaN from the list

string_collection.dropna()

0 Apple
1 Ball
2 Cat
4 Dog
dtype: object

(d) Let’s try with the titanic dataset

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline

df_titanic = pd.read_csv(‘titanic.csv’)

df_titanic.head()

df_titanic.isnull().any()

(d) Number of Null in the column(s)

print("Number of Null in age column:",df_titanic['age'].isnull().sum()) print("Number of Null in embark_town column:",df_titanic['embark_town'].isnull().sum())

Number of Null in age column: 177
Number of Null in embark_town column: 2

(e) Null values through heatmap

sns.heatmap(df_titanic.isnull(),yticklabels=False,cbar=False,cmap='viridis')

(ii) Filtering the missing data: There are two ways to filter out the missing values either by using dropna or notnull.

• dropna – will remove the row from the dataset/series
• notnull – still data will be in the dataset/series

NaN handling methods in pandas

Methods	Notes
isnull	returns boolean for specified column/variable
notnull	excluding the null values/rows
fillna	filling with the specified value
dropna	dropping row(s)

Usage

(a)Filtering Using Notnull/ Dropna rows

import pandas as pd import numpy as np from numpy import nan as NA data=pd.Series([100,250,NA,350,400,500,NA,950]) print(data) print("Apply dropna") print("=============") print(data.dropna() ) print("Apply notnull") print("=============") print(data[data.notnull()])

Output

0    100.0 1    250.0 2      NaN 3    350.0 4    400.0 5    500.0 6      NaN 7    950.0 dtype: float64 Apply dropna ============= 0    100.0 1    250.0 3    350.0 4    400.0 5    500.0 7    950.0 dtype: float64 Apply notnull ============= 0    100.0 1    250.0 3    350.0 4    400.0 5    500.0 7    950.0 dtype: float64

(iii) Filtering the NA from dataframe

import pandas as pd import numpy as np from numpy import nan as NA data=pd.DataFrame([[100,101,102],['Raj','John',NA],[NA,NA,NA],['Chennai','Bangalore','Delhi']]) print(data)

Output

         0          1      2 0      100        101    102 1      Raj       John    NaN 2      NaN        NaN    NaN 3  Chennai  Bangalore  Delhi

(iv) Cleaning NA

Cleand_data=data.dropna() print(Cleand_data)

Output

         0          1      2 0      100        101    102 3  Chennai  Bangalore  Delhi

So far we have discussed filtering the missing data, but cleaning is not only the solution. in a real-time scenario, we can not remove just like that without the opinion from Subject Matter Experts (SMEs). Need to fill in the data. there are various techniques are there. Let’s will discuss, a few of them in this article.

import pandas as pd import numpy as np from numpy import nan as NA data=pd.DataFrame([[100,101,102],['Raj','John','Jay'],[NA,NA,NA],['Chennai','Bangalore','Delhi']])

data.fillna(0)

Output

0	1	2
0	100	101	102
1	Raj	John	Jay
2	0	0	0
3	Chennai	Bangalore	Delhi

(v) Fill in the data from the previous row

import pandas as pd import numpy as np from numpy import nan as NA data=pd.DataFrame([['Raj','John','Jay'],[100,101,102],[NA,NA,NA],['Chennai','Bangalore','Delhi']]) print(data)

Output

        0          1      2 0      Raj       John    Jay 1      100        101    102 2      NaN        NaN    NaN 3  Chennai  Bangalore  Delhi

data.fillna(method='ffill')

0	Raj	John	Jay
1	100	101	102
2	100	101	102
3	Chennai	Bangalore	Delhi

Will see this from a dataframe point of view.

(vi) Removing Duplicates rows from the dataframe, just using drop_duplicates

import pandas as pd import numpy as np from numpy import nan as NA data=pd.DataFrame([['Raj','Chennai'],['John','Chennai'],['Jey','Bangalore'],['Mohan','Delhi'],['Raj','Channai']]) print(data)

Output

       0          1 0    Raj    Chennai 1   John    Chennai 2    Jey  Bangalore 3  Mohan      Delhi 4    Raj    Channai

(vii) Finding duplicates

data.duplicated()

0    False 1    False 2    False 3    False 4     True dtype: bool

data.drop_duplicates()

0	1
0	Raj	Chennai
1	John	Chennai
2	Jey	Bangalore
3	Mohan	Delhi

(viii) Replacing Values

import pandas as pd import numpy as np from numpy import nan as NA data=pd.DataFrame([['Raj','Chennai',0],['John','Chennai',2],['Jey','Bangalore',-1],['Mohan','Delhi',-3]]) print(data)

Output

0	Raj	Chennai	0
1	John	Chennai	2
2	Jey	Bangalore	0
3	Mohan	Delhi	0

With mean() in a dataset, Consider that the given auto-mpg has null values in the horsepower column, and some junk data (like ? ).

print(df_cars["horsepower"].isna().sum())

Output

19

So, the horsepower column has 19 null values, let’s handle this now.

df_cars.horsepower = df_cars.horsepower.str.replace('?','NaN').astype(float) df_cars.horsepower.fillna(df_cars.horsepower.mean(),inplace=True) df_cars.horsepower = df_cars.horsepower.astype(int) print("######################################################################") print("          After Cleaning and type convertion in the Data Set") print("######################################################################") df_cars.info()

Null values are replaced by mean of the horsepower column.

print(df_cars["horsepower"].isna().sum())

Output

0

Yes! We did it! awsome. Now we could consider the horsepower column is clean and error-free.

Data Transforming

(I) Filtering Outliers: In simple terms, we can analyze the data distribution identify the outliers, and remove them from the dataset to avoid overfitting or underfitting during model evaluation. Mathematically finding the outliers really challenging process, surely will use visualization techniques will support ease and understanding better.

import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns from IPython.display import display import statsmodels as sm from scipy import stats df_cars = pd.read_csv("auto-mpg.csv") df_cars.horsepower = df_cars.horsepower.str.replace('?','NaN').astype(float) df_cars.horsepower.fillna(df_cars.horsepower.mean(),inplace=True) df_cars.horsepower = df_cars.horsepower.astype(int) sns.boxplot(x=df_cars["horsepower"])

We could observe that there is an outlier (dots)
after the scale of 200 for the horsepower feature. Let’s remove the outliers
using mathematical ways.

z_scores = stats.zscore(df_cars["horsepower"])

abs_z_scores = np.abs(z_scores) print(abs_z_scores)

Output

[0.67155703 1.5895576  1.19612879 1.19612879 0.93384291 2.455101  3.03212994 2.900987   3.16327288 2.2452723  1.72070054 1.45841466  1.19612879 3.16327288 0.24644355 0.24644355 0.19398637 0.50872943  0.43004366 1.53164436 0.45627225 0.37758649 0.24644355 0.22567103  0.37758649 2.900987   2.50755818 2.76984406 2.32395806 0.43004366  0.37758649 0.24644355 0.01038626 0.11530061 0.01584233 0.11530061  0.43004366 0.11530061 1.5895576  1.85184348 1.27481455 1.19612879  1.98298642 1.72070054 1.85184348 0.14698527 0.84970107 0.11530061  0.43004366 0.48250084 0.37758649 0.90215825 0.74478672 1.03330119  0.92838683 1.16444413 0.90215825 0.24644355 0.63987237 1.32181565  0.37758649 0.48250084 1.5895576  1.85184348 1.19612879 1.27481455  1.19612879 2.71738688 1.32727172 1.45841466 2.2452723  0.19398637  1.19612879 0.67155703 0.93384291 1.19612879 0.19944245 0.74478672  0.45627225 0.92838683 0.48250084 0.32512931 0.19398637 0.63987237  0.43004366 1.85184348 1.19612879 1.06498585 0.85515714 1.19612879  2.455101   1.19612879 1.40595749 1.19612879 2.900987   3.16327288  1.85184348 0.01584233 0.11530061 0.11530061 0.43004366 0.24644355  1.53164436 1.19612879 1.64201478 1.72070054 1.98298642 0.11530061  0.43004366 0.84970107 0.27267214 0.37758649 0.50872943 0.06829951  0.37758649 1.06498585 3.29441582 1.45295859 0.77101531 0.3513579  0.19944245 1.19612879 0.14698527 0.46172832 1.98298642 0.24644355  0.01038626 0.11530061 0.11530061 0.98084401 0.63987237 1.03330119  0.77101531 0.11530061 0.14698527 0.01584233 0.93384291 1.19612879  1.19612879 0.93384291 1.19612879 0.5611866  0.98084401 0.69232954  1.37427283 1.13821554 0.77101531 0.77101531 0.77101531 0.19398637  0.29890072 0.98084401 0.24644355 0.01584233 0.84970107 0.84970107  1.72070054 1.06498585 1.19612879 1.14367161 0.14698527 0.01584233  0.14698527 0.24644355 0.14698527 0.14698527 0.64532844 0.77101531  0.5611866  0.11530061 0.69232954 0.22021496 0.87592966 0.19398637  0.19398637 0.90215825 0.37758649 0.24644355 0.43004366 0.16775779  0.27812821 1.34804424 0.48250084 0.61364378 0.32512931 0.66610096  0.5611866  0.93384291 1.19612879 0.40927115 1.24858596 0.11530061  0.01584233 0.61364378 0.37758649 1.37427283 1.16444413 0.90215825  1.34804424 0.11530061 0.69232954 0.14698527 0.24644355 0.87592966  0.90215825 0.77101531 0.84970107 0.06284343 1.19612879 0.43004366  0.09452809 0.40927115 1.98298642 1.06498585 0.67155703 1.19612879  0.95461542 0.63987237 1.2169013  0.22021496 0.90215825 1.06498585  0.14698527 1.06498585 0.67155703 0.14698527 0.01584233 0.11530061  0.16775779 1.98298642 1.72070054 2.2452723  1.1699002  0.69232954  0.43004366 0.77101531 0.40381508 1.08575836 0.5611866  0.98084401  0.69232954 0.19398637 0.14698527 0.14698527 1.47918718 1.0070726  1.37427283 0.90215825 1.16444413 0.14698527 0.93384291 0.90761432  0.01584233 0.24644355 0.50872943 0.43004366 0.11530061 0.37758649  0.01584233 0.50872943 0.14698527 0.40927115 1.06498585 1.5895576  0.90761432 0.93384291 0.95461542 0.24644355 0.19398637 0.77101531  0.24644355 0.01584233 0.50872943 0.19398637 0.03661485 0.54041409  0.27812821 0.75024279 0.87592966 0.95461542 0.27812821 0.50872943  0.43004366 0.37758649 0.14698527 0.67155703 0.64532844 0.88138573  0.80269997 1.32727172 0.98630008 0.54041409 1.19612879 0.87592966  1.03330119 0.63987237 0.63987237 0.71855813 0.54041409 0.87592966  0.37758649 0.90215825 0.90215825 1.03330119 0.92838683 0.37758649  0.27812821 0.27812821 0.37758649 0.74478672 1.16444413 0.90215825  1.03330119 0.37758649 0.43004366 0.37758649 0.37758649 0.69232954  0.37758649 0.77101531 0.32512931 0.77101531 1.03330119 0.01584233  1.03330119 1.47918718 1.47918718 0.98084401 0.98084401 0.98084401  0.01038626 0.98084401 1.11198695 0.7240142  0.11530061 0.43004366  0.01038626 0.84970107 0.53495802 0.53495802 0.32512931 0.14698527  0.53495802 1.2169013  1.05952977 1.16444413 0.98084401 1.03330119  1.11198695 0.95461542 1.08575836 1.03330119 1.03330119 0.79724389  0.01038626 0.77101531 0.77101531 0.11530061 0.79724389 0.63987237  0.74478672 0.3043568  0.40927115 0.14698527 0.01584233 0.43004366  0.50872943 0.43004366 0.43004366 0.43004366 0.50872943 0.53495802  0.37758649 0.32512931 0.01038626 0.79724389 0.95461542 0.95461542  1.08575836 0.90215825 0.43004366 0.77101531 0.90215825 0.98084401  0.98084401 0.98084401 0.14698527 0.50872943 0.32512931 0.19944245  0.22021496 0.53495802 0.37758649 0.48250084 1.37427283 0.53495802  0.66610096 0.58741519 0.69232954]

I can understand that this is really all, that’s fine let’s set up a threshold and continue.

filtered_entries = (abs_z_scores < 1.5) new_df = df_cars[filtered_entries]

print(new_df)

mpg  cylinders  displacement  horsepower  weight  acceleration   0    18.0          8         307.0         130    3504          12.0    2    18.0          8         318.0         150    3436          11.0    3    16.0          8         304.0         150    3433          12.0    4    17.0          8         302.0         140    3449          10.5    11   14.0          8         340.0         160    3609           8.0    ..    ...        ...           ...         ...     ...           ...    394  44.0          4          97.0          52    2130          24.6    395  32.0          4         135.0          84    2295          11.6    396  28.0          4         120.0          79    2625          18.6    397  31.0          4         119.0          82    2720          19.4    398   NaN          4         250.0          78    2500          18.5          model_year  origin                       name   0          70.0     1.0  chevrolet chevelle malibu   2          70.0     1.0         plymouth satellite   3          70.0     1.0              amc rebel sst   4          70.0     1.0                ford torino   11         70.0     1.0         plymouth 'cuda 340   ..          ...     ...                        ...   394        82.0     2.0                  vw pickup   395        82.0     1.0              dodge rampage   396        82.0     1.0                ford ranger   397        82.0     1.0                 chevy s-10   398         NaN     NaN                        NaN    [360 rows x 9 columns]

sns.boxplot(x=new_df["horsepower"])

Now, the box plot is very clear and has no more
outliers. Think about the power of python libraries here.

(II) Converting Type: We Will analyze the given dataset columns type, this is an essential activity before we do feature engineering and test training.

df_cars = pd.read_csv("auto-mpg.csv")

print("############################################") print("          Info Of the Data Set") print("############################################") df_cars.info()

Output

Observation:
1. we could observe that the features and its data type, along with count Null
2. horsepower and name features are, Object in the given data set

How to transform this into a meaningful way for our analysis. using simple astype.

df_cars.horsepower = df_cars.horsepower.str.replace('?','NaN').astype(float) df_cars.horsepower.fillna(df_cars.horsepower.mean(),inplace=True) df_cars.horsepower = df_cars.horsepower.astype(int) print("######################################################################") print("          After Cleaning and type convertion in the Data Set") print("######################################################################") df_cars.info()

Output

Observation:
1. we could observe that the features and its data type, along with count Null
2. we could observe that horsepower is now int type.

(III) Create Dummy Variables: In a real-time scenario, wehave to handle the categorical variable in intelligent ways so that we could accommodate them in the process of converting them into dummy variables and make use of them as independent variables. Let’s see the sample here.

df_cars.head(5)

Let’s convert them into a Categorical variable

df_cars['origin'] = df_cars['origin'].replace({1: 'america', 2: 'europe', 3: 'asia'}) df_cars.head()

cData = pd.get_dummies(df_cars, columns=['origin']) cData

(III) String Transforming: In some situations, we have to deal with string values in the given dataset and as data scientists, we are responsible for streamlining them for data analysis. Here is one classical sample most commonly facing.

pattern = ‘chevroelt|chevy|chevrolet’

mask = df_cars[‘name’].str.contains(pattern, case=False, na=False)

df_cars[mask].head()

Observe here that Chevrolet in different spellings, So during classification modeling this would give you headache and challenging your patience, Now follow me, how we can handle this.

# Correct name  df_cars['name'] = df_cars['name'].str.replace('chevroelt|chevrolet|chevy','chevrolet') df_cars['name'] = df_cars['name'].str.replace('maxda|mazda','mazda') df_cars['name'] = df_cars['name'].str.replace('mercedes|mercedes-benz|mercedes benz','mercedes') df_cars['name'] = df_cars['name'].str.replace('toyota|toyouta','toyota') df_cars['name'] = df_cars['name'].str.replace('vokswagen|volkswagen|vw','volkswagen')

The above code will streamline the brand names and your modeling will give perform better than earlier.

Let’s see how to string transformation works here.

pattern = 'chevrolet' mask = df_cars['name'].str.contains(pattern, case=False, na=False) df_cars[mask].head()

Hope you love this, Yes! I can understand.

Read more articles on our website about data handling techniques. Click here.

Conclusion

This is a long journey, so far we covered the possible and most frequent techniques in Data Handling techniques right from data collection, cleaning, and wrangling aspects, still, many more techniques are there and usage is dependent on the cases, With respect to Data Science, the data handling is a vital role and 60-65% of effort would require to fine-tune our data for modeling, So remember all these features we had discussed over here certainly help you a lot, Let me say break and will connect with you all on something interesting topics. Hope you liked my article on data handling techniques.

Thanks for your time, Good Luck! See you all soon. – Shanthababu