Рубрика: AI

The underpinnings of LLMs like OpenAI's GPT-3 or its successor GPT-4 lie in deep learning, a subset of AI, which leverages neural networks with three or more layers. These models are trained on vast datasets encompassing a broad spectrum of internet text. Through training, LLMs learn to predict the next word in a sequence, given the words that have come before. This capability, simple in its essence, underpins the ability of LLMs to generate coherent, contextually relevant text over extended sequences.

The potential applications are boundless—from drafting emails, creating code, answering queries, to even writing creatively. However, with great power comes great responsibility, and managing these behemoth models in a production setting is non-trivial. This is where LLMOps steps in, embodying a set of best practices, tools, and processes to ensure the reliable, secure, and efficient operation of LLMs.

The roadmap to LLM integration have three predominant routes:

1. Prompting General-Purpose LLMs:
   • Models like ChatGPT and Bard offer a low threshold for adoption with minimal upfront costs, albeit with a potential price tag in the long haul.
   • However, the shadows of data privacy and security loom large, especially for sectors like Fintech and Healthcare with stringent regulatory frameworks.
2. Fine-Tuning General-Purpose LLMs:
   • With open-source models like Llama, Falcon, and Mistral, organizations can tailor these LLMs to resonate with their specific use cases with just model tuning resource as expense.
   • This avenue, while addressing privacy and security qualms, demands a more profound model selection, data preparation, fine-tuning, deployment, and monitoring.
   • The cyclic nature of this route calls for a sustained engagement, yet recent innovations like LoRA (Low-Rank Adaptation) and Q(Quantized)-LoRa have streamlined the fine-tuning process, making it an increasingly popular choice.
3. Custom LLM Training:
   • Developing a LLM from scratch promises an unparalleled accuracy tailored to the task at hand. Yet, the steep requisites in AI expertise, computational resources, extensive data, and time investment pose significant hurdles.

Among the three, the fine-tuning of general-purpose LLMs is the most favorable option for companies. Creating a new foundation model may cost up to $100 million, while fine-tuning existing ones ranges between $100 thousand to $1 million. These figures stem from computational expenses, data acquisition and labeling, along with engineering and R&D expenditures.

LLMOps versus MLOps

Machine learning operations (MLOps) has been well-trodden, offering a structured pathway to transition machine learning (ML) models from development to production. However, with the rise of Large Language Models (LLMs), a new operational paradigm, termed LLMOps, has emerged to address the unique challenges tied to deploying and managing LLMs. The differentiation between LLMOps and MLOps are on several factors:

1. Computational Resources:
   • LLMs demand a substantial computational prowess for training and fine-tuning, often necessitating specialized hardware like GPUs to accelerate data-parallel operations.
   • The cost of inference further underscores the importance of model compression and distillation techniques to curb computational expenses.
2. Transfer Learning:
   • Unlike the conventional ML models often trained from scratch, LLMs lean heavily on transfer learning, starting from a pre-trained model and fine-tuning it for specific domain tasks.
   • This approach economizes on data and computational resources while achieving state-of-the-art performance.
3. Human Feedback Loop:
   • The iterative enhancement of LLMs is significantly driven by reinforcement learning from human feedback (RLHF).
   • Integrating a feedback loop within LLMOps pipelines not only simplifies evaluation but also fuels the fine-tuning process.
4. Hyperparameter Tuning:
   • While classical ML emphasizes accuracy enhancement via hyperparameter tuning, in the LLM arena, the focus also spans reducing computational demands.
   • Adjusting parameters like batch sizes and learning rates can markedly alter the training speed and costs.
5. Performance Metrics:
   • Traditional ML models adhere to well-defined performance metrics like accuracy, AUC, or F1 score, while LLMs have different metric set like BLEU and ROUGE.
   • BLEU and ROUGE are metrics used to evaluate the quality of machine-generated translations and summaries. BLEU is primarily used for machine translation tasks, while ROUGE is used for text summarization tasks.
   • BLEU measures precision, or how much the words in the machine generated summaries appeared in the human reference summaries. ROUGE measures recall, or how much the words in the human reference summaries appeared in the machine generated summaries.
6. Prompt Engineering:
   • Engineering precise prompts is vital to elicit accurate and reliable responses from LLMs, mitigating risks like model hallucination and prompt hacking.
7. LLM Pipelines Construction:
   • Tools like LangChain or LlamaIndex enable the assembly of LLM pipelines, which intertwine multiple LLM calls or external system interactions for complex tasks like knowledge base Q&A.

https://www.fiddler.ai/llmops

Understanding the LLMOps Workflow: An In-depth Analysis

Language Model Operations, or LLMOps, is akin to the operational backbone of large language models, ensuring seamless functioning and integration across various applications. While seemingly a variant of MLOps or DevOps, LLMOps has unique nuances catering to large language models' demands. Let's delve into the LLMOps workflow depicted in the illustration, exploring each stage comprehensively.

1. Training Data:
   • The essence of a language model lies in its training data. This step entails collecting datasets, ensuring they're cleaned, balanced, and aptly annotated. The data's quality and diversity significantly impact the model's accuracy and versatility. In LLMOps, emphasis is not just on volume but alignment with the model's intended use-case.
2. Open Source Foundation Model:
   • The illustration references an “Open Source Foundation Model,” a pre-trained model often released by leading AI entities. These models, trained on large datasets, serve as an excellent outset, saving time and resources, enabling fine-tuning for specific tasks rather than training anew.
3. Training / Tuning:
   • With a foundation model and specific training data, tuning ensues. This step refines the model for specialized purposes, like fine-tuning a general text model with medical literature for healthcare applications. In LLMOps, rigorous tuning with consistent checks is pivotal to prevent overfitting and ensure good generalization to unseen data.
4. Trained Model:
   • Post-tuning, a trained model ready for deployment emerges. This model, an enhanced version of the foundation model, is now specialized for a particular application. It could be open-source, with publicly accessible weights and architecture, or proprietary, kept private by the organization.
5. Deploy:
   • Deployment entails integrating the model into a live environment for real-world query processing. It involves decisions regarding hosting, either on-premises or on cloud platforms. In LLMOps, considerations around latency, computational costs, and accessibility are crucial, along with ensuring the model scales well for numerous simultaneous requests.
6. Prompt:
   • In language models, a prompt is an input query or statement. Crafting effective prompts, often requiring model behavior understanding, is vital to elicit desired outputs when the model processes these prompts.
7. Embedding Store or Vector Databases:
   • Post-processing, models may return more than plain text responses. Advanced applications might require embeddings – high-dimensional vectors representing semantic content. These embeddings can be stored or offered as a service, enabling quick retrieval or comparison of semantic information, enriching the way models' capabilities are leveraged beyond mere text generation.
8. Deployed Model (Self-hosted or API):
   • Once processed, the model's output is ready. Depending on the strategy, outputs can be accessed via a self-hosted interface or an API, with the former offering more control to the host organization, and the latter providing scalability and easy integration for third-party developers.
9. Outputs:
   • This stage yields the tangible result of the workflow. The model takes a prompt, processes it, and returns an output, which depending on the application, could be text blocks, answers, generated stories, or even embeddings as discussed.

Top LLM Startups

The landscape of Large Language Models Operations (LLMOps) has witnessed the emergence of specialized platforms and startups. Here are two startups/platforms and their descriptions related to the LLMOps space:

Comet

Comet streamlines the machine learning lifecycle, specifically catering to large language model development. It provides facilities for tracking experiments and managing production models. The platform is suited for large enterprise teams, offering various deployment strategies including private cloud, hybrid, and on-premise setups​.

Dify

Dify is an open-source LLMOps platform that aids in the development of AI applications using large language models like GPT-4. It features a user-friendly interface and provides seamless model access, context embedding, cost control, and data annotation capabilities. Users can effortlessly manage their models visually and utilize documents, web content, or Notion notes as AI context, which Dify handles for preprocessing and other operations​.

Portkey.ai

Portkey.ai is an Indian startup specializing in language model operations (LLMOps). With a recent seed funding of $3 million led by Lightspeed Venture Partners, Portkey.ai offers integrations with significant large language models like those from OpenAI and Anthropic. Their services cater to generative AI companies, focusing on enhancing their LLM operations stack which includes real-time canary testing and model fine-tuning capabilities​.

Undoubtedly, NVIDIA is dominating the generative AI industry in both the software and the hardware realm. Its rival AMD, on the other hand, has been trying several ways to catch up. On the hardware front, it has been trying to maintain the lead with MCM design, which NVIDIA recently decided to adopt as well.

But when it comes to software, AMD has made a huge leap with its recent bet on Nod.ai, an open source AI software firm. “The acquisition of Nod.ai is expected to significantly enhance our ability to provide AI customers with open software that allows them to easily deploy highly performant AI models tuned for AMD hardware,” said Vamsi Boppana, senior vice president, AI Group at AMD.

There has been an urgent need for open source GPU architecture for a very long time. All the hardware companies have been trying to build alternatives and close the gap with NVIDIA, which includes AMD and Intel.

For AMD, the deal might be the easiest and cheapest way forward as well. Founded in 2013 by Anush Elangovan and Harsh Menon, Silicon Valley-based Nod.ai, had raised $20 million in its last funding. According to reports, the AI software company might just be purchased by AMD, which is currently valued at $36.5 million.

Elangovan, one of the co-founders has worked on the first Google ARM Chromebook and has worked as the foundation of the ChromeOS team at Google. Menon, the CTO of Nod.ai, was an early employee at Zee.Aero, has an MS in Aerospace Engineering, and is said to have worked on flying cars.

Betting on open source

Nod.ai has been known for developing a portfolio of tools and systems for boosting AI applications on AMD hardware. The team at Nod.ai has agreed to develop leading software for Instinct data centres accelerators, the consumer grade Ryzen AI processors, and Radeon GPUs.

The best part about the company is its open source approach. AMD is reportedly also too interested in working on open source solutions for lowering “the barriers of entry for customers through developer tools, libraries and models,” as it said in its AI strategy announcement. This is in contrast to what NVIDIA has been doing with its software made for GPUs.

For the longest time, NVIDIA’s compute unified device architecture (CUDA) has been the biggest moat for the company. The only problem is that it is closed source and only works for NVIDIA GPU workloads. Though people have been finding several solutions to work around this restriction, CUDA still remains the best on NVIDIA GPUs, and since everyone is using its GPUs, the moat becomes even bigger.

Arguably, the biggest moat for hardware companies is software. Bryan Catanzaro, VP of applied deep learning research, NVIDIA, has stated in an interview that even though NVIDIA is known as a hardware company, “many people don’t know this, but NVIDIA has more software engineers than hardware engineers”. Thus, even though CUDA is closed source, it is still free to use.

But AMD is hell-bent on open source, and Nod.ai’s Elangovan, agrees. “Our journey as a company has cemented our role as the primary maintainer and major contributor to some of the world’s most important AI repositories, including SHARK, Torch-MLIR and OpenXLA/IREE code generation technology,” he said.

Added to all this is AMD’s Radeon open compute (ROC), the partnership might actually prove to be an alternative to NVIDIA’s moats.

The trillion-dollar rival

The software bet has been going on at AMD for some time now. In August, the company also announced the acquisition of Mipsology, a French AI startup, which has also been a long-standing AMD partner and developing AI software for the chipmaker, similar to Nod.ai.

In August, Boppana wrote, “The team will help develop our full AI software stack, expanding our open ecosystem of software tools, libraries, and models to pave the way for streamlined deployment of AI models running on AMD hardware.”

Both the acquisitions, Nod.ai and Mipsology, highlight AMD’s dedication towards challenging NVIDIA’s monopoly in the AI market. Nod.ai team members are also going to join the Mipsology and AMD AI group, which has around 1,500 employees at the moment.

Finally, it seems like AMD might be able to crack the moat that NVIDIA has in the AI market.

The post AMD’s Attempt to Break NVIDIA’s CUDA appeared first on Analytics India Magazine.

Last week, developer community platform Replit made their suite of AI products called Replit AI accessible to everyone, with default code completion and assistance features available to over 23 million developers.

The San Francisco-based startup is taking a significant step by integrating GhostWriter, a generative AI-powered ‘coding partner’ launched in October 2022, into its core platform, extending access to all its users under the banner of “AI for all.”

Replit also unveiled an upgraded version of its proprietary generative AI LLM for coding, replit-code-v1.5-3b, a cutting-edge 3B LLM with a strong focus on code-related tasks, trained on a vast dataset of 1 trillion tokens consisting of 30 programming languages and a developer-oriented subset of StackExchange.

Making AI Accessible for All

Many people, including students in India and elsewhere, lack the financial means to pay for services like GitHub Copilot, which typically costs $10 a month or similar amounts. Access to AI-driven workflows is essential because these skills will be expected by future employers or for starting one’s own business.

Replit believes in contributing to the open-source community. The goal is to ensure that AI becomes an indispensable tool, much like calculators for mathematics, as they strive to democratise AI in coding.

“AI plays an integral role in coding, and we envision a future where AI assistance is accessible to all developers, even if they cannot afford $20 a month for ChatGPT Plus,” Anshul Bhide, BizOps, India Head, told AIM in an exclusive interview.

Unlike Github Copilot or Amazon’s CodeWhisperer, their updated LLM replit-code-v1.5-3b is also open-sourced, serving as a solid foundation for anyone working on code completion models. “In terms of performance, it’s on par, if not better, than both Star Coder and Code Llama once fine-tuned,” he added.

Earlier this year, Google Cloud teamed up with Replit granting access to the former’s infrastructure and Vertex AI, promising to enhance productivity and enable complex software coding in a fraction of the time.

Compared to GitHub, Replit distinguishes itself with real-time collaboration, browser-based coding, and voice commands. The partnership underscores Google‘s efforts to compete with Microsoft in the developer community by forming strategic alliances with existing players in the AI space.

Additionally, the team used publicly available code from the developer community. “What’s interesting is that research suggests that the size of the dataset matters less than its quality,” he added. Training a smaller model on a high-quality curated dataset often yields similar results in terms of model performance. This is important as he suggested that there is a lack of quality data to train algorithms.

As far as the programming languages are concerned, Bhide said that they focused on the top 30 languages used by smart developers on our platform including Python, Java, JavaScript, C++, and others and training the model on these languages would enhance code completion and productivity.

Their proprietary LLM, the first model trained on 128 H100-80G GPUs released as open source, faced a significant challenge in obtaining these GPUs due to the ongoing chip crisis and high demand. Talking about the same, Bhide discussed the infrastructural changes needed for large-scale AI deployment, emphasising the shift from CPU to GPU usage.

This shift is driven by the increased accessibility of AI tools to a broader developer audience, resulting in a greater demand for GPU capacity for code completion and model inference. However, the challenges of securing these GPUs were highlighted, including fierce competition from tech giants and data center providers. This surge in demand, coupled with NVIDIA’s limited manufacturing capacity, has made H100 GPUs a precious and scarce resource for the foreseeable future, significantly impacting AI development and deployment.

India Expansion Plans

India is a crucial market for Replit, being the second-largest and one of the fastest-growing markets. The company’s product-market fit in India is strong, thanks to factors like the preference for mobile over desktop, fast 5G internet, and a growing interest in coding. Anshul also touched on the rise of auto-coding platforms and Replit’s role in enhancing developers’ productivity.

Within the framework of Replit’s India strategy, there is a strong emphasis on addressing latency concerns by establishing local servers. This strategic move aligns with the company’s commitment to catering to the specific needs of the Indian audience. Additionally, Replit is actively engaged in localisation efforts, including the launch of “100 Days of Code” in Hindi.

Moreover, Replit has observed the significant popularity of its mobile app among Indian students for coding purposes, even within college classrooms. This trend underscores the platform’s relevance and effectiveness within the Indian educational landscape.

“The ultimate goal is to provide access to Replit products for learning and skill development, potentially opening doors to employment opportunities and entrepreneurship on a global scale,” concluded Bhide.

Read more: Replit Knows India Better than Amazon, Microsoft

The post Replit’s Mad Obsession with ‘AI for All’ appeared first on Analytics India Magazine.

Image by Author

The generative AI revolution has captured the tech world's imagination. ChatGPT and tools like it seem to herald a new era of possibility, where AI can generate content, art, and even programming code on demand. Venture capital has flooded into generative startups, with total funding reaching hundreds of billions dollars. But amidst the excitement, some are beginning to wonder — is this a bubble ready to pop?

The pattern seems familiar. A hot new technology arrives and is immediately embraced as world-changing and transformative. Massive amounts of capital pour in, valuations hit the stratosphere, and hype overwhelms rational analysis. This was the dot-com bubble in the late 90s, where internet startups with no revenue or business models achieved dizzying market caps. And it all came crashing down in 2000.

What Happened in the Dot-com Bubble?

The dot-com bubble, also known as the Internet bubble, was a period of excessive speculation and investment in internet-based companies during the late 1990s. This economic euphoria was driven by the belief in the transformative potential of the internet. However, the bubble eventually burst, leading to a crash in stock prices and the collapse of many startups.

Many dot-com companies were built on flimsy business models. They lacked solid revenue streams or profitability, relying heavily on investor funding. The focus was often on capturing market share and user growth rather than generating profits.

As the dot-com companies struggled to turn a profit, reality struck. The initial excitement and optimism began to fade as it became clear that many of these companies were not sustainable in the long run. Investors started to question the viability of these businesses.

The dot-com bubble burst in the early 2000s. The stock prices experienced a significant drop, leading to the bankruptcy of numerous dot-com companies. The NASDAQ index, which had reached its peak in March 2000, dropped 76.81% by October of the same year . Big firms like Cisco, Intel, and Oracle lost more than 80% of their share value — Dot-com bubble — Wikipedia.

Image from Wikipedia How the Generative AI Bubble will Burst

The rapid growth and hype surrounding generative AI has all the makings of an economic bubble. Generative AI models like DALL-E 2 and GPT-4 have captured the public imagination and attracted billions in investment. But this enthusiasm may prove unsustainable.

Like all bubbles, the Generative AI craze is built on speculative expectations about future capabilities. Investors are betting these technologies will continue rapid advancements and find lucrative real-world applications. But there is a risk these expectations get ahead of reality.

Several factors could burst the bubble. One is the limitations of today's Generative AI. While impressive, the models still produce low-quality outputs too unreliable for many tasks. And training ever-larger models requires exponentially more data and computing power, raising questions about scalability.

As hype meets reality, valuations of generative startups may prove unrealistic. Funding could dry up amidst unmet milestones, lack of profits, and loss of novelty. Stock prices will likely plunge once growth stalls.

Past experience shows hot new technologies go through a hype cycle before real capabilities emerge. While Generative AI has promise, investors should beware of irrational exuberance. Sustainable value will require matching capabilities to appropriate use cases rather than treating it as a cure-all.

Concerns about the Generative AI Bubble

With several issues in need of overcoming, it’s likely that fears of an AI bubble will persist. The mass adoption of Generative AI is still in its relative infancy, despite the huge number of companies that have already utilized the technology. As more companies take up Generative AI, fears may actually worsen. If an AI bubble does occur, it will be because of the following reasons.

Slowdown in Adoption

There are already signs of a slowdown in the adoption of Generative AI. People are starting to prefer creative work from humans rather than relying solely on AI-generated content. This preference for human creativity could hinder the growth and widespread adoption of Generative AI.

Capital Requirements

Many startups in the AI space rely on API calling and pre-trained models due to the high capital requirements for training their own models. This lack of capital can limit the growth and innovation of startups in the Generative AI sector.

Economic Factors

The global recession that is predicted to occur could have a significant impact on the AI industry. Investors may become more cautious and start pulling money from the market, leading to a decrease in funding for AI startups.

Legal and Ethical Concerns

Generative AI raises legal and intellectual property issues surrounding ownership and control of the content it generates . There are also concerns about ethics and bias resulting from the data AI systems are trained on. These concerns could lead to increased regulation and limitations on the use of Generative AI, making it more difficult for businesses to innovate.

Conclusion

The future of the Generative AI industry remains uncertain, and there are concerns about the potential bursting of the Generative AI bubble. While it is difficult to predict when this might happen, many are eagerly awaiting its outcome.

One of the main issues surrounding Generative AI is the high level of investment required and the replicability of the technology. These factors contribute to the uncertainty surrounding the industry's sustainability and long-term success.

To mitigate the risks and potential downfall of the Generative AI bubble, it is crucial to shift the focus from creating fancy product demos to building practical business use cases. This approach would require time and effort to develop and implement, but it could help ensure the industry's stability and growth.

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in Technology Management and a bachelor's degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.

More On This Topic

• ChatGPT, GPT-4, and More Generative AI News
• Are Data Scientists Still Needed in the Age of Generative AI?
• Synthetic Data Platforms: Unlocking the Power of Generative AI for…
• Free From Google: Generative AI Learning Path
• Whose Responsibility Is It To Get Generative AI Right?
• This Week in AI, August 7: Generative AI Comes to Jupyter & Stack Overflow…

Adobe is best known for its line of products for visual creators. But with the latest portfolio of 11 features and tools announced at the company’s Adobe Max event, it appears the design giant is confidently tapping the zeitgeist collectively by catering to its creative fans for software and even hardware.

Adobe’s visual tools like Photoshop, Premiere Pro, and Acrobat are familiar to nearly everyone. But Adobe is also making its way into another business — AI-powered wearable.

The company introduced Project Primrose, an interactive dress that demonstrates the potential use of “flexible textile displays,” allowing the wearer to display patterns and images on their body like a programmable screen.

While Adobe has technically already teased this “smart display fabric” technology before, we’ve only previously seen it applied to a flat canvas and a small handbag. As a dress, the numerous scale-like displays look cool but nowhere near practical.

“In an unpredictable economic climate, where consumers now re-evaluate the products and services they buy each day, a brand’s key growth driver is the ability to show people you accurately understand their current needs,” Anjul Bhambhri, senior vice president of Adobe Experience Cloud platform engineering, said.

On the text side of the equation Adobe introduced three new services based on LLMs. The recently introduced Adobe Experience Manager, Adobe Journey Optimizer, Customer Journey Analytics and Marketo Engage users will be the first to be able to take advantage of Generative AI Services, thanks to latest native workflow integrations.

Firefly Upgraded

The IT juggernaut also released the second generation of its image generator, Firefly Image 2 model that powers popular features like Photoshop’s Generative Fill — alongside two fresh Firefly models — creators of vector images and design templates.

Adobe says the model’s latest version stands out as a virtuoso in image generation. It delivers higher-quality images and understands the nuances of high-frequency details better than its predecessor. The model can photorealistic images perfecting vivid colours and near-to-perfect photorealistic images.

Changing the face of design the service provider is banging the AI drum really hard. At the core of its approach is Firefly, its inhouse bundle of generative AI models. Since its release the company has injected the model(s) trained on Adobe Stock images, along with public domain and copyright-free images in most of its offerings.

Just like the original text-to-image model, Adobe says its Firefly Vector model is designed to be safe for commercial use. The Image 2 is available to try via the web-based beta and will come soon to Creative Cloud apps as well.

(Water)marking its Territory

Adobe has also vowed to add metadata to AI-generated images, a digital signature of authenticity. In the near future, AI-generated images will include cryptographically signed data, allowing a quick distinction between AI and human-made images.

Adobe said it has the Content Credentials cloud, a vault for your image files’ metadata. If an image is shared without its unique metadata, Adobe’s cloud will show the missing signature.

“Once digital content is signed with Content Credentials (in a platform that has leverages the C2PA open-standards and Content Credentials), tamper-evident metadata is attached, so it travels with the content wherever it goes,” a spokesperson for Adobe told The Register.

As Adobe continues to power its tools with more AI possibilities, it’s leaving behind a slow period. While many tech companies have big, unclear ideas about AI, Adobe is focusing on practical uses that its existing user base of millions really like. They’ve been working on Firefly to make images, and now it’s even better in its second version. They’ve also added new features for making sound, video, and 3-D pictures. It’s like Adobe is an artist, painting a clear picture while others are still dreaming in fuzzy colors.

The post Adobe is Striking the Right Creative Chord With AI appeared first on Analytics India Magazine.

Introduction

Hello AI&ML Engineers, as you all know, Artificial Intelligence (AI) and Machine Learning Engineering are the fastest growing fields, and almost all industries are adopting them to enhance and expedite their business decisions and needs; for the same, they are working on various aspects and preparing the data for the AIML platform with the help of SMEs and AIML Experts to build the solutions.

Without a doubt, they’re undergoing, per the recommendation specific to the life cycle, picking the suitable algorithm(s) and giving solutions in terms of predictions, either Regression (or) Clustering (or) Classification modeling. Things are not stopping there.

To give more clarity, end users or stakeholders are looking for more clarity on solutions and justifications. This grey area is the so-called Black-Box.

Now in industry, the expensive addon in this series is the so-called Explainable AI (XAI), and hope you heard about this terminology. This addon would give confidence to the Machine Learning (ML) models that we are developing, and it is very transparent. This encourages AIML adoption across many industries like Banking, Finance, Healthcare, Retail, Manufacturing, and huge Research use cases.

In this article, we’re going to understand the below topic precisely within the stipulated timeline without getting bored.

• What is explainable AI?
• Why XAI
• Explainability Techniques
• Theory behind XAI
• Need For Model Explainability
• Consequences Of Poor ML Predictions

What is explainable AI?

Explainable artificial intelligence (XAI) is a collection of well-defined processes and methods that allows users to understand and trust the output created by properly chosen machine learning algorithms based on the problem statements used to describe an AI model, its anticipated impacts, and potential biases.

This article will provide you with the required intangible view on explainability techniques for machine learning (ML), along with key explanation methods and their approaches, which are required for the stakeholders and consumers to understand the transparency and interpretability of algorithms beyond their scope.

Generally speaking, there are multiple questions on the benefits of AI and ML adoption and how we can increase the scope of current business challenges and fulfill the consumer’s expectations.

So, to answer these questions, Explainable (XAI) is on the ground and helping many industries, and explainability has become a prerequisite now.

Why explainable Artificial Intelligence?

As we know, AIML is an Integral part of the digital business decision and prediction standpoint, and here the key concern of business stakeholders is the lack of clarity and interpretability as existing ML solutions predominantly use black-box algorithms and are highly subjected to human bias as we all know this. Potentially to remove the gap between the thoughts, The XAI has been introduced in the ML life cycle, and it is taking the responsibility and to seal the expectation to explain and translate BLACK-BOX ALGORITHMS, which are used for stakeholder critical business decision-making process; it becomes to increase their adoption and alignments.

You must understand that the XAI is the most effective best practice to validate that AI and ML solutions are transparent, accountable, ethical, and reliable. So it addresses algorithm requirements, transparency, risk evolution, and mitigation.

The XAI is a set of techniques that will provide light on choosing the algorithms and helping to function the same at each stage of the “Machine Learning” solution, same time without forgetting open-handed the facilities to business questions based on the outcome of ML models in the WHY AND HOW pattern.

Below is the block diagram of the Classical and XAI approaches.

Big picture of explainable Artificial Intelligence

Quickly, we could say that explainability can be applied in two stages: before the Modelling and after the Modelling.

In straight Data-centric and (Pre) and Model-Specific (Post). The below diagram shows this very precisely.”

Explainability Techniques

The Explainability can start by dividing the major categories

• Model-Specific explainability
• Model-Agnostic explainability
• Model-Centric explainability
• Data-Centric explainability

Model-Specific Explainability: This type of Explainability method is strictly relevant to specific Machine Learning model algorithm(s). Example: Decision Tree models are only specific to the Decision Tree algorithm, which comes under the Model-Specific Explainability method.

Model-Agnostic Explainability: This type of Explanation to any type of Machine Learning model regardless of the algorithm being used. Generally, post-analysis methods would be used after the Machine Learning model training, which is not dependent on any particular algorithm like Model-Specific Explainability and is not informed about the internal model structure and weights, if any. This is a flexible one.

Model-Centric: Traditionally, most Explanation methods are Model-Centric, as these methods are used to explain how the features and target values are being adjusted, apply the various algorithms, and extract the specific set of outcomes.

Data-Centric: I would say these methods are used to understand the nature of the data; I meant it’s consistent, well appropriate for solving business problems. As we know data plays a crucial role in building prediction and classical modeling. Sametime is necessary to understand the algorithm’s behavior concerning the given dataset. If the data is inconsistent, there are huge chances of the failure of the ML model. Data Profiling, Monitoring Data-Drifts, and Data-Adversarial are a few specific data-centric explainability approaches.

Model Explainability Methods: There are different approaches available to provide model explainability.

• Knowledge extraction
  • Exploratory Data Analysis (EDA)
• Result visualization
  • Comparison analysis
• Influence-based
  • Sensitivity Analysis and Feature Selection Importance

Knowledge extraction methods: This is Exploratory Data Analysis (EDA) process, which we use to extract critical insights and statistical information from the dataset. This is a post-analysis method and a kind of Model-Agnostic explainability.

Structures Dataset

Statistical Information outcome from across the different data points from the given dataset

• Mean and Median values
• Standard Deviation
• Variance

Insights from dataset

• Boxplots
• Distribution plots
• Heatmaps
• PDF plot

Example-based methods: This is something for non-technical end-users, By providing the best way to explain the Model functioning.

Influence-based methods: In which the feature(s) play an important role in influencing the model outcome and its decision-making process; most of the basic models are supported by this method by giving feature importance in decision-making.

Result visualization methods: This is just comparing model outcomes using specific Plotting methods.

The theory behind Explainable Artificial Intelligence

There are major theories behind the XAI, and we must understand that on top of the other factors. Below are a few key items from that list, and let’s go over them in crystal space.

• Benchmarks
• Commitment
• Reliability
• Perceptions
• Experiences
• Controlling Abnormality

Benchmarks for Explainable ML systems: As we know, there should be some set of benchmarks for the process to make successful; of course, for XAI to have the below parameters to fulfil the expectations while we are adopting the same at the organization level.

Commitment and Reliability: XAI should provide a holistic explanation and reliability explanation, leading to ML models’ commitment. These 2 factors play a key role, especially in a detailed RAC (Root Cause Analysis).

Perceptions and Experiences: These two factors always make much difference while dealing with RAC (Root Cause Analysis) for model predictions. There should be an excellent way of human-friendly explanations, which are always expected in a succinct and abstract presentation. Overloaded details lead to complications, and the end user’s experience would impact.

Controlling Abnormality: In ML solutions, abnormality in data is a usual challenge; we all know that. So, we have to carefully observe the nature of the DATA which we’re introducing to the algorithm and, before that, have to engage with the EDA process to understand the Data upside down even after the outcome; the model explanation should include abnormality explainability; so that end-user is much comfortable understanding on model outcomes irrespective of continuous value or categorical value in our datasets.

Let’s focus on the consequences of poor predictions and the need to overcome model explainability in the real-time scenario.

Consequences of poor ML predictions

• Most of the time, models suffer due to the reasons below; as we know, this is due to poor prediction.
  • Bias in the ML Algorithms
  • Bias in the Dataset for prediction
• Drift in
  • Data
  • Concept

Both are purely dependent on external factors and this item would collapse the model at any level and unstable model in production

• Model outcomes
  • Overfitted
  • Underfitted

Hope you’re familiar with these two factors and we could easily identify them during EDA process and modeling with test and train data.

• Data commitments
  • Quality of Training Data and Lack of Enough
    Data

Quality of the Data would be the major root cause for poor ML prediction, So each data engineers are responsible for this during onboarding the data into the data platform. And make sure the source system has this expectation from D&A team and ML engineers.

• Irrelevant Features selections

Feature selection is the major activity in tabular data, As ML engineers we should analyse all the required fields and exclude them from the test and training process itself. If required certainly we could use Dimensional reduction techniques as well. But we have to make sure what is our Dependent variable and Independent variable.

Conclusion

Guys, We have discussed the XAI at a high level and hope you understand, we can say XAI is

• What and Why Explainable AI?
• Major explainability techniques and theory behind XAI
• Need for model explainability and consequences of poor ML predictions.

Here is the takeaway about XAI in the below points.

• ‘Explainability’ and ‘interpretability’ are frequently used interchangeably.
• This integral role played by AI and ML models has led to the growing concern of business stakeholders and consumers about the lack of transparency and interoperability, as this black box is favorably subjected to bias.
• This plays a critical role in industrial operations; model explainability is a prerequisite. (such as healthcare, finance, legal, and others)
• XAI is the most effective practice to guarantee that AI and ML solutions are transparent
• This is trustworthy, responsible, and ethical so that all regulatory requirements on algorithmic transparency, risk mitigation, and a fallback plan are addressed efficiently.
• AI and ML explainability techniques provide visibility into how algorithms are operated at different stages.
• XAI is allowing end-users to ask questions about the consequences of AI and ML models.

Thanks for your time; I will get back to you with another topic shortly, then Bye! See you again – Shanthababu, Cheers!

NoBroker, last week announced the launch of CallZen.AI. This AI-powered platform provides conversational insights from almost all Indian languages regardless of the context, whether it is calls, chats, or meetings.

This is a completely new foray for NoBroker and it comes at an ideal time when everyone is using AI powered tools. Their customers include the banking, finance, insurance, health tech and edtech sector with a freemium model. For professionals and enterprises, it has started a one-month trial model.

This tool provides multiple features like extracting insights from customer conversations, identifying conversation highlights, semantic analysis of the text in different languages to point to the same information, summarisation, and smart clustering. This removes the need for manual filtering of data.

A notable feature of CallZen is its capacity to pinpoint important moments during customer conversations and schedule call-backs accordingly. Aside from improving internal processes; it also facilitates CRM integration, allowing businesses to set up custom CRM updates, emails, and alerts based on specific call highlights.

Real-time alerting based on conversational triggers is another feature that sets CallZen.AI apart. It allows businesses to integrate moments and checklists into their existing systems, ensuring that they are immediately alerted to any violations or unusual conversations between agents and customers. Agents’ performance is closely monitored using metrics such as call score, customer satisfaction rate, handling time, hold detection, talk ratio, and customer sentiments.

Businesses can export reports and visualise data through charts, graphs, and trends, providing them with a clear and concise overview of the metrics that matter most to their operations.

NoBroker’s Tech Prowess

NoBroker, a tech-driven real estate startup, prioritises technology over human reliance to address challenges. AI is crucial to their success, with a team of over 35 data scientists. They emphasise automation through AI techniques such as vision and NLP, creating proprietary products like that of Callzen.AI. This product, initially for internal use, is now offered as a SaaS platform to other companies.

The startup’s tech team includes 1,000 agents, and they record and validate over 7,000 hours of call centre conversations daily. Utilising NLP, they identify positive and negative customer interactions. ‘Rentometer’ and ‘NB Estimate’ are their rent and price prediction engines, offering personalised recommendations considering factors like location and commute time.

Vision analytics handle property images, and their system ‘IRIS Internal’ verifies image authenticity. NoBroker’s proprietary technology uses AI and user behaviour analytics to detect and eliminate fraudulent listings, resulting in a platform where 99.99% of properties are genuine. They’ve also created a substantial data lake called ‘Starship,’ containing 500 GB of data, enabling queries and business intelligence decisions with information from call data to business operations.

Read more: Data Science Hiring Process at NoBroker

The post NoBroker Launches CallZen.AI To Improve Customer Service appeared first on Analytics India Magazine.

Image by Author

Data cleaning is a critical part of any data analysis process. It's the step where you remove errors, handle missing data, and make sure that your data is in a format that you can work with. Without a well-cleaned dataset, any subsequent analyses can be skewed or incorrect.

This article introduces you to several key techniques for data cleaning in Python, using powerful libraries like pandas, numpy, seaborn, and matplotlib.

Understanding the Importance of Data Cleaning

Before diving into the mechanics of data cleaning, let's understand its importance. Real-world data is often messy. It can contain duplicate entries, incorrect or inconsistent data types, missing values, irrelevant features, and outliers. All these factors can lead to misleading conclusions when analyzing data. This makes data cleaning an indispensable part of the data science lifecycle.

We’ll cover the following data cleaning tasks.

Image by Author Setup for Data Cleaning in Python

Before getting started, let's import the necessary libraries. We'll be using pandas for data manipulation, and seaborn and matplotlib for visualizations.

We’ll also import the datetime Python module for manipulating the dates.

import pandas as pd  import seaborn as sns  import datetime as dt  import matplotlib.pyplot as plt  import matplotlib.ticker as ticker

Loading and Inspecting Your Data

First, we'll need to load our data. In this example, we're going to load a CSV file using pandas. We also add the delimiter argument.

df = pd.read_csv('F:\KDNuggets\KDN Mastering the Art of Data Cleaning in Python\property.csv', delimiter= ';')

Next, it's important to inspect the data to understand its structure, what kind of variables we're working with, and whether there are any missing values. Since the data we imported is not huge, let’s have a look at the whole dataset.

# Look at all the rows of the dataframe  display(df)

Here’s how the dataset looks.

You can immediately see there are some missing values. Also, the date formats are inconsistent.

Now, let’s take a look at the DataFrame summary using the info() method.

# Get a concise summary of the dataframe  print(df.info())

Here’s the code output.

We can see that only the column square_feet doesn’t have any NULL values, so we’ll somehow have to handle this. Also, the columns advertisement_date, and sale_date are the object data type, even though this should be a date.

The column location is completely empty. Do we need it?

We’ll show you how to handle these issues. We’ll start by learning how to delete unnecessary columns.

Deleting Unnecessary Columns

There are two columns in the dataset that we don’t need in our data analysis, so we’ll remove them.

The first column is buyer. We don’t need it, as the buyer’s name doesn’t impact the analysis.

We’re using the drop() method with the specified column name. We set the axis to 1 to specify that we want to delete a column. Also, the inplace argument is set to True so that we modify the existing DataFrame, and not create a new DataFrame without the removed column.

df.drop('buyer', axis = 1, inplace = True)

The second column we want to remove is location. While it might be useful to have this information, this is a completely empty column, so let’s just remove it.

We take the same approach as with the first column.

df.drop('location', axis = 1, inplace = True)

Of course, you can remove these two columns simultaneously.

df = df.drop(['buyer', 'location'], axis=1)

Both approaches return the following dataframe.

Handling Duplicate Data

Duplicate data can occur in your dataset for various reasons and can skew your analysis.

Let’s detect the duplicates in our dataset. Here’s how to do it.

The below code uses the method duplicated() to consider duplicates in the whole dataset. Its default setting is to consider the first occurrence of a value as unique and the subsequent occurrences as duplicates. You can modify this behavior using the keep parameter. For instance, df.duplicated(keep=False) would mark all duplicates as True, including the first occurrence.

# Detecting duplicates  duplicates = df[df.duplicated()]  duplicates

Here’s the output.

The row with index 3 has been marked as duplicate because row 2 with the same values is its first occurrence.

Now we need to remove duplicates, which we do with the following code.

# Detecting duplicates  duplicates = df[df.duplicated()]  duplicates

The drop_duplicates() function considers all columns while identifying duplicates. If you want to consider only certain columns, you can pass them as a list to this function like this: df.drop_duplicates(subset=['column1', 'column2']).

As you can see, the duplicate row has been dropped. However, the indexing stayed the same, with index 3 missing. We’ll tidy this up by resetting indices.

df = df.reset_index(drop=True)

This task is performed by using the reset_index() function. The drop=True argument is used to discard the original index. If you do not include this argument, the old index will be added as a new column in your DataFrame. By setting drop=True, you are telling pandas to forget the old index and reset it to the default integer index.

For practice, try to remove duplicates from this Microsoft dataset.

Data Type Conversion

Sometimes, data types might be incorrectly set. For example, a date column might be interpreted as strings. You need to convert these to their appropriate types.

In our dataset, we’ll do that for the columns advertisement_date and sale_date, as they are shown as the object data type. Also, the date dates are formatted differently across the rows. We need to make it consistent, along with converting it to date.

The easiest way is to use the to_datetime() method. Again, you can do that column by column, as shown below.

When doing that, we set the dayfirst argument to True because some dates start with the day first.

# Converting advertisement_date column to datetime  df['advertisement_date'] = pd.to_datetime(df['advertisement_date'], dayfirst = True)    # Converting sale_date column to datetime  df['sale_date'] = pd.to_datetime(df['sale_date'], dayfirst = True)

You can also convert both columns at the same time by using the apply() method with to_datetime().

# Converting advertisement_date and sale_date columns to datetime  df[['advertisement_date', 'sale_date']] = df[['advertisement_date', 'sale_date']].apply(pd.to_datetime, dayfirst =  True)

Both approaches give you the same result.

Now the dates are in a consistent format. We see that not all data has been converted. There’s one NaT value in advertisement_date and two in sale_date. This means the date is missing.

Let’s check if the columns are converted to dates by using the info() method.

# Get a concise summary of the dataframe  print(df.info())

As you can see, both columns are not in datetime64[ns] format.

Now, try to convert the data from TEXT to NUMERIC in this Airbnb dataset.

Handling Missing Data

Real-world datasets often have missing values. Handling missing data is vital, as certain algorithms cannot handle such values.

Our example also has some missing values, so let’s take a look at the two most usual approaches to handling missing data.

Deleting Rows With Missing Values

If the number of rows with missing data is insignificant compared to the total number of observations, you might consider deleting these rows.

In our example, the last row has no values except the square feet and advertisement date. We can’t use such data, so let’s remove this row.

Here’s the code where we indicate the row’s index.

df = df.drop(8)

The DataFrame now looks like this.

The last row has been deleted, and our DataFrame now looks better. However, there are still some missing data which we’ll handle using another approach.

Imputing Missing Values

If you have significant missing data, a better strategy than deleting could be imputation. This process involves filling in missing values based on other data. For numerical data, common imputation methods involve using a measure of central tendency (mean, median, mode).

In our already changed DataFrame, we have NaT (Not a Time) values in the columns advertisement_date and sale_date. We’ll impute these missing values using the mean() method.

The code uses the fillna() method to find and fill the null values with the mean value.

# Imputing values for numerical columns  df['advertisement_date'] = df['advertisement_date'].fillna(df['advertisement_date'].mean())  df['sale_date'] = df['sale_date'].fillna(df['sale_date'].mean())

You can also do the same thing in one line of code. We use the apply() to apply the function defined using lambda. Same as above, this function uses the fillna() and mean() methods to fill in the missing values.

# Imputing values for multiple numerical columns  df[['advertisement_date', 'sale_date']] = df[['advertisement_date', 'sale_date']].apply(lambda x: x.fillna(x.mean()))

The output in both cases looks like this.

Our sale_date column now has times which we don’t need. Let’s remove them.

We’ll use the strftime() method, which converts the dates to their string representation and a specific format.

df['sale_date'] = df['sale_date'].dt.strftime('%Y-%m-%d')

The dates now look all tidy.

If you need to use strftime() on multiple columns, you can again use lambda the following way.

df[['date1_formatted', 'date2_formatted']] = df[['date1', 'date2']].apply(lambda x: x.dt.strftime('%Y-%m-%d'))

Now, let’s see how we can impute missing categorical values.

Categorical data is a type of data that is used to group information with similar characteristics. Each of these groups is a category. Categorical data can take on numerical values (such as "1" indicating "male" and "2" indicating "female"), but those numbers do not have mathematical meaning. You can't add them together, for instance.

Categorical data is typically divided into two categories:

1. Nominal data: This is when the categories are only labeled and cannot be arranged in any particular order. Examples include gender (male, female), blood type (A, B, AB, O), or color (red, green, blue).

1. Ordinal data: This is when the categories can be ordered or ranked. While the intervals between the categories are not equally spaced, the order of the categories has a meaning. Examples include rating scales (1 to 5 rating of a movie), an education level (high school, undergraduate, graduate), or stages of cancer (Stage I, Stage II, Stage III).

For imputing missing categorical data, the mode is typically used. In our example, the column property_category is categorical (nominal) data, and there’s data missing in two rows.

Let’s replace the missing values with mode.

# For categorical columns  df['property_category'] = df['property_category'].fillna(df['property_category'].mode()[0])

This code uses the fillna() function to replace all the NaN values in the property_category column. It replaces it with mode.

Additionally, the [0] part is used to extract the first value from this Series. If there are multiple modes, this will select the first one. If there's only one mode, it still works fine.

Here’s the output.

The data now looks pretty good. The only thing that’s remaining is to see if there are outliers.

You can practice dealing with nulls on this Meta interview question, where you’ll have to replace NULLs with zeros.

Dealing with Outliers

Outliers are data points in a dataset that are distinctly different from the other observations. They may lie exceptionally far from the other values in the data set, residing outside an overall pattern. They're considered unusual due to their values either being significantly higher or lower compared to the rest of the data.

Outliers can arise due to various reasons such as:

• Measurement or input errors
• Data corruption
• True statistical anomalies

Outliers can significantly impact the results of your data analysis and statistical modeling. They can lead to a skewed distribution, bias, or invalidate the underlying statistical assumptions, distort the estimated model fit, reduce the predictive accuracy of predictive models, and lead to incorrect conclusions.

Some commonly used methods to detect outliers are Z-score, IQR (Interquartile Range), box plots, scatter plots, and data visualization techniques. In some advanced cases, machine learning methods are used as well.

Visualizing data can help identify outliers. Seaborn's boxplot is handy for this.

plt.figure(figsize=(10, 6))  sns.boxplot(data=df[['advertised_price', 'sale_price']])

We use the plt.figure() to set the width and height of the figure in inches.

Then we create the boxplot for the columns advertised_price and sale_price, which looks like this.

The plot can be improved for easier use by adding this to the above code.

plt.xlabel('Prices')  plt.ylabel('USD')  plt.ticklabel_format(style='plain', axis='y')  formatter = ticker.FuncFormatter(lambda x, p: format(x, ',.2f'))  plt.gca().yaxis.set_major_formatter(formatter)

We use the above code to set the labels for both axes. We also notice that the values on the y-axis are in the scientific notation, and we can’t use that for the price values. So we change this to plain style using the plt.ticklabel_format() function.

Then we create the formatter that will show the values on the y-axis with commas as thousand separators and decimal dots. The last code line applies this to the axis.

The output now looks like this.

Now, how do we identify and remove the outlier?

One of the ways is to use the IQR method.

IQR, or Interquartile Range, is a statistical method used to measure variability by dividing a data set into quartiles. Quartiles divide a rank-ordered data set into four equal parts, and values within the range of the first quartile (25th percentile) and the third quartile (75th percentile) make up the interquartile range.

The interquartile range is used to identify outliers in the data. Here's how it works:

1. First, calculate the first quartile (Q1), the third quartile (Q3), and then determine the IQR. The IQR is computed as Q3 — Q1.
2. Any value below Q1 — 1.5IQR or above Q3 + 1.5IQR is considered an outlier.

On our boxplot, the box actually represents the IQR. The line inside the box is the median (or second quartile). The 'whiskers' of the boxplot represent the range within 1.5*IQR from Q1 and Q3.

Any data points outside these whiskers can be considered outliers. In our case, it’s the value of $12,000,000. If you look at the boxplot, you’ll see how clearly this is represented, which shows why data visualization is important in detecting outliers.

Now, let’s remove the outliers by using the IQR method in Python code. First, we’ll remove the advertised price outliers.

Q1 = df['advertised_price'].quantile(0.25)  Q3 = df['advertised_price'].quantile(0.75)  IQR = Q3 - Q1  df = df[~((df['advertised_price'] < (Q1 - 1.5 * IQR)) |(df['advertised_price'] > (Q3 + 1.5 * IQR)))]

We first calculate the first quartile (or the 25th percentile) using the quantile() function. We do the same for the third quartile or the 75th percentile.

They show the values below which 25% and 75% of the data fall, respectively.

Then we calculate the difference between the quartiles. Everything so far is just translating the IQR steps into Python code.

As a final step, we remove the outliers. In other words, all data less than Q1 — 1.5 * IQR or more than Q3 + 1.5 * IQR.

The '~' operator negates the condition, so we are left with only the data that are not outliers.

Then we can do the same with the sale price.

Q1 = df['sale_price'].quantile(0.25)  Q3 = df['sale_price'].quantile(0.75)  IQR = Q3 - Q1  df = df[~((df['sale_price'] < (Q1 - 1.5 * IQR)) |(df['sale_price'] > (Q3 + 1.5 * IQR)))]

Of course, you can do it in a more succinct way using the for loop.

for column in ['advertised_price', 'sale_price']:      Q1 = df[column].quantile(0.25)      Q3 = df[column].quantile(0.75)      IQR = Q3 - Q1      df = df[~((df[column] < (Q1 - 1.5 * IQR)) |(df[column] > (Q3 + 1.5 * IQR)))]

The loop iterates of the two columns. For each column, it calculates the IQR and then removes the rows in the DataFrame.

Please note that this operation is done sequentially, first for advertised_price and then for sale_price. As a result, the DataFrame is modified in-place for each column, and rows can be removed due to being an outlier in either column. Therefore, this operation might result in fewer rows than if outliers for advertised_price and sale_price were removed independently and the results were combined afterward.

In our example, the output will be the same in both cases. To see how the box plot changed, we need to plot it again using the same code as earlier.

plt.figure(figsize=(10, 6))  sns.boxplot(data=df[['advertised_price', 'sale_price']])  plt.xlabel('Prices')  plt.ylabel('USD')  plt.ticklabel_format(style='plain', axis='y')  formatter = ticker.FuncFormatter(lambda x, p: format(x, ',.2f'))  plt.gca().yaxis.set_major_formatter(formatter)

Here’s the output.

You can practice calculating percentiles in Python by solving the General Assembly interview question.

Conclusion

Data cleaning is a crucial step in the data analysis process. Though it can be time-consuming, it's essential to ensure the accuracy of your findings.

Fortunately, Python's rich ecosystem of libraries makes this process more manageable. We learned how to remove unnecessary rows and columns, reformat data, and deal with missing values and outliers. These are the usual steps that have to be performed on most any data. However, you’ll also sometimes need to combine two columns into one, verify the existing data, assign labels to it, or get rid of the white spaces.

All this is data cleaning, as it allows you to turn messy, real-world data into a well-structured dataset that you can analyze with confidence. Just compare the dataset we started with to the one we ended up with.

If you don’t see the satisfaction in this result and the clean data doesn’t make you strangely excited, what in the world are you doing in data science!?

Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn.

More On This Topic

• Mastering the Art of Data Storytelling: A Guide for Data Scientists
• 7 Steps to Mastering Data Cleaning and Preprocessing Techniques
• Data Cleaning with Python Cheat Sheet
• Introduction to Python Libraries for Data Cleaning
• Exploring Data Cleaning Techniques With Python
• Data storytelling — the art of telling stories through data