Рубрика: AI

Photo by Nino Souza

Long gone are times in business when all the data you needed was in your ‘little black book’. In this era of the digital revolution, not even the classical databases are enough.

Handling big data became a critical skill for businesses and, with them, data scientists. Big data is characterized by its volume, velocity, and variety, offering unprecedented insights into patterns and trends.

To handle such data effectively, it requires the usage of specialized tools and techniques.

What is Big Data?

No, it’s not simply lots of data.

Big data is most commonly characterized by the three Vs:

• Volume – Yes, the size of the generated and stored data is one of the characteristics. To be characterized as big, the data size must be measured in petabytes (1,024 terabytes) and exabytes (1,024 petabytes)
• Variety – Big data doesn’t only consist of structured but also semi-structured (JSON, XML, YAML, emails, log files, spreadsheets) and unstructured data (text files, images and videos, audio files, social media posts, web pages, scientific data such as satellite images, seismic waveform data, or raw experimental data), with the focus being on the unstructured data.
• Velocity – The speed of generating and processing data.

Big Data Tools and Techniques

All the big data characteristics mentioned impact the tools and techniques we use to handle big data.

When we talk about big data techniques, they are simply methods, algorithms, and approaches we use to process, analyze, and manage big data. On the surface, they are the same as in regular data. However, the big data characteristics we discussed call for different approaches and tools.

Here are some prominent tools and techniques used in the big data domain.

1. Big Data Processing

What is it?: Data processing refers to operations and activities that transform raw data into meaningful information. It tasks from cleaning and structuring data to running complex algorithms and analytics.

Big data is sometimes batch processed, but more prevalent is data streaming.

Key Characteristics:

• Parallel Processing: Distributing tasks across multiple nodes or servers to process data concurrently, speeding up computations.
• Real-time vs. Batch Processing: Data can be processed in real-time (as it's generated) or in batches (processing chunks of data at scheduled intervals).
• Scalability: Big data tools handle vast data by scaling out, adding more resources or nodes.
• Fault Tolerance: If the node fails, the systems will continue processing, ensuring data integrity and availability.
• Diverse Data Sources: Big data comes from many sources, be it structured databases, logs, streams, or unstructured data repositories.

Big Data Tools Used: Apache Hadoop MapReduce, Apache Spark, Apache Tez, Apache Kafka, Apache Storm, Apache Flink, Amazon Kinesis, IBM Streams, Google Cloud Dataflow

Tools Overview:

2. Big Data ETL

What is it?: ETL is Extracting data from various sources, Transforming it into a structured and usable format, and Loading it into a data storage system for analysis or other purposes.

Big data characteristics mean that the ETL process needs to handle more data from more sources. Data is usually semi-structured or unstructured, which is transformed and stored differently than structured data.

ETL in big data also usually needs to process data in real time.

Key Characteristics:

• Data Extraction: Data is retrieved from various heterogeneous sources, including databases, logs, APIs, and flat files.
• Data Transformation: Converting the extracted data into a format suitable for querying, analysis, or reporting. Involves cleaning, enriching, aggregating, and reformatting the data.
• Data Loading: Storing the transformed data into a target system, e.g., data warehouse, data lake, or database.
• Batch or Real-time: Real-time ETL processes are more prevalent in big data than batch processing.
• Data Integration: ETL integrates data from disparate sources, ensuring a unified view of data across an organization.

Big Data Tools Used: Apache NiFi, Apache Sqoop, Apache Flume, Talend

Tools Overview:

Big Data ETL Tools
Tool	Key Features	Advantages
Apache NiFi	• Data flow design via a web-based UI • Data provenance tracking • Extensible architecture with processors	• Visual interface: Easy to design data flows • Supports data provenance • Extensible with a wide range of processors
Apache Sqoop	• Bulk data transfer between Hadoop and databases • Parallel import/export • Compression and direct import features	• Efficient data transfer between Hadoop and relational databases • Parallel import/export • Incremental data transfer capabilities
Apache Flume	• Event-driven and configurable architecture • Reliable and durable data delivery • Native integration with Hadoop ecosystem	• Scalable and distributed • Fault-tolerant architecture • Extensible with custom sources, channels, and sinks.
Talend	• Visual design interface • Broad connectivity to databases, apps, and more • Data quality and profiling tools	• Wide range of connectors for various data sources • Graphical interface for designing data integration processes • Supports data quality and master data management

3. Big Data Storage

What is it?: Big data storage must store vast amounts of data generated at high velocities and in various formats.

The three most distinct ways to store big data are NoSQL databases, data lakes, and data warehouses.

NoSQL databases are designed for handling large volumes of structured and unstructured data without a fixed schema (NoSQL — Not Only SQL). This makes them adaptable to the evolving data structure.

Unlike traditional, vertically scalable databases, NoSQL databases are horizontally scalable, meaning they can distribute data across multiple servers. Scaling becomes easier by adding more machines to the system. They are fault-tolerant, have low latency (appreciated in applications requiring real-time data access), and are cost-efficient at scale.

Data lakes are storage repositories that store vast amounts of raw data in their native format. This simplifies data access and analytics, as all data is located in one place.

Data lakes are scalable and cost-efficient. They provide flexibility (data is ingested in its raw form, and the structure is defined when reading the data for analysis), support batch and real-time data processing, and can be integrated with data quality tools, leading to more advanced analytics and richer insights.

A data warehouse is a centralized repository optimized for analytical processing that stores data from multiple sources, transforming it into a format suitable for analysis and reporting.

It is designed to store vast amounts of data, integrate it from various sources, and allow for historical analysis since data is stored with a time dimension.

Key Characteristics:

• Scalability: Designed to scale out by adding more nodes or units.
• Distributed Architecture: Data is often stored across multiple nodes or servers, ensuring high availability and fault tolerance.
• Variety of Data Formats: Can handle structured, semi-structured, and unstructured data.
• Durability: Once stored, data remains intact and available, even in the face of hardware failures.
• Cost-Efficiency: Many big data storage solutions are designed to run on commodity hardware, making them more affordable at scale.

Big Data Tools Used: MongoDB (document-based), Cassandra (column-based), Apache HBase (column-based), Neo4j (graph-based), Redis (key-value store), Amazon S3, Azure Data Lake, Hadoop Distributed File System (HDFS), Google Big Lake, Amazon Redshift, BigQuery

Tools Overview:

4. Big Data Mining

What is it?: It’s discovering patterns, correlations, anomalies, and statistical relationships in large datasets. It involves disciplines like machine learning, statistics, and using database systems to extract insights from data.

The amount of data mined is vast, and the sheer volume can reveal patterns that might not be apparent in smaller datasets. Big data usually comes from various sources and is often semi-structured or unstructured. This requires more sophisticated preprocessing and integration techniques. Unlike regular data, big data is usually processed in real time.

Tools used for big data mining have to handle all this. To do that, they apply distributed computing, i.e., data processing is spread across multiple computers.

Some algorithms might not be suitable for big data mining, as it requires scalable parallel processing algorithms, e.g., SVM, SGD, or Gradient Boosting.

Big data mining has also adopted Exploratory Data Analysis (EDA) techniques. EDA analyzes datasets to summarize their main characteristics, often using statistical graphics, plots, and information tables. Because of that, we’ll talk about big data mining and EDA tools together.

Key Characteristics:

• Pattern Recognition: Identifying regularities or trends in large datasets.
• Clustering and Classification: Grouping data points based on similarities or predefined criteria.
• Association Analysis: Discovering relations between variables in large databases.
• Regression Analysis: Understanding and modeling the relationship between variables.
• Anomaly Detection: Identifying unusual patterns.

Big Data Tools Used: Weka, KNIME, RapidMiner, Apache Hive, Apache Pig, Apache Drill, Presto

Tools Overview:

5. Big Data Visualization

What is it?: It’s a graphical representation of information and data extracted from vast datasets. Using visual elements like charts, graphs, and maps, data visualization tools provide an accessible way to understand patterns, outliers, and trends in the data.

Again, the characteristics of big data data, such as size and complexity, make it different from regular data visualization.

Key Characteristics:

• Interactivity: Big data visualization requires interactive dashboards and reports, allowing users to drill down into specifics and explore data dynamically.
• Scalability: Large datasets need to be handled efficiently without compromising performance.
• Diverse Visualization Types: E.g., heat maps, geospatial visualizations, and complex network graphs.
• Real-time Visualization: Many big data applications require real-time data streaming and visualization to monitor and react to live data.
• Integration With Big Data Platforms: Visualization tools often integrate seamlessly with big data platforms.

Big Data Tools Used: Tableau, PowerBI, D3.js, Kibana

Tools Overview:

Conclusion

Big data is so similar to regular data but also completely different. They share the techniques for handling data. But due to big data characteristics, these techniques are the same only by their name. Otherwise, they require completely different approaches and tools.

If you want to get into big data, you’ll have to use various big data tools. Our overview of these tools should be a good starting point for you.
Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn.

More On This Topic

• Top AI and Data Science Tools and Techniques for 2022 and Beyond
• Knowledge Graph Conference, covering tools, techniques, case studies and…
• Working with Python APIs For Data Science Project
• How to Ace Data Science Interview by Working on Portfolio Projects
• Want to Join a Bank? Everything Data Scientists Need to Know About Working…
• Getting Deep Learning working in the wild: A Data-Centric Course

Recently, iFlytek, an AI company based in China, released its Spark AI model for the public. Unlike other countries, companies in China go through a lot of security assessments and require clearance from the government to release products to the public. But China has been bullish on generative AI and thus the company was quickly allowed to release its product for the public.

Interestingly, Liu Qingfeng, founder and chairman of iFlytek claimed that Huawei’s GPU capabilities are now on par with NVIDIA’s A100 GPUs. Though Huawei has never actually said that it is developing its GPUs for AI capabilities, Liu comments about the astounding capabilities of the hardware definitely confirms that the company is making strides to fight against the NVIDIA monopoly.

Adding to all of this, Liu from iFlytek said that the company wants to release a general purpose AI model by October to compete with OpenAI’s ChatGPT, and with GPT-4 by the first of 2024. According to him, these ambitious plans wouldn’t have been possible without Huawei’s enhanced GPUs given the computational capabilities of the country’s hardware were lagging behind NVIDIA all this while.

This was confirmed by Jensen Huang, the CEO of NVIDIA, in a recent interaction with AIM. “Huawei is definitely a very formidable company,” said Huang. “They just caught up with A100s and we have to acknowledge that Huawei is one of the most technologically advanced companies in the world. Even without technological access, it is incredible what they did with their latest phone.” He further explains that Huawei has been an extraordinary company even before all the geo-political tensions.

Huawei is building China’s self-reliance

The US sanctions on NVIDIA for not exporting its chips to China has been constraining the country’s development even further. To curb this, Huawei has its own plans.

In 2019, Huawei released the Ascend 910 AI processor which is based on Ascend-Max chipsets. Eric Xu, the rotating chairman of Huawei said, “Without a doubt, it has more computing power than any other AI processor in the world,” definitely taking a hit on NVIDIA’s A100s. Interestingly, Ascend 910 accelerator delivered 256 TeraFLOPs of power for tensor floating point operations with just 310 W of max power, compared to 312 TeraFlops of NVIDIA A100, with 400 W of max power. This Ascend 910 powers Huawei’s Atlas 900 Pod A2 AI training cluster, which is possibly being used by iFlytek for its AI efforts.

In 2020, Huawei announced that it would be entering the GPU market soon. Even though the TFlops are not the only parameters for comparing the capabilities of GPUs, Huawei’s GPUs still are comparatively coming close to what NVIDIA’s are capable of.

Along this, Huawei has also been venturing into the phone market and fulfilling China’s self-reliance dreams. For this, Huawei decided to build its own smaller chips for developing its own phones.

In 2022, Semiconductor Manufacturing International Corp (SMIC) successfully built a 7-nanometer processor, which is powered by Kirin 9000s chip. Interestingly, Huawei’s latest phone, Mate 60 Pro is built on top of this 7-nanometer processor. Huawei is building the country’s self-reliance dreams by partnering with others within the country.

Moreover, China has realised that it can actually compete against US companies such as NVIDIA and Apple by the support of Huawei. The government decided to ban the use of Apple iPhones for government officials and employees. The country realised that it is one of the biggest markets of Apple and generates around a fifth of the revenue for the company. Given the sanctions by the US through NVIDIA, China decided to end its reliance on any company in the US, and thus Apple is taking the hit, which does not really come as a surprise.

Apple of China

Huawei released Mate 60 Pro, just weeks before Apple was supposed to release iPhone 15 on September 12. The 7-nanometer based phone is still behind Apple’s 4-nanometer chip based iPhone, but it could definitely make a dent at the iPhone maker’s revenue given the ban. After the launch of Huawei’s new phone, the stock prices of Apple tumbled 6 percent between Wednesday and Friday last week, resulting in nearly $200 billion off the value of the company.

Though regular citizens of China can still buy the new iPhone in the country, the sanctions and bans definitely bring up the question for US companies operating in China. There is a possibility that the country might decide to ban its products completely in the near future, and Huawei might come out as the champion for the country in its AI, chipmaking, phone market, and thus the self-reliance mission.

All in all, even though NVIDIA is undoubtedly one of the leaders when it comes to AI and chipmaking, and Apple is definitely the leading phone company in China, there is very little reason to believe that Huawei can completely take away the market leadership position from these tech giants. But Huawei is definitely on the rise on many different fronts, and as confirmed by Huang, and the current updates and releases, Huawei might possibly become the Apple or NVIDIA of China.

The post NVIDIA, Apple Have Got a Real Competitor Now appeared first on Analytics India Magazine.

Indian IT company Infosys is likely to partner with NVIDIA to use its infrastructure and capabilities to build AI models and applications, alongside reskilling 3 lakh+ employees in AI, hinted NVIDIA CEO Jensen Huang in a recent interaction with AIM.

In a recent gathering of some of the brightest #AI & startup minds, it was insightful to hear tech leaders, @NandanNilekani, Infosys, & Jensen Huang, Nvidia, discuss the future of AI and how it will transform the future. Stay tuned for what's coming #InfosysTopaz pic.twitter.com/3MsBww16OI

— Infosys (@Infosys) September 10, 2023

This development comes on the heels of NVIDIA’s recently announced partnerships with Reliance and Tata where both of the Indian tech giants will be working with NVIDIA to build one of the largest AI infrastructure in India using NVIDIA’s GH200 Grace Hopper Superchip and NVIDIA DGX Cloud which is an AI supercomputing service in the cloud.

Like Infosys, TCS had previously made an announcement about its plans to enhance the skills of its 600,000+ employees through its partnership with NVIDIA.

On his recent visit to India Huang met with influential tech leaders in India, including Infosys co-founder Nandan Nilekani, startup innovators, AI advocates, and key players in India’s digital infrastructure. He expressed his excitement, saying that India is on the verge of becoming a major player in the global AI field.

Huang emphasized India’s strength in information technology and the potential for AI to accelerate the development of the nation’s IT industry. “IT is one of your natural resources. You produce it at an incredible scale. You’re incredibly good at it. You export it all over the world,” Huang said.

Earlier this year Infosys launched Topaz, an AI-focused suite with 12,000 use cases, enabling industry-specific solutions in intelligent automation, AI-driven customer service, and enhanced security. Topaz helps businesses adopt open-source LLMs to build narrow transformers, solving specific enterprise challenges and driving growth. Infosys Chief Executive Officer Salil Parekh had mentioned that there are 50 client projects where the company is using generative AI.

The post Infosys Likely to Partner with NVIDIA to Train 3 Lakh+ Employees in AI appeared first on Analytics India Magazine.

AI Apps are domain-infused, AI/ML-powered applications that continuously learn and adapt with minimal human intervention in helping non-technical users manage data and analytics-intensive operations to deliver well-defined operational outcomes.

I originally introduced the idea of a “Data Product Development Canvas” as one of the capstone deliverables (the other being the data science Hypothesis Development Canvas) for my “Thinking Like a Data Scientist” methodology. Several folks and students tested the canvas and gave me great feedback. The most critical feedback was to focus on the ultimate end deliverable: creating a user-friendly, AI-powered app, or “AI App,” enabling non-data scientists to leverage data and analytics to deliver a well-defined outcome.

My favorite “AI App” is the Uber app. The Uber app leverages data and analytics to provide a frictionless experience for riders and drivers, such as matching them based on location, time, and preferences, estimating fares and trip duration, optimizing routes and traffic conditions, and personalizing recommendations and promotions.

Other examples of “AI Apps” include:

• Apple Maps (Google Maps, Waze) leverages data and analytics to deliver a well-defined operational outcome. Apple Maps uses data to provide a convenient and efficient experience for users who need to navigate from one place to another, such as finding the best route, estimating travel time and distance, avoiding traffic jams and road closures, and discovering nearby places and services.
• Netflix leverages data and analytics to deliver personalized and relevant entertainment experiences for users. Netflix uses machine learning algorithms to analyze user preferences, behavior, and feedback and provide recommendations for movies and shows that users may like. Netflix also uses artificial intelligence to optimize its streaming quality, content production, and marketing strategies.
• Spotify leverages data and analytics to deliver customized and engaging music experiences for users. Spotify uses machine learning algorithms to understand user tastes, moods, and contexts and provide recommendations for songs, playlists, podcasts, and artists that users may enjoy. Spotify also uses artificial intelligence to create personalized playlists, discover new music, and enhance sound quality.
• Google Photos leverages data and analytics to organize and manage users’ photos and videos. Google Photos uses machine learning algorithms to recognize faces, objects, places, and events in users’ photos and videos and group them into albums, stories, and collages. Google Photos also uses artificial intelligence to edit, enhance, and share users’ photos and videos.

AI Apps require blending five critical disciplines – Data Management, Data Science, Application Development, Design Thinking, and Customer Experience – to create an AI App that can continuously learn and adapt to deliver meaningful, relevant, responsible, and ethical business or operational outcomes. AI Apps provide the following benefits, unlike one-off (orphaned) analytic models:

• AI Apps are operational assets that can continuously learn and adapt with minimal human intervention.
• AI Apps are economic assets that can be shared, reused, and refined across multiple use cases at decreasing marginal costs.
• AI Apps can be strung together to address complex value chain processes when the AI Apps are built on a common architecture that facilitates the sharing, reuse, and continuous refinement of the enterprise’s data, KPIs, features, ML models, and user interface.

Let’s dive into explaining the AI App Product Development canvas. There is a lot to explore!

AI Apps Development Canvas Version 2.0 – Page 1

Figure 1 shows page 1 of the AI App Development Canvas.

Figure 1: Data Product Development Canvas – Page 1

Page 1 of the AI App Development Canvas covers the following information:

(0) AI App Description. Product Name (Eventa?), Author, Completion or Update date, and Version number.

(1) Business Problem & Ideal Outcome. What is the business/operational problem or opportunity we are trying to address, and what are the Ideal Outcomes?

(2) Business Value. What are the potential benefits (financial, operational, customer, environmental, employee) from addressing this Business Problem?

(3) Potential Impediments. What are potential impediments (technology, people skills, organization, timing) in addressing this Business Problem?

(4) Business Entities. What are the business entities (human and device/equipment) around which we seek to uncover predicted propensities?

(5) Targeted Users and Desired Outcomes.Who are the intended users, and what are their desired outcomes? Note: This critical section will require considerably more space than what is provided in the canvas.

(6) Upstream System Dependencies. What is the upstream system, application, or organizational dependencies for the AI App?

(7) Downstream System Dependencies. What is the downstream system, application, or organizational dependencies for the AI App?

(8) Key Decisions. What are the targeted users’ critical decisions or actions enabled by the app?

(9) KPIs and Metrics. What are the KPIs and metrics against which the targeted users will measure the effectiveness of the AI Apps?

(10) Analytic (Predictive) Scores. What Analytic Scores are required to power the recommendations that support the targeted users’ critical decisions?

(11) Prescriptive Recommendations. What prescriptive recommendations are required to support the targeted users’ critical decisions?

(12) User Gains (Benefits). What benefits (or gains) would users experience in achieving the desired outcome?

(13) User Pains (Impediments). What impediments (pains) have users experienced to achieve the desired outcome?

AI Apps Development Canvas Version 2.0 – Page 2

Figure 2 shows Page 2 of the AI App Development Canvas.

Figure 2: Data Product Development Canvas – Page 2

Page 2 of the Data Product Development Canvas collects the following information:

(14) Data Sources. What data sources are required to support the app?

(15) Data transformations. What data transformations & enrichments are required for feature engineering?

(16) Machine Learning (ML) features. What ML features are required for the ML models?

(17) Model Performance. What are the ML model accuracy and processing time requirements?

(18) Model Monitoring/Observability. How will the model be instrumented and monitored to manage model drift and drive continuous model learning and refining?

(20) Data Pipeline Performance. What is user environmental, architecture, and technical requirements?

(21) API Requirements. What are API requirements for integrating the analytic results into the management and operational systems?

(22) App UI / UEX. How should analytic results (visualizations) be presented to the users so that they are understandable and actionable?

(23) Privacy. What personal data needs to be captured to deliver desired outcomes and their privacy considerations (GDPR, HIPAA, CCPA, FRCA)?

(24) App Feedback. How will user feedback and app performance be captured and fed back into the AI App and its enabling analytic models?

AI App Development Canvas Summary – Part 1

I will test the AI App Development Canvas to define the requirements and specifications for a “Local Events Marketing Optimization” app. This test will be part of my ongoing series on integrating GenAI into my “Thinking Like a Data Scientist” methodology, which I have discussed in my blog posts (listed below).

Let’s see what GenAI / Bing can do…

Integrating Generative AI into the “Thinking Like a Data Scientist” methodology blog series:

• Integrating GenAI into “Thinking Like a Data Scientist” Methodology – Part I https://www.datasciencecentral.com/integrating-genai-into-thinking-like-a-data-scientist-methodology-part-i/
• Integrating GenAI into “Thinking Like a Data Scientist” Methodology – Part II https://www.datasciencecentral.com/integrating-genai-into-thinking-like-a-data-scientist-methodology-part-ii/
• Integrating GenAI into “Thinking Like a Data Scientist” Methodology – Part III https://www.datasciencecentral.com/integrating-genai-into-thinking-like-a-data-scientist-methodology-part-iii/

Image by Author

Through your journey as a data scientist, you will come across hiccups, and overcome them. You will learn how one process is better than another, and how to use different processes depending on your task at hand.

These processes will work hand-in-hand, to ensure that your data science project goes as effectively as possible and plays a key component in your decision-making process.

What is Data Management?

One process is data management. Living in a data-driven world, data management is an important element for organizations to leverage their data assets and ensure they are effective.

It is the process of collecting, storing, organizing and maintaining data to ensure that it is accurate, accessible to those who need it and reliable throughout your data science project lifecycle. Just like any management process, it requires procedures that are backed and supported by policies and technologies.

The key components of data management in data science projects are:

• Data Collection and Acquisition
• Data Cleaning and Preprocessing
• Data Storage
• Data Security and Privacy
• Data Governance and Documentation
• Collaboration and Sharing

As you can see, there are a few key components. It may look daunting right now, but I will go through each one to give you an overview of what to expect as a data scientist.

Data Collection and Acquisition

Although there is a lot of data out there today, data collection will still be a part of your role as a data scientist. Data collection and acquisition is the process of gathering raw data from a variety of sources such as websites, surveys, databases and more. This phase is very important as the quality of your data has a direct impact on your outcome.

You will need to identify different data sources and find ones that fit your requirements. Ensure that you have the right permissions to access these data sources, the reliability of the data sources, and the format is aligned with your scope. You can collect the data through different methods such as manual data entry, data extraction, and more.

Throughout these steps, you want to ensure data integrity and accuracy.

Data Cleaning and Preprocessing

Once you have your data, the next step is cleaning it — which can take up a lot of your time. You will need to comb through the dataset, find any issues and correct them. Your end goal during this phase will be to standardize and transform your data so that it’s ready for analysis.

Data cleaning can help with handling missing values, duplicate data, incorrect data types, outliers, data format, transformation, and more.

Data Storage

Once you have cleaned through your data and it’s of good quality and ready for analysis — store it! You don’t want to lose all those hours you just put in to clean it and get it to the gold standard.

You will need to choose the best data storage solution for your project and organization, for example, databases or cloud storage. Again, this will all be based on data volume and complexity. You can also design architecture that can allow for efficient data retrieval and scalability.

Another tool you can implement is data versioning and archiving which allows you to maintain all historical data and any changes to help preserve the data assets and long-term access.

Data Security and Privacy

We all know how important data is in this day and age, so protect it at all costs! Data breaches and privacy violations can have severe consequences, and you don’t want to have to deal with this problem.

There are some steps that you can take to ensure data security and privacy, such as access control, encryption, regular audits, data lifecycle management, and more. You want to ensure whatever route you take to protect your data that it is compliant with data privacy regulations, such as GDPR.

Data Governance and Documentation

If you want to ensure data quality and accountability throughout the data lifecycle, data governance and documentation are essential to your data management process. This process involves having policies, processes and best practices in place to ensure that your data is well-managed and all your assets are protected. The main aim of this is to provide transparency and compliance.

All these policies and processes should be documented comprehensively to provide insight into how the data is structured, stored, and used. This builds trust within an organization, and how they use data to drive the decision-making process to steer away from risks and find new opportunities.

Examples of processes include creating comprehensive documentation, metadata, maintaining an audit trail and providing data lineage.

Collaboration and Sharing

Data science projects consist of collaborative workflows, and with this, you can imagine how messy it can get. You have one data scientist working on the same dataset that another data scientist is doing further cleaning on.

To ensure data management within the team, it is always good to communicate your tasks so that you do not overlap with one another, or one person has a better version of a dataset than someone else.

Collaboration within a data science team ensures that the data is accessible and valuable to different stakeholders. To improve collaboration and sharing within a data science team, you can have data-sharing platforms, use collaborative tools such as Tableau, put access controls in place, and allow feedback.

Data Management Tools and Technologies

Okay now that we’ve gone through the key components of data management, I will now create a list of data management tools and technologies that can help you in your data science project lifecycle.

Relational Database Management Systems (RDBMS):

• MySQL
• PostgreSQL
• Microsoft SQL Server

NoSQL Databases:

• MongoDB
• Cassandra

Data Warehouse

• Amazon Redshift
• Google BigQuery
• Snowflake

ETL (Extract, Transform, Load) Tools:

• Apache NiFi
• Talend
• Apache Spark

Data Visualization and Business Intelligence:

• Tableau
• Power BI

Version Control and Collaboration:

• Git
• GitHub

Data Security and Privacy:

• Varonis
• Privitar

Wrapping it up

Data management is an important element of your data science project. See it as the foundation that is holding your castle up. The better and more effective the data management process is, the better your outcome. I have provided a list of articles that you can read to learn more about data management.

Resources and Further Learning

• 5 Data Management Challenges with Solutions
• Top 5 Data Management Platforms
• Free Data Management with Data Science Learning with CS639
• Why is Data Management so Important to Data Science?

Nisha Arya is a Data Scientist, Freelance Technical Writer and Community Manager at KDnuggets. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.

More On This Topic

• Creating Good Meaningful Plots: Some Principles
• Machine Learning Model Development and Model Operations: Principles and…
• A First Principles Theory of Generalization
• Why is Data Management so Important to Data Science?
• Free Data Management with Data Science Learning with CS639
• Free 4 Week Data Science Course on AI Quality Management

Fresh off the press is the TIME AI 100 list. Among the tech titans stands Charlie Brooker, the man who suggested a different point of view of the technologies being built in Silicon Valley. The 52-year-old writer extraordinaire is often credited for his uncanny scripted Netflix series ‘Black Mirror’. Since its UK premiere in 2011, Brooker has consistently shattered the rose-tinted glasses through which we’ve long adored tech — since Meta was Facebook and X was Twitter. He has made us question: What if all this machine circus isn’t a blessing, but a curse?

Time and again since Brooker’s worst-case scenario lens hit thve air, he has managed to correctly predict the future in horrible ways. The debut episode of the latest season glances at how people might have to contend with managing their digital alter egos. The phenomenon went on to become the figurehead of the Hollywood writers’ strike — which was sparked by the anxiety swirling around the AI’s dear child ChatGPT taking away the writers’ livelihoods.

The striking writers are grappling with another pressing issue: How do we regulate these AI-generated doppelgängers?

Coming back to Black Mirror, Season 6, Episode 1, introduces us to Joan, a tech exec whose life becomes a biographical drama on a Netflix-ish platform named Streamberry, portrayed by none other than Salma Hayek. While the concept of a celebrity living in your shoes might sound enticing, it’s anything but for Joan. Every day, she’s haunted by her own and reel-Hayek’s actions, exposing her daily shenanigans and regrettable choices.

Things spiral downwards from there when she realises the Hayek playing Joan on screen is an AI generated replica of the actress who has sold the rights to use her face to the company behind the show-about-the-show.

Joan-esque Hollywood

Enter Soul Machines, a company that could transform this dystopia into reality. A 2021 report by The Verge revealed that this classic Black Mirror company, co-founded by Greg Cross, primarily creates harmless customer service avatars.

Much like Hayek in Brooker’s Netflix universe, Soul Machines has digitised NBA and K-pop icons, according to their website. The Information has even reported that “many stars and agents are quietly taking meetings with AI companies to explore their options.” While the company opens up new avenues for monetizing celebrity likenesses, it also exposes them to the risk of damaging their brand.

Echoes of the futuristic past

Very recently AI tools have started to mimic famous as well as historical figures and the parallel timeline in Black Mirror feels closer than ever.

SAG-AFTRA, the union representing over 160,000 actors, warns that generative AI and technologies alike could leave “principal performers and background actors vulnerable to having most of their work replaced by digital replicas.”

The wishful fantasy of hyper-personalised content tailored to individual tastes, generated by AI is not distant. Superstar Jennifer Lopez’s digital twin is already campaigning for cruise ads by mimicking her voice and appearance. The campaign boosted bookings at the same time stirring concerns about misuse. While JLo’s team has taken precautionary measures – the society has highly been using deepfakes awfully.

Often, Black Mirror’s storylines have seemed to foreshadow some of the darker developments in the Bay Area. Perhaps, in the not-so-distant future there’ll be a Joan reading an article on AIM about an episode titled “Joan Is Awful,” only for it to become a scene “Joan Is Awful” — reflecting the world’s obsession with being digitally real.

The post Black Mirror Feels Closer Than Ever appeared first on Analytics India Magazine.

ChatGPT, which is probably the most used tech tool in the history of humanity, has played a pivotal role in normalising AI; a field researchers have been digging into since the 1940sand ChatGPT has been instrumental in bringing it to the forefront of mainstream media. The big techs, billion-dollar startups and think tanks behind the conversational tech have left no stone unturned to publicise the commodity; to the extent that the century-old TIME magazine had to introduce a list of 100 visionaries reshaping the global landscape of AI. However, even with this comprehensive list, some well-deserving individuals didn’t receive the recognition they should have. Let’s take a look at some of such AI experts and their contributions.

Andrej Karpathy

When it comes to giving back to the open-source community, Andrej Karpathy, the computer vision genius, is the first name that pops into everyone’s mind. Back in January, he released NanoGPT, a fast repository for training and tuning medium-sized GPTs, building upon his earlier work with miniGPT for GPT language models. His latest contribution is baby Llama which he made by tuning NanoGPT to use Meta’s Llama 2 architecture instead of GPT-2.

The former AI director at Tesla came to fame for his immense contribution to create Optimus, Tesla’s groundbreaking humanoid robot, recently rejoined OpenAI. Karpathy also played a pivotal role as the head of Tesla Autopilot’s computer vision team. He is also known for his comprehensive educational resources, coding tutorials on YouTube and more.

Mira Murati
OpenAI has been the most influential name in AI since the team released ChatGPT in December 2022. This is evident as all the cofounders, namely Greg Brockman, Ilya Sutsvekar and Sam Altman are rightly placed in the list. Along with that we also have Jan Leike (Superalignment Co-Lead) and Anna Makanju (VP of Global Head). However, Mira Murati, the CTO of OpenAI is missing. Since her appointment in 2018 as the VP of Applied AI and Partnerships and eventually becoming the CTO in 2019, Murati has spearheaded the making and release of notable generative AI models like DALL.E-2, GPT-3, GPT-3.5, GPT-4 and more.

The 35-year-old completed mechanical engineering from Dartmouth College, interned at Goldman Sachs, and has a rich experience of working in engineering roles at Zodiac Aerospace and Tesla for the ‘Model X’. She later became VP of Product and Engineering at Leap Motion, a company specialising in hand motion-controlled technology for PCs and Macs, where she oversaw the launch of hand-tracking software for VR.

Jakub Pachocki

Talking about OpenAI, another name that has surprisingly not made it to the list is Jakub Pachocki, Principal of Research, who joined the company in 2017. He is the brain behind the much celebrated GPT-4. According to Altman, “we wouldn’t be here without him”. He has garnered immense recognition for technical vision and leadership, emphasising his pivotal role in GPT-4’s development. Pachocki got into AI when AlphaGo came out, and he saw that deep learning could push computers further. He played a big part in OpenAI’s development of a bot for Dota 2, where it trained by playing itself until it reached a pro level.

David Silver

Google DeepMind is often referred to as the champion of reinforcement learning and that is because of their principal research scientist David Silver who missed the well-deserving spot on the list. He is also a professor at the University of London. His research primarily revolves around the development of AI agents through RL techniques. He co-led the project that integrated deep learning and RL to achieve proficiency in playing Atari games directly from pixel data, and spearheaded the AlphaGo initiative, making it the first time for a program to defeat a top professional player in the complex abstract strategy game of Go.

Additionally, Silver’s leadership of the AlphaZero project resulted in an AI system autonomously mastering chess, shogi, and Go, surpassing the world’s strongest programs. Most recently, he co-led the AlphaStar project, which achieved grandmaster-level gameplay in StarCraft. His contributions have garnered him prestigious recognition, including the ACM Prize in Computing, Marvin Minsky award, Mensa Foundation Prize, and the Royal Academy of Engineering Silver Medal.

Ian Goodfellow

Notably absent from the list is Ian Goodfellow, the renowned creator of Generative Adversarial Nets, who currently serves as a research scientist at Google DeepMind. GANs were one of the early methods of generating images before diffusion models came in. Currently working as a research scientist at Google DeepMind, Goodfellow is a prominent figure in the AI field. He holds the distinction of being the first author of the textbook “Deep Learning” (2016), co-authored with AI experts Yoshua Bengio and Aaron Courville. The Standford graduate has held significant positions at OpenAI and Apple in the past.

At Google DeepMind, Goodfellow collaborates with Oriol Vinyals, a principal research scientist, with whom he has partnered on various research projects including one concerning the TensorFlow interface while both were at Google. Again in 2015, they worked together on research related to neural network optimisation problems.

Ashish Vaswani

It’s pretty safe to say that BIT Mesra graduate Ashish Vaswani played a significant role in the success of companies like OpenAI, who are renowned for their cutting edge NLP models. ChatGPT would not have come to existence it it wasnt for his contributions. Vaswani is one of the key contributors to the Transformer model, which eliminates the need for sequential processing in tasks involving sequences, relying solely on self-attention mechanisms. This model has greatly influenced the development of many other cutting-edge NLP models, such as BERT, GPT-2, GPT-3, GPT-3.5, and GPT-4. The Transformer model was introduced in the paper “Attention Is All You Need“, during his time at Google. He also cofounded and was the chief scientist of Adept AI Labs that works towards useful general intelligence. However, he left the company last year and is working on his new stealth startup. AI enthusiasts are rightly disappointed that he did not make it to the TIME list.

Russ Salakhutdinov

Often referred to as a hero of deeplearning, Canadian AI researcher Ruslan Salakhutdinov specialises in probabilistic graphical models and large-scale optimisation, with Geoff Hinton as his doctoral advisor during PhD. He’s famous for creating Bayesian Program Learning, addressing one-shot learning, which mimics how humans grasp concepts from a single example. He served as a director of ML at Apple for a sometime before joining Felix Smart in 2023 as a board director, a company that uses AI to take care for plants and animals. He is also a computer science professor at Carnegie Mellon University and has published over 42 machine learning papers since 2009, backed by funding from Google, Microsoft, and Samsung.

Daphne Kaller

Another important AI expert who wasn’t mentioned on the TIME AI list is Israeli American computer scientist Daphne Koller who is acclaimed for her groundbreaking contributions to machine learning and probabilistic models, as well as their application in the fields of biology and human health. However, she was featured as one of the 100 most influential people by TIME in 2013. But since then, she has made significant contributions in the field as AI came into life. Apart from that, she is also recognised for her efforts in democratising education as she also founded an online learning platform along with Andrew Ng who made it to the list.

Presently, Kaller leads Insitro, a biotech startup focusing on discovering improved medicines through the integration of machine learning and biology at scale. She played a pivotal role in advancing graphical models, focusing on both model structure and parameters, and merging statistical learning with relational modeling languages. Additionally, she developed fundamental techniques for inference and learning in temporal models, with her textbook on Probabilistic Graphical Models co-authored with Nir Friedman serving as a definitive reference in the field. In the realm of life sciences, Koller introduced Module Networks, harnessing modularity in gene regulation to create a powerful gene activity model. Her pioneering applications of ML to pathology showcased its ability to surpass human pathologists and emphasized the significance of stromal tissue in cancer prognosis. The ACM Prize in Computing recipient’s influential work in combining relational logic and probability revolutionised how uncertainty is managed in complex computer systems, spanning databases, image recognition, biological and medical models, and natural language processing.

Rana el Kaliouby

TIME magazine overlooked another significant individual, Rana el Kaliouby, the pioneering mind behind “Emotion AI,” who has fundamentally reshaped our interactions and exchanges within an ever more technology-driven global landscape. Driven by her upbringing in a technologically inclined family and inspired by Rosalind Picard’s book on Affective Computing, she infused AI with emotional intelligence. Through her work at MIT and the founding of Affectiva, she pioneered emotion technology, allowing computers to detect and respond to human emotions through facial expressions. This emotion recognition software holds a multitude of potential uses, spanning diverse fields such as linguistics and video content creation. Individuals with autism, who exhibit a broad spectrum of emotions outside the typical range, could potentially enhance their mood awareness with assistance from their parents or caregivers. Moreover, the realm of computer-generated facial imagery, including the prospect of android endeavours, stands to achieve heightened realism and subtlety for production purposes.

We will discuss the rest AI leaders and their contributions in the following part of the story.

The post Top 15 AI Leaders Who Were Left Out of The TIME AI 100 List (Part 1) appeared first on Analytics India Magazine.

Introduction to Structured Query Language

When it comes to managing and manipulating data in relational databases, Structured Query Language (SQL) is the biggest name in the game. SQL is a major domain-specific language which serves as the cornerstone for database management, and which provides a standardized way to interact with databases. With data being the driving force behind decision-making and innovation, SQL remains an essential technology demanding top-level attention from data analysts, developers, and data scientists.

SQL was originally developed by IBM in the 1970s, and became standardized by ANSI and ISO in the late 1980s. All types of organizations — from small businesses to universities to major corporations — rely on SQL databases such as MySQL, SQL Server, and PostgreSQL to handle large-scale data. SQL's importance continues to grow with the expansion of data-driven industries. Its universal application makes it a vital skill for various professionals, in the data realm and beyond.

SQL allows users to perform various data-related tasks, including:

• Querying data
• Inserting new records
• Updating existing records
• Deleting records
• Creating and modifying tables

This tutorial will offer a step-by-step walkthrough of SQL, focusing on getting started with extensive hands-on examples.

Step 1: Setting Up Your SQL Environment

Choosing a SQL Database Management System (DBMS)

Before diving into SQL queries, you'll need to choose a database management system (DBMS) that suits your project's needs. The DBMS serves as the backbone for your SQL activities, offering different features, performance optimizations, and pricing models. Your choice of a DBMS can have a significant impact on how you interact with your data.

• MySQL: Open source, widely adopted, used by Facebook and Google. Suitable for a variety of applications, from small projects to enterprise-level applications.
• PostgreSQL: Open source, robust features, used by Apple. Known for its performance and standards compliance.
• SQL Server Express: Microsoft's entry-level option. Ideal for small to medium applications with limited requirements for scalability.
• SQLite: Lightweight, serverless, and self-contained. Ideal for mobile apps and small projects.

Installation Guide for MySQL

For the sake of this tutorial, we will focus on MySQL due to its widespread usage and comprehensive feature set. Installing MySQL is a straightforward process:

1. Visit MySQL's website and download the installer appropriate for your operating system.
2. Run the installer, following the on-screen instructions.
3. During the setup, you will be prompted to create a root account. Make sure to remember or securely store the root password.
4. Once installation is complete, you can access the MySQL shell by opening a terminal and typing mysql -u root -p. You'll be prompted to enter the root password.
5. After successful login, you'll be greeted with the MySQL prompt, indicating that your MySQL server is up and running.

Setting Up a SQL IDE

An Integrated Development Environment (IDE) can significantly enhance your SQL coding experience by providing features like auto-completion, syntax highlighting, and database visualization. An IDE is not strictly necessary for running SQL queries, but it is highly recommended for more complex tasks and larger projects.

• DBeaver: Open source and supports a wide range of DBMS, including MySQL, PostgreSQL, SQLite, and SQL Server.
• MySQL Workbench: Developed by Oracle, this is the official IDE for MySQL and offers comprehensive tools tailored for MySQL.

After downloading and installing your chosen IDE, you'll need to connect it to your MySQL server. This usually involves specifying the server's IP address (localhost if the server is on your machine), the port number (usually 3306 for MySQL), and the credentials for an authorized database user.

Testing Your Setup

Let's make sure that everything is working correctly. You can do this by running a simple SQL query to display all existing databases:

SHOW DATABASES;

If this query returns a list of databases, and no errors, then congratulations! Your SQL environment has been successfully set up, and you are ready to start SQL programming.

Step 2: Basic SQL Syntax and Commands

Creating a Database and Tables

Before adding or manipulating data, you will first need both a database and one table, at minimum. Creating a database and a table is accomplished by:

CREATE DATABASE sql_tutorial;  USE sql_tutorial;  CREATE TABLE customers (    id INT PRIMARY KEY AUTO_INCREMENT,     name VARCHAR(50),    email VARCHAR(50)  );

Manipulating Data

Now you are ready for data manipulation. Let's have a look at the basic CRUD operations:

• Insert: INSERT INTO customers (name, email) VALUES ('John Doe', 'john@email.com');
• Query: SELECT * FROM customers;
• Update: UPDATE customers SET email = 'john@newemail.com' WHERE id = 1;
• Delete: DELETE FROM customers WHERE id = 1;

Filtering and Sorting

Filtering in SQL involves using conditions to selectively retrieve rows from a table, often using the WHERE clause. Sorting in SQL arranges the retrieved data in a specific order, typically using the ORDER BY clause. Pagination in SQL divides the result set into smaller chunks, displaying a limited number of rows per page.

• Filter: SELECT * FROM customers WHERE name = 'John Doe';
• Sort: SELECT * FROM customers ORDER BY name ASC;
• Paginate: SELECT * FROM customers LIMIT 10 OFFSET 20;

Data Types and Constraints

Understanding data types and constraints is crucial for defining the structure of your tables. Data types specify what kind of data a column can hold, such as integers, text, or dates. Constraints enforce limitations to ensure data integrity.

• Integer Types: INT, SMALLINT, TINYINT, etc. Used for storing whole numbers.
• Decimal Types: FLOAT, DOUBLE, DECIMAL. Suitable for storing numbers with decimal places.
• Character Types: CHAR, VARCHAR, TEXT. Used for text data.
• Date and Time: DATE, TIME, DATETIME, TIMESTAMP. Designed for storing date and time information.

CREATE TABLE employees (      id INT PRIMARY KEY AUTO_INCREMENT,      first_name VARCHAR(50) NOT NULL,      last_name VARCHAR(50) NOT NULL,      birth_date DATE,      email VARCHAR(50) UNIQUE,      salary FLOAT CHECK (salary > 0)    );

In the above example, the NOT NULL constraint ensures that a column cannot have a NULL value. The UNIQUE constraint guarantees that all values in a column are unique. The CHECK constraint validates that the salary must be greater than zero.

Step 3: More Advanced SQL Concepts

Joining Tables

Joins are used to combine rows from two or more tables based on a related column between them. They are essential when you want to retrieve data that is spread across multiple tables. Understanding joins is crucial for complex SQL queries.

• INNER JOIN: SELECT * FROM orders JOIN customers ON orders.customer_id = customers.id;
• LEFT JOIN: SELECT * FROM orders LEFT JOIN customers ON orders.customer_id = customers.id;
• RIGHT JOIN: SELECT * FROM orders RIGHT JOIN customers ON orders.customer_id = customers.id;

Joins can be complex but are incredibly powerful when you need to pull data from multiple tables. Let's go through a detailed example to clarify how different types of joins work.

Consider two tables: Employees and Departments.

-- Employees Table  CREATE TABLE Employees (    id INT PRIMARY KEY,    name VARCHAR(50),    department_id INT  );    INSERT INTO Employees (id, name, department_id) VALUES  (1, 'Winifred', 1),  (2, 'Francisco', 2),  (3, 'Englebert', NULL);    -- Departments Table  CREATE TABLE Departments (    id INT PRIMARY KEY,    name VARCHAR(50)  );    INSERT INTO Departments (id, name) VALUES  (1, 'R&D'),  (2, 'Engineering'),  (3, 'Sales');  

Let's explore different types of joins:

-- INNER JOIN  -- Returns records that have matching values in both tables    SELECT E.name, D.name   FROM Employees E  INNER JOIN Departments D ON E.department_id = D.id;    -- LEFT JOIN (or LEFT OUTER JOIN)  -- Returns all records from the left table,  -- and the matched records from the right table    SELECT E.name, D.name   FROM Employees E  LEFT JOIN Departments D ON E.department_id = D.id;    -- RIGHT JOIN (or RIGHT OUTER JOIN)  -- Returns all records from the right table  -- and the matched records from the left table    SELECT E.name, D.name   FROM Employees E  RIGHT JOIN Departments D ON E.department_id = D.id;  

In the above examples, the INNER JOIN returns only the rows where there is a match in both tables. The LEFT JOIN returns all rows from the left table, and matching rows from the right table, filling with NULL if there is no match. The RIGHT JOIN does the opposite, returning all rows from the right table and matching rows from the left table.

Grouping and Aggregation

Aggregation functions perform a calculation on a set of values and return a single value. Aggregations are commonly used alongside GROUP BY clauses to segment data into categories and perform calculations on each group.

• Count: SELECT customer_id, COUNT(id) AS total_orders FROM orders GROUP BY customer_id;
• Sum: SELECT customer_id, SUM(order_amount) AS total_spent FROM orders GROUP BY customer_id;
• Filter group: SELECT customer_id, SUM(order_amount) AS total_spent FROM orders GROUP BY customer_id HAVING total_spent > 100;

Subqueries and Nested Queries

Subqueries allow you to perform queries within queries, providing a way to fetch data that will be used in the main query as a condition to further restrict the data that is retrieved.

SELECT *    FROM customers    WHERE id IN (      SELECT customer_id      FROM orders      WHERE orderdate > '2023-01-01'    );

Transactions

Transactions are sequences of SQL operations that are executed as a single unit of work. They are important for maintaining the integrity of database operations, particularly in multi-user systems. Transactions follow the ACID principles: Atomicity, Consistency, Isolation, and Durability.

BEGIN;    UPDATE accounts SET balance = balance - 500 WHERE id = 1;    UPDATE accounts SET balance = balance + 500 WHERE id = 2;    COMMIT;

In the above example, both UPDATE statements are wrapped within a transaction. Either both execute successfully, or if an error occurs, neither execute, ensuring data integrity.

Step 4: Optimization and Performance Tuning

Understanding Query Performance

Query performance is crucial for maintaining a responsive database system. An inefficient query can lead to delays, affecting the overall user experience. Here are some key concepts:

• Execution Plans: These plans provide a roadmap of how a query will be executed, allowing for analysis and optimization.
• Bottlenecks: Identifying slow parts of a query can guide optimization efforts. Tools like the SQL Server Profiler can assist in this process.

Indexing Strategies

Indexes are data structures that enhance the speed of data retrieval. They are vital in large databases. Here's how they work:

• Single-Column Index: An index on a single column, often used in WHERE clauses; CREATE INDEX idx_name ON customers (name);
• Composite Index: An index on multiple columns, used when queries filter by multiple fields; CREATE INDEX idx_name_age ON customers (name, age);
• Understanding When to Index: Indexing improves reading speed but can slow down insertions and updates. Careful consideration is needed to balance these factors.

Optimizing Joins and Subqueries

Joins and subqueries can be resource-intensive. Optimization strategies include:

• Using Indexes: Applying indexes on join fields improves join performance.
• Reducing Complexity: Minimize the number of tables joined and the number of rows selected.

SELECT customers.name, COUNT(orders.id) AS total_orders    FROM customers    JOIN orders ON customers.id = orders.customer_id    GROUP BY customers.name    HAVING orders > 2;

Database Normalization and Denormalization

Database design plays a significant role in performance:

• Normalization: Reduces redundancy by organizing data into related tables. This can make queries more complex but ensures data consistency.
• Denormalization: Combines tables to improve read performance at the cost of potential inconsistency. It's used when read speed is a priority.

Monitoring and Profiling Tools

Utilizing tools to monitor performance ensures that the database runs smoothly:

• MySQL's Performance Schema: Offers insights into query execution and performance.
• SQL Server Profiler: Allows tracking and capturing of SQL Server events, helping in analyzing performance.

Best Practices in Writing Efficient SQL

Adhering to best practices makes SQL code more maintainable and efficient:

• Avoid SELECT *: Select only required columns to reduce load.
• Minimize Wildcards: Use wildcards sparingly in LIKE queries.
• Use EXISTS Instead of COUNT: When checking for existence, EXISTS is more efficient.

SELECT id, name   FROM customers   WHERE EXISTS (      SELECT 1       FROM orders       WHERE customer_id = customers.id  );

Database Maintenance

Regular maintenance ensures optimal performance:

• Updating Statistics: Helps the database engine make optimization decisions.
• Rebuilding Indexes: Over time, indexes become fragmented. Regular rebuilding improves performance.
• Backups: Regular backups are essential for data integrity and recovery.

Step 5: Performance & Security Best Practices

Performance Best Practices

Optimizing the performance of your SQL queries and database is crucial for maintaining a responsive and efficient system. Here are some performance best practices:

• Use Indexes Wisely: Indexes speed up data retrieval but can slow down data modification operations like insert, update, and delete.
• Limit Results: Use the LIMIT clause to retrieve only the data you need.
• Optimize Joins: Always join tables on indexed or primary key columns.
• Analyze Query Plans: Understanding the query execution plan can help you optimize queries.

Security Best Practices

Security is paramount when dealing with databases, as they often contain sensitive information. Here are some best practices for enhancing SQL security:

• Data Encryption: Always encrypt sensitive data before storing it.
• User Privileges: Grant users the least amount of privileges they need to perform their tasks.
• SQL Injection Prevention: Use parameterized queries to protect against SQL injection attacks.
• Regular Audits: Conduct regular security audits to identify vulnerabilities.

Combining Performance and Security

Striking the right balance between performance and security is often challenging but necessary. For example, while indexing can speed up data retrieval, it can also make sensitive data more accessible. Therefore, always consider the security implications of your performance optimization strategies.

Example: Secure and Efficient Query

-- Using a parameterized query to both optimize  -- performance and prevent SQL injection    PREPARE secureQuery FROM 'SELECT * FROM users WHERE age > ? AND age < ?';  SET @min_age = 18, @max_age = 35;  EXECUTE secureQuery USING @min_age, @max_age;  

This example uses a parameterized query, which not only prevents SQL injection but also allows MySQL to cache the query, improving performance.

Moving Forward

This getting started guide has covered the fundamental concepts and popular practical applications of SQL. From getting up and running to mastering complex queries, this guide should have provided you with the skills you need to navigate data management through the use of detailed examples and with a practical approach. As data continues to shape our world, mastering SQL opens the door to a variety of fields, including data analytics, machine learning, and software development.

As you progress, consider extending your SQL skill set with additional resources. Sites like w3schools SQL Tutorial and SQL Practice Exercises on SQLBolt provide additional study materials and exercises. Additionally, HackerRank's SQL problems provide goal-oriented query practice. Whether you're building a complex data analytics platform or developing the next generation of web applications, SQL is a skill you will definitely be using regularly. Remember that the journey to SQL mastery traverses a long road, and is a journey that is enriched by consistent practice and learning.

Matthew Mayo (@mattmayo13) holds a Master's degree in computer science and a graduate diploma in data mining. As Editor-in-Chief of KDnuggets, Matthew aims to make complex data science concepts accessible. His professional interests include natural language processing, machine learning algorithms, and exploring emerging AI. He is driven by a mission to democratize knowledge in the data science community. Matthew has been coding since he was 6 years old.

More On This Topic

• Getting Started with Python Data Structures in 5 Steps
• Getting Started with SQL Cheatsheet
• Getting Started with 5 Essential Natural Language Processing Libraries
• Getting Started with Distributed Machine Learning with PyTorch and Ray
• Getting Started with Reinforcement Learning
• Getting Started with Automated Text Summarization

While AI has the potential to address the most complex global issues, it is crucial to use it responsibly and take into account the negative consequences of its application to mitigate harm. When companies jump onto the bandwagon of embracing emerging technologies without considering the broader social, economic, cultural, and political environments, they may jeopardise privacy and security while worsening existing inequalities. So, let’s delve into some of the top courses and certification programs to learn about ethics in AI.

Responsible AI Governance Badge Program

The EqualAI Badge Program, in partnership with the World Economic Forum, readies senior corporate leaders in AI-driven companies to establish a reputation for responsible and inclusive practices. Participants will gain expertise in creating and upholding responsible AI governance, become part of a supportive network of senior executives who share their values, and obtain certification through the EqualAI badge for mastering AI governance best practices.

AI Ethics

This Oxford University course on AI ethics covers fundamental concepts and broader philosophical considerations regarding AI’s ethical implications in our daily lives. The program starts by defining AI and distinguishing it from other machine learning methods, while also addressing the urgency of AI ethics. It then explores the ethics of AI creation, questioning its moral permissibility and the conditions for an AI to have moral significance. Next, the course delves into the philosophical aspects of designing ethical AI, including rule-based, bottom-up, and top-down approaches. It also examines the potential threats posed by AI to humanity and the feasibility of aligning our goals with AI’s objectives. Finally, the program scrutinizes the ethics surrounding specific AI applications, such as fully automated war drones, autonomous vehicles, robots, and AI-driven healthcare diagnostics.

Ethics of AI

The Ethics of AI is a free online course offered by the University of Helsinki, designed for those interested in the ethical dimensions of AI. The course seeks to educate individuals about AI ethics, exploring the boundaries of ethical AI development and encouraging ethical considerations in AI endeavours. Throughout the program, participants will work around the ethical dilemmas associated with responsible AI use and advancement, familiarise themselves with the ethical queries and principles relevant to modern AI and apply ethical theories and concepts practically to integrate them with AI applications.

Ethics of AI: Safeguarding Humanity

This MIT-led course equips you with the skills to handle ethical dilemmas in AI development and usage. It delves into AI’s ethical dimensions, spotlighting issues like machine bias and ethical hazards while prompting you to weigh your personal and organizational obligations. Over a three-day span, you’ll tackle the ethical facets of implementing AI at your workplace, gaining insights into harnessing AI for the greater good of humanity.

The Ethics of AI

In this three-week online masterclass by the London School of Economics, participants will apply moral concepts like fairness, transparency, and inequality to real-world scenarios to effectively navigate ethical dilemmas, posed by AI. The course explores how AI is applied in various business contexts, such as hiring and employee oversight, and its implications for issues like discrimination and power imbalances. Participants will develop practical skills for immediate application, engaging in live sessions, ethical investigations in AI, and connecting with a global network of peers. By the end of the program, attendees will have a toolkit to navigate AI’s ethical dilemmas, critical thinking skills to debate key AI ethics issues, and an understanding of AI’s impact on inequality, resource distribution, and power dynamics in the workplace. This three-week online masterclass requires a commitment of 6-8 hours per week.

Certified Ethical Emerging Technologist

In this comprehensive program spanning five courses, our team of AI pioneers, ethical experts, and researchers will guide you in mastering the essential facets of ethics in data-driven technology. These modules cover fundamental ethical principles, industry-standard frameworks, how to identify and mitigate ethical risks, adept communication on ethical dilemmas, and the establishment of organizational governance crucial for fostering ethical, trusted, and inclusive data-driven innovations. Upon completing all five courses, you will have a know-how to bridge the gap between theory and practical application and how to apply ethical principles, frameworks, regulations, and standards within the realm of data-driven technologies, adept at recognizing and addressing ethical hazards in the entire lifecycle of such technologies, skilled in communicating effectively with a diverse range of stakeholders about ethical safeguards and risk mitigation strategies, and proficient in crafting, implementing, and assessing organisational policies and governance structures essential for maintaining ethical data-driven technologies.

Ethics in the Age of AI Specialisation

LearnQuest’s four-part course series covers a range of essential topics. Participants will first grasp the concept of predictive models and their practical applications in the business world. Next, they’ll explore the pervasive use of learning algorithms in daily life. The course will also delve into the potentially biased impact of algorithms on human behaviour and strategies to mitigate such bias. Lastly, participants will learn to pinpoint vulnerabilities within public data sets and assess violations of algorithmic privacy.

AI Ethics: Global Perspectives

This course aims to explain the societal consequences of technology and empower both individuals and organisations to engage in ethical and responsible utilisation of AI and data. Geared towards present and prospective data scientists, policymakers, and business executives, it introduces fresh monthly lectures centred on data and AI issues. Each module features a video presentation along with supplementary materials like videos, readings, and podcasts to enhance learning.

Read more: LLMs are an Ethical Nightmare

The post Top 8 Courses & Certifications on AI Ethics appeared first on Analytics India Magazine.

NVIDIA recently announced it is set to release TensorRT-LLM in coming weeks, an open source software that promises to accelerate and optimize LLM inference.

TensorRT-LLM encompasses a host of optimizations, pre- and post-processing steps, and multi-GPU/multi-node communication primitives, all designed to unlock unprecedented performance levels on NVIDIA GPUs.

Notably, this software empowers developers to experiment with new LLMs, offering peak performance and customization capabilities without necessitating expertise in C++ or NVIDIA CUDA.

Naveen Rao, Vice President of Engineering at Databricks, lauded TensorRT-LLM, describing it as “easy to use, feature-packed with streaming of tokens, in-flight batching, paged-attention, quantization, and more.” He emphasized that it delivers state-of-the-art performance for LLMs on NVIDIA GPUs, ultimately benefiting customers with cost savings.

Performance benchmarks demonstrate the significant improvements brought by TensorRT-LLM on the latest NVIDIA Hopper architecture. For instance, the H100 alone is 4x faster than A100. Adding TensorRT-LLM and its benefits, including in-flight batching, result in an 8X total increase to deliver the highest throughput.

Furthermore, TensorRT-LLM demonstrated its ability to accelerate inference performance for Meta’s 70-billion-parameter Llama 2 model by a staggering 4.6x when compared to A100 GPUs.

Today’s LLMs are incredibly versatile, serving a multitude of tasks with varying output sizes. TensorRT-LLM addresses this challenge with in-flight batching, an optimized scheduling technique that allows for the concurrent execution of requests.

With the rapid innovation in the LLM ecosystem and the emergence of larger, more advanced models, the need for multi-GPU coordination and optimization has become paramount. TensorRT-LLM leverages tensor parallelism, a model parallelism technique, to efficiently scale LLM inference across multiple GPUs and servers. This automation eliminates the need for developers to manually split models and manage execution across GPUs.

TensorRT-LLM also equips developers with a wealth of open-source NVIDIA AI kernels, including FlashAttention and masked multi-head attention, to optimize models as they evolve.

​​To access TensorRT-LLM, developers can apply for early access through the NVIDIA Developer Program

The post NVIDIA Introduces TensorRT-LLM To Accelerate LLM Inference on H100 GPUs appeared first on Analytics India Magazine.