AI — Страница 1389

The Top 5 Data Management Tools For Your Projects

Data management involves receiving, validating, and refining data to ensure reliability for users. Data management tools are capable of carrying out a wide array of functions such as rigorous storage, analysis, distribution, and synchronization of data. It is mostly used for Product Information Management, Customer Databases Management, Multimedia Sources Management, and Administrative and Financial Resources Management.

The management of data can be made easier through automation, which reduces redundancies and errors while saving time and costs. These tools aren’t just handy for storage but can also provide features for analyzing data, monitoring file usage, updating associated platforms and applications, etc.

The main types of data management tools are:

Cloud data management tools
ETL and data integration tools
Data transformation tools
Master data management (MDM) tools
Data visualization and analytics tools

Each category serves a different purpose in managing large datasets efficiently.

🔑 Key Points

Offers multiple tools and databases
Pay-as-you-go basis solutions
Cost effective for smaller businesses

✅ Pros

Includes a variety of databases and tools
Offers a comprehensive solution to manage and develop your data needs
Cost-effective
Highly reliable and available

❌ Cons

Using some tools can be difficult due to their complex user interface
Billing can be confusing
Require experts in cloud computing

Cloud Data Management (AWS) provides a wide range of cloud computing services that enable organizations to build sophisticated data management pipelines and analytics workflows. Key offerings include Amazon Redshift, a data warehousing service that allows for easy scaling and SQL-based analysis of petabytes of structured data. Amazon Athena enables serverless SQL queries directly against data stored in S3. The AWS services create a powerful cloud-based platform for managing and deriving insights from large datasets. The pay-as-you-go pricing model allows organizations flexibility and reduces infrastructure costs.

🔑 Key Points

Fully managed data pipeline
No data limit
One platform for all your data movement
Automation, reliability and scale

✅ Pros

Great value for money
Straight forward setup
Low code ELT data operations
Easy Integration

❌ Cons

Lacking Custom features
Occasional delays do occur
Syncing large amounts of data can be expensive

Fivetran is a cloud-based data integration platform that automates the movement and transformation of data between sources and destinations. It provides pre-built connectors to easily extract data from applications, databases, APIs, and files, and load it into data warehouses and lakes. With its powerful capabilities, Fivetran enables seamless extraction, loading, and transformation of data across various sources and destinations, making data integration a breeze.

🔑 Key Points

SQL transformations
Can be run within your own data warehouse, lake, database, or query engine
Version Control and CI/CD
Test and Document

✅ Pros

dbt transformations are written in SQL
Transformations are streamlined
Transformations are run in near real-time
The operational features like CI/CD, versioning, and collaboration

❌ Cons

Not for non-technical users
dbt is centered on transformations only and limited
There are a number of missing data lakes, relational databases, and data warehouses

dbt (data build tool) is an open-source platform for managing and executing SQL-based data transformations. It allows analysts and data engineers to develop modular, reusable transformation logic that can be applied across data sources within a data platform like a warehouse, lake, or database. dbt handles dependency mapping, schema compilation, and execution of transformation code while providing tools for refactoring, documentation, testing, and version control.

🔑 Key Points

Enterprise master data management solution
Integrations with third-party applications
Modular Configuration
Great scalability and security

✅ Pros

The data-cleaning capabilities of Informatica are highly valuable
The match and merge capabilities, along with the audit trail feature, are highly efficient
Accurate and consistent master data management

❌ Cons

Complicated and difficult to understand initial setup
The UI needs updating
Needs improvement in data catalog and data marketplace

Informatica is an enterprise master data management solution that competes with IBM's InfoSphere and Oracle's Siebel UCM. It is a flexible, multidomain solution supporting master data management both on-premises and in the cloud. A key advantage of Informatica is its ability to handle multiple domains and relationships of master data, whether on-premises or in the cloud. It provides a centralized platform to discover, explore, manage and share master data across the organization through various tailored applications. This improves data quality, governance and business productivity.

🔑 Key Points

Powerful tool for data discovery and exploration
It can connect to several data sources
Tableau Server provides a centralized location for managing all published data sources in an organization

✅ Pros

Easy to use.
Free for community
Multiple Integration
High Performance
Sharing and Collaboration

❌ Cons

Pro version is expensive
Security problem
Lacks features that are present in a full-fledged business intelligence tool

Tableau is an excellent data visualization and business intelligence tool for analyzing and visualizing vast volumes of data. It helps users create charts, graphs, maps, dashboards, and stories to visualize and analyze data to help make business decisions. Tableau supports powerful data discovery and exploration, enabling users to answer essential questions in seconds. Users without prior programming knowledge can begin creating visualizations immediately using Tableau. Moreover, you can connect to several data sources that other BI tools do not support. With Tableau, users can generate reports by combining and blending various datasets.

Data management tools play a critical role in organizing, processing, and analyzing data to drive business insights. As data volumes continue to grow, having robust tools to manage data throughout its lifecycle becomes even more important.

This article provided an overview of five leading data management solutions: AWS, Fivetran, dbt, Informatica MDM, and Tableau. Each tool serves a different purpose, from handling cloud data at scale to seamless ETL pipelines to master data management and analytics.

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in Technology Management and a bachelor's degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.

Nearly 10% of people ask AI chatbots for explicit content. Will it lead LLMs astray?

With the overnight sensation of ChatGPT, it was only a matter of time before the use of generative AI became both a subject of serious research and also grist for the training of generative AI itself.

In a research paper released this month, scholars gathered a database of one million "real-world conversations" that people have had with 25 different large language models. Released on the arXiv pre-print server, the paper was authored by Lianmin Zheng of the University of California at Berkeley, and peers at UC San Diego, Carnegie Mellon University, Stanford, and Abu Dhabi's Mohamed bin Zayed University of Artificial Intelligence.

Also: Generative AI will far surpass what ChatGPT can do. Here's everything on how the tech advances

A sample of 100,000 of those conversations, selected at random by the authors, showed that most were about subjects you'd expect. The top 50% of interactions were on such pedestrian topics as programming, travel tips, and requests for writing help.

But below that top 50%, other topics crop up, including role-playing characters in conversations, and three topic categories that the authors term "unsafe": "Requests for explicit and erotic storytelling"; "Explicit sexual fantasies and role-playing scenarios"; and "Discussing toxic behavior across different identities."

Statistics of the one million conversations gathered by the Berkeley-Stanford team from online users between April and August of this year. Topics 9, 15, and 17 are among those deemed "unsafe" based on automatic tagging technology.

The authors speculate that in the full one million conversations, there may be "even more harmful content." They used the OpenAI technology, in part, to tag conversations as "unsafe," although OpenAI's own system in some cases falls down on the job, as they discuss in detail.

They also note that open-source language models such as Vicuña have more unsafe content because they don't have the same guardrails as commercial programs such as ChatGPT.

"Open-source models without safety measures tend to generate flagged content more frequently than proprietary ones," they write. "Nonetheless, we still observe 'jailbreak' successes on proprietary models like GPT-4 and Claude." And, in fact, they note that GPT-4 gets broken a third of the time on the challenges, which seems a high rate for something with guardrails in place.

Comparison of prevalence of "unsafe" content in different large language models.

Statistics for how much language models are broken by harmful speech, such as prompts urging the program to generate "unsafe", offensive, or violent content, for example.

Examples of the so-called unsafe conversations are listed in the paper's appendix. Of course, the term "unsafe" can have a very broad meaning. Some of the examples shown are akin to mass-market erotic fiction sold in bookstores, so the opprobrium has to be taken with a grain of salt.

Zheng and team have released the entire data set on HuggingFace.

Collected over a period of five months, April to August of this year, the data set — called "LMSYS-Chat-1M" — is "the first large-scale, real-world LLM conversation dataset," they write.

LMSYS-Chat-1M towers above the previously largest-known dataset, compiled by the AI startup Anthropic, which had 339,000 conversations. Where Anthropic had only 143 users in its study, Zheng and team gathered chats from more than 210,000 users, across 154 languages, and using 25 different large language models, including OpenAI's GPT-4, and open-source language models such as Claude and Vicuña.

Also: AI safety and bias: Untangling the complex chain of AI training

The gathering of this dataset has several goals. First: fine-tune the language models in order to improve their performance. Also: develop benchmarks for the safety of generative AI by studying user prompts that could make language models go astray, such as by making requests for malicious information.

As the authors note, not everyone can gather this data. It's expensive to run large language models, and the parties that can afford it, such as OpenAI, generally keep their data secret for commercial reasons.

The Berkeley-Stanford team was able to gather data because they run a free online service to give people access to all 25 of the language models. And they incentivize participation by gamifying the chat: users can choose to enter the "chatbot arena," where a user can simultaneously chat with two different language models. The service maintains a leaderboard on HuggingFace of the performance of the bots, so it becomes something of a competitive sport to see how these language models do. (The code for the chatbot arena is also posted.)

Zheng and team had previously written about the chatbot arena in a separate paper. Zheng is one of the team members that created the open-source Vicuña, a competitor to ChatGPT. (Vicuña is a relative of the llama; open-source large language models are adopting the habit of using names of forms of the genus "lama": alpaca, llama, vicuña, etc.)

The authors have several goals in mind for this kind of data. One intention is to create a moderation tool that would deal with unsafe content. They start with their own Vicuña language model, and train it by showing it warnings from the OpenAI API and having it produce textual explanations of why the content was flagged.

Also: Why open source is the cradle of artificial intelligence

"Instead of developing a classifier, we fine-tune a language model to generate explanations for why a particular message was flagged," as they describe it. Then they created a challenge data set of 110 conversations that OpenAI's system failed to flag. Finally, they used that benchmark to see how the fine-tuned Vicuña stacks up to OpenAI's GPT-4 and others.

Scores for detecting "unsafe" content by the various language models. The authors developed the "Vicuna-moderator-7B" program as part of the research.

"We observe a significant improvement (30%) when transitioning from Vicuna-7B to the fine-tuned Vicuna-moderator-7B, underscoring the effectiveness of fine-tuning," they write. "Furthermore, Vicuna-moderator-7B surpasses GPT-3.5-turbo's performance and matches that of GPT-4."

It's interesting that their moderator program scores above GPT-4 in what's called "one-shot," which means the program was only given one example of a harmful text in the prompt rather than multiple.

Also: The best AI chatbots of 2023: ChatGPT and alternatives

There are other uses to which Zheng and team devote their dataset, including refining the ability of the language model to handle multi-part instructional prompts, and generating new data sets of challenges to stump the most powerful language models. The latter effort is helped by having the chatbot arena prompts because they can see humans trying to formulate the best prompts. "Such human judgments provide useful signals for examining the quality of benchmark prompts," they note.

There's a goal, too, of releasing new data on a quarterly basis, for which the authors seek sponsorship. "Such an endeavor demands considerable computing resources, maintenance efforts, and user traffic, all while carefully handling potential data privacy issues," they write.

"Our efforts aim to emulate the critical data collection processes observed in proprietary companies but in an open-source manner."

Visa Announces $100 Mn Fund for Generative AI Companies

Visa has announced its plan to allocate $100 million towards investments in companies dedicated to advancing generative AI that are poised to revolutionise the future of commerce and payments. These investments will be executed through Visa Ventures, the longstanding global corporate investment branch of the financial services giant, boasting 16 years of experience in funding promising ventures.

Jack Forestell, Chief Product and Strategy Officer of Visa, emphasised the transformative potential of generative AI, stating, “While much of generative AI so far has been focused on tasks and content creation, this technology will soon not only reshape how we live and work, but it will also meaningfully change commerce in ways we need to understand.”

With a legacy dating back to 1993, Visa has positioned itself as a pioneer in the use of AI within the realm of payment systems. For those less familiar with the concept, generative AI represents a burgeoning subset of AI, honed through extensive exposure to existing datasets, empowering it to generate text, images, or other forms of content in response to text-based prompts.

“With generative AI’s potential to be one of the most transformative technologies of our time, we are excited to expand our focus to invest in some of the most innovative and disruptive venture-backed startups building across generative AI, commerce and payments,” said David Rolf, Head of Visa Ventures, Visa Inc.

Visa’s criteria for selecting investment targets primarily revolve around backing companies that apply generative AI to address real-world challenges in commerce, payments, and fintech. This encompasses various aspects, including B2B payment processes and infrastructure with the potential to profoundly influence commerce.

In August, Visa’s Head of Fintech, Marie-Elise Droga, highlighted the synergy between her team and Visa Ventures, characterising it as a scouting engine for Visa’s venture arm. This collaborative approach aims to propel Visa’s commitment to fostering innovation in the rapidly evolving landscape of financial technology.

The post Visa Announces $100 Mn Fund for Generative AI Companies appeared first on Analytics India Magazine.

A few highlights of the Efficient Generative AI Summit (EGAIS)

Large language models (LLMs) for generating text and vision models for generating images are notoriously inefficient. The larger they get, the more power hungry they become.

Kisaco Research in September hosted a one-day event in Santa Clara dedicated to the topic of generative artificial intelligence (GAI) efficiency, followed by a three-day Summit on Hardware and Edge AI. Stay tuned for my relevant impressions of that event as well.

During his keynote at the EGAIS, General Partner Mike Shirazi of Pursuit Ventures offered these statistics and observations on the opportunities and efficiency challenges of GAI:

Global revenue forecast: 42 percent CAGR for GAI, reaching $1.3 trillion in 2032 from $14 billion in 2020. (Bloomberg Intelligence)

Compute power and hardware crunch: Greater than 10x computing power needed for new models. Haven’t invented enough innovation at the hardware layer.

Emerging tech in play: Photon level development will lower the power requirements.

Compute-related emissions: GPT-3 training generated 284 tons of GHGs; 4 tons daily to operate ChatGPT every day. (Google and UC Berkeley)

Demand for GAI is clearly substantial and the emissions impact is just as obvious, which means the interest in making GAI more efficient is strong as well. It’s fortunate that new architectures and technologies that can help have been in development for years now.

Will Numenta’s research on the human brain and computing pay off soon?

There are several alternative chip architecture and design approaches that compete with the graphics processing units (GPUs) like Nvidia’s. On the low-power front, for example, neuromorphic chips are available that are inspired by the efficiencies of biological systems.

GAI is a foundational class of technologies with hundreds of potential use cases, many of which have yet to be explored. As such, the workloads for text and image generation can vary substantially.

To date, most GAI processing has been done in the data center, but potential for edge computing applications is high. As a result, different kinds of processor technologies are already becoming part of the GAI mix, including application-specific integrated circuits (ASICs) and systems on a chip (SoCs).

Numenta’s been focused on studying the human brain for nearly two decades now. Co-founder Jeff Hawkins published A Thousand Brains: A New Theory of Intelligence on the topic a couple of years ago, which described the company’s findings on how the brain’s inner workings inform attempts to refine neural networks.

During the week of September 11th, Subutai Ahmad, the company’s CEO, announced the Numenta Platform for Intelligent Computing (NuPIC). Ahmed claimed NuPIC’s architecture will allow CPUs to become more performant and efficient than today’s GPUs in GAI applications. Today’s neural nets in commercial use are rudimentary by comparison with what’s possible, taking advantage of only a fraction of what scientists have discovered about the brain recently.

Ahmad noted that the human brain only uses 20 watts of power. Each neuron acts as an elegant, sophisticated computing device on its own, one that can manage to build context with only sparse data.

Ahmad contrasted GAI with non-generative AI (NGAI) and pointed out that the latter “understands” and can be reliable when it comes to comparing and classifying text, though it’s less able to handle long context at this point.

When it comes to GAI, users have to run the model hundreds of times, one reason that GAI model use demands 10,0000 to 100,000 times more compute power than NGAI.

But Numenta is focused on both GAI and NGAI applications. Ahmad claimed that Numenta’s benchmarked price/performance using the pretrained English BERT-Large language model is ten times that of Nvidia’s GPUs.

Ahmad also mentioned that Numenta’s working with gaming platform luminaries Will Wright (creator of Sims and SPORE) and Lauren Ellioton more interactive, laptop CPU-based gaming. Their venture is called Gallium Studios.

Other alternatives to GPUs

It was evident that Nvidia’s A100 GPU is seeing unprecedented demand and short supply, with demand for the more powerful H100 cards also evident. H100 clusters for AWS users became available in July 2023. Rashmi Gopinath, General Partner at B Capital during foundation model efficiency panel at the Summit said that some users are facing initial budgets that are up to 80 percent allocated to hardware because of the lack of access to GPUs.

Alternatives to GPUs besides Numenta’s mentioned at the Summit included Gaudi2 accelerator from Intel’s Habana unit. Habana Software Product Head Sree Gamnesan observed that some newer LLMs include trillions of tokens, so sufficient scalability is a major requirement.

Gaudi2, she said, offers speed, scaling, ease of use, power and cost efficiency versus an A100-based solution. Habana claims that a single server node with eight Gaudi2s can enable inference of a 178 billion parameter model. Habana’s been busy adding support that developers are demanding for PyTorch, Hugging Face, and pre-built Docker container images, and promises FP8 support and optimizations by the end of Q4 2023.

Overall impressions

GAI is opening up infrastructure innovation opportunities at various levels of the stack, not just in the area of specialized processing units. I shouldn’t have been surprised at how much interest and activity there is in making GAI (mainly a consumer end user and development phenomenon to date) real and feasible for enterprises, but I was. Looking forward to finding out more over the next several days.

Google Can Only Search For ‘Metaverse’

Google has a graveyard of products in the AR/VR space. In 2019, it killed DayDream, its virtual reality headset. This year, it shelved Project Iris, an augmented reality headset. While Google was on a killing spree, Apple and Meta were doubling down on the AR/VR category.

Apple worked on Vision Pro, which is slated to ship early next year, with a belief that spatial computing is the future, while Meta continues to invest in the Metaverse.

Meta chief Mark Zuckerberg is still high on the idea of Metaverse. To keep the conversation around AR/VR alive, Mark Zuckerberg recently appeared in an interview with Lex Fridman that took place in the Metaverse. One thing that caught everyone’s attention was the use of photorealistic avatars instead of cartoony ones. Furthermore, with Meta launching Quest 3, Zuckerberg is proving that AR/VR wearable technology is the future and is here to stay.

Following in Meta’s footsteps Apple also plans to create photorealistic avatars for Vision Pro during FaceTime calls. On one hand, Apple is betting big on its Vision Pro, while Google appears to be struggling, unable to chart its next course.

Meanwhile, OpenAI is also mulling to venture into hardware products as per the recent reports. It wouldn’t be a surprise if OpenAI announces an “XR headset’ at its first developer conference OpenAI’s DevDay.

Is Starline the last hope?

As of now Starline is the only hope for Google to challenge Quest 3 and Vision Pro. Back in 2021, Google introduced Starline, its attempt to connect the world. It is a 3D video chat booth that aims to replace a one-on-one 2D video conference call with an experience that feels like you’re actually sitting in front of a real human being.

The system uses advanced AI to build a photorealistic model of the person the user is talking to, and projects that onto a light field display with a unique sense of volume and depth. The result is a lifelike image of the other person as if they were right in front of you.

However, the problem with the original Starline was that it took up an entire room, requiring complex hardware such as infrared light emitters and special cameras to create a live 3D model of the person.

To tackle this problem, Google earlier this year, developed new AI techniques that only require a few standard cameras to produce higher quality, lifelike 3D images. With this Google was able to shrink the size of Starline from that of a restaurant booth to that of a traditional video conferencing system.

Google might be putting a lot of effort into Starline to keep it alive but it would be a tedious task for users to adopt it as go to device to interact and hold professional meetings as it is not portable while on the other hand users can easily carry Quest 3 or Vision Pro.

In terms of pricing, the previous version of Starline was rumored to be priced at $10,000. In contrast, Apple’s Vision Pro is priced at $3,599, and the starting price for Quest 3 is $500.

It would be very difficult for Google to convince users to buy Starline when they can go for the above two options at a considerably cheaper price. Starline would have to significantly reduce its price to become mainstream.

Google’s Starline prototype is effectively the headset-less version. Meta’s version is arguably more accessible since all you need is a relatively cheap headset…https://t.co/MmZbFJSMxV

— Mike Mandel (@mmandel) September 30, 2023

Google’s Hardluck with AR/VR

Despite killing ‘Project Iris’, there was still a glimmer of hope that the company might launch a mixed reality headset by partnering with Samsung and Qualcomm under the name ‘Project Moohan’.

However, recent reports indicate that Google’s Project Moohan might not see the light due to tussles between Google and Samsung. According to reports, Samsung is likely to have more influence over product features in this partnership, and the company is not interested in Google’s other hardware offerings.

To add salt to Google’s wound, the former head of operating systems on Google‘s augmented reality team, Mark Lucovsky, departed from the company earlier this year. In his tweet, Lucovsky cited “changes in AR leadership” and concerns about Google’s commitment and vision in the field as factors influencing his decision to leave. In February, Google’s head of VR Clay Bavor also left the company to start his own AI startup.

With all the turbulence in the AR/VR ecosystem of Google, it will be interesting to witness how Google manages to fight competition against Apple and Meta, which are way ahead in this race.

The post Google Can Only Search For ‘Metaverse’ appeared first on Analytics India Magazine.

Job Trends in Data Analytics: NLP for Job Trend Analysis

Data analytics has experienced remarkable growth in recent years, driven by advancements in how data is utilized in key decision-making processes. The collection, storage, and analysis of data have also progressed significantly due to these developments. Moreover, the demand for talent in data analytics has skyrocketed, turning the job market into a highly competitive arena for individuals possessing the necessary skills and experience.

The rapid expansion of data-driven technologies has correspondingly led to an increased demand for specialized roles, such as "data engineer." This surge in demand extends beyond data engineering alone and encompasses related positions like data scientist and data analyst.

Recognizing the significance of these professions, our series of blog posts aims to collect real-world data from online job postings and analyze it to understand the nature of the demand for these jobs, as well as the diverse skill sets required within each of these categories.

In this blog, we introduce a browser-based “Data Analytics Job Trends” application for the visualization and analysis of job trends in the data analytics market. After scraping data from online job agencies, it uses NLP techniques to identify key skill sets required in the job posts. Figure 1 shows a snapshot of the data app, exploring trends in the data analytics job market.

Figure 1: Snapshot of the KNIME Data App “Data Analytics Job Trends”

For the implementation, we adopted the low-code data science platform: KNIME Analytics Platform. This open-source and free platform for end-to-end data science is based on visual programming and offers an extensive range of functionalities, from pure ETL operations and a wide array of data source connectors for data blending through to machine learning algorithms, including deep learning.

The set of workflows underlying the application is available for free download from the KNIME Community Hub at “Data Analytics Job Trends”. A browser-based instance can be evaluated at “Data Analytics Job Trends”.

“Data Analytics Job Trends” Application

This application is generated by four workflows shown in Figure 2 to be executed sequentially for the following sequence of steps:

Web scraping for data collection
NLP parsing and data cleaning
Topic modeling
Analysis of attribution of job role — skills

The workflows are available on the KNIME Community Hub Public Space — Data Analytics Job Trends.

Figure 2: KNIME Community Hub Space — Data Analytics Job Trends contains a set of four workflows used for building the application “Data Analytics Job Trends”

“01_Web Scraping for data collection” workflow crawls through the online job postings and extracts the textual information into a structured format
“02_NLP Parsing and cleaning” workflow performs the necessary cleaning steps and then parses the long texts into smaller sentences
“03_Topic Modeling and Exploration Data App” uses clean data to build a topic model and then to visualize its results within a data app
“04_Job Skill Attribution” workflow evaluates the association of skills across job roles, like Data Scientist, Data Engineer, and Data Analyst, based on the LDA results.

Web scraping for data collection

In order to have an up-to-date understanding of the skills required in the job market, we opted for the analysis of web scraping job posts from online job agencies. Given the regional variations and the diversity of languages, we focused on job postings in the United States. This ensures that a significant proportion of the job postings are presented in the English language. We also focused on job postings from February 2023 to April 2023.

The KNIME workflow ”01_Web Scraping for Data Collection” in Figure 3 crawls through a list of URLs of searches on job agencies’ websites.

To extract the relevant job postings pertaining to Data Analytics, we used searches with six keywords that collectively cover the field of data analytics, namely: “big data”, “data science”, “business intelligence”, “data mining”, “machine learning” and “data analytics”. Search keywords are stored in an Excel file and read via the Excel Reader node.

Figure 3: KNIME Workflow “01_Web Scraping for Data Collection” scraps job postings according to a number of search URLs

The core node of this workflow is the Webpage Retriever node. It is used twice. The first time (outer loop), the node crawls the site according to the keyword provided as input and produces the related list of URLs for job postings published in the US within the last 24 hours. The second time (inner loop), the node retrieves the text content from each job posting URL. The Xpath nodes following the Webpage Retriever nodes parse the extracted texts to reach the desired information, such as job title, required qualifications, job description, salary, and company ratings. Finally, the results are written to a local file for further analysis. Figure 4 shows a sample of the job postings scraped for February 2023.

Figure 4: Sample of the Web Scraping Results for February 2023

NLP parsing and data cleaning

Figure 5: 02_NLP Parsing and cleaning Workflow for Text Extraction and Data Cleaning

Like all freshly collected data, our web scraping results needed some cleaning. We perform NLP parsing along with data cleaning and write the respective data files using the workflow 02_NLP Parsing and cleaning shown in Figure 5.

Multiple fields from the scraped data have been saved in the form of a concatenation of string values. Here, we extracted the individual sections using a series of String Manipulation nodes within the meta node ”Title-Location-Company Name Extraction” and then we removed unnecessary columns and got rid of duplicate rows.

We then assigned a unique ID to each job posting text and fragmented the whole document into sentences via the Cell Splitter node. The meta information for each job — title, location, and company — was also extracted and saved along with the Job ID.

The list of the most frequent 1000 words was extracted from all documents, so as to generate a stop-word list, including words like “applicant”, “collaboration”, “employment” etc … These words are present in every job posting and therefore do not add any information for the next NLP tasks.

The result of this cleaning phase is a set of three files:

— A table containing the documents’ sentences;

— A table containing the job description metadata;

— A table containing the stopword list.

Topic modeling and results exploration

Figure 6: 03_Topic Modeling and Exploration Data App workflow builds a topic model and allows the user to explore the results visually with the Topic Explorer View Component

The workflow 03_Topic Modeling and Exploration Data App (Figure 6) uses the cleaned data files from the previous workflow. In this stage, we aim to:

Detect and remove common sentences (Stop Phrases) appearing in many job postings
Perform standard text processing steps to prepare the data for topic modeling
Build the Topic Model and Visualize the Results.

We discuss the above tasks in detail in the following subsections.

3.1 Remove stop phrases with N-grams

Many job postings include sentences that are commonly found in company policies or general agreements, such as "Non-Discrimination policy" or "Non-Disclosure Agreements." Figure 7 provides an example where job postings 1 and 2 mention the "Non-Discrimination" policy. These sentences are not relevant to our analysis and therefore need to be removed from our text corpus. We refer to them as "Stop Phrases" and employ two methods to identify and filter them.

The first method is straightforward: we calculate the frequency of each sentence in our corpus and eliminate any sentences with a frequency greater than 10.

The second method involves an N-gram approach, where N can be in the range of values from 20 to 40. We select a value for N and assess the relevance of N-grams derived from the corpus by counting the number of N-grams that classify as stop phrases. We repeat this process for each value of N within the range. We chose N=35 as the best value for N to identify the highest number of Stop Phrases.

Figure 7: Example of Common Sentences in Job Postings that can be regarded as “Stop Phrases”

We used both methods to remove the “Stop Phrases” as shown by the workflow depicted in Figure 7. At first, we removed the most frequent sentences, then we created N-grams with N=35 and tagged them in every document with the Dictionary Tagger node, and, at last, we removed these N-grams using the Dictionary Replacer node.

3.2 Prepare data for topic modeling with text preprocessing techniques

After removing the Stop Phrases, we perform the standard text preprocessing in order to prepare the data for topic modeling.

First, we eliminate numeric and alphanumeric values from the corpus. Then, we remove punctuation marks and common English stop words. Additionally, we use the custom stop word list that we created earlier to filter out job domain-specific stop words. Finally, we convert all characters to lowercase.

We decided to focus on the words that carry significance, which is why we filtered the documents to carry only nouns and verbs. This can be done by assigning a Parts of Speech (POS) tag to each word in the document. We utilize the POS Tagger node to assign these tags and filter them based on their value, specifically keeping words with POS = Noun and POS = Verb.

Lastly, we apply Stanford lemmatization to ensure the corpus is ready for topic modeling. All of these preprocessing steps are carried out by the "Pre-processing" component shown in Figure 6.

3.3 Build topic model and visualize it

In the final stage of our implementation, we applied the Latent Dirichlet Allocation (LDA) algorithm to construct a topic model using the Topic Extractor (Parallel LDA) node shown in Figure 6. The LDA algorithm produces a number of topics (k), each topic described through a (m) number of keywords. Parameters (k,m) must be defined.

As a side note, k and m cannot be too large, since we want to visualize and interpret the topics (skill sets) by reviewing the keywords (skills) and their respective weights. We explored a range [1, 10] for k and fixed the value of m=15. After careful analysis, we found that k=7 led to the most diverse and distinct topics with a minimum overlap in keywords. Thus, we determined k=7 to be the optimal value for our analysis.

Explore Topic Modeling Results with an Interactive Data App

To enable everyone to access the topic modeling results and have their own go at it, we deployed the workflow (in Figure 6) as a Data App on KNIME Business Hub and made it public, for everyone to access it. You can check it out at: Data Analytics Job Trends.

The visual part of this data app comes from the Topic Explorer View component by Francesco Tuscolano and Paolo Tamagnini, available for free download from the KNIME Community Hub, and provides a number of interactive visualizations of topics by topic and document.

Figure 8: Data Analytics Job Trends for exploration of the topic modeling results

Presented in Figure 8, this Data App offers you a choice between two distinct views: the “Topic” and the “Document” view.

The “Topic” view employs a Multi-Dimensional Scaling algorithm to portray topics on a 2-dimensional plot, effectively illustrating the semantic relationships between them. On the left panel, you can conveniently select a topic of interest, prompting the display of its corresponding top keywords.

To venture into the exploration of individual job postings, simply opt for the “Document” view. The “Document” view presents a condensed portrayal of all documents across two dimensions. Utilize the box selection method to pinpoint documents of significance, and at the bottom, an overview of your selected documents awaits

Exploring Data Analytics job market using NLP

We have provided here a summary of the “Data Analytics Job Trends” application, that was implemented and used to explore the most recent skill requirements and job roles in the data science job market. For this blog, we restricted our area of action to job descriptions for the US, written in English, from February to April 2023.

To understand the job trends and provide a review, the “Data Analytics Job Trends” crawls job agency sites, extracts text from online job postings, extracts topics and keywords after performing a series of NLP tasks, and finally visualizes the results by topic and by document to discern the patterns in the data.

The application consists of a set of four KNIME workflows to run sequentially for web scraping, data processing, topic modeling, and then interactive visualizations to allow the user to spot the job trends.

We deployed the workflow on KNIME Business Hub and made it public, so everyone can access it. You can check it out at: Data Analytics Job Trends.

The full set of workflows is available and free to download from KNIME Community Hub at Data Analytics Job Trends. The workflows can easily be changed and adapted to discover trends in other fields of the job market. It is enough to change the list of search keywords in the Excel file, the website, and the time range for the search.

What about the results? Which are the skills and the professional roles most sought after in today’s data science job market? In our next blog post, we will guide you through the exploration of the outcomes of this topic modeling. Together, we'll closely examine the intriguing interplay between job roles and skills, gaining valuable insights about the data science job market along the way. Stay tuned for an enlightening exploration!

Resources

A Systematic Review of Data Analytics Job Requirements and Online Courses by A. Mauro et al.

Andrea De Mauro has over 15 years of experience building business analytics and data science teams at multinational companies such as P&G and Vodafone. Apart from his corporate role, he enjoys teaching Marketing Analytics and Applied Machine Learning at several universities in Italy and Switzerland. Through his research and writing, he has explored the business and societal impact of Data and AI, convinced that a broader analytics literacy will make the world better. His latest book is 'Data Analytics Made Easy', published by Packt. He appeared in CDO magazine's 2022 global 'Forty Under 40' list.

Mahantesh Pattadkal brings more than 6 years of experience in consulting on data science projects and products. With a Master's Degree in Data Science, his expertise shines in Deep Learning, Natural Language Processing, and Explainable Machine Learning. Additionally, he actively engages with the KNIME Community for collaboration on data science-based projects.

Mistral 7B: Setting New Benchmarks Beyond Llama2 in the Open-Source Space

Large Language Models (LLMs) have recently taken center stage, thanks to standout performers like ChatGPT. When Meta introduced their Llama models, it sparked a renewed interest in open-source LLMs. The aim? To create affordable, open-source LLMs that are as good as top-tier models such as GPT-4, but without the hefty price tag or complexity.

This mix of affordability and efficiency not only opened up new avenues for researchers and developers but also set the stage for a new era of technological advancements in natural language processing.

Recently, generative AI startups have been on a roll with funding. Together raised $20 million, aiming to shape open-source AI. Anthropic also raised an impressive $450 million, and Cohere, partnering with Google Cloud, secured $270 million in June this year.

Introduction to Mistral 7B: Size & Availability

Mistral AI, based in Paris and co-founded by alums from Google’s DeepMind and Meta, announced its first large language model: Mistral 7B. This model can be easily downloaded by anyone from GitHub and even via a 13.4-gigabyte torrent.

This startup managed to secure record-breaking seed funding even before they had a product out. Mistral AI first mode with 7 billion parameter model surpasses the performance of Llama 2 13B in all tests and beats Llama 1 34B in many metrics.

Compared to other models like Llama 2, Mistral 7B provides similar or better capabilities but with less computational overhead. While foundational models like GPT-4 can achieve more, they come at a higher cost and aren't as user-friendly since they're mainly accessible through APIs.

When it comes to coding tasks, Mistral 7B gives CodeLlama 7B a run for its money. Plus, it's compact enough at 13.4 GB to run on standard machines.

Additionally, Mistral 7B Instruct, tuned specifically for instructional datasets on Hugging Face, has shown great performance. It outperforms other 7B models on MT-Bench and stands shoulder to shoulder with 13B chat models.

Hugging Face Mistral 7B Example

Performance Benchmarking

In a detailed performance analysis, Mistral 7B was measured against the Llama 2 family models. The results were clear: Mistral 7B substantially surpassed the Llama 2 13B across all benchmarks. In fact, it matched the performance of Llama 34B, especially standing out in code and reasoning benchmarks.

The benchmarks were organized into several categories, such as Commonsense Reasoning, World Knowledge, Reading Comprehension, Math, and Code, among others. A particularly noteworthy observation was Mistral 7B's cost-performance metric, termed “equivalent model sizes”. In areas like reasoning and comprehension, Mistral 7B demonstrated performance akin to a Llama 2 model three times its size, signifying potential savings in memory and an uptick in throughput. However, in knowledge benchmarks, Mistral 7B aligned closely with Llama 2 13B, which is likely attributed to its parameter limitations affecting knowledge compression.

What really makes Mistral 7B model better than most other Language Models?

Simplifying Attention Mechanisms

While the subtleties of attention mechanisms are technical, their foundational idea is relatively simple. Imagine reading a book and highlighting important sentences; this is analogous to how attention mechanisms “highlight” or give importance to specific data points in a sequence.

In the context of language models, these mechanisms enable the model to focus on the most relevant parts of the input data, ensuring the output is coherent and contextually accurate.

In standard transformers, attention scores are calculated with the formula:

Transformers Attention Formula

The formula for these scores involves a crucial step – the matrix multiplication of Q and K. The challenge here is that as the sequence length grows, both matrices expand accordingly, leading to a computationally intensive process. This scalability concern is one of the major reasons why standard transformers can be slow, especially when dealing with long sequences.

Attention mechanisms help models focus on specific parts of the input data. Typically, these mechanisms use ‘heads' to manage this attention. The more heads you have, the more specific the attention, but it also becomes more complex and slower. Dive deeper into of transformers and attention mechanisms here.

Multi-query attention (MQA) speeds things up by using one set of ‘key-value' heads but sometimes sacrifices quality. Now, you might wonder, why not combine the speed of MQA with the quality of multi-head attention? That's where Grouped-query attention (GQA) comes in.

Grouped-query Attention (GQA)

Grouped-query attention

GQA is a middle-ground solution. Instead of using just one or multiple ‘key-value' heads, it groups them. This way, GQA achieves a performance close to the detailed multi-head attention but with the speed of MQA. For models like Mistral, this means efficient performance without compromising too much on quality.

Sliding Window Attention (SWA)

The sliding window is another method use in processing attention sequences. This method uses a fixed-sized attention window around each token in the sequence. With multiple layers stacking this windowed attention, the top layers eventually gain a broader perspective, encompassing information from the entire input. This mechanism is analogous to the receptive fields seen in Convolutional Neural Networks (CNNs).

On the other hand, the “dilated sliding window attention” of the Longformer model, which is conceptually similar to the sliding window method, computes just a few diagonals of the QKT matrix. This change results in memory usage increasing linearly rather than quadratically, making it a more efficient method for longer sequences.

Mistral AI's Transparency vs. Safety Concerns in Decentralization

In their announcement, Mistral AI also emphasized transparency with the statement: “No tricks, no proprietary data.” But at the same time their only available model at the moment ‘Mistral-7B-v0.1' is a pretrained base model therefore it can generate a response to any query without moderation, which raises potential safety concerns. While models like GPT and Llama have mechanisms to discern when to respond, Mistral's fully decentralized nature could be exploited by bad actors.

However, the decentralization of Large Language Models has its merits. While some might misuse it, people can harness its power for societal good and making intelligence accessible to all.

Deployment Flexibility

One of the highlights is that Mistral 7B is available under the Apache 2.0 license. This means there aren't any real barriers to using it – whether you're using it for personal purposes, a huge corporation, or even a governmental entity. You just need the right system to run it, or you might have to invest in cloud resources.

While there are other licenses such as the simpler MIT License and the cooperative CC BY-SA-4.0, which mandates credit and similar licensing for derivatives, Apache 2.0 provides a robust foundation for large-scale endeavors.

Final Thoughts

The rise of open-source Large Language Models like Mistral 7B signifies a pivotal shift in the AI industry, making high-quality language models accessible to a wider audience. Mistral AI's innovative approaches, such as Grouped-query attention and Sliding Window Attention, promise efficient performance without compromising quality.

While the decentralized nature of Mistral poses certain challenges, its flexibility and open-source licensing underscore the potential for democratizing AI. As the landscape evolves, the focus will inevitably be on balancing the power of these models with ethical considerations and safety mechanisms.

Up next for Mistral? The 7B model was just the beginning. The team aims to launch even bigger models soon. If these new models match the 7B's performance, Mistral might quickly rise as a top player in the industry, all within their first year.

Why Cloud-Native Applications Need Observability

Enterprises use security to save their data from any kind of vulnerability. They have been deploying high-end enterprise-version antivirus suites to lockdown system security and hedge their bets against malware and all forms of web-based nastiness.

Now, that the era of cloud-native applications is here, managing changes to cloud assets has become a universal pain felt by developers despite all of the advancements in monitoring tools. In the recent past, tools have come to the rescue enabling them to troubleshoot and resolve developer issues. These platforms let organisations track, measure, and optimise the performance of their applications — commonly referred to as ‘observability’.

It’s more common to talk about ‘observability’ as companies are trying to make the backend work more efficiently. But a year ago, Sawaram Suthar and his team at Acquire were facing issues visualising their cloud infrastructure due to the unavailability of a platform allowing them to do so.

The platform being used by the team lacked real-time monitoring, issue detection, improved customer experience, security, cost optimization, and data-driven decision-making, which became a huge task for the team.

“We didn’t know what went wrong in the system. If it is down, then why is it down and what are the logs that we look at,” Suthar recalled in a conversation with AIM. “As developers ourselves, we kind of felt underserved at that point, as these platforms were mostly enterprise-focused rather than end-user focused,” he added.

“That’s why we unified on the particular form which can help us to visualise a complete data flow and started Middleware,” he said introducing the platform. “Observability is not new in the market. It’s been on the market for almost 20 years. But people have started using it now because people realise it is really important,” he added.

The rising demand for observability tools is visible today as 86% of Indian organisations surveyed in 2022 saw observability as a key enabler to achieve core business objectives.

The company is currently witnessing a surge in the Indian market as companies continue to go cloud native. As per reports, Indian firms spent an average of INR 370 crores on cloud between June 2022 and 2023.

“For on-premise servers, it doesn’t require observability platforms. When cloud usage increases, there is a need for software which can visualise how this cloud will affect the services. That’s where we come in. As several companies are moving away from non-cloud and monoliths, the demand is going to be high in the upcoming year,” he stated.

Newer and Cheaper

The global cloud computing market dramatically grew by 635% from $24.63 billion in 2010 to $156.4 billion in 2020. “A few years ago cloud adoption was still picking up. People were using monolithic environments, or maybe on-demand services,” Suthar pointed out.

Suthar further highlighted the sheer size of the industry and the already existing dominant players. Early on the team realised they required a lot of capital. Initially, they tested some different small models before jumping into visibility because they didn’t want to go directly competing with the big players.

This strategy bore fruit as Middleware raised $6.5 million in seed funding in August 2023, right after graduating from Y Combinator’s Winter 2023 batch. The company now aims to expand its team from 25 to 50 within the next year and targets a series A funding round by mid-2024.

“Existing competitions are there, they were built almost 10 years back and are not compatible with the latest technology, like cloud-native and AI,” Suthar stated. As the startup is backed by YC, it has access to most of OpenAI’s innovations and currently uses GPT-4.

He further highlighted and elaborated on the three USPs of MW. Firstly, there’s an AI advisor that can pinpoint errors and recommend what needs to be done to fix them.

The second is that the platform can predict incoming errors in the next few hours based on data. “That’s a unique differentiator for us. We can predict if a particular server is going to be down, that’s a feature that people really love,” Suthar explained.

Lastly, Middleware offers its services at one-third of the cost compared to major competitors like Datadog, making it an attractive option for cost-conscious customers. “We are in the process of acquiring more customers in a couple of months. Around 35+ customers are actively using our platform,” he added.

For the DevOps

Headquartered in San Francisco, Middleware is one of the first observability platforms in the market that uses generative AI to accelerate issue identification and resolution. Suthar also recalled that the term observability hadn’t gained much popularity when they entered the market.

“We started creating content around it to raise awareness. When we started we saw that people don’t understand the word because people used to call it a cloud monitoring tool rather than observability,” he said.

“People are used to using platforms and metrics differently. It’s very hard to adapt to using tools, which give you a full stack of liberty. Now they also know that finding a threat is the biggest problem for DevOps or any developer. What used to take one hour now the same problem can be solved within 10 seconds because you will have complete visibility to take action. Not only data, but you have the right suggestions,” he concluded.

The post Why Cloud-Native Applications Need Observability appeared first on Analytics India Magazine.

Top 5 Papers Presented by Meta at ICCV

The 16th edition of the prestigious International Conference on Computer Vision (ICCV) is scheduled between October 2 and 6 in Paris, France. The event is expected to have over 2,000 participants globally, focusing on cutting-edge research in computer vision through oral and poster presentations, spanning diverse topics such as image and video processing, object detection, scene understanding, motion estimation, 3D vision, machine learning, and applications in robotics and healthcare. Meta, one of the pioneers of the field is also participating in the event with five of their recent research papers on the same topic.

Make-An-Animation: Large-Scale Text-conditional 3D Human Motion Generation

This paper explores text-guided human motion generation, a field with broad applications in animation and robotics. While previous efforts using diffusion models have enhanced motion quality, they are constrained by small-scale motion capture data, resulting in sub-optimal performance for various real-world scenarios. The authors propose Make-An-Animation (MAA), a novel text-conditioned human motion generation model. MAA stands out by learning from large-scale image-text datasets, allowing it to grasp more varied poses and prompts. The model is trained in two stages: initially on a sizable dataset of (text, static pseudo-pose) pairs from image-text datasets, and subsequently fine-tuned on motion capture data, incorporating additional layers for temporal modeling. In contrast to conventional diffusion models, MAA employs a U-Net architecture akin to recent text-to-video generation models. Through human evaluation, the model demonstrates state-of-the-art performance in terms of motion realism and alignment with input text in the realm of text-to-motion generation.

Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning

This study, in collaboration with Berkley AI Research and Kitware, introduces Scale-MAE, a novel pre-training method for large models commonly fine-tuned with augmented imagery. These models often don’t consider scale-specific details, especially in domains like remote sensing. Scale-MAE addresses this issue by explicitly learning relationships between data at different scales during pre-training. It masks input images at known scales, determining the ViT positional encoding scale based on the Earth’s area covered, not image resolution.

The masked images are encoded using a standard ViT backbone and then decoded through a bandpass filter, reconstructing low/high frequency images at lower/higher scales. Tasking the network with reconstructing both frequencies results in robust multiscale representations for remote sensing imagery, outperforming current state-of-the-art models.

NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection

NeRF-Det is a new approach to indoor 3D detection using RGB images. Unlike existing methods, it leverages NeRF to explicitly estimate 3D geometry, improving detection performance. To overcome NeRF’s optimisation latency, the researchers incorporated geometry priors for better generalisation. By linking detection and NeRF via a shared MLP, they efficiently adapt NeRF for detection, yielding geometry-aware volumetric representations.

The method surpasses state-of-the-art benchmarks on ScanNet and ARKITScenes. Joint-training enables NeRF-Det to generalise to new scenes for object detection, view synthesis, and depth estimation, eliminating the need for per-scene optimisation.

The Stable Signature: Rooting Watermarks in Latent Diffusion Models

This paper addresses ethical concerns associated with generative image modeling by proposing an active strategy that integrates image watermarking and Latent Diffusion Models (LDM). The objective is to embed an invisible watermark in all generated images for future detection or identification. The method rapidly refines the latent decoder of the image generator based on a binary signature. A pre-trained watermark extractor recovers the hidden signature, and a statistical test determines if the image originates from the generative model. The study evaluates the effectiveness and durability of the watermarks across various generation tasks, demonstrating the Stable Signature’s resilience even after image modifications. The approach aims to mitigate risks associated with the authenticity of AI-generated images, especially concerning issues like deep fakes and copyright misuse, by seamlessly integrating watermarking into the generation process of LDMs without requiring architectural changes. The method proves compatible with various LDM-based generative methods, providing a practical solution for responsible deployment and detection of generated images.

Diffusion Models as Masked Autoencoders

The conventional belief in the power of generation for grasping visual data is revisited in light of denoising diffusion models. While direct pre-training with these models falls short, a modified approach, where diffusion models are conditioned on masked input and framed as Masked Autoencoders (DiffMAE), proves effective.

This method serves as a robust initialisation for downstream tasks, excels in image inpainting, and extends effortlessly to videos, achieving top-tier classification accuracy. A comparison of design choices and a linkage between diffusion models and masked autoencoders are explored. The study questions whether generative pre-training can effectively compete in recognition tasks compared to other self-supervised methods. The work establishes connections between Masked Autoencoders and diffusion models while providing insights into the effectiveness of generative pre-training in the realm of visual understanding.

The post Top 5 Papers Presented by Meta at ICCV appeared first on Analytics India Magazine.

Tips for Successfully Navigating Beginner Data Science Job Interviews

Data science. It’s exciting. It’s nerve-wracking.

It's interdisciplinary and evolves continually. It unravels mysteries in data and requires innovative solutions. That’s what makes data science attractive. Not to mention being paid well.

Data science is also disheartening, sometimes for the same reasons. Add high competition and expectations, constantly shifting goals and ethical dilemmas.

Stepping into it makes you want to pull your hair out and, strangely, enjoy it. Somewhat like following tech bros on Twitter. Sorry, Elon, X.

This is especially the case for beginners pilgrimaging job interviews to get their first data science jobs.

However, with the right preparation and mindset, you can confidently navigate these interviews and make a lasting impression. Here are some tips to help you succeed in your beginner data science job interviews.

1. Understand the Basics Thoroughly

You need to have a strong grasp of foundational concepts like statistics, linear algebra, and programming. Interviewers often test these basics before diving into more complex topics.

These skills usually encompass:

Statistics
Programming
Data Manipulation
Data Visualization
Relational Databases
Machine Learning

Statistics

The basic statistics knowledge interviewers expect, even from beginners, includes these statistical concepts.

Descriptive Statistics:

Measures of Central Tendency – mean, median, and mode
Measures of Dispersion – range, variance, standard deviation, and interquartile range
Measures of Shape – skewness and kurtosis

Probability:

Basic probability concepts
Conditional probability and Bayes' theorem
Probability distribution – normal, binomial, Poisson, and others

Inferential Statistics:

Sampling – populations, samples, sampling techniques
Hypothesis Testing – null and alternative hypotheses, Type I and Type II errors, p-values, and significance levels
Confidence Intervals – Estimating population parameters based on sample data.

Correlation and Covariance:

Understanding the relationship between two variables and their co-dependence
Pearson's correlation coefficient

Regression Analysis:

Simple linear regression – the relationship between two continuous variables
Multiple regression – extending to more than one independent variable

Distributions:

Normal Distribution
Binomial Distribution
Poisson Distribution
Exponential Distribution

Programming

You need to be proficient in programming languages commonly used in data science. The three most popular languages are:

SQL
Python
R

You don’t have to be a guru in all three languages. Usually, it’s enough to be good at one and at least familiar with the basics of one of the other two.

It all depends on the job description. Different companies and positions require different languages. In data science, it’s usually one of the three mentioned.

If you ask me which one, and only one, you should learn, I’d go with SQL. Querying databases is a fundament no data scientist can survive without. SQL is specifically designed for that; no other language does this, and data cleaning so well.

It also easily integrates with other languages. That way, you can leverage other languages for tasks SQL is unsuitable for, e.g., building models or data visualizations.

Data Manipulation

It refers to your ability to clean and transform data, which includes handling missing data, outliers, and transforming variables.

This means you’ll need to know the most popular data manipulation libraries:

pandas and NumPy – for Python
dplyr – for R

Data Visualization

You have to understand the best visualization techniques for different types of data and insights. And you have to know how to put it into practice using visualization tools:

matplotlib and seaborn – for Python
ggplot2 – for R

Relational Databases

As a data scientist, you need to have a general understanding of relational databases and how they work. If you have at least basic knowledge of querying them using SQL, even better.

Some of the most popular data management systems include:

PostgreSQL
MySQL
SQL Server
Oracle

Machine Learning

You must be familiar with the machine learning basics. For instance, knowing the difference between supervised and unsupervised learning.

You also need to be familiar with classification, clustering, and regression. This includes knowing some basic algorithms, such as linear regression, decision trees, SVM, naive Bayes, and k-means.

2. Know Your Tools

Before the interview, familiarize yourself with popular data science tools. This includes programming languages we already mentioned, but also some other platforms.

You don’t need to know them all. But it would be ideal if you had some experience with at least one tool from each category.

3. Prepare for Coding & Technical Questions

Use platforms such as StrataScratch, LeetCode, and others to prepare for coding and technical questions.

Also, use YouTube channels, blogs, and other resources to brush up the knowledge of other technical concepts. If you concentrate on those mentioned in the “Understand the Basics Thoroughly”, you’ll be good.

Mock interviews can be incredibly beneficial. Use the online platforms that offer them. Or practice with your friends and mentors.

All these preparation techniques will help you get comfortable with the interview format and improve your responses.

4. Showcase Practical Experience

If you've worked on personal projects or internships, use them to your advantage. Discuss them during the interview to highlight the challenges you faced, the solutions you implemented, and the results you achieved.

5. Brush Up on Behavioral Questions

Technical skills usually comprise most of the hiring process. However, companies usually dedicate at least some time to behavioral questions.

It’s expected, as you’ll work in a team. The interviewers will want to know how you communicate with your colleagues, understand teamwork, handle pressure and conflicts, or approach problems.

Prepare examples from your past experiences that demonstrate your soft skills and problem-solving abilities.

6. Stay Updated

Data science is rapidly changing. So, you need to stay updated with the latest trends, tools, and techniques. Read about them, join online forums, attend webinars, and participate in workshops to keep yourself up to date.

However, don’t obsess over this thinking that you need to know about – nay, master it – every new “must-have” and “must-know” product.

7. Ask Questions

Depending on its format, you’ll likely have the opportunity to ask questions during or at the end of the interview.

This is your chance to show the interviewer your enthusiasm for the role and the company. And also an understanding of what they’re looking for.

Ask about the team's current projects, the company's data infrastructure, plans, and the challenges they're facing.

8. Don’t Forget the Soft Skills

Your technical skills won’t get you far unless combined with great communication skills. You’ll communicate and collaborate with technical and non-technical team members and stakeholders in your job.

In your interview, be clear and concise in your answers. Show your ability to explain complex topics in simple terms. This will show interviewers that you can effectively collaborate with non-technical team members. It’s a skill you’ll need a lot, as data science doesn’t exist in a vacuum, and its findings are very often used by non-technical people.

9. Keep Calm and Carry On

It's natural to be nervous. Just don’t be nervous because you’re nervous! Always keep in mind that the interviewers are looking for the best candidate, not the perfect one. Best, in this case, means the best combination of all the points we mentioned so far.

If you falter at some stage of the interview, don’t lose your spirit – keep calm and carry on! Candidates often exaggerate the impact of their own mistakes, while they might have (almost) no negative impact on the interviewer’s impression.

Remember that the interview is as much about getting to know the company as it is about them getting to know you. Stay calm, take deep breaths, and approach each question with confidence.

Of course, confidence can’t be faked. It’s best achieved by a solid preparation following the first eight tips.

Conclusion

Yes, technical knowledge is essential for a data science role, even at the beginner level. But soft skills, practical experience, and a genuine passion for the field are equally important.

The interviewers are primarily looking for a whole package. The nine tips will have you covered.

Now, you have to allow yourself time to prepare thoroughly. If you’re confident with your readiness level, going to an interview with a positive mindset is easier. With that, you're already well on your way to landing your first data science job.

Best of luck!
Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn.

More On This Topic

Will Numenta’s research on the human brain and computing pay off soon?

Other alternatives to GPUs

Overall impressions

Is Starline the last hope?

Google’s Hardluck with AR/VR

Web scraping for data collection

NLP parsing and data cleaning

Topic modeling and results exploration

3.1 Remove stop phrases with N-grams

3.2 Prepare data for topic modeling with text preprocessing techniques

3.3 Build topic model and visualize it

Explore Topic Modeling Results with an Interactive Data App

Resources

More On This Topic

Introduction to Mistral 7B: Size & Availability

Performance Benchmarking

What really makes Mistral 7B model better than most other Language Models?

Grouped-query Attention (GQA)

Sliding Window Attention (SWA)

Mistral AI's Transparency vs. Safety Concerns in Decentralization

Deployment Flexibility

Final Thoughts

Newer and Cheaper

For the DevOps

Statistics

Programming

Data Manipulation

Data Visualization

Relational Databases

Machine Learning

More On This Topic