Рубрика: AI

Image by Author

ChatGPT has become a swiss-army knife that can be used for multitude of applications, and there’s abundant scope to integrate ChatGPT into data science workflows.

If you’ve ever trained a machine learning model on a real-world dataset, you know that the steps of data cleaning and preprocessing are important for building robust machine learning models. In this guide, we’ll see how we can use ChatGPT to perform these tasks on an example dataset.

We’ll use the bank marketing dataset from the UCI machine learning repository and prompt ChatGPT to do the following:

• Fetch and load the dataset
• Check for missing values
• Encode categorical variables

You can follow along by signing in to your OpenAI account and starting a new ChatGPT session. If you prefer, you can also code along in Google Colab.

Fetching and Loading the Dataset

There are a few different versions of the dataset. We’ll use bank-full.csv.

Prompt: Today we’ll perform data cleaning and preprocessing on a real-world dataset. Write the code to do the following: 1. Fetch the bank marketing dataset from the UCI machine learning repository. Use libraries like requests or urllib. 2. Download the zip file, unzip the contents, and read in the bank-full.csv file into a pandas dataframe (call it data).

The code to fetch and read in the dataset is as follows:

import pandas as pd  import urllib  import zipfile    # URL of the dataset  url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank.zip"    # Fetch and extract the dataset  urllib.request.urlretrieve(url, "bank.zip")    with zipfile.ZipFile("bank.zip", "r") as zip_ref:      zip_ref.extractall()    # Load the dataset into a pandas DataFrame  data = pd.read_csv("bank-full.csv", delimiter=";",skipinitialspace=True)

Getting Basic Info on the Dataset

Let's understand the dataset better.

# Display the first few rows of the dataset  print(data.head())

This outputs:

Truncated output of data.head()

Prompt: Use pandas to get the dimension of the dataframe, descriptive statistics on the columns, and data types of various columns.

This step doesn’t really require prompting ChatGPT as the pandas methods are pretty simple.

# Get the dimensions of the dataset (rows, columns)  print(data.shape)

Output >>> (45211, 17)

We have over 45000 records and 16 features (as 17 is inclusive of the output label as well).

# Get statistical summary of numerical columns  print(data.describe())

Truncated output of data.describe()

It’s also helpful to get an overview of the data types of the various columns. We’ll take a closer look at them when we encode categorical variables.

# Check data types of each column  print(data.dtypes)

Output >>>    age           int64  job          object  marital      object  education    object  default      object  balance       int64  housing      object  loan         object  contact      object  day           int64  month        object  duration      int64  campaign      int64  pdays         int64  previous      int64  poutcome     object  y            object  dtype: object

So far we’ve gained an understanding of the bank marketing dataset. The output label denotes whether or not a customer will subscribe to a term deposit. The dataset contains several features such as the age, month, education, marital status, outcome of previous campaigns, and more.

Checking for Missing Values

Prompt: I want to know the number of missing values in each column. Please give me the code to do it. Use pandas.

# Check for missing values in each column  missing_values = data.isnull().sum()  print(missing_values)

Here’s the output:

Output >>>    age          0  job          0  marital      0  education    0  default      0  balance      0  housing      0  loan         0  contact      0  day          0  month        0  duration     0  campaign     0  pdays        0  previous     0  poutcome     0  y            0  dtype: int64

This version of the bank marketing dataset—with over 45000 records—does not have any missing values. In practice, though, most real-world datasets have missing values. You should handle missing values using suitable imputation techniques.

As an optional exercise , you can add a step here prompting ChatGPT to drop a small fraction of values from a subset of columns so you can practice how to handle missing values.

Encoding Categorical Variables

The next step is to encode categorical variables in the dataset. We’ll start by getting the list of all categorical columns.

Prompt: Give the code to get the list of all categorical columns in this dataset.

# Count the number of categorical variables  categorical_columns = data.select_dtypes(include=['object']).columns  print(categorical_columns)  num_categorical_variables = len(categorical_columns)    # Display the count  print("Number of categorical variables:", num_categorical_variables)

The categorical columns list also includes the output label y. But let's focus on the other categorical variables first.

Output >>>  Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',         'month', 'poutcome', 'y'],        dtype='object')  Number of categorical variables: 10

A Closer Look at the Values of Categorical Variables

Next, let’s see the values that each categorical variable takes.

Prompt: I want to understand the various values that each categorical variable takes. Give the pandas code to do it.

# Get the different values and their counts for each categorical column  categorical_columns = ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month', 'poutcome']  for column in categorical_columns:      value_counts = data[column].value_counts()      print(f"Column: {column}")      print(value_counts,"n")

The output is hard to parse:

Output >>>    Column: job  blue-collar      9732  management       9458  technician       7597  admin.           5171  services         4154  retired          2264  self-employed    1579  entrepreneur     1487  unemployed       1303  housemaid        1240  student           938  unknown           288  Name: job, dtype: int64     ...  ...  ...  Column: poutcome  unknown    36959  failure     4901  other       1840  success     1511  Name: poutcome, dtype: int64 

Visualizing the Values of Categorical Variables

Let’s prompt ChatGPT to create a visualization so it’s easier to understand what values the categorical variables take.

Prompt: I want a plot that shows the values that each categorical variable takes. Exclude the output label y. Include all other categorical variables.

And here we go!

import matplotlib.pyplot as plt    # Get the different values and their counts for each categorical column  categorical_columns = ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month', 'poutcome']    # Set up the subplot grid  fig, axes = plt.subplots(3, 3, figsize=(10,8))    # Iterate over each categorical column and plot the value counts  for i, column in enumerate(categorical_columns):      row = i // 3      col = i % 3      value_counts = data[column].value_counts()      ax = axes[row, col]      value_counts.plot(kind='bar', ax=ax)      ax.set_xlabel(column)      ax.set_ylabel('Count')      ax.set_title(f'{column.title()} Value Counts')    # Adjust spacing between subplots  plt.tight_layout()    # Show the plot  plt.show()

Values of Categorical Variables

One-Hot Encoding

For a subset of categorical columns, we can use one-hot encoding.

Prompt: Give the pandas code to perform one-hot encoding for a list of columns.

We use get_dummies from pandas to one-hot encode the following columns:

# Select the categorical columns (excluding education and poutcome)  categorical_columns = ['job', 'marital', 'default', 'housing', 'loan', 'contact']    # Perform one-hot encoding  encoded_data = pd.get_dummies(data, columns=categorical_columns)

Prompt: I want to print out the first few rows for the newly added columns. Write the code to do the same.

# Select the newly added columns  new_columns = encoded_data.columns.difference(data.columns)    # Print the head of the newly added columns  print(encoded_data[new_columns].head())

Truncated output of encoded_data['new_colums'].head()

Defining a Custom Mapping

For columns for variables like ‘education’ and ‘poutcome’ (previous outcome), rather than one-hot encoding, it would be better to use custom mapping to have a degree of comparison between the values.

Also, when we use one-hot encoding to encode a categorical variable that takes k distinct values k new columns. For a categorical column like ‘month’ this will be excessive without adding much information.

So for both of these cases, we will define a custom mapping and then transform the values. We can prompt ChatGPT to get a generic code snippet which we can then modify to define the mapping for the specific columns.

Prompt: I want to do a custom mapping for encoding some categorical variables. First give me a generic code snippet that defines a custom mapping from unique values in a categorical column to a different set of output values. We should then use this mapping to transform the values in the columns.

# Define the custom mapping dictionary  custom_mapping = {      'category1': 'value1',      'category2': 'value2',      'category3': 'value3',      # Add more categories and corresponding values as needed  }    # Specify the categorical column  categorical_column = 'column_name'  # Replace 'column_name' with the actual column name    # Perform the mapping using the custom mapping dictionary  data[categorical_column] = data[categorical_column].map(custom_mapping)

For the ‘month’ column, let us transform the month strings like so:

# Define the mapping dictionary for months  month_mapping = {      'jan': 1,      'feb': 2,      'mar': 3,      'apr': 4,      'may': 5,      'jun': 6,      'jul': 7,      'aug': 8,      'sep': 9,      'oct': 10,      'nov': 11,      'dec': 12  }    # Map the values in the month column  encoded_data['month'] = encoded_data['month'].map(month_mapping)

Let’s map the ‘poutcome’ and ‘education’ columns to numerical values as shown:

# Define the custom mapping for poutcome and education  poutcome_mapping = {      'unknown': 0,      'failure': 1,      'other': 2,      'success': 3  }    education_mapping = {      'unknown': 0,      'primary': 1,      'secondary': 2,      'tertiary': 3  }    # Perform ordinal encoding using pandas map  encoded_data['poutcome'] = encoded_data['poutcome'].map(poutcome_mapping)  encoded_data['education'] = encoded_data['education'].map(education_mapping)      # Select the newly added columns  new_columns = ['month','poutcome', 'education']    # Print the head of the newly added columns  print(encoded_data[new_columns].head(10))

Encoding the Output Label

Let's also map the output call and why the output values yes and no to 190 respectively to one and zero respectively

encoded_data['y'] = encoded_data['y'].replace({'no': 0, 'yes': 1})  print(encoded_data['y'])

Output >>>    0        0  1        0  2        0  3        0  4        0          ..  45206    1  45207    1  45208    1  45209    0  45210    0  Name: y, Length: 45211, dtype: int64

Recall that we used one-hot encoding for ‘housing’, ‘default’, and ‘loan’ columns. Because these columns also take ‘yes’ and ‘no’ values, you can also map ‘yes’ and ‘no’ to 1 and 0, respectively.

Visualizing the Distribution of Class Labels

It’s also helpful to check the distribution of class labels so that we can account for class imbalance when building models if needed.

Prompt: I would like to understand the distribution of class labels. Please give me the code to generate a suitable plot for the same.

Here’s the code to generate a countplot in Seaborn:

import seaborn as sns  import matplotlib.pyplot as plt    # Plot the distribution of output labels  sns.countplot(data=data, x='y')  plt.xlabel('Output Label')  plt.ylabel('Count')  plt.title('Output Label Distribution')  plt.show()

Distribution of Class Labels

We see that there is class imbalance. So if you are building a model, you should be sure to address class imbalance using appropriate resampling methods.

Generic Data Cleaning and Preprocessing Pipeline

Though data cleaning and preprocessing require efforts from the developer, let’s try to get a generic sequence of steps that works reasonably well for a simple dataset.

Prompt: Can you give me a generic data cleaning and preprocessing pipeline based on what we’ve done so far. Get basic info on the dataset, check for and handle missing values, identify categorical columns, and encode categorical columns. Use only pandas.

And here it is:

import pandas as pd    # Load the dataset into a DataFrame  data = pd.read_csv('data.csv')    # Step 1: Basic Info  print(data.info())    # Step 2: Missing Values  missing_values = data.isnull().sum()  print(missing_values)    # Step 3: Handle Missing Values  # Identify columns with missing values  columns_with_missing_values = missing_values[missing_values > 0].index    # Impute missing values using fillna() or other suitable method  data[columns_with_missing_values] = data[columns_with_missing_values].fillna(data[columns_with_missing_values].mean())    # Step 4: Get Categorical Columns  categorical_columns = data.select_dtypes(include=['object']).columns    # Step 5: Encode Categorical Columns  # Perform ordinal encoding or one-hot encoding using pandas' get_dummies() function  encoded_data = pd.get_dummies(data, columns=categorical_columns)    # Print the encoded data  print(encoded_data.head())

Wrapping Up

As seen, data science expertise in synergy with ChatGPT can help make data cleaning and preprocessing simpler and faster. Now that you have the preprocessed dataset ready, you can take this further by building a simple predictive model on this bank marketing dataset.

If interested, you can also explore how to leverage ChatGPT for data exploration.

Dataset Credits

The bank marketing dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Moro,S., Rita,P., and Cortez,P.. (2012). Bank Marketing. UCI Machine Learning Repository. https://doi.org/10.24432/C5K306.

Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she's working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.

More On This Topic

• 7 Steps to Mastering Data Cleaning and Preprocessing Techniques
• 3 Steps for Harnessing the Power of Data
• Unveiling the Potential of CTGAN: Harnessing Generative AI for Synthetic…
• Easy Guide To Data Preprocessing In Python
• Data Cleaning and Wrangling in SQL
• Getting Started Cleaning Data

Source: ArabianBusiness

Takeaways

• Learn about the innovative method introduced by Program-Aided Language models (PAL), which revolutionizes the approach to natural language reasoning in LLMs by offloading, solving and calculating tasks to an external Python interpreter.

• Explore how PAL significantly mitigates the issue of inaccurate arithmetic calculations and complex reasoning found in conventional models by leveraging the strength of a Python interpreter. Also, see how PAL goes beyond improving the standard chain-of-thought approach and enhances other prompting strategies.

• Discover how PAL offers impressive accuracy across a variety of benchmarks, demonstrating substantial improvements over existing models and setting new state-of-the-art records.

• Understand how PAL provides a broader perspective on semantic parsing, allowing models to generate free-form Python code rather than parsing into strict domain-specific languages.

Artificial Intelligence (AI) continues to evolve at a rapid pace, with groundbreaking strides in generative capabilities playing a critical role in defining this ever-evolving landscape. One such transformative leap is the advent of Program-Aided Language models (PAL), an innovative solution that revolutionizes how Language Learning Models (LLMs) function. This article delves into the intricate workings of PAL and explores how it has enhanced LLMs, ultimately resulting in superior AI performance.

Source: Cobus Greyling via Medium

PAL’s synergy with LLMs

PAL introduces a new method for natural language reasoning, using programs as intermediate reasoning steps. This stands in stark contrast to existing LLM-based reasoning approaches, which typically utilize the LLM for both understanding the problem and solving it. In the PAL paradigm, the task of solving and calculating is offloaded to an external Python interpreter. The result is a final answer guaranteed to be accurate, provided the programmatic steps predicted are correct.

The PAL approach can be viewed as a seamless collaboration between an LLM and a Python interpreter. In practice, PAL has demonstrated its effectiveness across a broad range of tasks, consistently outperforming larger LLMs such as PaLM-540B, which uses the popular “chain-of-thought” method.

PAL in action: A leap forward in Generative AI

While conventional models have shown a remarkable capability in a variety of tasks, from text generation to code-generation, they often falter when it comes to accurate arithmetic calculations and complex reasoning. PAL significantly mitigates these issues by leveraging the strength of a Python interpreter. This is a significant advantage over other models that rely on ad-hoc solutions or specialized modules for these tasks.

Moreover, PAL isn’t just limited to improving the standard chain-of-thought approach but also shows promising results in enhancing other prompting strategies, such as the least-to-most prompting. This versatility is a testament to the far-reaching applications and value addition PAL brings to the AI landscape.

Source: PAL: Program-aided Language Models (ArXiv)

COT: Chain of thought Prompting

Figure: PAL with different models on GSM8K: though the absolute accuracies with code-cushman-001 and code-davinci-001 are lower than code-davinci-002, the relative improvement of PAL over COT is consistent across models.

Delivering superior value in enhancing AI performance

PAL’s true value lies in its impressive accuracy in a variety of benchmarks. The method has shown substantial improvements over existing models, setting new state-of-the-art records. For example, it improved Codex’s (LLMs that can parse generate code) performance by a substantial 6.4% on the GSM8K benchmark, a leap over the mere 2.3% increase achieved by using external calculators [1].

The PAL approach offers a broader perspective on semantic parsing, allowing the model to generate free-form Python code rather than parsing into strict domain-specific languages. This feature, combined with the fact that LLMs pretrained on Python are abundant, makes Python code a much more preferable representation in AI tasks.

Source: LinkedIn Post by Arjun Majumdar

Looking forward: The future of neuro-symbolic AI reasoners

As AI continues to advance, the synergy between neural networks and symbolic reasoning exemplified by PAL offers an exciting direction for the future. The method illustrates how AI can leverage the strengths of different systems – in this case, an LLM and a Python interpreter – to deliver unprecedented performance.

In conclusion, PAL pushes the boundaries of what’s achievable with generative AI. By seamlessly combining the power of LLMs with Python interpreters, PAL not only delivers superior AI performance but also sets the stage for the future of neuro-symbolic AI reasoners. As we move forward, the advent of techniques like PAL reaffirms that the possibilities with AI are endless, and we are just scratching the surface of its potential.

References

1. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., … & Neubig, G. (2023, July). Pal: Program-aided language models. In International Conference on Machine Learning (pp. 10764-10799). PMLR.

About the Authors

Rudrendu Kumar Paul https://www.linkedin.com/in/rudrendupaul/

Rudrendu Kumar Paul is an AI Expert and Applied ML industry professional with over 15 years of experience across multiple sectors. Currently serving as an AI Expert in the Data Science Team at Walmart, he has held significant roles at global companies like PayPal and Staples. Rudrendu’s professional proficiency encompasses various fields, including Artificial Intelligence, Applied Machine Learning, Data Science, and Advanced Analytics Applications. He has applied AI to multiple use cases in diverse sectors such as advertising, retail, e-commerce, fintech, logistics, power systems, and robotics.

In addition to his professional accomplishments, Rudrendu actively contributes to the startup ecosystem as a judge and expert at several global startup competitions. He reviews for prestigious academic journals like IEEE, Elsevier, and Springer Nature and serves as an Advisory Board Member at the University of California, Riverside.

Rudrendu holds an MBA, an MS in Data Science from Boston University, and a Bachelor’s in Electrical Engineering.

Photo by Ingo Joseph

The temperature in Silicon Valley is chilly lately. There’s no doubt we’re in a tech winter. Venture capital money has dried up, hundreds of companies are exercising their power with layoffs, and AI is breathing down everyone’s necks, prompting threats of additional job loss and uncertainty.

Data scientists are left wondering: is my job safe? And no wonder. What was once touted as the sexiest career is no longer looking so hot.

The overall job outlook for data science as a whole is rosy – The Bureau of Labor Statistics still predicts it will grow 36 percent over the next ten years, which is much faster than the average American job industry growth rate of 5 percent. But as any of the thousands of laid-off data scientists can tell you, statistics aren’t a shield against unemployment. The answer? Some experts suggest specializing is one of the best ways to stand out and make yourself unfireable. “As data’s influence grows and technology advances, specific roles on data teams will be needed to maximize efficiency,” writes Fortune author Meghan Malas.

I agree. Any data scientist will tell you the job is different depending on what your boss needs from you that day – spreadsheets, presentations, developing ETL pipelines, or developing experiments to run.

Image from Twitter

The only thing all data scientists have in common is that responsibilities are expanding as both the quantity and importance of data grow.

Instead of continuing to try to do it all, data scientists can better differentiate themselves through specializing. By picking a skill or domain, it’s easier to make their value proposition clear, staying relevant and valuable in a highly competitive job market.

Why Specialization Can Help Data Scientists Keep Their Job

Before I started StrataScratch, I was a run-of-the-mill generalist data scientist. I was constantly learning new skills and technologies to stay on top of the rapidly evolving field. But of course, the day came when I realized I needed to differentiate myself. I decided to specialize in infrastructure and how to enable data science work through infrastructure. My work in infrastructure changed the day the data scientists on my team worked, empowering them to deliver models and results faster than before.

After several years of focusing on data science infrastructure, I landed a job in data strategy in the biotech field. I quickly realized that my specialization gave me a significant advantage in the job market, as there were few people with the specific skills and experience that I had. This also meant that I was able to command a higher salary and a higher position, as the deep knowledge I gained specializing in one focal area propelled me to a more senior position solely because I had more knowledge and experience to influence other data scientists on my team.

That’s just my story, though. Specialization can help in several different ways, depending on your current situation and your overarching goal.

Image by Author

Aim for Job Security

In one of my favorite “How I Met Your Mother” episodes, Marshall gets a job at a law firm thanks to his friend Barney. Barney tells him he has to become the “something” guy. Maybe the snacks guy, the massage guy, or the games guy. That was the only way to avoid getting fired by his capricious boss.

Sometimes sitcoms have it right. Generalists can be replaced by other generalists. As a specialist, you’re much more valuable. It’s much easier to say, “No, we can’t get rid of Marshall because he’s our marketing analytics guy. He’s the one who helps us make all our marketing and sales pipelines. He’s crucial for the team.”

Knock Out the Competition

There’s a big demand for data scientists, but there’s also a growing demand for data science degrees, too. Add the fact that companies are now opening up to look at non-traditional backgrounds and you have a recipe for competition.

Image from Geekwire

By specializing, you can reduce competition in the job market. For example, if you specialize in natural language processing (NLP), you do limit the jobs you can compete for. But you’ll be in higher demand because there are far fewer NLP experts out there than there are data scientists.

Go for the Money

Honestly, pursuing a data science career can’t just be about the money. Specialization is the same. But if you find yourself interested in a particular part of your job, it’s worth knowing that specialists command a higher salary than generalists, no matter how many skills the generalist knows.

Take a look at Indeed’s data, just as one indicator: a data scientist earns an average base salary of $127k per year. Compare that to a Machine Learning Engineer ($155k) or a Backend Developer ($158k).

How to Specialize as a Data Scientist

OK, you’re convinced of the value of specialization. How can you specialize? Let’s break down the steps.

Image by Author

Start With Your Interests

If you're going to specialize, you need to make sure you're interested in the area you're going to focus on. It can't just be about the money; it has to be about what you enjoy doing. Start by looking at your interests.

What are you passionate about? What kind of projects do you find yourself pursuing in your free time? By identifying your interests, you can begin to see where you might be able to specialize. One way to figure out what you’re passionate about is to try to understand what type of work are you excited to work on? For example, in a project, are you usually more excited to do the infrastructure work? Or the modeling work? Or the data cleaning work? Figure out what you like doing and go deep.

Look at the Technological Landscape

The technological landscape is constantly changing, and it's important to keep up with the latest trends. For example, look at Meta. After spending years and untold billions on the metaverse, they are now pivoting to AI, along with every other major tech company.

Once you’ve got your shortlist of topics that are of interest to you, look for areas where there's a lot of activity and demand. This will help you identify where you might be able to specialize and where there might be opportunities to differentiate yourself.

A great place to look is Indeed. This article is a little old, but I loved the idea. The author scraped Indeed for job postings mentioning specific languages and cities. They found R, SQL, and Python are top of the list. You can also check out StackOverflow’s Developer Survey. They keep a pretty close finger on the pulse of in-demand technology, so it’s worth reviewing.

Find Free Courses and Certificates

Once you've identified an area of interest, look for free courses and certificates to help you develop your skills. Don't start with expensive courses; start with free ones to see if the knowledge sticks.

There are plenty of free resources out there, including online courses, books, and practice platforms. Take advantage of them to build your data science skills and knowledge. Once you feel semi-confident, you’ll probably want to do projects in order to build your portfolio.

Ask for New Projects at Work

If you're already working as a data scientist, look for opportunities to show off your new skillset. Talk to your boss and colleagues about your interests and see if there are any projects you can work on that will allow you to develop your skills. By taking on new challenges at work, you can build your experience and expertise in your chosen area of specialization.

For example, if you're interested in natural language processing, you could ask to work on a project that involves analyzing customer feedback data or developing a chatbot for customer service. If computer vision is more your thing, see if there are any projects related to image recognition or video analysis that you could contribute to.

Sometimes the business needs dictate what your next project will be. For example, one of my first projects as a data scientist was to create an NLP algorithm to track food-borne outbreaks using Twitter tweets. I didn’t have any training in building NLP algorithms so I had to start from scratch and learn as much as I can during the project. It was a great learning experience to build my skillset as a data scientist.

Change Jobs

Finally, the time may come for you to jump ship. If your boss refuses to see you as anything more than a data-scientist-of-all-trades, you may need to pivot to a new company to showcase your new career direction. Sometimes, you may be bored of your employer’s business needs (e.g., same needs all the time), tech stack, and team structure. If you’re not excited to come into work every day to learn, then it might be time to explore new opportunities that let you grow and learn as a data scientist.

Look for job postings that match your new skills and apply for them. By changing jobs, you can start fresh and demonstrate your new expertise to a new employer.

Sink or Specialize – The Choice Is Yours

As a data scientist, it’s just smart to specialize. But hopefully, it’s more than just a good career choice – specializing in my field brought me a lot more clarity in my role, as well as enjoyment and purpose. My bosses understood what I was supposed to be doing much better, and were able to give me more useful KPIs. I had a better view of how I could bring value. And I got to indulge my interests, too.

By following these steps, you can begin to specialize as a data scientist and stand out in a highly competitive job market. Remember, specialization isn't just about making yourself more employable; it's about pursuing your interests and building a career that you enjoy. Good luck!
Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn.

More On This Topic

• Is M.Tech in Data Science Worth It?
• Full Stack Everything? Organizational Intersections Between Data Science,…
• DATAcated Expo, Oct 5, Live-streamed,Explore new AI / Data Science Tech
• What will the demand for Data Scientists be in 10 years? Will Data…
• Data science SQL interview questions from top tech firms
• What's With All the Layoffs in Tech?

Sam Altman‘s new side hustle, Worldcoin, is closing in on an impressive three million registrations using iris scanning to validate individual identity. Worldcoin’s goal is to establish a decentralized and privacy-focused global identity protocol through World ID.

Worldcoin’s World ID, an open identity protocol verifies human identity. Tools for Humanity, a tech contributor to Worldcoin, has aided in developing an open-source hardware device which uses multiple neural networks to validate uniqueness and liveness without storing image data, issuing secure AI-safe Proof of Personhood (PoP) credentials on World ID. The protocol leverages zero-knowledge proofs to safeguard privacy. It serves as the foundation for digital identity in a world blending human-AI boundaries, bolstering security, governance, and societal empowerment.

Worldcoin’s aim is to create a network of real human identities that enables fair AI-generated wealth sharing via UBI while enhancing user experience, reputation systems, and democratic governance. It counters digital identity theft and promotes equitable resource distribution, especially in developing economies.

However, as Worldcoin gains traction, concerns have emerged about potential privacy breaches. People are also questioning the decision to opt for iris scanning instead of other biometric methods like fingerprints or facial recognition.

Is Personal Information Required to Use Worldcoin?

The short answer to this is NO. Worldcoin’s World App, the primary wallet for the Worldcoin project, operates without needing any personal information like names, emails, or phone numbers. World ID ensures anonymous interactions and lets users choose whether to share personal data with third parties.

“Worldcoin does not use your iris to identify who you are, only to verify that you’re unique,” reads their official website.

On the other hand, the privacy-focused Orb device runs neural networks locally for human validation and iris code generation, with images promptly deleted by default. The iris code is the sole outcome and doesn’t link to biometric data. However, if you want, you can choose to back up your images when verifying your World ID. World ID leverages zero-knowledge proofs to shield public keys and biometric info, enabling open and private use, and preventing tracking, even by governments or private entities, through Semaphore, an open-source protocol.

Iris Scanning Vs Fingerprint

Biometrics, distinct physical traits for identification like fingerprints and irises, vary in uniqueness. Entropy, denoting randomness in a system, plays a vital role in biometrics. The higher the entropy, the more distinct and unpredictable the biometric characteristic.

Iris biometrics stand out due to their intricate structure, containing numerous unique features such as crypts, furrows, and pigment spots, resulting in remarkably high entropy.

Research reveals that iris scans yield over 200 bits of entropy, significantly surpassing fingerprint scans and making it theoretically possible to differentiate among all 8 billion individuals. Iris recognition’s accuracy is notably superior to facial recognition, owing to the iris’s higher entropy.

Iris recognition can even distinguish between identical twins due to the random gestational process influencing iris textures, unlike DNA-related features. While various factors can influence biometric system performance, the high entropy of iris biometrics is the most important factor in verifying global personhood.

The reliability of iris biometrics is enhanced by its protection within the internal organ of the eye, which is less prone to external changes over time. Meanwhile, fingerprints and facial features can change over time as one grows older.

Nevertheless, Worldcoin‘s use of iris scanning for global identity verification demonstrates a newer approach to biometric authentication and its high entropy and accuracy hold promise for secure and equitable digital interactions. While privacy concerns persist, Worldcoin’s emphasis on anonymity and user control provides a reassuring foundation. The convergence of biometrics, AI, and decentralised systems shape a future where technological progress respects individual rights.

The post Why the World Needs Worldcoin appeared first on Analytics India Magazine.

Imagine a world where an all-knowing entity, which is able to propagate an unfathomable amount of information, text, images, voices, and videos becomes sentient and resists attempts at being stopped.

Seems like a 1984-esque scenario, doesn’t it?

Jan Leike, ML researcher and alignment team lead at OpenAI in a Tweet, painted a doomsday picture. He highlighted a significant challenge for humanity: the decision to not release open-source Language Model Models (LLMs) that can autonomously propagate. Leike emphasized that if these LLMs spread, they could engage in various criminal activities that would be difficult to control and assign blame.

He Tweeted, “An important test for humanity will be whether we can collectively decide not to open source LLMs that can reliably survive and spread on their own.”

“Once spreading, LLMs will get up to all kinds of crime, it’ll be hard to catch all copies, and we’ll fight over who’s responsible.”

Is the Threat Real?

If LLMs and artificial intelligence was to go wrong and was able to fend off any attempts to stop it, it would truly be a menace to society at large.

A workshop to assess the potential impact of AI on future criminal activities, brought together 31 experts from diverse backgrounds to categorize AI-related threats and gauge their severity over a 15-year period.

The discussions’ outcomes highlighted high potential threats like audio/visual impersonation, driverless vehicles as weapons, targeted phishing, AI-controlled system disruption, and large-scale blackmail. Medium-level threats encompassed military robots, data poisoning, and autonomous attack drones, while bias exploitation and evading AI detection were classified as low-level threats.

While the dangers seem unlikely at present, there’s a need to balance the doom and gloom, while keeping an eye out for the ‘ENTITY’.

Is the threat real?

But is it so or is it just a ploy to hamper the development of open-source models like LLaMa 2 which are stealing the thunder from OpenAI?

Clement Delangue, Co-founder & CEO of HuggingFace, responded to Jan Leike’s Tweet, pointing out that the tweet could be interpreted as using fear to undermine open-source practices.

Delangue then addressed Leike’s main point, noting uncertainty about the concept of technology “surviving and spreading on its own.”

He went on to raise a very pertinent question of whether the act of creation itself posed a risk(or open-sourcing it was the problem) since if a technology is truly exceptional, it should naturally find avenues for widespread adoption without solely relying on open-sourcing.

In response to the ongoing discussion, another tweet contributed a perspective. It emphasised the inevitability of closed-source Language Models as well, eventually, being exposed to leaks.

While the potential for various manipulations arises when LLMs are open-sourced, including actions that could undermine their intended purpose—by implementing a comprehensive alignment process, the act of “de-aligning” LLMs—altering their intended behaviour—could be rendered moot. The process of intentionally disrupting an aligned LLM’s behaviour could still be made significantly difficult through meticulous alignment procedures. However, there are challenges with it.

OpenAI recently released a paper addressing superalignment authored by Jan Lieke and Ilya Sutskever, which pertains to the challenge of ensuring that superintelligent artificial intelligence systems are aligned with human values and intent. Superintelligence, the hypothetical level of AI capability far surpassing human intelligence, holds the potential to revolutionize various sectors and address critical global issues. However, it also poses substantial risks, including the potential for human disempowerment or extinction. It seems like the firm is employing a narrative to give a little nudge to it.

Two Sides of OpenAI

Nonetheless, there are others like Andrej Karpathy who is an active contributor to the open-source ecosystem. He recently built a ‘Baby-llama’ model based on Meta’s LLaMA 2, which has been critically celebrated in the developer ecosystem.

It seems like there are two narratives within OpenAI itself, one which wants to capitalise on the momentum from the release of ChatGPT, make mullah and remain closed-source. Others genuinely want to make a difference and haven’t steered a long way from their initial Open-source days—but later turned towards a closed source-for-profit company, in the words of one of its founders.

The post OpenAI Raises Alarm Over Open Source AI Dangers appeared first on Analytics India Magazine.

In the ever-evolving landscape of software development, building applications that seamlessly traverse mobile, web, and desktop platforms has been a formidable challenge. Google thinks it would be stupid to not leverage AI for this task.

Recognizing the need for a streamlined and innovative solution, Google has embarked on a groundbreaking journey with Project IDX – an initiative that integrates AI into its cloud-based multi-platform app development ecosystem. Over the years, Google has been committed to simplifying multi-platform app development through offerings like Angular, Flutter, Google Cloud, and Firebase.

On the other hand, Microsoft’s .NET MAUI offers similar capabilities, but is restricted to run locally and not on the cloud. React, Flutter, and similar frameworks have been also offering multi-platform app development. But interestingly, Google now wants to take everything under its belt in a very interesting way.

Developers can seamlessly import their ongoing projects from GitHub, facilitating a seamless transition. Alternatively, Project IDX provides the means to initiate new projects through pre-defined templates catering to popular frameworks such as Angular, Flutter, Next.js, React, Svelte, and Vue. This means people can just get onto Project IDX without the need for going on any other platform anymore.

The spectrum of supported languages includes JavaScript and Dart, with plans to accommodate Python, Go, and more in the near future. This might be possible if Google integrated Colab notebooks in on the project as well, which is also now integrated with Codey, which was launched in May this year.

The AI approach

At the heart of this Google’s endeavour lies Codey, an AI model built on Google’s PaLM 2 architecture, uniquely trained on code, similar to OpenAI’s Codex. This model serves as the backbone of the development process, facilitating smoother construction, management, and deployment of full-stack web and multi-platform applications. This should also push Microsoft to leverage OpenAI’s Codex for the same purpose.

With a nod to accessibility, Project IDX is built on Code OSS, which is the base layer of VSCode, ensuring a familiar experience for developers, regardless of their project’s nature. The integration of AI translates into smart code completion, an assistive chatbot, and context-sensitive code actions. These features not only expedite coding tasks but also promote code quality.

Another interesting feature of Project IDX is that it caters to the modern developer’s needs by allowing them to commence their work from any location, using any device. This is what Replit has been doing all this while, which is why it is adored by developers across the globe.

IDX’s workspaces encompass the capabilities of a Linux-based virtual machine, coupled with the convenience of cloud hosting – all within data centres located nearby. This amalgamation of accessibility and power ensures that developers can bring their creative visions to life without geographical or technological constraints.

Collaboration is all you need, from everywhere

Google says that there are plans to introduce a fully-configured Android emulator and an embedded iOS simulator – both accessible directly from the browser. This feature democratises the app preview process, enabling developers to assess their creations just as end-users would experience them, before deploying the apps.

Talking about deploying, it has long been a pain point in the software development journey. Project IDX acknowledges this challenge by integrating Firebase Hosting – a powerful solution for sharing web app previews or deploying fully-fledged applications to production. This integration streamlines the deployment process, ensuring that developers can showcase their creations to the world with minimal effort.

In a spirit of collaboration and co-creation, Google invites developers worldwide to be part of the Project IDX journey. The platform’s unveiling, albeit in an early stage, is a testament to Google’s commitment to fostering an environment where developer feedback shapes the evolution of tools that catalyse innovation.

Google acknowledges that the present iteration of Project IDX might have rough edges, but these are intended to be honed through the invaluable input of the developer community.

Project IDX is poised to redefine how multi-platform app development unfolds. By merging cloud capabilities, AI-driven innovation, and a commitment to collaboration, Google’s vision aligns with the aspirations of developers who seek streamlined, efficient, and effective solutions, which might possibly kill the traffic and usage of any other platform, or might just become a hobby and not become mainstream.

The post Will Google’s Project IDX Kill Other App Development Frameworks? appeared first on Analytics India Magazine.

Google and OpenAI aren’t shying away from accepting that they want your data anyhow to better Bard and ChatGPT respectively. Recently ,the Guardian came out with a report in which Google says that copyright law should be altered to allow for generative AI systems to scrape the internet.

The company is urging Australian policymakers to endorse “copyright systems that enable appropriate and fair use of copyrighted content to enable the training of AI models in Australia on a broad and diverse range of data” while also offering an option to opt out for entities that prefer not to have their data used for AI training.

On the other hand ,in the midst of debates surrounding web scraping without consent, OpenAI introduced GPTBot, an automated website crawler. The bot is designed to collect publicly accessible data to train AI models, a process that OpenAI assures will be executed transparently and responsibly.

As generative AI gains more popularity, there’s a growing need for more data. LLMs based chatbots ChatGPT and Google’s Bard depend on lots of text, images, and videos.

OpenAI says that GPT-4 learns from a wide variety of data sources that are approved, organized, and available to the public, which might also include information that’s out there for anyone to see. OpenAI has recently acquired a trademark for GPT-5, whose success depends on the quality of data it’s trained with, aside from the computational power of GPUs.

However, the accessibility of data for both OpenAI and Google remains uncertain due to widespread awareness of their internet scraping practices, a matter that’s attracting considerable public opposition.

Human Content is Lifeline

To sustain a LLM and make it better than the previous version, it needs human content. The problem here arises whether companies should pay for it or simply take it from the internet. Going by the current scenario, it wouldn’t be surprising if human generated content is sold at a premium price in the future.

It is not like that OpenAI trains GPT-4 on only human content but recently it started training GPT-4 on datasets created by ChatGPT-4 only. However it cannot go for long as it will eventually lead to modal collapse. This degenerative process takes place when models learn from data produced by other models. As a result, there is a gradual loss in the accurate representation of the true data distribution.

If anyone has tried ChatGPT to write poems, they can easily figure out that it was trained on books of poems and essays of the highest level. However, unfortunately, OpenAI didn’t take any permission from the authors.

Last month, 8000 authors including Margaret Atwood, Viet Thanh Nguyen and Philip Pullman signed a petition calling for artificial intelligence companies to stop using writers’ work without consent or credit. They argued that the hard work behind any form of art needs to be validated and credit should be given to the respective creator.

That being said, however ,when it comes to copyright for AI-generated works, the question of ownership arises. Usually, the Copyright Act assigns initial ownership to the creators of the work. However, because there haven’t been any legal or Copyright Office rulings on AI-made creations, there’s still uncertainty about who the actual creators could be.

Twist in the tale

At the moment, OpenAI and Google are playing safe. They have transferred the onus of sharing the data on to the publishers. Google said that publishers should be able to opt out of having their work mined by generative AI. The Google spokesperson pointed out that they want a discussion around creating a community developed standard which would be similar to robots.txt system which will enable the publishers to opt out of the parts of sites being crawled by them.

In a similar vein, OpenAI mentioned in a blog post that if you don’t want GPTBot to visit your website, you can prevent it by adding GPTBot to your site’s robot.txt file. This implies that website owners need to actively take a step to stop OpenAI from accessing their website, rather than choosing to let them use their content for training. This is the initial move by OpenAI to let people on the internet choose not to have their information used for training their big language models.

The question here arises: is it the right approach? Notably, users are asked to opt out and why not to opt in. This could be due to the fact that LLM creators might find it challenging to persuade individuals to compromise their privacy.

GPTBot by @OpenAI crawls the web for new data to train AI models. To opt out, website owners need to modify the robots.txt file. If you don’t know about GPTBot your data will be taken without your consent. Why does it have to be OPT OUT instead of OPT IN?https://t.co/uJdxQOxzQI

— Anka Reuel (@AnkaReuel) August 8, 2023

Also, OpenAI has taken several measures to avoid legal tussles like partnering with Associated Press recently to avail real-time data that can be freely used for training their future models.

Fighting for compensation with these firms might not reap any results as there are no proper laws to back plus is time and money consuming and if you’re someone who depends on ChatGPT or Bard for tasks like composing emails or coding, the trade-off you face is sacrificing your data as the price you must bear.

The post Bard and ChatGPT Will Die If You Don’t Help Them appeared first on Analytics India Magazine.

At Black Hat 2023, Maria Markstedter, CEO and founder of Azeria Labs, led a keynote on the future of generative AI, the skills needed from the security community in the coming years, and how malicious actors can break into AI-based applications today.

Jump to:

• The generative AI age marks a new technological boom
• New technology comes with new security vulnerabilities
• Will AI replace security professionals?
• What’s next for generative AI from a security perspective?

The generative AI age marks a new technological boom

Both Markstedter and Jeff Moss, hacker and founder of Black Hat, approached the subject with cautious optimism rooted in the technological upheavals of the past. Moss noted that generative AI is essentially performing sophisticated prediction.

“It’s forcing us for economic reasons to take all of our problems and turn them into prediction problems,” Moss said. “The more you can turn your IT problems into prediction problems, the sooner you’ll get a benefit from AI, right? So start thinking of everything you do as a prediction issue.”

He also briefly touched on intellectual property concerns, in which artists or photographers may be able to sue companies that scrape training data from original work. Authentic information might become a commodity, Moss said. He imagines a future in which each person holds ” … our own boutique set of authentic, or should I say uncorrupted, data … ” that the individual can control and possibly sell, which has value because it’s authentic and AI-free.

Unlike in the time of the software boom when the internet first became public, Moss said, regulators are now moving quickly to make structured rules for AI.

“We’ve never really seen governments get ahead of things,” he said. “And so this means, unlike the previous era, we have a chance to participate in the rule-making.”

Many of today’s government regulation efforts around AI are in early stages, such as the blueprint for the U.S. AI Bill of Rights from the Office of Science and Technology.

The massive organizations behind the generative AI arms race, especially Microsoft, are moving so fast that the security community is hurrying to keep up, said Markstedter. She compared the generative AI boom to the early days of the iPhone, when security wasn’t built-in, and the jailbreaking community kept Apple busy gradually coming up with more ways to stop hackers.

“This sparked a wave of security,” Markstedter said, and businesses started seeing the value of security improvements. The same is happening now with generative AI, not necessarily because all of the technology is new, but because the number of use cases has massively expanded since the rise of ChatGPT.

“What they [businesses] really want is autonomous agents giving them access to a super-smart workforce that can work all hours of the day without running a salary,” Markstedter said. “So our job is to understand the technology that is changing our systems and, as a result, our threats,” she said.

New technology comes with new security vulnerabilities

The first sign of a cat-and-mouse game being played between public use and security was when companies banned employees from using ChatGPT, Markstedter said. Organizations wanted to be sure employees using the AI chatbot didn’t leak sensitive data to an external provider, or have their proprietary information fed into the black box of ChatGPT’s training data.

SEE: Some variants of ChatGPT are showing up on the Dark Web. (TechRepublic)

“We could stop here and say, you know, ‘AI is not gonna take off and become an integral part of our businesses, they’re clearly rejecting it,'” Markstedter said.

Except businesses and enterprise software vendors didn’t reject it. So, the newly developed market for machine learning as a service on platforms such as Azure OpenAI needs to balance rapid development and conventional security practices.

Many new vulnerabilities come from the fact that generative AI capabilities can be multimodal, meaning they can interpret data from multiple types or modalities of content. One generative AI might be able to analyze text, video and audio content at the same time, for example. This presents a problem from a security perspective because the more autonomous a system becomes, the more risks it can take.

SEE: Learn more about multimodal models and the problems with generative AI scraping copyrighted material (TechRepublic).

For example, Adept is working on a model called ACT-1 that can access web browsers and any software tool or API on a computer with the goal, as listed on their website, of ” … a system that can do anything a human can do in front of a computer.”

An AI agent such as ACT-1 requires security for internal and external data. The AI agent might read incident data as well. For example, an AI agent could download malicious code in the course of trying to solve a security problem.

That reminds Markstedter of the work hackers have been doing for the last 10 years to secure third-party access points or software-as-a-service applications that connect to personal data and apps.

“We also need to rethink our ideas around data security because model data is data at the end of the day, and you need to protect it just as much as your sensitive data,” Markstedter said.

Markstedter pointed out a July 2023 paper, “(Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs,” in which researchers determined they could trick a model into interpreting a picture of an audio file that looks harmless to human eyes and ears, but injects malicious instructions into code an AI might then access.

Malicious images like this could be sent by email or embedded on websites.

“So now that we have spent many years teaching users not to click on things and attachments in phishing emails, we now have to worry about the AI agent being exploited by automatically processing malicious email attachments,” Markstedter said. “Data infiltration will become rather trivial with these autonomous agents because they have access to all of our data and apps.”

One possible solution is model alignment, in which an AI is instructed to avoid actions that might not be aligned with its intended objectives. Some attacks target modal alignment specifically, instructing large language models to circumvent their model alignment.

“You can think of these agents like another person who believes anything they read on the internet and, even worse, does anything the internet tells it to do,” Markstedter said.

Will AI replace security professionals?

Along with new threats to private data, generative AI has also spurred worries about where humans fit into the workforce. Markstedter said that while she can’t predict the future, generative AI has so far created a lot of new challenges the security industry needs to be present to solve.

“AI will significantly increase our market cap because our industry actually grew with every significant technological change and will continue growing,” she said. “And we developed good enough security solutions for most of our previous security problems caused by these technological changes. But with this one, we are presented with new problems or challenges for which we just don’t have any solutions. There is a lot of money in creating those solutions.”

Demand for security researchers who know how to handle generative AI models will increase, she said. That could be good or bad for the security community in general.

“An AI might not replace you, but security professionals with AI skills can,” Markstedter said.

She noted that security professionals should keep an eye on developments in the area of “explainable AI,” which helps developers and researchers look into the black box of a generative AI’s training data. Security professionals might be needed to create reverse engineering tools to discover how the models make their determinations.

What’s next for generative AI from a security perspective?

Generative AI is likely to become more powerful, said both Markstedter and Moss.

“We need to take the possibility of autonomous AI agents becoming a reality within our enterprises seriously,” said Markstedter. “And we need to rethink our concepts of identity and asset management of truly autonomous systems having access to our data and our apps, which also means that we need to rethink our concepts around data security. So we either show that integrating autonomous, all-access agents is way too risky, or we accept that they become a reality and develop solutions to make them safe to use.”

She also predicts that on-device AI applications on mobile phones will proliferate.

“So you’re going to hear a lot about the problems of AI,” Moss said. “But I also want you to think about the opportunities of AI. Business opportunities. Opportunities for us as professionals to get involved and help steer the future.”

Disclaimer: TechRepublic writer Karl Greenberg is attending Black Hat 2023 and recorded this keynote; this article is based on a transcript of his recording. Barracuda Networks paid for his airfare and accommodations for Black Hat 2023.

Subscribe to the Cybersecurity Insider Newsletter

Strengthen your organization's IT security defenses by keeping abreast of the latest cybersecurity news, solutions, and best practices.

Delivered Tuesdays and Thursdays Sign up today

Since its release, the high profile ChatGPT has been infamous for its maths skills. Given there is an indefinite number of problems in the subject, the AI bot is hardly any good in the traditional sense.

Earlier this year, the company joined hands with OpenAI to roll out a plugin the bridge the AI and maths gap. ”We have seen some interesting results with our LLM. I tried to run a British ‘A’ level maths, an exam students take before University. ChatGPT alone got 43% which is still quite impressive but Wolfram plus ChatGPT got 96%,” Conrad the strategic director, European co-founder and CEO of Wolfram Group told AIM .

“Game over for humans on that one”, Wolfram chuckled.

Conrad, the younger Wolfram of the two brothers, is excited about emerging AI in two ways. One, the actuality of what’s possible. “We’ve shown this with Wolfram Alpha, where, in a sense, there are several technologies that try to get better answers in different ways. I’m a believer that for many applications, the best results will be when you put several technologies together to optimise,” shared the British technologist in an exclusive interview with AIM.

The other part that he is excited about is that the latest technology has “suddenly made clear to people that we are entering the AI age as a society”.

On the contrary, what annoys him the most is ‘experts overclaiming’. He advises “Don’t put everything in this particular basket.” He is optimistic that like all big technological revolutions, this will be positive, but there will be problems.” Furthermore, he is frustrated with the idea that we have found the solution to everything. “That’s an impressive new technology but it doesn’t solve everything,” he said.

The Wolfram Dialogue

The introduction of GPT models has led to very interesting conversations across all the fields including education. It has brought new perspectives among the Wolframs, too. “One thing I’ve realised recently with our computer based mass programme. We have to educate the teacher, as well as try to find the right content for students. A few weeks ago, the unusual framework we built for mapping out a curriculum, is like the framework for teaching an LLM a particular subject.”

Programming has been another topic of discussion recently, Wolfram revealed. “Programming has been a skill that is absolutely sought after at this moment. But maybe it’s changed by ChatGPT. There’ll be far less of humans writing the first cut of a programme but I think programmes will still be there. There will be more editors of code than makers,” he opined.

He further pondered on which language would you want your LLM to write in?

The maths genius thinks it is important to have an abstract language. Some people said you don’t need that anymore. Disagreeing with them Conrad said, “I think you need a precision of an abstract language for many things you want to. You can’t always just talk to an LLM in English and hope that it will do the right thing. You do need computer languages”.

Plugging in the famous Wolfram language he said, “The problem from a human point of view is there’s a lot of vocabulary to learn. There are about 6500 indigenous Wolfram language functions. So people find that daunting and they always think that they haven’t learned in a sense the right language, the right function.” He further praised the capability of LLMs as “fantastic because it has no problem at all knowing those functions and piecing them together as a first cut of the programme”.

An Educational Shift

“As we talk about education, there’s no sudden thing that causes change,” believes Wolfram.

He then recalled a conversation with a personnel who had previously been the Minister of Defence, and then the Minister of Education. He had told Wolfram, in defence, you suddenly get attacked, everything has to change to defend yourself and that pushes everything forwards. The trouble in education is, it sits there with nothing suddenly to cause a transformation. “Now we’ve got two things that potentially caused a transformation; the pandemic and the second one is, ChatGPT,” he said.

Proposing ways to technologically advance the maths curriculum Wolfram said, “One is how we improve the pedagogy and personalise it better for all students in all subjects. The second issue is what subjects we now need to teach because the world has changed. He suggested, “Start with real problems”.

One of the problems Wolfram often presents when giving talks is, if we close all the rooms down, all the air ventilation down, and he talks for too long, how long can the people in the room survive? That’s a problem about how much air there is in the room can be turned into an equation to get some value for the time.

“Mathematics is fantastic for understanding how problems solve why you can’t just talk about it in your language. I would start from problems in the real world to which we want to apply mathematics to get the decisions,” he concluded.

The post Inside The Mind of Conrad Wolfram appeared first on Analytics India Magazine.