Рубрика: AI

When you do model training, you send data through the network multiple times. Think of it like wanting to become the best basketball player. You aim to improve your shooting, passing, and positioning to minimize errors. Similarly, machines use repeated exposure to data to recognize patterns.

This article will focus on a fundamental concept called backward propagation. After reading, you’ll understand:

1. What backward propagation is and why it’s important.

2. Gradient Descent and its type.

3. Backward propagation in Machine Learning.

Let’s delve into backpropagation and its significance.

What is backpropagation and why does it matter in neural networks?

In Machine Learning, machines take actions, analyze mistakes, and try to improve. We give the machine an input and ask for a forward pass, turning input into output. However, the output may differ from our expectations.

Neural Networks are supervised learning systems, meaning they know the correct output for any given input. Machines calculate the error between the ideal and actual output from the forward pass. While a forward pass highlights prediction mistakes, it lacks intelligence if machines don’t correct these errors. For learning in depth about machine learning & Neural networks you can join multiple data science courses available online. Algorithms’ insight into ML and neural networks and their practical application is important to understand.

After the forward pass, machines send back errors as a cost value. Analyzing these errors involves updating parameters used in the forward pass to transform input into output. This process, sending cost values backward toward the input, is called “backward propagation.” It’s crucial because it helps calculate gradients used by optimization algorithms to learn parameters.

What is the time complexity of a backpropagation algorithm?

The time complexity of a backpropagation algorithm, which refers to how long it takes to perform each step in the process, depends on the structure of the neural network. In the early days of deep learning, simple networks had low time complexity. However, today’s more complex networks, with many parameters, have much higher time complexity. The primary factor influencing time complexity is the size of the neural network, but other factors like the size of the training data and the amount of data used also play a role.

Essentially, the number of neurons and parameters directly impacts how backpropagation operates. The time complexity of the forward pass (the movement of input data through the layers) increases as the number of neurons involved grows. Similarly, in the backward pass (when parameters are updated to repair errors), additional parameters result in increased temporal complexity.

Gradient descent

Gradient Descent is like training to be a great cricket player who excels at hitting a straight drive. During the model training, you repeatedly face balls of the same length to master that specific stroke and reduce the room for errors. Likewise, gradient descent is an algorithm that aims to minimize the error-proneness, or cost function, to produce the most accurate result possible. Artificial Intelligence uses this Gradient Descent data to train a model. Software model training in depth is covered in many online full stack developer courses. Learning from Online material will give a good hands-on experience in Model Training in ML and software architecture.

But, Before starting training, you need the right equipment. Just as a cricketer needs a ball, you need to know the function you want to minimize (the cost function), its derivatives, and the current inputs, weight, and bias. The goal is to get the most accurate output, and in return, you get the values of the weight and bias with the smallest margin of error.

Gradient Descent is a fundamental algorithm in many machine-learning models. Its purpose is to find the minimum of the cost function, representing the lowest point or deepest valley. The cost function helps identify errors in the predictions of a machine learning model.

Using calculus, you can find the slope of a function, which is the derivative of the function concerning a value. Knowing the slope for each weight guides you toward the lowest point in the valley. The Learning Rate, a hyper-parameter, determines how much you adjust each weight during the iteration process. It involves trial and error, often improved by providing the neural network with more datasets. A well-functioning Gradient Descent algorithm should decrease the cost function with each iteration, and when it can’t decrease further, it is considered converged.

There are different types of gradient descents.

Batch gradient descent

It calculates the error but updates the model only after evaluating the entire dataset. It is computationally efficient but may not always achieve the most accurate results.

Stochastic gradient descent

It updates the model after every training example, showing detailed improvement until convergence.

Mini-batch gradient descent

It is a deep learning technique that combines batch and stochastic gradient descent. The dataset is separated into small groups and analyzed separately.

Backpropagation algorithm in machine learning?

Backpropagation is a type of learning in machine learning. It falls under supervised learning, where we already know the correct output for each input. This helps calculate the loss function gradient, showing how the expected output differs from the actual output. In supervised learning, we use a training data set with clearly labeled data and specified desired outputs.

The pseudocode in the backpropagation algorithm?

The backpropagation algorithm pseudocode serves as a basic blueprint for developers and researchers to guide the backpropagation process. It provides high-level instructions, including code snippets for essential tasks. While the overview covers the basics, the actual implementation is usually more intricate. The pseudocode outlines sequential steps, including core components of the backpropagation process. It can be written in common programming languages like Python.

Conclusion

Backpropagation, also known as backward propagation, is an important phase in neural networks’ training. It calculates gradients of the cost function concerning learnable parameters. It’s a significant topic in Artificial Neural Networks (ANN). Thanks for reading so far, I hope you found the article informative.

Background

Retrieval Augmented Generation(RAG) is an approach for enhancing existing LLMs with external knowledge sources, to provide more relevant and contextual answers. In a RAG, the retrieval component fetches additional information that grounds the response to specific sources and the information is then fed to the LLM prompt to ground the response from the LLM(the augmentation phase). Relative to other techniques (such as fine tuning), RAG is cheaper. It also has the advantage of reducing hallucinations by providing the LLM with additional context – making RAG a popular approach today for LLM tasks such as recommendations, text extraction, sentiment analysis etc.

If we break this idea down further, based on the user intent, we typically query a vector database. Vector databases use a continuous vector space to capture the relationship between two concepts using a proximity based search. Vector databases find and retrieve data based on the similarity of data points, represented as vectors in a multi-dimensional space.

An overview of vector databases

In a vector database, data—whether it be text, images, audio, or any other type of information—is transformed into vectors. A vector is a numeric representation of data in a high-dimensional space. Each dimension corresponds to a feature of the data, and the value in each dimension reflects the intensity or presence of that feature. Proximity-based searches in vector databases involve querying these databases by using another vector and searching for vectors that are “close” to it in the vector space. The proximity between vectors is often determined by distance metrics such as Euclidean distance, cosine similarity, or Manhattan distance.

When you perform a search in a vector database, you provide a query that the system converts into a vector. The database then calculates the distance or similarity between this query vector and the vectors already stored in the database. Those vectors that are closest to the query vector (according to the chosen metric) are considered the most relevant results.

The use of proximity-based searches in vector databases is particularly powerful for tasks like Recommendation Systems, Information Retrieval and Anomaly detection etc

This approach enables systems to operate more intuitively and respond more effectively to user queries by understanding the context and deeper meanings within the data, rather than relying solely on surface-level matches.

However, vector databases have some limitations based on proximity searches for instance data quality, the ability to handle dynamic knowledge and transparency.

Limitations of RAG

Depending on the size of the document, there are three broad classes of RAG: if the document is small, it can be accessed in context; if the document is large (or there are multiple documents), smaller chunks are generated at the time of the query and these chunks are indexed and accessed to respond to the query.

Despite its success, RAG has some shortcomings.

There are two main metrics to measure the performance of RAGs, Perplexity and Hallucination,

Perplexity represents the number of equally-likely next-word alternatives available during text generation. I.e. the degree to which a language model is “perplexed” by its choices.Hallucination is a statement made by AI that is untrue or imagined.

While RAG helps to reduce hallucination, it does not eliminate it. If you have a small, concise document you can reduce perplexity (because the LLM has few options) and reduce hallucination (if you ask it only about whats in the document). The flipside is of course that a single, small document lends to a trivial application. For more complex applications, you need a way to provide more context.

For example, consider the word ‘bark’ – we could have at least two different ciontexts

Tree context: “The rough bark of the oak tree protected it from the cold.”

Dog context: “The neighbor’s dog will bark loudly whenever someone passes by their house.”

One way to provide more context is to combine the RAG with a knowledge graph (a GRAPHRAG)

In a knowledge graph, these words would be connected to their relevant contexts and meanings. For example, “bark” would have connections to nodes representing both “tree” and “dog”. Additional connections could indicate common actions (e.g., “protect” for tree bark, “make noise” for dog bark) or attributes (e.g., “rough” for tree bark, “loud” for dog bark). This structured information allows a language model to select the appropriate meaning based on other words in the sentence or the overall topic of the conversation.

In next sections, we will see the limitations of RAG and how GRAPHRAG addresses these limitations

Image source:

https://medium.com/neo4j/enhancing-the-accuracy-of-rag-applications-with-knowledge-graphs-ad5e2ffab663

HCLSoftware announced that it has been selected by the State Bank of India (SBI) for its MarTech solution as part of SBI’s digital transformation programme.

As part of the five-year agreement, HCLSoftware will deploy the HCL Unica platform to enable SBI to digitally transform its customer interaction framework and provide hyper-personalized communication across the bank’s diverse digital marketing channels, while adhering to the Digital Personal Data Protection Act (DPDPA) and other stringent security requirements.

HCL Unica, with its advanced Customer Data Platform, AI capabilities and comprehensive campaign management tools, would leverage real-time data to significantly improve SBI’s ability to engage with its customers. It will help facilitate complex, multi-channel digital marketing campaigns, enhancing customer engagement precision and relevance.

“The partnership underscores the strength of the innovative capabilities of HCL Software to deliver digital transformation at scale. We are proud that HCL Unica would enable one of the largest banking transformations in the world and help SBI deliver superior customer engagement and experience,” said Rajiv Shesh, Chief Revenue Officer, HCLSoftware.

HCL Unica’s powerful Customer Data Platform will organize and aggregate SBI’s customer data from various touchpoints, creating a unified view that facilitates deeper insights and targeted marketing initiatives.

The post SBI to Leverage HCL Unica to Digitally Transform Customer Engagement appeared first on Analytics India Magazine.

The advancements in large language models have significantly accelerated the development of natural language processing, or NLP. The introduction of the transformer framework proved to be a milestone, facilitating the development of a new wave of language models, including OPT and BERT, which exhibit profound linguistic understanding. Furthermore, the inception of GPT, or Generative Pre-trained Transformer models, introduced a new paradigm with autoregressive modeling and established a robust method for language prediction and generation. The advent of language models like GPT-4, ChatGPT, Mixtral, LLaMA, and others has further fueled rapid evolution, with each model demonstrating enhanced performance in tasks involving complex language processing. Among existing methods, instruction tuning has emerged as a key technique for refining the output of pre-trained large language models, and the integration of these models with specific tools for visual tasks has highlighted their adaptability and opened doors for future applications. These extend far beyond the traditional text-based processing of LLMs to include multimodal interactions.

Furthermore, the convergence of natural language processing and computer vision models has given rise to VLMs, or Vision Language Models, which combine linguistic and vision models to achieve cross-modal comprehension and reasoning capabilities. The integration and advent of visual and linguistic models have played a crucial role in advancing tasks that require both language processing and visual understanding. The emergence of revolutionary models like CLIP has further bridged the gap between vision tasks and language models, demonstrating the feasibility and practicality of cross-modal applications. More recent frameworks like LLaMA and BLIP leverage tailored instruction data to devise efficient strategies that demonstrate the potent capabilities of the model. Additionally, combining large language models with image outputs is the focus of recent multimodal research, with recent methods being able to bypass direct generation by utilizing the image retrieval approach to produce image outputs and interleaved texts.

With that being said, and despite the rapid advancements in vision language models facilitating basic reasoning and visual dialogue, there still exists a significant performance gap between advanced models like GPT-4, and vision language models. Mini-Gemini is an attempt to narrow the gap that exists between vision language models and more advanced models by mining the potential of VLMs for better performance from three aspects: VLM-guided generation, high-quality data, and high-resolution visual tokens. To enhance visual tokens, the Mini-Gemini framework proposes to utilize an additional visual encoder for high-resolution refinement without increasing the count of visual tokens. The Mini-Gemini framework further constructs a high-quality dataset in an attempt to promote precise comprehension of images and reasoning-based generation. Overall, the Mini-Gemini framework attempts to mine the potential of vision language models, and aims to empower existing frameworks with image reasoning, understanding, and generative capabilities simultaneously. This article aims to cover the Mini-Gemini framework in depth, and we explore the mechanism, the methodology, the architecture of the framework along with its comparison with state of the art frameworks. So let’s get started.

Mini-Gemini: Accelerating Multi-Modality VLMs

Over the years, large language models have evolved, and they now boast of remarkable multi-modal capabilities, and are becoming an essential part of current vision language models. However, there exists a gap between the multi-modal performance of large language models and vision language models with recent research looking for ways to combine vision with large language models using images and videos. For vision tasks itself, image resolution is a crucial element to explicitly despite the surrounding environment with minimal visual hallucinations. To bridge the gap, researchers are developing models to improve the visual understanding in current vision language models, and two of the most common approaches are: increasing the resolution, and increasing the number of visual tokens. Although increasing the number of visual tokens with higher resolution images does enhance the visual understanding, the boost is often accompanied with increased computational requirements and associated costs especially when processing multiple images. Furthermore, the capabilities of existing models, quality of existing data, and applicability remains inadequate for an accelerated development process, leaving researchers with the question, “how to accelerate the development of vision language models with acceptable costs”?

The Mini-Gemini framework is an attempt to answer the question as it attempts to explore the potential of vision language models from three aspects: VLM-guided generation or expanded applications, high-quality data, and high-resolution visual tokens. First, the Mini-Gemini framework implements a ConvNet architecture to generate higher-resolution candidates efficiently, enhancing visual details while maintaining the visual token counts for the large language model. The Mini-Gemini framework amalgamates publicly available high-quality datasets in an attempt to enhance the quality of the data, and integrates these enhancements with state of the art generative and large language models with an attempt to enhance the performance of the VLMs, and improve the user experience. The multifaceted strategy implemented by the Mini-Gemini framework enables it to explore hidden capabilities of vision language models, and achieves significant advancements with evident resource constraints.

In general, the Mini-Gemini framework employs an any to any paradigm since it is capable of handling both text and images as input and output. In particular, the Mini-Gemini framework introduces an efficient pipeline for enhancing visual tokens for input images, and features a dual-encoder system comprising of twin encoders: the first encoder is for high-resolution images, while the second encoder is for low-quality visual embedding. During inference, the encoders work in an attention mechanism, where the low-resolution encoder generates visual queries, while the high-resolution encoder provides key and values for reference. To augment the data quality, the Mini-Gemini framework collects and produces more data based on public resources, including task-oriented instructions, generation-related data, and high-resolution responses, with the increased amount and enhanced quality improving the overall performance and capabilities of the model. Furthermore, the Mini-Gemini framework supports concurrent text and image generation as a result of the integration of the vision language model with advanced generative models.

Mini-Gemini : Methodology and Architecture

At its core, the Mini-Gemini framework is conceptually simple, and comprises three components.

1. The framework employs dual vision encoders to provide low-resolution visual embeddings and high resolution candidates.
2. The framework proposes to implement patch info mining to conduct mining at patch level between low-resolution visual queries, and high-resolution regions.
3. The Mini-Gemini framework utilizes a large language model to marry text with images for both generation and comprehension simultaneously.

Dual-Vision Encoders

The Mini-Gemini framework can process both text and image inputs, with the option to handle them either individually or in a combination. As demonstrated in the following image, the Mini-Gemini framework starts the process by employing bilinear interpolation to generate a low-resolution image from its corresponding high-resolution image.

The framework then processes these images and encodes them into a multi-grid visual embedding in two parallel image flows. More specifically, the Mini-Gemini framework maintains the traditional pipeline for low-resolution flows and employs a CLIP-pretrained Visual Transformer to encode the visual embeddings, facilitating the model to preserve the long-range relation between visual patches for subsequent interactions in large language models. For the high-resolution flows, the Mini-Gemini framework adopts the CNN or Convolution Neural Networks based encoder for adaptive and efficient high resolution image processing.

Patch Info Mining

With the dual vision encoders generating the LR embeddings and HR features, the Mini-Gemini framework proposes to implement patch info mining with the aim of extending the potential of vision language models with enhanced visual tokens. In order to maintain the number of visual tokens for efficiency in large language models, the Mini-Gemini framework takes the low-resolution visual embeddings as the query, and aims to retrieve relevant visual cues from the HR feature candidates, with the framework taking the HR feature map as the key and value.

As demonstrated in the above image, the formula encapsulates the process of refining and synthesizing visual cues, which leads to the generation of advanced visual tokens for the subsequent large language model processing. The process ensures that the framework is able to confine the mining for each query to its corresponding sub region in the HR feature map with the pixel-wise feature count, resulting in enhanced efficiency. Owing to this design, the Mini-Gemini framework is able to extract the HR feature details without enhancing the count of visual tokens, and maintains a balance between computational feasibility and richness of detail.

Text and Image Generation

The Mini-Gemini framework concatenates the visual tokens and input text tokens as the input to the large language models for auto-regressive generation. Unlike traditional vision language models, the Mini-Gemini framework supports text-only as well as text-image generation as input and output, i.e. any to any inference, and it is the result of this outstanding image-text understanding and reasoning capabilities, the Mini-Gemini is able to generate high quality images. Unlike recent works that focus on the domain gap between text embeddings of the generation models and large language models, the Mini-Gemini framework attempts to optimize the gap in the domain of language prompts by translating user instructions into high quality prompts that produce context relevant images in latent diffusion models. Furthermore, for a better understanding of instruction finetuning, and cross modality alignment, the Mini-Gemini framework collects samples from publicly available high quality datasets, and uses the GPT-4 turbo framework to further construct a 13K instruction following dataset to support image generation.

Mini-Gemini : Experiments and Results

To evaluate its performance, the Mini-Gemini framework is instantiated with the pre-trained ConvNext-L framework for the HR vision encoder, and with a CLIP-pre-trained Vision Transformer for the LR vision encoder. To ensure training efficiency, the Mini-Gemini framework keeps the two vision encoders fixed, and optimizes the projectors of patch info mining in all stages, and optimizes the large language model during the instruction tuning stage itself.

The following table compares the performance of the Mini-Gemini framework against state of the art models across different settings, and also takes in consideration private models. As it can be observed, the Mini-Gemini outperforms existing frameworks across a wide range of LLMs consistently at normal resolution, and demonstrates superior performance when configured with the Gemma-2B in the category of efficient models. Furthermore, when larger large language models are employed, the scalability of the Mini-Gemini framework is evident.

To evaluate its performance on high resolution and extended visual tokens, the experiments are performed with an input size of 672 for the LR vision encoder, and 1536 for the visual encoder. As mentioned earlier, the main purpose of the HR visual encoder is to offer high-resolution candidate information. As it can be observed, the Mini-Gemini framework delivers superior performance when compared against state of the art frameworks.

Furthermore, to assess the visual comprehension prowess of the Mini-Gemini framework in real-world settings, developers apply the model to a variety of reasoning and understanding tasks as demonstrated in the following image. As it can be observed, the Mini-Gemini framework is able to solve a wide array of complex tasks thanks to the implementation of patch info mining, and high-quality data. But what’s more impressive is the fact that the Mini-Gemini framework demonstrates a keen addition to detail that extends beyond mere recognition prowess, and describes intricate elements intricately.

The following figure provides a comprehensive evaluation of the generative abilities of the Mini-Gemini framework.

When compared against recent models like ChatIllusion and AnyGPT, the Mini-Gemini framework demonstrates stronger multi-modal understanding abilities, allowing it to generate text to image captions that align with the input instructions better, and results in image to text answers with stronger conceptual similarity. What’s more impressive is the fact that the Mini-Gemini framework demonstrates remarkable proficiency in generating high-quality content using multi-model human instructions only with text training data, a capability that illustrates Mini-Gemini’s robust semantic interpretation and image-text alignment skills.

Final Thoughts

In this article we have talked about Mini-Gemini, a potent and streamlined framework for multi-modality vision language models. The primary aim of the Mini-Gemini framework is to harness the latent capabilities of vision language models using high quality data, strategic design of the framework, and an expanded functional scope. Mini-Gemini is an attempt to narrow the gap that exists between vision language models and more advanced models by mining the potential of VLMs for better performance from three aspects: VLM-guided generation, high-quality data, and high-resolution visual tokens. To enhance visual tokens, the Mini-Gemini framework proposes to utilize an additional visual encoder for high-resolution refinement without increasing the count of visual tokens. The Mini-Gemini framework further constructs a high-quality dataset in an attempt to promote precise comprehension of images and reasoning-based generation. Overall, the Mini-Gemini framework attempts to mine the potential of vision language models, and aims to empower existing frameworks with image reasoning, understanding, and generative capabilities simultaneously.

Indian SaaS giant Zoho, the first bootstrapped Saas company to reach 100M users, is the preferred choice for tech companies in Africa. The enterprise suite of products from Zoho has been adopted over similar workplace products such as Google Workspace with cost playing a major role in this decision.

Zoho is not only a cost-effective option for tech companies, but the company has allowed the acceptance of regional currencies, which has helped with higher adoption rates. By allowing customers to forgo limitations associated with dollar transactions, transaction processes have become smoother.

Zoho co-founder and CEO, Sridhar Vembu, highlighted the acceptance of local currencies in Latin America too.

Source: X

Expansion in Africa

The company has also recruited local talent to help have a strong foothold in the region. As per a report, seven startups in Nigeria, Kenya and South Africa have adopted Zoho’s products over the past year.

Starting operations in 2019, with two salespersons in South Africa and Nigeria, the company now has 60 employees across Africa.

“A lot of companies are adopting digital, either for the first time or they’re on that path of making their businesses more efficient using technology,” said Praval Singh, vice president of marketing and customer experience at Zoho.

Global and Rural Growth

The Middle East and Africa region have been the fastest growing market for Zoho with revenue contribution of close to 10% of the total global revenue. The company is also looking to actively expand in these regions by doubling hirings.

The company is also actively expanding in Tier 2 and 3 cities in India, and recently opened an R&D facility in Kottarakara, a small town in Kerala. The first rural R&D centre was set up in another small town Tenkasi, in Tamil Nadu.

Keeping the expansion plans alive, the company’s IT enterprise wing, ManageEngine, recently confirmed that the company has invested $10M in NVIDIA, Intel, and AMD GPUs.

Zoho now serves over 700k businesses across 150 countries.

The post African Tech Companies Prefer Zoho Enterprise over Google Workspace appeared first on Analytics India Magazine.

Image by Author

I once spoke with a guy who bragged that, armed only with some free LinkedIn courses and an outdated college Intro to SQL course, he’d managed to bag a six-figure job in data science. Nowadays, most people struggling to get a good data science job will agree that’s unlikely to happen. Does that mean the data science job category is a popped bubble – or worse, that it hasn’t yet burst, but is about to?

In short, no. What’s happened is that data science used to be an undersaturated field, easy to get into if you used the right keywords on your resume. Nowadays, employers are a little more discerning and often have specific skill sets in mind that they’re looking for.

The bootcamps, free courses, and ‘Hello World’ projects don’t cut it anymore. You need to prove specific expertise and nail your data science interview, not just drop buzzwords. Not only that, but the shine of “data scientist” has worn off a little. For a long time, it was the sexiest job out there. Now? Other fields, like AI and machine learning, are just a bit sexier.

That all being said, there are still more openings in data science than there are applicants, and reliable indicators say the field is growing, not shrinking.

Not convinced? Let’s look at the data.

The Big Picture

Over the course of this article, I’ll drill down into multiple graphs, charts, figures, and percentages. But let’s start with just one percentage from one outstandingly reputable source: The Bureau of Labor Statistics.

The BLS predicts that there will be a 35 percent change in employment from 2022 to 2032 for data scientists. In short, in 2032, there will be about a third more jobs in data science than there were in 2022. For comparison, the average growth rate for all jobs is 3 percent. Keep that number in mind as you go through the rest of this article.

The BLS does not think that data science is a bubble waiting to burst.

The Layoffs

Now we can start getting into a bit of the nitty gritty. The first signs people point to as signs of a popped or impending bubble pop are the mass layoffs in data science.

It’s true that the numbers don’t look good. Starting in 2022 and continuing through 2024, the tech sector in general experienced 430k layoffs. It’s difficult to tease out data science-specific data from those numbers, but the best guesses are that around 30 percent of those were in data science and engineering.

Source: https://techcrunch.com/2024/04/05/tech-layoffs-2023-list/

However, that’s not a burst bubble of data science. It’s a little smaller in scope than that – it’s a pandemic bubble popping. In 2020, as more people stayed home, profits rose, and money was cheap, FAANG and FAANG-adjacent companies scooped up record numbers of tech workers, only to lay many of them off just a few years later.

If you zoom out and look at the broader picture of hirings and layoffs, you’ll be able to see that the post-pandemic slump is a dip in an overall rising line, which is even now beginning to recover:

Source: https://www.statista.com/

You can clearly see the huge dip in tech layoffs during 2020 as the market tightened, and then the huge spike starting in Q1 of 2022 as layoffs began. Now, in 2024, the number of layoffs is smaller than in 2023.

The Job Openings

Another scary stat often touted is that FAANG companies shuttered their job openings by 90% or more. Again, this is most in reaction to a widely high number of job openings during the pandemic.

That being said, job openings in the tech sector are still lower than they were pre-pandemic. Below, you can see an adjusted chart showing demand for tech jobs relative to February 2020. It’s clear to see that the tech sector took a blow it’s not recovering from any time soon.

Source: https://www.hiringlab.org/2024/02/20/labor-market-update-tech-jobs-below-pre-pandemic-levels/

However, let’s look a little closer at some real numbers. Looking at the chart below, while job openings are indubitably down from their 2022 peak, the overall number of openings is actually increasing – up 32.4% from the lowest point.

Source: https://www.trueup.io/job-trend

The Narrative

If you look at any labor and news reports online, you’ll see there’s a bit of an anti-remote, anti-tech backlash happening at the moment. Meta, Google, and other FAANG companies, spooked by the bargaining power that employees enjoyed during the pandemic heights, are now pushing for return-to-office mandates (data science jobs and other tech jobs are often remote) and laying off large quantities of employees somewhat unnecessarily, judging by their revenue and profit reports.

Just to give one example, Google’s parent company Alphabet laid off over 12,000 employees over the course of 2023 despite growth across its ad, cloud, and services divisions.

This is just one facet with which to examine the data, but part of the reason companies are doing these layoffs is more to do with making the board happy rather than any decreased need for data scientists.

The Demand

I find that people believing we’re in a data science bubble are most often those who don’t really know what data scientists do. Think of that BLS stat and ask yourself: why does this well-informed government agency believe that there’s strong growth in this sector?

It’s because the need for data scientists cannot go away. While the names might be changed – AI expert or ML Cloud Specialist rather than Data Scientist – the skills and tasks that data scientists perform can’t be outsourced, dropped, decreased, or automated.

For example, predictive models are essential for businesses to forecast sales, predict customer behavior, manage inventory, and anticipate market trends. This enables companies to make informed decisions, plan strategically for the future, and maintain competitive advantages.

In the financial sector, data science plays a crucial role in identifying suspicious activities, preventing fraud, and mitigating risks. Advanced algorithms analyze transaction patterns to detect anomalies that may indicate fraud, helping protect businesses and consumers alike.

NLP enables machines to understand and interpret human language, powering applications like chatbots, sentiment analysis, and language translation services. This is critical for improving customer service, analyzing social media sentiment, and facilitating global communication.

I could list dozens more examples demonstrating that data science is not a fad, and data scientists will always be in demand.

Why Does It Feel Like We're In A Bubble?

Revisiting my anecdote from earlier, part of the reason it feels like we’re in a bubble that is either popping or about to pop is the perception of data science as a career.

Back in 2011, Harvard Business Review famously called it the sexiest job of the decade. In the intervening years, companies hired more “data scientists” than they knew what to do with, often unsure about what data scientists actually did.

Now, a decade and a half later, the field is a little wiser. Employers understand that data science is a broad field, and are more interested in hiring machine learning specialists, data pipeline engineers, cloud engineers, statisticians, and other specialties that broadly fall under the data science hat but are more specialized.

This also helps explain why this idea of walking into six figure job straight out of a bachelor's degree used to be the case — since employers didn't know better — but now is impossible to do. The lack of “easy” data science jobs makes it feel like the market is tighter. It's not; data shows job openings are still high and demand is still greater than the graduates coming out with appropriate degrees. But employers are more discerning and unwilling to take a chance on untried college grads with no demonstrated experience.

The Need For Data Science Has Not Decreased Or Been Replaced

Finally, you can take a look at the tasks that data scientists do and ask yourself what companies would do without those tasks getting done.

If you don't know much about data science, you might guess that companies can simply “automate” this work, or even go without. But if you know anything about the actual tasks data scientists do, you understand that the job is, currently, irreplaceable.

Think of how things were in the 2010s: that guy I talked about, with just a basic understanding of data tools, catapulted himself into a lucrative career. Things aren’t like that anymore, but this recalibration isn’t a sign of a bursting bubble as some believe. Instead, it’s the field of data science maturing. The entry-level data science field may be oversaturated, but for those with specialized skills, deep knowledge, and practical experience, the field is wide open.

Furthermore, this narrative of a “bubble” is fueled by a misunderstanding of what a bubble actually represents. A bubble occurs when the value of something (in this case, a career sector) is driven by speculation rather than actual intrinsic worth. However, as we covered, the value proposition of data science is tangible and measurable. Companies need data scientists, plain and simple. There’s no speculation there.

There’s also a lot of media sensationalism surrounding the layoffs in big tech. While these layoffs are significant, they reflect broader market forces rather than a fundamental flaw in the data science discipline. Don’t get caught up in the headlines.

Finally, it’s also worth noting that the perception of a bubble may stem from how data science itself is changing. As the field matures, the differentiation between roles becomes more pronounced. Job titles like data engineering, data analysis, business intelligence, machine learning engineering, and data science are more specific, and require a more niche skill set. This evolution can make the data science job market appear more volatile than it is, but in reality, companies just have a better understanding of their data science needs and can recruit for their specialities.

Final Thoughts

If you want a job in data science, go for it. There’s very little chance we’re actually in a bubble. The best thing you can do is, as I’ve indicated, pick your specialty and develop your skills in that area. Data science is a broad field, spilling over into different industries, languages, job titles, responsibilities, and seniorities. Select a specialty, train the skills, prep for the interview, and secure the job.

Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Nate writes on the latest trends in the career market, gives interview advice, shares data science projects, and covers everything SQL.

More On This Topic

• The Generative AI Bubble Will Burst Soon
• Dot com to Dot AI: The New Tech Bubble?
• Stop Learning Data Science to Find Purpose and Find Purpose to…
• Data Science Minimum: 10 Essential Skills You Need to Know to Start…
• KDnuggets™ News 22:n06, Feb 9: Data Science Programming…
• Data Science Definition Humor: A Collection of Quirky Quotes…

Large Language Models (LLMs) have revolutionized the field of natural language processing (NLP) by demonstrating remarkable capabilities in generating human-like text, answering questions, and assisting with a wide range of language-related tasks. At the core of these powerful models lies the decoder-only transformer architecture, a variant of the original transformer architecture proposed in the seminal paper “Attention is All You Need” by Vaswani et al.

In this comprehensive guide, we will explore the inner workings of decoder-based LLMs, delving into the fundamental building blocks, architectural innovations, and implementation details that have propelled these models to the forefront of NLP research and applications.

The Transformer Architecture: A Refresher

Before diving into the specifics of decoder-based LLMs, it's essential to revisit the transformer architecture, the foundation upon which these models are built. The transformer introduced a novel approach to sequence modeling, relying solely on attention mechanisms to capture long-range dependencies in the data, without the need for recurrent or convolutional layers.

The original transformer architecture consists of two main components: an encoder and a decoder. The encoder processes the input sequence and generates a contextualized representation, which is then consumed by the decoder to produce the output sequence. This architecture was initially designed for machine translation tasks, where the encoder processes the input sentence in the source language, and the decoder generates the corresponding sentence in the target language.

Self-Attention: The Key to Transformer's Success

At the heart of the transformer lies the self-attention mechanism, a powerful technique that allows the model to weigh and aggregate information from different positions in the input sequence. Unlike traditional sequence models, which process input tokens sequentially, self-attention enables the model to capture dependencies between any pair of tokens, regardless of their position in the sequence.

The self-attention operation can be broken down into three main steps:

1. Query, Key, and Value Projections: The input sequence is projected into three separate representations: queries (Q), keys (K), and values (V). These projections are obtained by multiplying the input with learned weight matrices.
2. Attention Score Computation: For each position in the input sequence, attention scores are computed by taking the dot product between the corresponding query vector and all key vectors. These scores represent the relevance of each position to the current position being processed.
3. Weighted Sum of Values: The attention scores are normalized using a softmax function, and the resulting attention weights are used to compute a weighted sum of the value vectors, producing the output representation for the current position.

Multi-head attention, a variant of the self-attention mechanism, allows the model to capture different types of relationships by computing attention scores across multiple “heads” in parallel, each with its own set of query, key, and value projections.

Architectural Variants and Configurations

While the core principles of decoder-based LLMs remain consistent, researchers have explored various architectural variants and configurations to improve performance, efficiency, and generalization capabilities. In this section, we'll delve into the different architectural choices and their implications.

Architecture Types

Decoder-based LLMs can be broadly classified into three main types: encoder-decoder, causal decoder, and prefix decoder. Each architecture type exhibits distinct attention patterns, as illustrated in Figure 1.

Encoder-Decoder Architecture

Based on the vanilla Transformer model, the encoder-decoder architecture consists of two stacks: an encoder and a decoder. The encoder uses stacked multi-head self-attention layers to encode the input sequence and generate latent representations. The decoder then performs cross-attention on these representations to generate the target sequence. While effective in various NLP tasks, few LLMs, such as Flan-T5, adopt this architecture.

Causal Decoder Architecture

The causal decoder architecture incorporates a unidirectional attention mask, allowing each input token to attend only to past tokens and itself. Both input and output tokens are processed within the same decoder. Notable models like GPT-1, GPT-2, and GPT-3 are built on this architecture, with GPT-3 showcasing remarkable in-context learning capabilities. Many LLMs, including OPT, BLOOM, and Gopher, have widely adopted causal decoders.

Prefix Decoder Architecture

Also known as the non-causal decoder, the prefix decoder architecture modifies the masking mechanism of causal decoders to enable bidirectional attention over prefix tokens and unidirectional attention on generated tokens. Like the encoder-decoder architecture, prefix decoders can encode the prefix sequence bidirectionally and predict output tokens autoregressively using shared parameters. LLMs based on prefix decoders include GLM130B and U-PaLM.

All three architecture types can be extended using the mixture-of-experts (MoE) scaling technique, which sparsely activates a subset of neural network weights for each input. This approach has been employed in models like Switch Transformer and GLaM, with increasing the number of experts or total parameter size showing significant performance improvements.

Decoder-Only Transformer: Embracing the Autoregressive Nature

While the original transformer architecture was designed for sequence-to-sequence tasks like machine translation, many NLP tasks, such as language modeling and text generation, can be framed as autoregressive problems, where the model generates one token at a time, conditioned on the previously generated tokens.

Enter the decoder-only transformer, a simplified variant of the transformer architecture that retains only the decoder component. This architecture is particularly well-suited for autoregressive tasks, as it generates output tokens one by one, leveraging the previously generated tokens as input context.

The key difference between the decoder-only transformer and the original transformer decoder lies in the self-attention mechanism. In the decoder-only setting, the self-attention operation is modified to prevent the model from attending to future tokens, a property known as causality. This is achieved through a technique called “masked self-attention,” where attention scores corresponding to future positions are set to negative infinity, effectively masking them out during the softmax normalization step.

Architectural Components of Decoder-Based LLMs

While the core principles of self-attention and masked self-attention remain the same, modern decoder-based LLMs have introduced several architectural innovations to improve performance, efficiency, and generalization capabilities. Let's explore some of the key components and techniques employed in state-of-the-art LLMs.

Input Representation

Before processing the input sequence, decoder-based LLMs employ tokenization and embedding techniques to convert the raw text into a numerical representation suitable for the model.

Tokenization: The tokenization process converts the input text into a sequence of tokens, which can be words, subwords, or even individual characters, depending on the tokenization strategy employed. Popular tokenization techniques for LLMs include Byte-Pair Encoding (BPE), SentencePiece, and WordPiece. These methods aim to strike a balance between vocabulary size and representation granularity, allowing the model to handle rare or out-of-vocabulary words effectively.

Token Embeddings: After tokenization, each token is mapped to a dense vector representation called a token embedding. These embeddings are learned during the training process and capture semantic and syntactic relationships between tokens.

Positional Embeddings: Transformer models process the entire input sequence simultaneously, lacking the inherent notion of token positions present in recurrent models. To incorporate positional information, positional embeddings are added to the token embeddings, allowing the model to distinguish between tokens based on their positions in the sequence. Early LLMs used fixed positional embeddings based on sinusoidal functions, while more recent models have explored learnable positional embeddings or alternative positional encoding techniques like rotary positional embeddings.

Multi-Head Attention Blocks

The core building blocks of decoder-based LLMs are multi-head attention layers, which perform the masked self-attention operation described earlier. These layers are stacked multiple times, with each layer attending to the output of the previous layer, allowing the model to capture increasingly complex dependencies and representations.

Attention Heads: Each multi-head attention layer consists of multiple “attention heads,” each with its own set of query, key, and value projections. This allows the model to attend to different aspects of the input simultaneously, capturing diverse relationships and patterns.

Residual Connections and Layer Normalization: To facilitate the training of deep networks and mitigate the vanishing gradient problem, decoder-based LLMs employ residual connections and layer normalization techniques. Residual connections add the input of a layer to its output, allowing gradients to flow more easily during backpropagation. Layer normalization helps to stabilize the activations and gradients, further improving training stability and performance.

Feed-Forward Layers

In addition to multi-head attention layers, decoder-based LLMs incorporate feed-forward layers, which apply a simple feed-forward neural network to each position in the sequence. These layers introduce non-linearities and enable the model to learn more complex representations.

Activation Functions: The choice of activation function in the feed-forward layers can significantly impact the model's performance. While earlier LLMs relied on the widely-used ReLU activation, more recent models have adopted more sophisticated activation functions like the Gaussian Error Linear Unit (GELU) or the SwiGLU activation, which have shown improved performance.

Sparse Attention and Efficient Transformers

While the self-attention mechanism is powerful, it comes with a quadratic computational complexity with respect to the sequence length, making it computationally expensive for long sequences. To address this challenge, several techniques have been proposed to reduce the computational and memory requirements of self-attention, enabling efficient processing of longer sequences.

Sparse Attention: Sparse attention techniques, such as the one employed in the GPT-3 model, selectively attend to a subset of positions in the input sequence, rather than computing attention scores for all positions. This can significantly reduce the computational complexity while maintaining reasonable performance.

Sliding Window Attention: Introduced in the Mistral 7B model , sliding window attention (SWA) is a simple yet effective technique that restricts the attention span of each token to a fixed window size. This approach leverages the ability of transformer layers to transmit information across multiple layers, effectively increasing the attention span without the quadratic complexity of full self-attention.

Rolling Buffer Cache: To further reduce memory requirements, especially for long sequences, the Mistral 7B model employs a rolling buffer cache. This technique stores and reuses the computed key and value vectors for a fixed window size, avoiding redundant computations and minimizing memory usage.

Grouped Query Attention: Introduced in the LLaMA 2 model, grouped query attention (GQA) is a variant of the multi-query attention mechanism that divides attention heads into groups, each group sharing a common key and value matrix. This approach strikes a balance between the efficiency of multi-query attention and the performance of standard self-attention, providing improved inference times while maintaining high-quality results.

Model Size and Scaling

One of the defining characteristics of modern LLMs is their sheer scale, with the number of parameters ranging from billions to hundreds of billions. Increasing the model size has been a crucial factor in achieving state-of-the-art performance, as larger models can capture more complex patterns and relationships in the data.

Parameter Count: The number of parameters in a decoder-based LLM is primarily determined by the embedding dimension (d_model), the number of attention heads (n_heads), the number of layers (n_layers), and the vocabulary size (vocab_size). For example, the GPT-3 model has 175 billion parameters, with d_model = 12288, n_heads = 96, n_layers = 96, and vocab_size = 50257.

Model Parallelism: Training and deploying such massive models require substantial computational resources and specialized hardware. To overcome this challenge, model parallelism techniques have been employed, where the model is split across multiple GPUs or TPUs, with each device responsible for a portion of the computations.

Mixture-of-Experts: Another approach to scaling LLMs is the mixture-of-experts (MoE) architecture, which combines multiple expert models, each specializing in a specific subset of the data or task. The Mixtral 8x7B model is an example of an MoE model that leverages the Mistral 7B as its base model, achieving superior performance while maintaining computational efficiency.

Inference and Text Generation

One of the primary use cases of decoder-based LLMs is text generation, where the model generates coherent and natural-sounding text based on a given prompt or context.

Autoregressive Decoding: During inference, decoder-based LLMs generate text in an autoregressive manner, predicting one token at a time based on the previously generated tokens and the input prompt. This process continues until a predetermined stopping criterion is met, such as reaching a maximum sequence length or generating an end-of-sequence token.

Sampling Strategies: To generate diverse and realistic text, various sampling strategies can be employed, such as top-k sampling, top-p sampling (also known as nucleus sampling), or temperature scaling. These techniques control the trade-off between diversity and coherence of the generated text by adjusting the probability distribution over the vocabulary.

Prompt Engineering: The quality and specificity of the input prompt can significantly impact the generated text. Prompt engineering, the art of crafting effective prompts, has emerged as a crucial aspect of leveraging LLMs for various tasks, enabling users to guide the model's generation process and achieve desired outputs.

Human-in-the-Loop Decoding: To further improve the quality and coherence of generated text, techniques like Reinforcement Learning from Human Feedback (RLHF) have been employed. In this approach, human raters provide feedback on the model's generated text, which is then used to fine-tune the model, effectively aligning it with human preferences and improving its outputs.

Advancements and Future Directions

The field of decoder-based LLMs is rapidly evolving, with new research and breakthroughs continuously pushing the boundaries of what these models can achieve. Here are some notable advancements and potential future directions:

Efficient Transformer Variants: While sparse attention and sliding window attention have made significant strides in improving the efficiency of decoder-based LLMs, researchers are actively exploring alternative transformer architectures and attention mechanisms to further reduce computational requirements while maintaining or improving performance.

Multimodal LLMs: Extending the capabilities of LLMs beyond text, multimodal models aim to integrate multiple modalities, such as images, audio, or video, into a single unified framework. This opens up exciting possibilities for applications like image captioning, visual question answering, and multimedia content generation.

Controllable Generation: Enabling fine-grained control over the generated text is a challenging but important direction for LLMs. Techniques like controlled text generation and prompt tuning aim to provide users with more granular control over various attributes of the generated text, such as style, tone, or specific content requirements.

Conclusion

Decoder-based LLMs have emerged as a transformative force in the field of natural language processing, pushing the boundaries of what is possible with language generation and understanding. From their humble beginnings as a simplified variant of the transformer architecture, these models have evolved into highly sophisticated and powerful systems, leveraging cutting-edge techniques and architectural innovations.

As we continue to explore and advance decoder-based LLMs, we can expect to witness even more remarkable achievements in language-related tasks, as well as the integration of these models into a wide range of applications and domains. However, it is crucial to address the ethical considerations, interpretability challenges, and potential biases that may arise from the widespread deployment of these powerful models.

By staying at the forefront of research, fostering open collaboration, and maintaining a strong commitment to responsible AI development, we can unlock the full potential of decoder-based LLMs while ensuring they are developed and utilized in a safe, ethical, and beneficial manner for society.