The Future of Serverless Inference for Large Language Models

Recent advances in large language models (LLMs) like GPT-4, PaLM have led to transformative capabilities in natural language tasks. LLMs are being incorporated into various applications such as chatbots, search engines, and programming assistants. However, serving LLMs at scale remains challenging due to their substantial GPU and memory requirements.

Approaches to overcome this generally fall into two main categories:

  1. Model Compression Techniques

These techniques aim to reduce the size of the model while maintaining accuracy. Common approaches include:

  • Pruning – Removing redundant or less important parameters from the model. This creates a sparse model with fewer parameters.
  • Quantization – Using lower precision numbers like int8 or bfloat16 to represent weights instead of fp32 or fp16. This reduces memory footprint.
  • Knowledge distillation – Training a smaller “student” model to mimic a large “teacher” model. The smaller model is then used for inference.
  1. Selective Execution

Rather than compressed models, these techniques selectively execute only parts of the model per inference:

  • Sparse activations – Skipping computation on zero activations.
  • Conditional computation – Executing only certain layers conditioned on the input.

On complementary side wrt to the software architect side; to enable faster deployment of LLMs researchers have proposed serverless inference systems. In serverless architectures, LLMs are hosted on shared GPU clusters and allocated dynamically based on demand. This allows efficient utilization of GPUs and reduces costs for developers. Prominent implementations include Amazon SageMaker, Microsoft Azure ML, and open-source options like KServe.

Despite the promise of serverless LLMs, existing systems exhibit high latency overheads that degrade user experience in interactive applications:

  1. Costly checkpoint downloads: LLMs have large memory footprints, often gigabytes to terabytes in size. Downloading checkpoints from remote storage is time-consuming, taking over 20 seconds even with optimized networks.
  2. Inefficient checkpoint loading: Even with local SSD storage, loading checkpoints into GPU memory takes tens of seconds due to factors like tensor deserialization and allocation. This adds significant delays beyond container startup time.

To address these issues, researchers at MIT CSAIL proposed ServerlessLLM, an innovative system that achieves low-latency serverless inference for LLMs. ServerlessLLM enhances locality by exploiting the abundant yet underutilized capacity and bandwidth in multi-tier server storage for LLM deployment.

Overview of LLM serverless inference systems

Overview of LLM serverless inference systems

Key Innovations in ServerlessLLM ServerlessLLM incorporates several novel designs to slash LLM loading times in serverless environments:

  1. Rapid checkpoint loading
  • Loading-optimized checkpoint format that enables fast sequential reading and efficient in-memory tensor addressing.
  • Multi-tier checkpoint loading pipeline that maximizes bandwidth utilization across network, SSDs, DRAM, and GPU memory through techniques like direct I/O, pinned memory transfer, and parallelism.
  1. Live migration for locality-driven inference
  • Token-based migration that only transmits essential prompt tokens over the network, avoiding slow snapshot transfer.
  • Two-phase migration that allows uninterrupted inference by asynchronously recomputing cache states on the destination server before transferring final tokens.
  1. Latency-optimized server allocation
  • Accurate models to estimate checkpoint loading times from each tier and migration times for a server.
  • Locality-aware scheduler that selects servers minimizing expected startup latency using the above models.

These optimizations allow ServerlessLLM to reduce LLM loading times by 4-8X and end-to-end startup times by over 25X compared to existing systems like PyTorch, TensorFlow, and KServe.

Let's dive deeper into how ServerlessLLM achieves these significant performance gains.

Accelerating Checkpoint Loading

The first major bottleneck addressed by ServerlessLLM is the high latency of loading LLM checkpoints from storage into GPU memory.

To enable rapid checkpoint loading, ServerlessLLM introduces:

  1. Loading-optimized checkpoint format

Standard checkpoints used by frameworks like PyTorch are designed for model training and debugging. But for serverless inference, checkpoints are read-only and accessed repeatedly.

To optimize for such read-intensive usage, ServerlessLLM converts checkpoints into a format with two key properties:

  • Sequential chunk-based reading: Tensors are grouped into per-GPU binary files, facilitating large sequential reads.
  • Efficient tensor addressing: An index maps tensor names to memory offsets, allowing direct in-memory restoration without deserialization.
  1. Multi-tier checkpoint loading pipeline

ServerlessLLM leverages the tiered architecture of GPU servers, with storage media like SSDs and networking connecting to GPUs via PCIe, NVMe, etc.

The system incorporates a multi-stage pipeline to maximize bandwidth utilization across all tiers:

  • In-memory data chunks are allocated using pinned memory for fast GPU transfer.
  • Direct I/O is used for efficient SSD reads without caching overheads.
  • Multiple threads read different storage chunks in parallel.
  • Inter-stage coordination occurs via asynchronous task queues.

Together, this enables saturating the bandwidth capacity of even the fastest tiers like NVMe RAID. Experiments reveal that ServerlessLLM achieves 6-8X faster loading than PyTorch/TensorFlow, reducing startup times for large LLMs from over a minute to under 10 seconds.

Locality-Driven LLM Inference via Live Migration

With accelerated loading, ServerlessLLM faces a new challenge – how to leverage pre-loaded checkpoints for locality without interrupting ongoing inferences on busy servers?

ServerlessLLM introduces a novel technique – live migration of LLM inference across GPU servers. This allows seamlessly transferring execution to servers with local checkpoints available.

Key enablers of live LLM migration:

  1. Token-based migration

Rather than snapshotting the entire model state, ServerlessLLM only migrates the minimal prompt tokens over the network. This transfers orders of magnitude less data than snapshots.

  1. Two-phase migration

Destination server asynchronously precomputes cache states from prompt tokens. Once ready, source server transfers final tokens before releasing resources. This prevents inference stalls.

Experiments reveal that token-based migration slashes migration times from tens of seconds to under a second even for long sequences. Live migration is crucial to prevent queuing delays when achieving locality-driven allocation.

Latency-Optimized Model Scheduling

To minimize end-to-end latency, ServerlessLLM enhances the scheduler to optimize server selection considering locality. This involves:

  1. Fine-grained loading time estimator

Models predict loading times from network, SSD caches, and memory for each server using metrics like queue delays, model sizes, and measured bandwidth.

  1. Accurate migration time predictor

The scheduler estimates migration times for servers using the number of prompt and output tokens. It tracks inference progress asynchronously to avoid overhead.

  1. Locality-aware allocation

For each inference request, the scheduler evaluates estimated loading and migration times across servers. It selects the server minimizing expected startup latency.

The scheduler also maintains server task queues and leverages a strongly consistent store for fault tolerance. Together, these innovations reduce scheduling overheads while maximizing locality benefits.

Evaluating ServerlessLLM Performance

Comprehensive experiments benchmark the end-to-end effectiveness of ServerlessLLM against existing systems using real-world models like OPT-175B and workloads modeled after Azure traces.

Key results:

  • Microbenchmarks: ServerlessLLM accelerates checkpoint loading by 3.6-8.2X over PyTorch/TensorFlow. It fully saturates storage bandwidth, even for cutting-edge NVMe RAID.
  • Scheduling: ServerlessLLM reduces allocation latency by 4-12X compared to random scheduling, highlighting benefits of locality-awareness. Live migration prevents queuing delays.
  • End-to-end serving: For large models like OPT-30B, ServerlessLLM improves 99th percentile latency by 28-200X over systems like KServe and Ray Serve. It also enhances resource efficiency.

These substantial gains demonstrate ServerlessLLM's ability to overcome bottlenecks in existing serverless implementations and unlock the power of LLMs for interactive services.

The optimizations introduced in ServerlessLLM, like multi-tier loading, live migration, and latency-driven scheduling, can help inform the design of future serverless architectures. The system's ability to slash loading and startup times unblocks the scalable deployment of large language models for practical applications.

Looking Ahead: Ongoing Challenges

While a significant leap forward, ServerlessLLM represents only the first step in optimizing serverless inference for massive LLMs. Several open problems remain, including:

  • Predicting real-time model demand to guide provisioning and pre-loading
  • Intelligently placing checkpoints across servers to maximize cache hits
  • Efficiently scaling scheduling algorithms to handle larger clusters
  • Ensuring fairness in resource allocation across models and developers
  • Generalizing innovations like live migration to other serverless workloads

Addressing these areas can help build on the promise of serverless LLMs and make their capabilities even more accessible. Beyond system-level optimizations, reducing the egregious carbon footprint and potential harms of large models also remains an urgent priority.

ServerlessLLM demonstrates that tremendous headroom exists for innovation in next-generation serverless architectures for AI workloads. As LLMs continue ballooning in size and popularity, solutions like ServerlessLLM that unlock their scalability will grow even more impactful. The confluence of systems and machine learning research can introduce new paradigms in serving, sharing, and scaling AI models safely and sustainably.

Tech Giants Pledge to Train Millions of Workers in AI

Tech Giants Pledge to Train Millions of Workers in AI January 26, 2024 by Alex Woodie

(Peshkova/Shutterstock)

Getting workers up to speed on the latest artificial intelligence technology has become a priority for many of the world’s top companies. That includes the tech giants, which recently have announced new AI training and education initiatives aimed at getting workers ready to build AI apps and use AI tools in the workplace. The best part is many of the training programs are free.

IBM in September unveiled an ambitious plan to train 2 million workers in AI by the end of 2026. IBM aims to deliver the training through universities as well as SkillsBuild, its free educational program that aims to enhance tech skills of high school and adult learners in underrepresented communities.

Anybody can sign up for SkillsBuild classes, which span more than 1,000 courses in 20 different languages. IBM currently offers nearly 30 hours of AI training spread across several courses on the SkillsBuild website, including courses on AI fundamentals, a how-to course for building a custom chatbot, and a course on ethics.

Big Blue also is working with universities around the world to build and expand AI curriculum. The company says it’s making its network of AI experts available to universities to help build the courses. It’s also providing colleges with lectures, “immersive skilling experiences,” and “self-directed AI learning paths” to help accelerate the education of students. Certificates will be available upon completion, IBM says.

In October, a Microsoft UK executive announced plans to train an additional 1 million people in AI by 2025. Speaking at the Microsoft Envision event, Clare Barclay, CEO of Microsoft UK, said:

Amazon plans to train 2 million people in AI by 2025 through its Skills Builder program (Image courtesy Amazon)

“By 2025 we will support one million people to gain the AI skills they need to start, or move into, a career in technology. And in doing so, we will open up the opportunities of AI to diverse talent across the UK.”

Microsoft UK plans to deliver the AI training through its Get On program, which it originally launched three years earlier.

Not to be outdone, Amazon in November announced plans to train 2 million people in AI by 2025. The ecommerce giant plans to deliver the training to both school-age students and adults through its AI Ready program.

Amazon currently is offering eight free courses on AI and generative AI topics on AI Ready, which is an extension of existing courses offered through its Amazon Web Services subsidiary. The courses are open to anyone and are free. The courses include:

  • Introduction to Generative Artificial Intelligence;
  • Generative AI Learning Plan for Decision Makers;
  • Introduction to Amazon CodeWhisperer;
  • Foundations of Prompt Engineering;
  • Low-Code Machine Learning on AWS;
  • Building Language Models on AWS;
  • Amazon Transcribe—Getting Started; and
  • Building Generative AI Applications Using Amazon Bedrock.

The need for AI skills is plainly obvious, according to Amazon, which points out that 73% of people in a recent survey say hiring people with AI skills is a priority. However, three-quarters of these respondents say they are unable to meet their demand for AI skills.

This AI skills gap is helping to drive salaries upwards for folks with the requisite skills. Amazon says employers are paying people with AI skills 47% more than those who are not upskilled in AI.

Prompt engineers are currently among the highest paid workers in all of tech. A recent survey by EPCGroup found that the average salary for prompt engineers, who are experts at manipulating the prompts that are used as input to large language models such as GPT-4, is about $300,000. Other in-demand AI jobs include machine learning engineers, who earn an average of $160,000, and AI research scientists, who earn an average of $132,00, the study found.

Related

About the author: Alex Woodie

Alex Woodie has written about IT as a technology journalist for more than a decade. He brings extensive experience from the IBM midrange marketplace, including topics such as servers, ERP applications, programming, databases, security, high availability, storage, business intelligence, cloud, and mobile enablement. He resides in the San Diego area.

For the age of the AI PC, here comes a new test of speed

pc140034

New PCs now available with the Core Ultra chip on display at Intel's event.

This year will be the year of the "AI PC," at least according to the CEOs of both Intel and Advanced Micro Devices (AMD), who have signaled their intent to bring meaningful artificial intelligence capabilities to the personal computer along with their partners Dell, HP, Inc., and Lenovo, among others.

In anticipation of a wave of laptops and desktops with special AI abilities, the computer industry's leading organization for rating the performance of neural networks this week announced the formation of a new effort to benchmark AI on personal computers.

Also: Nvidia makes the case for the AI PC at CES 2024

The new MLPerf Client benchmark will rate how well chips and systems perform with generative AI on desktops, laptops, and workstations, said the MLCommons, the industry consortium that has published various AI benchmarks for five years, in a blog post.

Intel and AMD hope that new AI capabilities will juice desktop sales.

The initial benchmarks will focus on measuring how fast computers make predictions, commonly known as "inference," as opposed to "training," which is when a neural net is first developed. Most client computing devices do not possess the requisite power to perform the training of neural networks, so inference is a logical place to start.

The first benchmark test of inference will involve tasks using Meta's open-source Llama 2 large language model.

Co-chairs of the MLPerf Client working group include Ramesh Jaladi, senior director of engineering in the IP Performance group at Intel; Yannis Minadakis, partner GM, software development at Microsoft; and Jani Joki, director of performance benchmarking at NVIDIA.

The new benchmark is timely as PC executives are pinning their hopes for a revival of the moribund PC market on local processing of AI as an adjunct to the cloud-based processing that currently dominates AI. PC sales fell 15% last year in unit terms, the worst year in PC history, according to research firm Gartner.

Also: Intel spotlights AI in new Core Ultra, 5th-gen Xeon chips

In a media event last month, Intel CEO Pat Gelsinger touted the company's newest desktop chip, Core Ultra, code-named "Meteor Lake," as an AI platform. For the first time in any Intel desktop processor, the chip includes a dedicated accelerator unit for AI, called a neural processing unit (NPU).

That NPU circuitry will enable "a new level of power-efficient AI acceleration with 2.5x better power efficiency than the previous generation" of Core chip, according to Intel.

The timing of a first benchmark release is likely months away, the MLCommons told ZDNET via email. Other benchmark reports have typically been published twice yearly.

Artificial Intelligence

90% of IT leaders say it’s tough to integrate AI with other systems

IT worker on laptop

Over three-quarters of organizations are using multiple artificial intelligence (AI) models and 90% report they are experiencing difficulties integrating AI with other systems, according to the 2024 Connectivity Benchmark Report by MuleSoft.

Also: So you want to work in AI but don't know where to start? Follow these 5 steps

MuleSoft's ninth annual Connectivity Benchmark Report, in collaboration with Vanson Bourne and Deloitte Digital, was produced from interviews with 1,050 IT leaders (management positions or above) across the globe (public and private sector with at least 1,000 employees). The report's executive summary suggests:

  • The new normal: AI inflection point amplifies the need for a coherent IT strategy. Eighty-seven percent of IT leaders report that the nature of digital transformation is changing. AI further complexifies the tech landscape, with 991 apps in the average enterprise. IT budgets increase to meet the surging demand.
  • AI adoption explodes, integration and security concerns are the biggest barriers. The AI genie is out of the bottle, with over three-quarters of organizations reporting they use multiple AI models. As many as 90% say difficulty integrating AI with other systems is a barrier, followed by 79% reporting security concerns.
  • IT leaders acknowledge that data silos and systems fragility are holding their companies back. Almost universal, 98% of IT leaders report facing challenges regarding digital transformation. Key drivers are the persistence of data silos at 81% and the fragility of tightly coupled and highly dependent systems at 72%.
  • Automation is still a source of contention between IT and the business. Business users benefit greatly from the automation of their work (1.9 hours per employee per week) and demand more flexibility to automate. However, the majority of IT departments still need to figure out how to enable this in a secure and governed way. Two-thirds (66%) of automation projects have IT as the sole gatekeeper.
  • Virtually every company (99%) runs on APIs, but few have turned them into a strategic lever. APIs remain a powerhouse for productivity and revenue, with one-third of revenue attributed to API-related offerings. However, managing and securing data at scale has become increasingly complex with API sprawl. This sprawl is why one-quarter of all APIs go ungoverned

Let's dig a bit deeper into the major findings of the report, focusing on the first three findings — AI's inflection point, AI's explosive adoption, and the negative impact of data silos.

The new normal: AI inflection point amplifies the need for a coherent IT strategy — Top 3 insights

  1. Organizations rely on IT leaders to guide the implementation of AI and automation. Only 75% of respondents state that their IT leaders communicate an upfront strategy.
  2. IT budgets increase to accommodate surging demand for new projects. The average budget is $10.5 million over the past 12 months, and 77% expect IT staff headcount to increase even further in the coming year.
  3. AI has created more complexity for integration strategies. Organizations now use an average of 991 applications across their digital estate, with an average lifespan of four years.

The report notes that IT leaders feel the nature of digital transformation is changing — and this shift means the role of the CIO and other IT leaders is becoming more critical than ever. The nature of the customer experience and their expectations are also changing. Customers expect more, but data silos stand in the way.

Also: Have 10 hours? IBM will train you in AI fundamentals — for free

Nearly three-quarters (70%) of customer experiences are digital, but only 26% of organizations report providing a complete connected user experience across all channels. To further add to the challenge facing IT, organizations must balance nearly 1,000 applications to create a cohesive experience for end users, with each app having less than a four-year lifespan.

But there is good news for IT — budgets will increase to meet demand. IT teams have spent an average of $10.5 million on IT staff during the past 12 months. And over three-quarters (77%) expect their IT staff headcount to increase even further to meet rising project demand. More budget will also lead to greater demand for IT services. The number of projects asked of IT teams has risen by 39% from 2023.

AI adoption explodes, and integration and security concerns are the biggest barriers — Top 3 insights

  1. Eighty percent of organizations report already using multiple AI models. And AI adoption will increase by 78% within the next three years.
  2. AI is viewed as a solution to growing backlogs if used correctly: 85% of IT leaders say the application of AI has increased developers' productivity in their organization.
  3. But barriers remain. Over 90% report that integration issues impede AI adoption, 79% have security concerns, and 64% are concerned with ethical AI usage and adoption. Only 2% report there are no significant barriers to utilizing their data for AI.

AI adoption continues to explode and so do IT project backlogs. As backlogs grow, IT leaders look to AI to help. IT leaders are balancing AI adoption as ethical concerns, and blockers persist. While AI adoption is growing rapidly, a recent report shows that 64% of IT leaders are concerned with ethical AI usage and adoption.

Also: Is prompt engineer displacing data scientist as the 'sexiest job of the 21st century'?

The report also notes that AI success is dependent on integration. While most respondents believe AI increases productivity, it is dependent on integrated data. Nearly half of IT teams plan on adopting an AI solution in the coming year, but only 28% of apps are connected. Sixty-two percent feel their organization is ill-equipped to fully harmonize their data systems to leverage AI technologies. The concerns around integration are twofold: the difficulty integrating generative AI features with other software systems, and the need for integration between existing systems

Data silos and systems fragility are holding companies back — Top 3 insights

  1. Eighty-one percent of respondents report that data silos are hindering their digital transformation efforts. Seventy-two percent feel their current IT infrastructure makes their systems overly dependent on one another.
  2. IT teams struggle to integrate efficiently: 98% report facing challenges regarding digital transformation. Skill gaps and compliance concerns top the list of challenges
  3. The inability to connect data-derived insights into user experiences is the most commonly cited integration challenge for 75% of organizations.

The report reveals that 81% of IT leaders point to data silos as the biggest hindrance to digital transformation. Integration remains a top priority for implementing a digital transformation strategy, while new technologies also emerge and create further challenges.

With only 72% of applications being integrated/connected, IT teams must focus on increasing productivity to close this gap. All teams are seeking better integration. Teams across the organization depend on the efficient use of data from separate systems, according to IT leaders who report that nearly every department requires more integration.

Nearly all organizations (98%) report facing challenges when it comes to digital transformation, including lack of skill set and experience (38%), risk management and compliance (37%), legacy infrastructure (34%), migrating legacy software (34%), and integrating siloed apps and data.

Since 2018, IT teams have faced an average 36% increase each year in the number of expected projects. The report suggests the path to efficiency is via better integration plans and execution. Organizations that have adopted an integration strategy have reported a vast array of benefits, including boosted customer experience and more significant return on investment.

Also: 6 skills you need to become an AI prompt engineer

The Connectivity Benchmark 2024 Report also deep dives into automation. Automation is still a source of contention between IT and the business. Business users want to self-serve, but 66% of automations are still governed by IT. The growth in robotic process automation remains steady, with 31% of teams opting for the solution. This is a substantial rise from 13% in 2021.

Automation continues to demonstrate its value by, among other things, driving operational efficiency (reported by 56% of automation users) and productivity (reported by 52%). The report says IT continues to struggle with an automation backlog. About two-thirds of IT developers implement and govern automations for business users.

However, only 22% of IT leaders feel their strategy to enable non-technical business users to integrate apps and data sources powered by APIs is up to date. There is also a move towards citizen developers to help IT to address the growing backlog — low-code and no-code solutions must be part of the innovation and development strategy.

Also: 6 AI tools to supercharge your work and everyday life

The report concludes by reminding us that almost every company runs on APIs, but few have turned them into a strategic lever. The explanation for this reality has a lot to do with company revenues. API-related offerings take the credit for a large portion of revenue. Thirty-three percent of all revenue is from APIs and API-related offerings. Also, APIs have contributed to increased revenue for 39% of respondents — and decreased operational costs by 35%. If you're interested, you can learn more about the 2024 Connectivity Benchmark Report.

Featured

The Only Free Course You Need To Become a Professional Data Engineer

The Only Free Course You Need To Become a Professional Data Engineer
Image by Author

There are many courses and resources available on machine learning and data science, but very few on data engineering. This raises some questions. Is it a difficult field? Is it offering low pay? Is it not considered as exciting as other tech roles? However, the reality is that many companies are actively seeking data engineering talent and offering substantial salaries, sometimes exceeding $200,000 USD. Data engineers play a crucial role as the architects of data platforms, designing and building the foundational systems that enable data scientists and machine learning experts to function effectively.

Addressing this industry gap, DataTalkClub has introduced a transformative and free bootcamp, "Data Engineering Zoomcamp". This course is designed to empower beginners or professionals looking to switch careers, with essential skills and practical experience in data engineering.

Course Overview: A Roadmap to Data Engineering Mastery

This is a 6-week bootcamp where you will learn through multiple courses, reading materials, workshops, and projects. At the end of each module, you will be given homework to practice what you've learned.

  1. Week 1: Introduction to GCP, Docker, Postgres, Terraform, and environment setup.
  2. Week 2: Workflow orchestration with Mage.
  3. Week 3: Data warehousing with BigQuery and machine learning with BigQuery.
  4. Week 4: Analytical engineer with dbt, Google Data Studio, and Metabase.
  5. Week 5: Batch processing with Spark.
  6. Week 6: Streaming with Kafka.

The Only Free Course You Need To Become a Professional Data Engineer
Image from DataTalksClub/data-engineering-zoomcamp Syllabus

The syllabus contains 6 modules, 2 workshops, and a project that covers everything needed for becoming a professional data engineer.

Module 1: Mastering Containerization and Infrastructure as Code

In this module, you will learn about the Docker and Postgres, starting with the basics and advancing through detailed tutorials on creating data pipelines, running Postgres with Docker, and more.

The module also covers essential tools like pgAdmin, Docker-compose, and SQL refresher topics, with optional content on Docker networking and a special walk-through for Windows subsystem Linux users. In the end, the course introduces you to GCP and Terraform, providing a holistic understanding of containerization and infrastructure as a code, essential for modern cloud-based environments.

Module 2: Workflow Orchestration Techniques

The module offers an in-depth exploration of Mage, an innovative open-source hybrid framework for data transformation and integration. This module begins with the basics of workflow orchestration, progressing to hands-on exercises with Mage, including setting it up via Docker and building ETL pipelines from API to Postgres and Google Cloud Storage (GCS), and then into BigQuery.

The module's blend of videos, resources, and practical tasks ensures a comprehensive learning experience, equipping learners with the skills to manage sophisticated data workflows using Mage.

Workshop 1: Data Ingestion Strategies

In the first workshop you will master building efficient data ingestion pipelines. The workshop focuses on essential skills like extracting data from APIs and files, normalizing and loading data, and incremental loading techniques. After completing this workshop, you will be able to create efficient data pipelines like a senior data engineer.

Module 3: Data Warehousing

The module is an in-depth exploration of data storage and analysis, focusing on Data Warehousing using BigQuery. It covers key concepts such as partitioning and clustering, and dives into BigQuery's best practices. The module progresses into advanced topics, particularly the integration of Machine Learning (ML) with BigQuery, highlighting the use of SQL for ML, and providing resources on hyperparameter tuning, feature preprocessing, and model deployment.

Module 4: Analytics Engineering

The analytics engineering module focuses on building a project using dbt (Data Build Tool) with an existing data warehouse, either BigQuery or PostgreSQL.

The module covers setting up dbt in both cloud and local environments, introducing analytics engineering concepts, ETL vs ELT, and data modeling. It also covers advanced dbt features such as incremental models, tags, hooks, and snapshots.

In the end, the module introduces techniques for visualizing transformed data using tools like Google Data Studio and Metabase, and it provides resources for troubleshooting and efficient data loading.

Module 5: Proficiency in Batch Processing

This module covers batch processing using Apache Spark, starting with introductions to batch processing and Spark, along with installation instructions for Windows, Linux, and MacOS.

It includes exploring Spark SQL and DataFrames, preparing data, performing SQL operations, and understanding Spark internals. Finally, it concludes with running Spark in the cloud and integrating Spark with BigQuery.

Module 6: The Art of Streaming Data with Kafka

The module begins with an introduction to stream processing concepts, followed by in-depth exploration of Kafka, including its fundamentals, integration with Confluent Cloud, and practical applications involving producers and consumers.

The module also covers Kafka configuration and streams, addressing topics like stream joins, testing, windowing, and the use of Kafka ksqldb & Connect. Additionally, it extends its focus to Python and JVM environments, featuring Faust for Python stream processing, Pyspark — Structured Streaming, and Scala examples for Kafka Streams.

Workshop 2: Stream Processing with SQL

You will learn to process and manage streaming data with RisingWave, which provides a cost-efficient solution with a PostgreSQL-style experience to empower your stream processing applications.

Project: Real-World Data Engineering Application

The objective of this project is to implement all the concepts we have learned in this course to construct an end-to-end data pipeline. You will be creating to create a dashboard consisting of two tiles by selecting a dataset, building a pipeline for processing the data and storing it in a data lake, building a pipeline for transferring the processed data from the data lake to a data warehouse, transforming the data in the data warehouse and preparing it for the dashboard, and finally building a dashboard to present the data visually.

Enroll Now

2024 Cohort Details

  • Registration: Enroll Now
  • Start date: January 15, 2024, at 17:00 CET
  • Self-paced learning with guided support
  • Cohort folder with homeworks and deadlines
  • Interactive Slack Community for peer learning

Prerequisites

  • Basic coding and command line skills
  • Foundation in SQL
  • Python: beneficial but not mandatory

Expert Instructors Leading Your Journey

  • Ankush Khanna
  • Victoria Perez Mola
  • Alexey Grigorev
  • Matt Palmer
  • Luis Oliveira
  • Michael Shoemaker

Conclusion

Join our 2024 cohort and start learning with an amazing data engineering community. With expert-led training, hands-on experience, and a curriculum tailored to the needs of the industry, this bootcamp not only equips you with the necessary skills but also positions you at the forefront of a lucrative and in-demand career path. Enroll today and transform your aspirations into reality!

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in Technology Management and a bachelor's degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.

More On This Topic

  • The only Jupyter Notebooks extension you truly need
  • 9 Skills You Need to Become a Data Engineer
  • Essential Books You Need to Become a Data Engineer
  • Everything you need to become a SAS Certified Machine Learning Engineer
  • Become a Data Science Professional in Five Steps
  • Should You Become a Freelance Artificial Intelligence Engineer?

SQL Simplified: Crafting Modular and Understandable Queries with CTEs

SQL Simplified: Crafting Modular and Understandable Queries with CTEs
Image by Author

In the world of data, SQL still stands as the lingua franca for interacting with databases.

Still today it stands as one of the most used languages to deal with data and is still considered a must-have for any good data professional.

However, anyone who has worked with complex SQL queries knows they can quickly turn into unwieldy beasts—difficult to read, maintain, or reuse.

This is why today it is not enough to know SQL, we need to be good at crafting queries. And this is actually a type of art.

This is where Common Table Expressions (CTEs) come into play, transforming the art of query writing into a more structured and approachable craft.

So let’s discover together how to code readable and reusable queries.

1. The Basics of CTEs

If you are wondering what is a CTE, you are in the right article.

A Common Table Expression (CTE) is a temporary result set that is defined within the execution scope of a single SQL statement.

They are temporal tables that can be referenced many times within a single query and typically are used to simplify complex joins and subqueries, with the ultimate goal of increasing the readability and organization of SQL code.

So they are powerful tool for breaking down complex queries into simpler parts.

Here’s why you should consider using CTEs:

  • Modularity: You can break down complex logic into readable chunks.
  • Readability: It makes understanding the flow of SQL queries easier.
  • Reusability: CTEs can be referenced multiple times in a single query, avoiding repetition.

2. Using the WITH Clause to generate modular queries

The magic begins with the WITH clause, which precedes your main query and defines different temporal tables (CTEs) with aliases.

Therefore, we always need to start our query with the “WITH” command to start defining out own CTEs. By using CTEs, we can break any complex SQL query into:

— Small temporal tables that compute related variables.

— A final table that only takes those variables that we want as our output.

And this is precisely the MODULAR approach we want in any code!

SQL Simplified: Crafting Modular and Understandable Queries with CTEs
Image by Author

So using CTEs in our queries allows us to:

— Execute a temporal table ONCE and reference it MULTIPLE times.

— Improve readability and simplify complex logic.

— Promote code reusability and modular design.

3. Case Study — Analyzing Airbnb Data

In order to understand this better, we can take a practical example of Airbnb listings in Barcelona.

Imagine we want to analyze the performance of listings by neighborhood and compare it with the city’s overall performance. You'd need to pull together information about neighborhoods, individual apartments, hosts, and pricing.

To exemplify this, we will use the InsideAirbnb table of Barcelona, which looks like follows:

SQL Simplified: Crafting Modular and Understandable Queries with CTEs

A naive approach might lead you to create nested subqueries that quickly become a maintenance nightmare like the one as follows:

Code by Author

Instead, we can utilize CTEs to compartmentalize our query into logical sections—each defining a piece of the puzzle.

  • Neighborhood Data: Create a CTE to summarize data by neighborhood.
  • Apartment and Host Information: Define CTEs for details about apartments and hosts.
  • City-wide Metrics: Another CTE to gather city-level statistics for comparison.
  • Final Assembly: Combine the CTEs in a final SELECT statement to present the data cohesively.

SQL Simplified: Crafting Modular and Understandable Queries with CTEs
Image by Author

And we would end up with the following query:

Code by Author 4. Advantages of a Modular Approach

By using CTEs, we turn a potentially monstrous single query into an organized set of data modules. This modular approach makes the SQL code more intuitive and adaptable to changes.

If new requirements emerge, you can adjust or add CTEs without overhauling the entire query.

5. Reusing CTEs for Comparative Analysis

Once you’ve established your CTEs, you can reuse them to perform comparative analysis. For instance, if you want to compare neighborhood data against city-wide metrics, you can reference your CTEs in a series of JOIN operations.

This not only saves time but also keeps your code efficient, as you won’t have to repeat the same query twice!

Final Thoughts

CTEs are a testament to the principle that a little structure goes a long way in programming. By adopting CTEs, you can write clearer, more maintainable, and reusable SQL queries.

It streamlines the query development process and makes it easier to communicate complex data retrieval logic to others.

Remember, the next time you find yourself about to embark on writing a multi-join, nested subquery monster, consider breaking it down with CTEs.

Your future self—and anyone else who might read your code—will thank you.

Josep Ferrer is an analytics engineer from Barcelona. He graduated in physics engineering and is currently working in the Data Science field applied to human mobility. He is a part-time content creator focused on data science and technology. You can contact him on LinkedIn, Twitter or Medium.

More On This Topic

  • The Resilient ML Stack is Modular
  • European AI Act: The Simplified Breakdown
  • Beyond Skynet: Crafting the Next Frontier in AI Evolution
  • 4 Useful Intermediate SQL Queries for Data Science
  • 5 Tricky SQL Queries Solved
  • Solving 5 Complex SQL Problems: Tricky Queries Explained

Ola founder’s Krutrim becomes India’s first AI unicorn

Ola founder’s Krutrim becomes India’s first AI unicorn Manish Singh 11 hours

Krutrim, an AI startup founded by Ola founder Bhavish Aggarwal, said it has raised a funding round that values it at $1 billion. The startup, founded last year, is the fastest to become a unicorn in India, it claimed in a press statement. It’s also the first Indian AI startup to become a unicorn, it said.

Matrix Partners India — which has also backed Aggarwal’s other two startups, ride-hailing platform Ola and EV startup Ola Electric — led the $50 million “first round” in Krutrim. TechCrunch reported last year that Aggarwal was in talks to raise $50 million for his new AI venture.

Krutrim, which means “artificial” in Sanskrit, is building a large language model that has been trained on local Indian languages in addition to English. The startup plans to launch a voice-enabled conversational AI assistant that understands and speaks multiple Indian languages, the startup said.

It plans to make a beta version of its eponymous chatbot available to consumers next month and follow it with rolling out APIs to developers and enterprises. On its website, Krutrim says it also plans to develop an in-house capability to manufacture chips optimised for AI compute. TechCrunch reported earlier that Aggarwal’s new venture will explore developing and designing chips.

Krutrim is Aggarwal’s third venture. Ola, his first, leads the ride-hailing market in India and is eyeing profitability. Ola Electric leads the two-wheeler EV market in India and recently filed the paperwork for a $662 million initial public offering.

“India has to build its own AI, and at कृत्रिम, we are fully committed towards building the country’s first complete AI computing stack,” Aggarwal said in a statement. “We are thrilled to announce the successful closure of our first funding round, which not only validates the potential of कृत्रिम ’s innovative AI solutions but also underscores the confidence investors have in our ability to drive meaningful change out of India for the world.”

The investment in Krutrim comes at a time when investors globally are rushing to identify and back AI breakthrough, banking on the thesis that advances in AI will make countless industries more efficient and startups at the forefront will deliver generational returns.

Despite being home to one of the world’s largest startup ecosystems, India has yet to make a material impact in the AI race. Indian contenders have yet to emerge and challenge the dominance of large language model titans such as OpenAI’s ChatGPT, Amazon-backed Anthropic, or Google’s Bard.

Indian powerhouse Reliance partnered with Nvidia in September, revealing plans to build a large language model that is trained on India’s diverse languages. But the firm — run by Mukesh Ambani, Asia’s richest person — has yet to launch its AI offering.

Peak XV and Lightspeed India recently backed Sarvam, an AI startup that is also building a large language model.

Training a large language model has proven to be a very expensive endeavour. OpenAI has raised over $11 billion to date, whereas Anthropic has also raised billions from investors including Google and Amazon. xAI, Elon Musk’s AI startup, is in talks to raise up to $6 billion at a valuation of about $20 billion, Financial Times reported Friday.

Where is India in the generative AI race?

Krutrim Secures $50 Million, Becomes India’s First AI Unicorn

Ola Copies OpenAI’s ChatGPT, Calls it Krutrim AI

Krutrim, India’s own AI company, dedicated to constructing a comprehensive AI computing stack, has successfully closed its first funding round, securing a substantial USD 50 million in equity at a valuation of USD 1 billion.

The funding round, led by prominent investors including Matrix Partners India, will be instrumental in accelerating the company’s mission to revolutionize the AI landscape, drive innovation, and expand its reach globally, the company said in its blog post.

India needs to build it own AI, and we @Krutrim are committing ourselves to building the country's complete AI computing stack. We are excited to announce the successful closure of our first funding round making कृत्रिम – India's fastest Unicorn and also the first AI unicorn 🙂… pic.twitter.com/sORXgTKjki

— Bhavish Aggarwal (@bhash) January 26, 2024

This achievement positions Krutrim as India’s fastest Unicorn and the country’s first AI unicorn, marking a significant milestone for both the company and the Indian tech ecosystem.

Bhavish Aggarwal, Founder, Krutrim said, “India has to build its own AI, and at कृत्रिम (Krutrim), we are fully committed towards building the country’s first complete AI computing stack.”

“As we embark on this exciting journey, we are proud to contribute to the vision of ‘Viksit Bharat’, where innovation and technological prowess will propel our nation towards reclaiming its status of the ‘Vishwaguru’,” he added.

Introduced last month, Krutrim is touted as “India’s first full-stack AI” solution. It is trained on 2 trillion tokens and can understand over 20 Indian languages and generate content in about 10 languages, including Marathi, Hindi, Bengali, Tamil, Kannada, Telugu, Odia, Gujarati, and Malayalam.

Bhavish has consistently brought cutting edge tech innovation to India at scale with Ola and Ola Electric – and now excitingly with Krutrim to power the journey of ‘Viksit Bharat’ digitally. We are incredibly privileged to deepen our partnership with Bhavish and Krutrim,”said Avnish Bajaj, Founder and MD, Matrix Partners India.

The post Krutrim Secures $50 Million, Becomes India’s First AI Unicorn appeared first on Analytics India Magazine.

NSF Forms AI Resource Pilot Program With Microsoft, OpenAI and Others

Microsoft, NVIDIA and OpenAI are among the companies working with the U.S. National Science Foundation and other federal agencies on the National AI Research Resource pilot program to connect researchers to AI research infrastructure. The program supports advancements in “safe, secure and trustworthy AI,” according to the announcement from the NSF.

What is the National AI Research Resource?

The National AI Research Resource is a pilot program to provide access to resource-intensive AI training and operation infrastructure like Microsoft Azure and privacy and security resources. The pilot program will “strengthen and democratize access to critical resources necessary to power responsible AI discovery and innovation,” according to the announcement.

“By investing in AI research through the NAIRR pilot, the United States unleashes discovery and impact and bolsters its global competitiveness,” said NSF Director Sethuraman Panchanathan in the announcement.

One of the stated goals of NAIRR is to incubate “commercially viable AI applications and solutions” through industry collaboration.

SEE: Security researchers identified a GPU vulnerability that could let attackers eavesdrop on conversations with generative AI (TechRepublic)

The NAIRR pilot program focuses on four areas:

  • NAIRR Open will enable various open and diverse AI resources.
  • NAIRR Secure will support AI research that requires privacy and security-preserving resources and is co-led by the National Institutes of Health and U.S. Department of Energy.
  • NAIRR Software will enable and study inter-operable use of AI software, platforms, tools and services used for NAIRR pilot resources.
  • NAIRR Classroom will be an outreach program for communities.

NAIRR is part of the plan outlined in the executive order regarding AI development released by President Joe Biden in October 2023.

Which tech companies are participating in NAIRR?

Some of the tech companies, nonprofits and philanthropic organizations which have provided resources to NAIRR are:

  • AWS
  • Anthropic
  • MD
  • Databrinks
  • Google
  • Hewlett Packard Enterprise
  • Hugging Face
  • IBM
  • Intel
  • Meta
  • Microsoft
  • NVIDIA
  • OpenAI

The complete list of 25 organizations can be found here.

The government partners include NASA, DARPA, NIST and the following:

  • National Institutes of Health.
  • National Oceanic and Atmospheric Administration.
  • U.S. Department of Agriculture.
  • U.S. Department of Defense.
  • U.S. Department of Energy.
  • U.S. Department of Veterans Affairs.
  • U.S. Patent and Trademark Office.

How could NAIRR impact businesses or enterprise use of AI?

The leading U.S. AI tech organizations will have a pathway through which to work directly with government-supported researchers and vice versa. This speaks to the government’s interest in working with businesses that use and create AI, and to the adoption of AI in the mainstream. The U.S. is the top ranked nation in AI adoption, according to the Harvard Business Review, followed by China, the U.K., Japan and Germany.

“We hope that by contributing our Data Intelligence Platform to NAIRR, we will enable the next generation of students to create new breakthroughs and build new businesses in AI,” wrote Jonathan Frankle, chief scientist, Databricks, in the announcement.

How to get involved

NAIRR is accepting applications from private sector and nonprofit organizations interested in becoming partners. Interested researchers can apply for access to limited NAIRR resources at the NAIRR pilot portal now. A call for proposals from interested researchers will open in spring 2024 with the full suite of NAIRR resources.

Why Falcon Sucks

Why Falcon Sucks

In June, the first time when UAE’s TII launched Falcon, its own LLM, it was on top of the Hugging Face Open LLM Leaderboard, championing ahead of Meta’s Llama 2 on several benchmarks. Undoubtedly, people had their doubts as the researchers did not have a paper to back up the claims so well.

Cut to the present, the team has still not released the model’s paper. The open source champion with a 40 billion parameter size, now also has a 180 billion parameter model, which though is open source, does not have many users because of its massive compute requirements. It seems as though there is something wrong with the model.

The Open LLM Leaderboard currently has a bunch of models that are fine-tuned on top of Mistral, Llama 2, and even Chinese models such as Yi-24B and Qwen-7B, but there is no model of Falcon.

Is it a boycott?

The foremost thing here is to understand that the model is not born out of the US or Europe, which are leading the LLM race with big-tech companies. But that did not stop the developer community from experimenting with the model. For the last months, Falcon was touted as the replacement of OpenAI and Google because of its open nature, but now it’s nowhere to be seen.

Maxime Labonne, the open source researcher and the creator of several models and LLM Courses, told AIM that though he experimented with Falcon, he finds Llama and Mistral’s models a lot easier for tokenisation on different European languages. “I find it surprising when people still use it,” he said.

Labonne emphasised on the part that it was good enough when it was released as there were no other alternatives apart from the first version of Meta’s LLaMA, and had a slightly different licence as well. Interestingly, even that was not as open source as people wanted a model to be.

“Now Llama 2 is a lot better than Falcon, and Mistral is a lot better than Llama 2. It makes no sense to use Falcon anymore,” he added.

Moreover, when it comes to Indic models, Llama is touted as the best option. Mistral, though good for European languages, struggles to tokenise Indic languages. Adarsh Shirawalmath, the creator Kannada Llama aka Kan-Llama, said that the team decided to not use Mistral or Falcon because it is very hard to fine-tune them with Indic language tokens.

That is why most of the current Indic language models are built on top of Llama 2. When asked about Falcon to the creators of BharatGPT, they said that the good part about it is that it is built with an academic, government, and private partnership. That is what IIT Bombay is also aiming for with BharatGPT, a large open source model.

Come of age, in a few months

Recently, researchers from IIT Patna released a dataset for healthcare in India, focusing on Indic languages as well. AIM asked Aman Chadha, the lead researcher about why the team did not use Falcon, but used all other models such as Llama, Zephyr, and others. “Falcon was great when it came out. But with Mistral and Zephyr and a bunch of them, they seem to have done incredibly well in the area that we wanted,” he explained.

“Falcon at this point is actually a little on the outdated side,” he added. “There’s been so much in terms of new models coming such as Tiny Llama. While Falcon is great depending on what the targeted use cases are, we look at what’s out there and how performance based models are and accordingly choose what you want to go out with,” Chadha asserted.

Undoubtedly, the sheer size of Falcon with 180 billion parameters seems like a good open source alternative to GPT-4 and others, as the conversation increasingly is moving towards edge use cases and multilingual models, Falcon seems to be getting too heavy and outdated, specially for the Indian landscape and multilingual tasks.

The post Why Falcon Sucks appeared first on Analytics India Magazine.