Рубрика: AI

Alation Adds GenAI to Data Catalog October 13, 2023 by Alex Woodie

(thodonal88/Shutterstock)

Alation is rolling out a new offering dubbed ALLIE AI that’s aimed at helping data stewards better organize data when it’s being ingested into the company, as well as to streamline and simplify how customers ultimately find what they’re looking for in Alation’s data catalog.

As the original creator of the data catalog and progenitor of the product category, Alation knows a thing or two about data curation and democratizing access to information in large, complex enterprises. It also knows how important having a structured process for ingesting data, assigning governance policies, and building indexes. Without it, we might as well return to the data dark ages.

With yesterdays’ launch of ALLIE AI, Alation is taking its data governance game up a notch and into the world of generative AI. As part of the new offering, Alation is training a large language model (LLM) on a customers’ own private data set to help with both frontend and backend data governance tasks.

On the frontend, or the data ingestion and curation stage, ALLIE AI will leverage the LLM’s built-in language understanding and generation capabilities to automatically document new data assets as they are brought into the customer’s private data respiratory and define the data governance policies. At the same time, ALLIE AI will suggest which of customer’s data stewards have the specialized knowledge to oversee and guide the data onboarding and sign off on the data governance polices that ALLIE AI automatically generates.

During the data access stage, such as when data analysts or data scientists are using the catalog to explore and access enterprise data sets, ALLIE AI leverages its natural language understanding capabilities to help data analysts and other users to access the data they need, without requiring knowledge of SQL or other specialist skills. Like most data catalogs, Alation uses traditional keyword search and indexing techniques to streamline user access to data, and the addition of an LLM’s vector search capability can provide better search results.

As the LLM learns about the data in the customer’s environment, it will get better at connecting users with data they’re looking for, said Jonathan Bruce, Alation’s vice president of product management.

“We expect to see a duality there for a significant amount of time. Matching by meaning is extremely useful for many of our customers in the way that it’s just going to collapse down the time to relevance, if you will,” he said. “It’s going to de-muddle search results so that customers can get what they want sooner in a way that doesn’t require them to work as much.”

Eventually, the LLM will enable Alation customers to have a natural language conversation about data within the catalog, Bruce said. We’re not there yet, but that’s the goal.

“If you’ve ever used ChatGPT, the first question you ask, it can gives you a general answer, but you can actually kind of tease out something more specific with some additional interactions,” he said. “That’s part of what we would do down the line.”

Alation’s goal is absolutely not to replace human stewards and curators, Bruce said. Humans will always be a part of the equation, he said. However, there’s quite a bit room for additional automation and accelerating the data onboarding process, and GenAI can deliver that.

“It’s about getting the catalog to a point whereby they’re able to hit that maturity curve sooner,” Bruce said. “That’s a critical for our customers, because they’re looking to wire in more and more data sources more frequently, and the human overhead to do that curation manually gets in the way of the speed of business they want to operate.”

(Dilok Klaisataporn/Shuttesrtock)

Many of Alation’s large customers, such as Cisco, employ a large number of data stewards to guide the ingest of new data into the company and its catalog in an orderly and repeatable manner. Each data steward brings strengths in different data specialties, and one way that GenAI can help is by gradually learning what those specialties are, so it can suggest which data steward oversees the creation of the data policies that govern who can access that data, and for what purposes.

“I think effectively what we’re doing, what we’re applying here, is allowing ALLIE AI to actually generate that content, but generate that content in a way that you don’t completely remove that human element,” Bruce told Datanami. “Humans can still review and provide feedback and then ultimately allow us to learn within the confines of my customer tenancy.”

Each ALLIE AI customers gets their own LLM, which ensures that the GenAI is trained on data sets and terms that are unique to the customer while also preventing sensitive data from inadvertently leaking outside the customers’ domain.

“It’s all within the customer tenancy,” Bruce says. “You have your own language model. It evolves within your own data, and that’s a really important part of how we afford that segmentation That is absolutely fundamental.”

Related

About the author: Alex Woodie

Alex Woodie has written about IT as a technology journalist for more than a decade. He brings extensive experience from the IBM midrange marketplace, including topics such as servers, ERP applications, programming, databases, security, high availability, storage, business intelligence, cloud, and mobile enablement. He resides in the San Diego area.

Artificial Intelligence (AI) is emerging as a formidable force, revolutionizing how we conceive, create, and deliver software solutions. As technology advances at an unprecedented pace, the role of AI in this domain has become increasingly significant. It’s no longer just a buzzword; it’s a fundamental tool that promises to reshape the entire software development process. And, unless you have been hiding away somewhere, distant from all of humankind, you would realize that the software development market is on the rise.

With our ceaselessly developing dependence on everything advanced, including the rising reliance of organizations and consumers on software, it checks out that this market is encountering sensational development. Be that as it may, what is fascinating in this setting is the role artificial intelligence has come to play in programming. As a matter of fact, an ever-increasing number of organizations are now utilizing Artificial Intelligence tools to work on the quality of software development and foster new kinds of software. For example, virtual assistant bots, self-driving vehicles, etc.

In this blog, I’ll try to delve into the multifaceted role of AI in software product development, exploring how it drives innovation, improves efficiency, and ultimately changes the way we think about software engineering.

At any rate, Artificial Intelligence continues to be a significant innovation in programming development and improvement. Thus, as Artificial Intelligence-fueled tools become progressively common, they enable software developers to be more effective and valuable while working with first-rate degrees of advancement in software apps. In any case, if you are still vacillating about hiring a company offering software product development for your business, permit us to give you a closer look into some of AI’s key benefits.

How AI stands to help businesses

• Reduced costs: Thanks to the reductions in the time and effort required to develop software enabled by AI, companies alleviate the costs associated with their software development project. Besides that, such tools can also help developers identify and manage software bugs early in the development lifecycle. This also helps companies reduce costs — since fixing bugs at later stages in the development lifecycle is more expensive.
• Automation of repetitive tasks: So many functions in software development are necessary but relatively mundane and repetitive and do not necessarily need human intervention. This is where AI comes in — you can use it to generate code for routine tasks. It can also be used for automatic code reviews. And let us not forget that AI can also help generate documentation for code and software systems, thus making sure that all documentation remains up-to-date and consistent with the codebase.
• Accelerated development pace: First, AI helps accelerate the speed of development work via automation of tasks and reducing the risk of errors that come with it. Then there are also AI-driven code completion and suggestion tools that assist developers with real-time recommendations and code snippets, significantly speeding up the coding process. Plus, AI can analyze data and user behavior patterns to help developers make informed decisions about feature development, resulting in quicker iterations of the software.
• Enhanced software testing: Many companies are also using AI to implement more comprehensive, efficient, and effective software testing. This includes automating the testing process by running many test cases and scenarios, which is particularly helpful and equally valuable for regression testing.

AI is already playing a super important role in software development. And as technology continues to evolve, we reckon AI will play an even more significant role in this market. What do you think?

Image by Author What is Rust Burn?

Rust Burn is a new deep learning framework written entirely in the Rust programming language. The motivation behind creating a new framework rather than using existing ones like PyTorch or TensorFlow is to build a versatile framework that caters well to various users including researchers, machine learning engineers, and low-level software engineers.

The key design principles behind Rust Burn are flexibility, performance, and ease of use.

Flexibility comes from the ability to swiftly implement cutting-edge research ideas and run experiments.

Performance is achieved through optimizations like leveraging hardware-specific features such as Tensor Cores on Nvidia GPUs.

Ease of use stems from simplifying the workflow of training, deploying, and running models in production.

Key Features:

• Flexible and dynamic computational graph
• Thread-safe data structures
• Intuitive abstractions for simplified development process
• Blazingly fast performance during training and inference
• Supports multiple backend implementations for both CPU and GPU
• Full support for logging, metric, and checkpointing during training
• Small but active developer community

Getting Started

Installing Rust

Burn is a powerful deep learning framework that is based on Rust programming language. It requires a basic understanding of Rust, but once you've got that down, you'll be able to take advantage of all the features that Burn has to offer.

To install it using an official guide. You can also check out GeeksforGeeks guide for installing Rust on Windows and Linux with screenshots.

Image from Install Rust

Installing Burn

To use Rust Burn, you first need to have Rust installed on your system. Once Rust is correctly set up, you can create a new Rust application using cargo, Rust's package manager.

Run the following command in your current directory:

cargo new new_burn_app

Navigate into this new directory:

cd new_burn_app

Next, add Burn as a dependency, along with the WGPU backend feature which enables GPU operations:

cargo add burn --features wgpu

In the end, compile the project to install Burn:

cargo build

This will install the Burn framework along with the WGPU backend. WGPU allows Burn to execute low-level GPU operations.

Example Code

Element Wise Addition

To run the following code you have to open and replace content in src/main.rs:

use burn::tensor::Tensor;  use burn::backend::WgpuBackend;    // Type alias for the backend to use.  type Backend = WgpuBackend;    fn main() {      // Creation of two tensors, the first with explicit values and the second one with ones, with the same shape as the first      let tensor_1 = Tensor::::from_data([[2., 3.], [4., 5.]]);      let tensor_2 = Tensor::::ones_like(&tensor_1);        // Print the element-wise addition (done with the WGPU backend) of the two tensors.      println!("{}", tensor_1 + tensor_2);  }

In the main function, we have created two tensors with WGPU backend and performed addition.

To execute the code, you must run cargo run in the terminal.

Output:

You should now be able to view the outcome of the addition.

Tensor {    data: [[3.0, 4.0], [5.0, 6.0]],    shape:  [2, 2],    device:  BestAvailable,    backend:  "wgpu",    kind:  "Float",    dtype:  "f32",  }

Note: the following code is an example from Burn Book: Getting started.

Position Wise Feed Forward Module

Here is an example of how easy it is to use the framework. We declare a position-wise feed-forward module and its forward pass using this code snippet.

use burn::nn;  use burn::module::Module;  use burn::tensor::backend::Backend;    #[derive(Module, Debug)]  pub struct PositionWiseFeedForward<B: Backend> {      linear_inner: Linear<B>,      linear_outer: Linear<B>,      dropout: Dropout,      gelu: GELU,  }    impl PositionWiseFeedForward<B> {      pub fn forward(&self, input: Tensor<B, D>) -> Tensor<B, D> {          let x = self.linear_inner.forward(input);          let x = self.gelu.forward(x);          let x = self.dropout.forward(x);            self.linear_outer.forward(x)      }  }

The above code is from the GitHub repository.

Example Projects

To learn about more examples and run them, clone the https://github.com/burn-rs/burn repository and run the projects below:

• MNIST: Train a model on either CPU or GPU using various backends.
• MNIST Inference Web: Model inference in the browser.
• Text Classification: Train a transformer encoder from scratch on GPU.
• Text Generation: Build and train autoregressive transformer from scratch on GPU.

Pre-trained Models

To build your AI application, you can use the following pre-trained models and fine-tune them with your dataset.

• SqueezeNet: squeezenet-burn
• Llama 2: Gadersd/llama2-burn
• Whisper: Gadersd/whisper-burn
• Stable Diffusion v1.4: Gadersd/stable-diffusion-burn

Conclusion

Rust Burn represents an exciting new option in the deep learning framework landscape. If you are already a Rust developer, you can leverage Rust's speed, safety, and concurrency to push the boundaries of what's possible in deep learning research and production. Burn sets out to find the right compromises in flexibility, performance, and usability to create a uniquely versatile framework suitable for diverse use cases.

While still in its early stages, Burn shows promise in tackling pain points of existing frameworks and serving the needs of various practitioners in the field. As the framework matures and the community around it grows, it has the potential to become a production-ready framework on par with established options. Its fresh design and language choice offer new possibilities for the deep learning community.

Resources

• Documenatiatin: https://burn-rs.github.io/book/overview.html
• Website: https://burn-rs.github.io/
• GitHub: https://github.com/burn-rs/burn
• Demo: https://burn-rs.github.io/demo

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in Technology Management and a bachelor's degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.

More On This Topic

• Using Datawig, an AWS Deep Learning Library for Missing Value Imputation
• Know your data much faster with the new Sweetviz Python library
• XGBoost Explained: DIY XGBoost Library in Less Than 200 Lines of Python
• Overview of Albumentations: Open-source library for advanced image…
• Simple Text Scraping, Parsing, and Processing with this Python Library
• skops: A New Library to Improve Scikit-learn in Production

The advent of generative AI is empowering everyone alike – organizations, small businesses, individuals, students, and medical professionals, to name a few. The last couple of years have been revolutionary for artificial intelligence innovation and transformation. How will 2024 shape up for AI, AI tools, and related professionals? Let’s analyze the trends that are most likely to be observed and practiced while being braced up for 2024.

1. Custom Tailored Content

Be it e-commerce entertainment or any other industry – personalization of content has become crucial for the success of businesses and individuals. Generative AI is all set to play a pathbreaking role in the development of tailored content as per the target audience. When Data, text, audio, or videos, are personalized, it creates a bigger impact on the TG.

2. Enhanced Creativity

Artists and creators will enjoy the power to create out-of-the-world creations. The process of creative creation will highly be augmented with AI tools. Composing music, ideating new ideas, and drawing new designs will become easier than ever before.

3. Converse like Humans

Chatbots and virtual assistants already come with advanced features like human-like interactions and the year 2024 is going to be revolutionary for them. Context-aware and high-on-emotional chatbots are going to rise in leaps and bounds with the use of natural language processing.

4. Healthcare Heroes

The advent of generative AI in the healthcare domain has significantly improved patient care. There have been discoveries of drugs, genomics research, medical imaging, and prediction of outstanding lifesaving remedies at reduced costs. The healthcare domain will see notable innovation and practice of patient care initiatives empowered by generative AI.

5. Entertainment Videos

Deepfake technology, with its pros and cons, is just a slice of the entire pie. Filmmakers and video creators will have access to a host of new features to create impactful and meaningful videos for their audiences. The art of storytelling will find new possibilities starting with special effects, synthetic actors, single-take scenes, and fast video editing.

6. Like what you See

Apart from mere image design, generative AI is here to transform the way products look and appear. Designers and engineers will heave a sign as their time-consuming task of creating innovative and effective product designs will be shouldered by generative AI tools. This trend is going to produce newfangled and visually appealing products, benefitting both consumers and sellers.

7. Are you Internet-secured?

With rapid usage of artificial intelligence and its sub-sets like machine learning, NLP, and computer vision, cybercriminals have also advanced their game. AI-driven cybersecurity will become a necessity and not merely a luxury or option. Detecting and mitigating cyber risks before they become fatal for the system will be the go-to approach for organizations. The idea of red teaming will become more and more significant for organizations to function unhindered and unpaused.

8. Environmental Concerns

Using AI to simulate climate scenarios and optimizing energy consumption to generate renewable sources of energy is a practice that will gain prominence in the coming years. Several climate models will be developed to study present adverse climate situations and innovate solutions to control the damaging effects on the climate.

9. Education for all

AI has remarkable plans to improve the quality of education by enhancing the overall learning system. The rise of AI-enabled tutors and the development of educational content will call for agility and adaptability among students and learners.

10. Ethical AI

As generations become more used to the usage of artificial intelligence, ethical concerns are bound to emanate. The focus on privacy policies and training data bias will only increase to a mountainous level.

11. Bank on AI

Taking control of financial data enables users to gain control over their expenses and the opportunity to make the best use of their money. A plethora of financial services will be offered to customers with the help of an AI-driven Open Banking system using the personalized data of individual users. This would be beneficial for financial institutions as well as consumers.

12. Precision Agriculture

Optimizing agricultural practices using AI is a no-brainer. Farmers and agricultural professionals are able to make informed decisions for their crops and land using data derived from AI models. The year 2024 will witness unprecedented growth in the agriculture sector with huge contributions from AI-enabled data studies.

Wrapping it up

Generative AI is poised to revolutionize industries and every aspect of our lives in 2024 and the coming years. Be it custom-tailored content to climate change mitigation to learning materials for students to medical innovation for patient care – generative AI applications are here to provide efficient and robust solutions. It is important for businesses and individuals to embrace this growing technology being wary of the risks involved and addressing them in collaboration.

Companies, more often, pay attention to automation and innovation over proficiency and productivity. However, firms can maintain a balance between both due to the extensive usage of AI and data science programs.

Here are the stats that show the impact of AI and data science in diverse sectors:

• By the end of 2023, over half of US-based healthcare providers intend to implement AI tools like RPA in their healthcare facilities, according to Gartner.
• It is estimated that by 2030, there will be 13.7 billion self-driving cars on the road, up from 20.3 million in 2021. By 2030, 10% of all vehicles are expected to be driverless.
• The healthcare industry’s market for big data analytics might be worth $67.82 billion by 2025.
• According to Statista Research Department, 68% of global travel brands made sizable investments in business intelligence as well as predictive analytics capabilities in 2019.

Applications of AI and data science have created a standardized method for executing business functions faster and more effectively. Thus, making a career in data science and AI is beyond just rewarding.

Additionally, by actively participating in decision-making, customer engagement, market research, product innovation, and marketing strategies, it has deeply sunk itself within the organization.

This article will help you know the future of AI and data science, with a focus on the trends that will dominate the sector in the next decade.

Introduction to artificial intelligence and data science

Both AI and data science have emerged as the most in-demand fields that have totally simplified the working of employees and helped organizations to become more productive. Let’s know about them more in-depth below:

Data science

Data science is the process of extracting raw data by applying the right mathematical formulas and scientific methods and turning them into structured data.

It utilizes several tools and techniques to get more business insights and convert them into actionable solutions. Choosing a data science career means you need to perform steps such as data mining, data cleansing, data manipulation, data aggregation, and data analysis.

Artificial intelligence (AI)

The theory as well as the development of computer systems that can carry out tasks that ordinarily require human intelligence is known as artificial intelligence.

AI is the subset of data science that is frequently considered the proxy for the human brain. It utilizes smart systems to provide business process automation, productivity, and efficiency. Here are a few real-life AI applications:

• Voice assistance
• Chatbots
• Automated recommendations
• Image recognition

As a result, many sectors that have incorporated AI and data science are now reaping the benefits that we will cover in the upcoming section.

Advantages of integrating AI and data science

Since the integration of AI and data science, various parts of society have changed, including everything from grocery shopping to using public transportation to commute.

Below, we have listed a few benefits triggered by the integration of AI and data science:

• Automation of exhaustive human tasks has assisted the workforce to focus on different functions.
• Boosts productivity and efficiency in Insurance, Healthcare, Marketing, Pharma, and various industries.
• Innovation systems to connect with consumers and analyze their requirements.
• Predicting disasters beforehand.
• Reduces human errors.

Future of data science

From the data explosion to the expansion of the Internet of Things (IoT) and social media, the future of data science is predicted to witness a few big innovations in the previous ten years.

According to experts, the advent of machines will result in growth in usage, and the utility of computer systems will increase in the next decade.

In addition, experts claim that social media use will increase with users using large amounts of data online. Social media will be used by consumers for business, entertainment, etc. As per some analysts, machine learning algorithms will also experience a steep rise.

Future of artificial intelligence

Artificial Intelligence enables the machine to act like a human brain. It carries out several business functions without the need for human intervention such as consumer interaction and raising brand awareness on social media.

Many researchers think AI will surpass humans in nearly all cognitive tasks. By automating tasks like managing employee or patient records, doing market research, and engaging with potential clients, among others, AI applications are revolutionizing diverse sectors.

Now that you are aware of the future of data science and AI systems, we will look at how to develop a successful career in the respective fields.

Path to build a career in data science

Businesses are mining an abundance of data and turning it into useful information. They have data scientists working for them.

Jobs in this field are plentiful as a result of the rising need for data science experts. Data engineer, data analyst, etc. are some of the job roles for which recruiters are hiring.

With each passing day, there is a greater need for a professional and skilled data scientist. All you can do is earn a bachelor’s or master’s degree in the data science field. Or you can consider taking data science certification programs to grow your skillset.

Path to build a career in AI

The rapid use of AI applications across several industries has created a wealth of artificial intelligence applications. So, a career in AI seems potential. The following list of popular AI job positions includes Engineers in big data, AI data analysis, and machine learning.

If you want to get into the AI field, the ideal way is to enroll in a good AI certification course that would help you gain the abilities required to manage challenging AI-related activities effectively.

Which field is the best career for you?

This is your life. So, it’s important to make the right career choice that will, in the future, surely bring you enthusiasm, and encourage you to follow your passion to succeed in a field you pick. After all, when you work dedicatedly and enjoy performing your tasks, you learn and attract more new opportunities.

Image by Author

A few months ago, we learnt about Falcon LLM, which was founded by the Technology Innovation Institute (TII), a company part of the Abu Dhabi Government’s Advanced Technology Research Council. Fast forward a few months, they’ve just got even bigger and better — literally, so much bigger.

Falcon 180B: All You Need to Know

Falcon 180B is the largest openly available language model, with 180 billion parameters. Yes, that’s right, you read correctly — 180 billion. It was trained on 3.5 trillion tokens using TII's RefinedWeb dataset. This represents the longest single-epoch pre-training for an open model.

But it’s not just about the size of the model that we’re going to focus on here, it’s also about the power and potential behind it. Falcon 180B is creating new standards with Large language models (LLMs) when it comes to capabilities.

The models that are available:

• Falcon 180B
• Falcon 180B Chat

The Falcon-180B Base model is a causal decoder-only model. I would recommend using this model for further fine-tuning your own data.

The Falcon-180B-Chat model has similarities to the base version but goes in a bit deeper by fine-tuning using a mix of Ultrachat, Platypus, and Airoboros instruction (chat) datasets.

Training

Falcon 180B scaled up for its predecessor Falcon 40B, with new capabilities such as multiquery attention for enhanced scalability. The model used 4096 GPUs on Amazon SageMaker and was trained on 3.5 trillion tokens. This is roughly around 7,000,000 GPU hours. This means that Falcon 180B is 2.5x faster than LLMs such as Llama 2 and was trained on 4x more computing.

Wow, that’s a lot.

Data

The dataset used for Falcon 180B was predominantly sourced (85%) from RefinedWeb, as well as being trained on a mix of curated data such as technical papers, conversations, and some elements of code.

Benchmark

The part you all want to know — how is Falcon 180B doing amongst its competitors?

Falcon 180B is currently the best openly released LLM to date (September 2023). It has been shown to outperform Llama 2 70B and OpenAI’s GPT-3.5 on MMLU. It typically sits somewhere between GPT 3.5 and GPT 4.

Image by HuggingFace Falcon 180B

Falcon 180B ranked 68.74 on the Hugging Face Leaderboard, making it the highest-scoring openly released pre-trained LLM where it surpassed Meta’s LLaMA 2 which was at 67.35.

How to use Falcon 180B?

For the developer and natural language processing (NLP) enthusiasts out there, Falcon 180B is available on the Hugging Face ecosystem, starting with Transformers version 4.33.

However, as you can imagine due to the model’s size, you will need to take into consideration hardware requirements. To get a better understanding of the hardware requirements, HuggingFace ran tests needed to run the model for different use cases, as shown in the image below:

Image by HuggingFace Falcon 180B

If you would like to give it a test and play around with it, you can try out Falcon 180B through the demo by clicking on this link: Falcon 180B Demo.

Falcon 180B vs ChatGPT

The model has some serious hardware requirements which are not easily accessible to everybody. However, based on other people's findings on testing both Falcon 180B against ChatGPT by asking them the same questions, ChatGPT took the win.

It performed well on code generation, however, it needs a boost on text extraction and summarization.

Wrapping it up

If you’ve had a chance to play around with it, let us know what your findings were against other LLMs. Is Falcon 180B worth all the hype that’s around it as it is currently the largest publicly available model on the Hugging Face model hub?

Well, it seems to be as it has shown to be at the top of the charts for open-access models, and models like PaLM-2, a run for their money. We’ll find out sooner or later.
Nisha Arya is a Data Scientist, Freelance Technical Writer and Community Manager at KDnuggets. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.

Nisha Arya is a Data Scientist and Freelance Technical Writer. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.

More On This Topic

• Falcon LLM: The New King of Open-Source LLMs
• Want to Use Your Data Skills to Solve Global Problems? Here’s What You Need…
• I Used ChatGPT (Every Day) for 5 Months. Here Are Some Hidden Gems That…
• Here’s Why You Need Python Skills as a Machine Learning Engineer
• PyCaret 2.3.5 Is Here! Learn What’s New
• Here Are the AI Tools I Use Along With My Skills to Make $10,000 Monthly —…

Image by Author

In response to changing technological, organizational, and business needs, data architecture has evolved over the last decade or so. But has this evolution been significant enough? Most organizations typically have a centralized data architecture. Which, by design, consolidates data under a single umbrella, often managed by a dedicated data team.

While effective in ensuring security and better governance, centralized data architecture has its limitations in terms of scalability, flexibility, and accessibility amongst others.

Enter Data Mesh, a concept (almost) analogous to microservices in software architecture. Data Mesh aims to decentralize data management just the way microservices focus on decentralizing application components. It distributes data ownership and accountability among domain-specific teams, acknowledging data as a strategic asset, best managed at its source.

In this article, we'll explore Data Mesh, its key principles, factors to consider, and challenges associated with the adoption of a data mesh architecture.

What Is a Data Mesh?

The concept of a Data Mesh was first introduced by Zhamak Dehghani, in the article "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh" which outlines the principles and concepts behind the data mesh. This article and subsequent discussions within the data communities played a significant role in popularizing the data mesh architecture.

A Data Mesh is a contemporary approach to data architecture and management that departs from traditional centralized data models. It introduces a decentralized structure for organizing, distributing, and utilizing an organization's data assets.

In a data mesh, data ownership and responsibilities are distributed among domain-specific teams or data product teams, granting them autonomy in managing their data within their respective domains.

This decentralized approach aims to address the limitations associated with centralized data models, such as scalability challenges, data silos, and slow response times to changing data needs. By empowering domain-specific teams to independently manage their data, a data mesh promotes a culture of data autonomy, agility, and accountability within an organization. It also the efficient handling of diverse data sources while maintaining a focus on data quality and relevance.

Key Principles in the Data Mesh Architecture

Data Mesh architecture is built upon a set of principles designed to address the challenges of scaling and managing data within and across organizations. These principles provide a foundation for a decentralized and more scalable approach to data management.

Image by Author

Domain-Oriented Ownership

In a data mesh, data ownership is decentralized and distributed among various domains or business units within the organization. Each domain is responsible for the data generated and used within its specific area of expertise or functionality. This principle recognizes that domain experts are best equipped to understand and manage the data within their respective domains.

Domain-oriented ownership improves data quality and accuracy because those closest to the data source have a deep understanding of its context and can ensure its integrity. It also promotes a sense of ownership and responsibility for data, encouraging domain teams to maintain high data standards.

Data as a Product

Data in a data mesh is treated as a product rather than a byproduct of business operations. Each domain is responsible for delivering well-defined data products that are designed, packaged, and made available for consumption by other domains within the organization. These data products have clear definitions, access mechanisms, and service-level agreements (SLAs).

Treating data as a product encourages data producers to focus on delivering high-quality and valuable data to consumers. It also ensures that data products are designed with user needs in mind, making data more accessible and usable for a broader range of stakeholders.

Self-Serve Data Infrastructure

Data Mesh promotes the development of self-serve data infrastructure that empowers data consumers such as data analysts, data scientists, business users to access and process data independently. This infrastructure includes data catalogs, data discovery mechanisms, and data processing pipelines that enable consumers to find, understand, and utilize data without heavy reliance on centralized data engineering teams.

Self-serve data infrastructure reduces bottlenecks and accelerates data access empowering a broader range of users to work with data. It democratizes data within the organization, making it more accessible and enabling faster insights and decision-making.

Federated Computational Governance

To maintain data quality, security, and compliance in a decentralized data architecture, data mesh employs federated computational governance. Each domain defines and enforces its own governance policies tailored to the specific needs of its data. While there may be global standards and guidelines, individual domains have the autonomy to govern their data assets.

This balances the need for global data standards with the flexibility required by individual domains. It allows domains to adapt governance practices to their unique data challenges while ensuring that data remains secure, compliant, and of high quality.

These four key principles of data mesh, therefore, collectively aim to address the challenges of scaling data operations in large organizations by promoting:

• decentralization,
• data product thinking,
• self-service, and
• effective governance.

By implementing these principles, organizations can unlock the full potential of their data assets, improve collaboration between domain teams, and make data a more valuable and accessible resource for all stakeholders.

Implementing a Data Mesh? Here Are Factors to Consider

Transitioning to a data mesh often involves a significant cultural shift within an organization. A data mesh encourages collaboration, shared ownership, and data product thinking, aligning data practices more closely with the organization's evolving culture and values. Here are some factors that organizations might consider when implementing a data mesh.

Business Goals and Strategy

Any major shift in data architecture should align with the organization's broader business goals and strategic objectives.

Implementing a data mesh should be seen as a strategic enabler, enhancing the organization's ability to leverage data effectively to achieve its overall goals and objectives.

Existing Infrastructure

Organizations must evaluate and consider their current data infrastructure and investments when evaluating the feasibility of a data mesh.

Transitioning to a data mesh may require adjustments to the existing technology stack and infrastructure, making it essential to align these aspects with the new approach.

Data Complexity and Scale

When organizations face growing data complexity and scale, they must consider alternative data management approaches. A data mesh offers scalability and adaptability, especially when dealing with increasingly complex and large-scale data environments.

So a data mesh is a good choice when the volume, variety, or velocity of data makes it difficult to manage centrally, or when data requirements are diverse across different business units or domains.

Data Governance and Compliance

Maintaining data quality, privacy, security, and compliance is a challenging aspect of data management, particularly in decentralized environments.

A data mesh strategy must address these complexities effectively, ensuring data governance practices and regulatory requirements are met.

Data Accessibility and Ownership

In organizations with distributed data sources and diverse domains, traditional centralized data management may not suffice. Implementing a data mesh aligns data ownership with domain-specific teams, empowering them to take responsibility for their data, which can be particularly valuable in such environments.

Also, to facilitate data-driven decision-making throughout the organization, it's crucial to make data more accessible. A data mesh democratizes data access, allowing a wider range of users to access and utilize data, leading to improved decision-making across various departments or teams.

Challenges in Adopting a Data Mesh Architecture

Moving from a centralized data architecture to a data mesh is not without challenges. In this section, we delve into some of them—from governance to monitoring.

Data Governance

In a data mesh, data governance becomes more complex because data is distributed across multiple domains and teams. Ensuring consistent data quality, privacy, security, and compliance standards across these domains can be challenging:

• Establishing clear data ownership and responsibility for data governance tasks, such as defining data schemas and access controls, can be a challenge when multiple teams are involved.
• Developing and enforcing data governance policies and practices that align with the decentralized nature of a data mesh requires careful planning.

Data Discoverability

In a decentralized data mesh, discovering and accessing data can be challenging. Ensuring that data is properly cataloged, tagged, and documented is essential for enabling data discoverability. Here are some strategies:

• Implementing effective metadata management practices to provide context and descriptions for datasets, making it easier for users to understand the available data resources.
• Developing and maintaining a data catalog or metadata repository that allows users to search for and find relevant datasets efficiently.

Data Ownership

A clear and consistent definition of data ownership and accountability for each data domain and data product is crucial in a data mesh. Determining who is responsible for maintaining, updating, and curating the data can be challenging, especially when there are multiple stakeholders. Organizations can address this challenge by:

• Ensuring that data owners have the necessary authority and resources to manage their data domains effectively.
• Establishing mechanisms for resolving conflicts or disputes related to data ownership and responsibilities.

Monitoring and Observability

In a data mesh, monitoring the health, performance, and reliability of data pipelines and data products can be complex. Some strategies include:

• Implementing robust monitoring and observability tools and practices to track data quality, latency, and usage across different domains.
• Developing alerting and reporting mechanisms to quickly identify and address issues that may affect data availability or reliability.

We’ve highlighted some challenges in the implementation of a data mesh. These are more of checkpoints that organizations should be aware of when moving to a decentralized data mesh architecture.

Conclusion

Data Mesh, therefore, is a paradigm shift in data architecture, offering solutions to the challenges of centralized models. We discussed how distributing data ownership, promoting data product thinking, and enabling self-service access are beneficial. However, successful implementation requires careful consideration of cultural and technological factors, and a proactive approach to data governance.
Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she's working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.

Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she's working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.

More On This Topic

• New Computing Paradigm for AI: Processing-in-Memory (PIM) Architecture
• Data Mesh & Its Distributed Data Architecture
• KDnuggets™ News 22:n07, Feb 16: How to Learn Math for Machine…
• Data Mesh Architecture: Reimagining Data Management
• KDnuggets News, May 18: 5 Free Hosting Platform For Machine Learning…
• Opening Keynote Speaker for Big Data London (21-22 September 2022)…

Generative AI represents a paradigm shift in the way we approach creativity and innovation. However, given banking is one of the most highly regulated industries, generative AI adoption could be tricky. Srikanth Gopalakrishnan Head of the India Technology Centre at Deutsche Bank Group, during the ongoing Cypher2023 event, India’s largest AI conference, said that we should tread carefully as we venture into this new frontier, taking into account the ethical challenges it poses and strive for a measured and responsible adoption.

As we leverage the capabilities of generative AI, all the while safeguarding the unique attributes of human creativity, we can unlock a future where technology and human ingenuity harmoniously coexist. This harmonious coexistence, according to Gopalakrishnan, promises a more dynamic and enhanced society.

“In regulated industries like banking or the medical industry, there is a lot of homework that has been done and a lot of R&D that’s being done in order for us to get to where we think the benefits of AI are going to be good,” Gopalakrishnan said.

Explainability is key

Gopalakrishnan said for banks to leverage this technology, they need to be 100 percent sure that the technology works. Given Large Language Models (LLMs) still hallucinate, leveraging the technology for customer-facing domains could prove to be tricky.

“You can’t just say invest in the stock and leave it at that. The ‘trust me’ is not going to work. The ‘trust me’ must also have explainability. And that is why certain industries (like banking) will have to be extra careful about how to take this forward in a viable fashion.

“We still need explainability and we’ve been talking to Google, NVIDIA and others, and they are all working on essentially marking up the explainability aspect within the code that they have. First of all, it needs to give confidence to us internally, give confidence to customers and the regulators as well,” he said.

However, he feels there are a few areas in banking where generative AI can be adopted without much to worry about regulation. Customer service is one of those areas, Gopalakrishnan said.

“I believe the volume of business can actually go up in terms of how to actually interact and reach out to customers. We are talking about bots being available, personalised responses being provided.”

Other areas where banks can leverage generative AI are risk management and fraud detection.

Future-proofing of jobs

Moreover, in this talk, Gopalakrishnan also touches upon the ongoing discussion of the impact of generative on jobs.

“We have a large number of graduates joining us every year and I have been talking to them; because someone who is relatively new to the industry, they might have to redefine themselves at least three to four times in their career,” he said.

“So it’s important for us to start understanding that when you are talking about AI, and the formation that will bring about the traditional jobs, repetitive tasks are going to go away, in fact, they are already going away.”

Even jobs that require a certain level of intelligence from a human to be able to make a decision will also probably go away, Gopalakrishnan said.

“However, people’s jobs’ will not go away in the sense that you will be out of a job.” The ability to learn and relearn is essential, regardless of career stage.

The post Cypher2023: Are Banks Ready for Generative AI? appeared first on Analytics India Magazine.