Data Maturity: The Cornerstone of AI-Enabled Innovation

Data Maturity: The Cornerstone of AI-Enabled Innovation
Photo by Google DeepMind

In the relentless pursuit of innovation and securing a competitive advantage, businesses are progressively harnessing the power of artificial intelligence (AI) as a transformative tool. The promise of AI to streamline operations, elevate decision-making processes, and unveil concealed patterns within data has spurred its swift integration across industries, especially in retail, manufacturing, and distribution.

However, despite the compelling possibilities, achieving the maximum benefits of AI hinges on a robust groundwork of data maturity. Unfortunately, numerous enterprises encounter challenges in attaining this maturity due to various factors. These challenges often encompass:

  • Fragmented data silos
  • Poor data quality
  • Limited transparency about data assets and skills
  • Organizational inertia toward reestablishing a balance between technology as an enabler vs. provider of business data needs

In this article, I will highlight prescriptive strategies to overcome these challenges toward establishing a robust data foundation for scaling differentiated AI capabilities.

The State of AI Today as Industry Leaders Forging Ahead

Retail, manufacturing, and distribution leaders use AI's power to achieve remarkable results, from optimizing supply chains to predicting customer behavior. Generative AI is gaining mainstream traction. A recent Fortune/Deloitte CEO survey found widespread CEO interest in the potential of generative AI. In a recent survey, 79% of chief executive officers expressed optimism in technology's potential to enhance operational efficiency, with over half anticipating the emergence of new avenues for growth. A substantial portion revealed ongoing efforts in evaluating and experimenting with generative AI, underscoring a proactive approach to harnessing cutting-edge advancements in the business landscape.

Industry leaders with the highest levels of AI maturity have demonstrated differentiated capabilities to drive sales and optimize operations. For instance, Amazon's AI-powered recommendation engine, which suggests products based on a customer's past purchases and browsing history, has been instrumental in driving sales. Similarly, Walmart has successfully used AI algorithms for inventory management and demand forecasting, which means the retail giant is deploying AI to ensure that products are available when and where its customers need them.

Yet, according to Gartner's AI Maturity Model, 52% of mid-to-large US organizations are still experimenting with AI.

Data Maturity: The Cornerstone of AI-Enabled Innovation

Chief data officers who play an essential role in adopting AI and supporting digital transformation and are responsible for data strategy and governance within organizations see data quality as one of the largest obstacles to fully leveraging AI capabilities, according to a recent AWS Survey of over 300 CDOs.

Let's look at the data maturity challenges that impact AI adoption and how to overcome them.

Data Maturity: The Missing Link to AI Scalability

Despite the undeniable potential of AI, many enterprises need help scaling AI-enabled use cases due to data-related hurdles. As organizations embark on ambitious AI initiatives, they often encounter significant roadblocks that hinder timely implementation and widespread adoption. Organizations must prioritize data maturity to navigate these challenges and fully realize the potential of AI.

Data maturity refers to an organization's ability to effectively manage, govern, and utilize its data assets. It encompasses data quality, governance, integration, and analytics capabilities. A lack of data maturity can lead to several challenges that hinder AI adoption and scalability, such as:

  • Data Silos and Fragmentation: Data scattered across disparate systems and formats creates data silos, which can prevent holistic utilization company-wide.
  • Data Quality Issues: Inaccurate, incomplete, or inconsistent data can lead to flawed AI models and unreliable insights.
  • Data Governance Gaps: Without proper data governance practices, businesses can face issues related to data security, privacy, and compliance.
  • Limited Data Analytics Capabilities: The inability to extract meaningful insights from data can hamper the development and application of AI.

These challenges underscore the critical role of data maturity in enabling AI scalability. To overcome these hurdles, enterprises must adopt a comprehensive data management and governance approach.

Prescriptive Strategies to Overcome Key Challenges

DataArt offers enterprises comprehensive strategies and solutions to augment data maturity. We propel our partners toward a software ecosystem where data is democratized, agile, and purpose-driven, surmounting obstacles hindering AI adoption. By fostering a culture of data ownership, empowerment, and innovation, enterprises are better positioned to harness AI's transformative potential and drive scalable, AI-enabled use cases, placing themselves at the cutting edge of a future defined by data-driven excellence and sustained growth.

The Confluence of Data Mesh and Data Product

The emergence of the Data Mesh and Data Product strategies heralds a transformative paradigm shift in the global economy. Data Mesh, a novel architectural approach, advocates decentralizing data ownership and management, fostering domain-driven data architecture across a single enterprise. This strategy aims to alleviate the bottlenecks of centralized data lakes or warehouses by distributing data ownership to domain-specific teams. Through this work of data distribution, Data Mesh empowers teams to curate, own, and evolve their data products, promoting agility and scalability while maintaining data governance and quality.

Data Maturity: The Cornerstone of AI-Enabled Innovation
Figure 1: Data Mesh framework to enable fast value realization through business domain-driven data products.

Simultaneously, the Data Product strategy further solidifies the foundation for AI scalability. It champions the conceptualization, creation, and management of data as products that cater to specific user needs within an organization. Each data product encapsulates valuable insights, prepared datasets, or analytical tools tailored for consumption by diverse stakeholders. This approach fosters a culture of data ownership and empowers teams to innovate, collaborate, and derive actionable insights from curated data products, accelerating AI adoption.

For example, a customer segmentation analytical data product can be further used to create churn data products, and both can be used for marketing purposes to generate hyper-personalized content for customers. Without a data product or data product marketplace, teams would have to spend time building these analytical capabilities from scratch. Instead, each new use case can reuse and repurpose existing data products, reducing development time and producing more consistent outputs.

Data Democratization and Effective Data Governance

As companies across industries seek more effective ways to manage their data, several factors must be carefully considered. Data democratization involves making data accessible and understandable to stakeholders, like data scientists, business analysts, domain experts, management, and executives. Additionally, companies must ensure their data is readily available, legible but also secure, and compliant, with transparent standards and controls. Implementing the correct security and compliance measures will help businesses safeguard data integrity, privacy, and regulatory adherence.

This evolution represents a sea change in how organizations harness data. Historically, IT departments were responsible for building their company's data-related modules, like warehouses and analytical data products. It can become a technology facilitator rather than solely controlling data access and provisioning by implementing an AI-powered approach to data democratization. With a deployed AI-powered system, IT can focus its resources on empowering users to independently navigate and derive insights from their company's data. Enabling this transition requires a fundamental shift in IT's role, moving from gatekeepers to partners in fostering collaboration and innovation.

Data curation plays a pivotal role in ensuring the quality, relevance, and usability of data assets within an organization. However, maintaining it is often a challenge due to the sheer volume and variety of data sources, functional silos, and manual effort. This is one of the areas that can be improved with AI. AI-driven tools and algorithms can automate data processing tasks, enabling faster curation, data cleaning, and normalization, reducing manual efforts. AI algorithms can recognize patterns within data and contextualize information, facilitating more accurate curation and categorization.

By adopting and implementing these strategies, enterprises can establish a strong data maturity foundation, enabling them to harness AI's power effectively and scale AI-enabled use cases across their businesses. In addition, DataArt can help companies establish or improve core foundational capabilities that connect technology, people, and processes, such as:

  • Breaking Down Data Silos: Integrating data from disparate sources into a centralized repository, ensuring data consistency and accessibility.
  • Establishing Data Governance: Implementing a framework that defines data ownership, access controls, data quality standards, and data usage policies.
  • Enhancing Data Quality: Implementing data quality checks, cleansing processes, and enrichment techniques to improve data accuracy and completeness.
  • Fostering Data Literacy: Training employees on data management principles, data analysis techniques, and data-driven decision-making to enhance organizational data utilization.
  • Investing in Data Infrastructure: Upgrading data infrastructure to handle the growing volume, velocity, and variety of data, ensuring efficient data storage, processing, and analysis.
  • Embracing DataOps: Implementing DataOps practices to automate data management processes, enabling rapid data delivery and continuous improvement.
  • Leveraging Cloud-Based Data Solutions: Utilizing cloud-based data platforms to gain scalability, flexibility, and cost-efficiency in data management.
  • Continuous Monitoring and Improvement: Monitoring data quality, governance compliance, and usage patterns to identify and address emerging challenges.

Conclusion

Data maturity is not just a technical requirement; it is a strategic imperative for enterprises seeking to unlock the transformative potential of AI. By addressing the critical challenges associated with data maturity, enterprises can pave the way for a future shaped by data-driven insights and AI-powered innovation.

Oleg Royz is a Vice President, Retail and Distribution at DataArt.

More On This Topic

  • The Data Maturity Pyramid: From Reporting to a Proactive…
  • Discover What It Takes to Scale Innovation & Data Science
  • Learn Data Science and Business Analytics to Drive Innovation and Growth
  • AI + No-Code: The Viral Combo Redefining Developer Innovation
  • A Microsoft Engineer's Guide to AI Innovation and Leadership
  • Navigating Data Science Job Titles: Data Analyst vs. Data Scientist…

Hugging Face Launches Chat Assistants, Takes on OpenAI’s GPT Store

Hugging Face, open-source hub for AI models, has unveiled a new feature – the Hugging Chat Assistant. This latest addition allows users to effortlessly create their personalised chat assistant in just two clicks, akin to the functionality seen in OpenAI’s GPT models.

Introducing Hugging Chat Assistant! 🤵 Build your own personal Assistant in Hugging Face Chat in 2 clicks! Similar to @OpenAI GPTs, you can now create custom versions of @huggingface Chat! 🤯
An Assistant is defined by
🏷️ Name, Avatar, and Description
🧠 Any available open… pic.twitter.com/9XaReKgg9m

— Philipp Schmid (@_philschmid) February 2, 2024

Similar to OpenAI’s GPTs, Hugging Face’s Chat Assistants enable users to craft custom versions of the chat interface, offering a range of customisation options. An Assistant on the platform is characterised by its name, avatar, and description, along with the flexibility to choose from various open-source language models such as Llama2 or Mistral.

What sets Hugging Face’s Chat Assistants apart is their open-source nature, providing users with the freedom to utilise and share the feature without any subscription fee, in contrast to the $20 subscription charged by OpenAI for ChatGPT Plus.

However, Compared to OpenAI’s custom GPTs, it’s still missing RAG, web search, actions, and a GPT builder.This puts it below the custom GPT level for now, but it’s all in the roadmap for the open-source competitor.

Ten days ago I posted about GPT Store being a bit sad 😢:
What if we could build an open source alternative, with the full power of the Community?
So last Friday we launched Hugging Chat Assistants, and the adoption has been impressive:
– 4,000 Assistants have been created on… pic.twitter.com/X7Duqc40ss

— Julien Chaumond (@julien_c) February 5, 2024

In just a few days since its launch, the Hugging Face Chat Assistants platform has rapidly grown, boasting over 4,000 Assistants. Furthermore, Hugging Face plans to introduce seven additional open-source stores in the near future, promising further expansion and diversity in the available options for users.

The post Hugging Face Launches Chat Assistants, Takes on OpenAI’s GPT Store appeared first on Analytics India Magazine.

AI chip company TensTorrent partners with C-DAC for Digital India’s futureLABS

Tenstorrent Inc., a leading AI computing company, has revealed its collaboration with C-DAC in alignment with the Digital India futureLABS initiative, particularly focusing on RISC-V compute and design & innovation ecosystem.

This partnership signifies Tenstorrent’s commitment to advancing the semiconductor industry’s progress, aligning with the Government of India’s objectives.

Digital India futureLABS, is conceived to move up the value-chain & fortify domestic R&D by creating a collaborative ecosystem for development of IPs, Standards and catalyzing next-generation Electronics System Design ecosystem in the country by enabling co-development opportunities & partnerships amongst Government, Startups, R&D organisations and large enterprises.

Digital India futureLABS will be coordinated by C-DAC to provide the dedicated thrust to following key growth areas- Automotive, Compute, Communication, Strategic Electronics and Industrial Electronics/ IoT.

“India is witnessing immense growth. What India is going to do over the next five to ten years is going to set the tone for the future. Tenstorrents’ chiplet ecosystems, open hardware platforms, RISC-V and artificial intelligence (AI) are a perfect match to reduce the cost of manufacturing at scale.

“It is the right time for India to take advantage of the technology offerings from Tenstorrent and disrupt the status quo and incumbents as it catapults to the numero uno position. We are very happy to be part of this innovation and looking forward to the journey,” Keith Witek, Chief Operating Officer at Tenstorrent said.

The post AI chip company TensTorrent partners with C-DAC for Digital India’s futureLABS appeared first on Analytics India Magazine.

The Surge in Tech Layoffs 2024: Who’s to Blame?

The Surge in Tech Layoffs 2024: Who’s to Blame?
Image by Unsplash

A month into 2024 and I’ve never seen my LinkedIn profile so busy with ‘Open to Work’ banners and posts. 2023 was already tough, with a lot of companies announcing layoffs, and what’s even sadder is that the layoffs were quite brutal. Some were told ‘You’re fired, see you later’, and others weren’t even paid what they were owed — it was all a mess.

They say the start of 2024 always has the remaining 2023 energy in it — so the layoffs in 2024 continue.

Naturally seeing tech layoffs will make you very uncomfortable. You start to feel unsure about the security of your job and if you should do the smart thing and start looking for something new. But then the thought of entering another tech company who may restructure or go bust too.

Yes, your head is probably spinning on what to do.

The Reason Behind the Tech Layoffs

According to Crunchbase, at the end of Jan. 19, 2024, at least 2,215 U.S. tech sector employees were laid off. In 2024 as a whole. at least 6,505 workers at U.S.-based tech companies have lost their jobs so far in the year — and it’s just started!

So what is going on?

The Tech Pandemic

During the Covid-19 pandemic, the tech industry was booming like crazy! You saw every person and their dog transitioning into the tech industry, with very generous salaries and nice benefit packages. Jobs were opening up left, right and centre and people were loving it — they finally found their dream job!

People were stuck at home, WFH became the new best thing, and companies had more money as they were not spending so much on office space, team meetings, etc. The hiring spree starts, with companies loving the fact that their team is expanding!

The tech hiring spree was short-lived with people getting laid off in 2022, 2023, and now 2024.

365 data science did some digging using data analysis to figure out the root cause of all the tech layoffs, and this is what they found:

The Surge in Tech Layoffs 2024: Who’s to Blame?
Image by 365 data science

As you can see from the visualization, the majority of layoffs were within HR and Talent Resourcing — which is no shock as at one point they were very much needed for the hiring spree. The second majority was software engineers, then marketing, and so on.

The data also showed that between 2022-2023, 56% of layoffs were women, 48% were aged between 30-40, 89% were US-based, and 60% of them held a bachelor's degree.

How do you win right?

Is AI Taking Over?

It’s what we’re all thinking. It can’t be a coincidence that in the past few years, AI has really taken off and people are starting to lose their jobs. Or is it?

Take your local supermarket for example, in the past few months I have seen more automated self-checkouts than I have ever before. The till sections have been heavily reduced and replaced with self-checkout tills.

If we go back to 365datascience’s findings, the majority of layoffs were within HR and Talent Resourcing. Some of the day-to-day tasks within these sectors have been heavily replaced with automation. Companies are integrating a lot of tools and software that are being used to handle manual tasks done within HR. Is this the reason why HR and Talent Resourcing was at the top of the list?

But what does that mean for the other sectors? Why are they getting laid off?

Personally, I believe the other sectors are dealing with the aftermath of the tech hiring spree during the pandemic and as we are going through the possibilities of a recession, companies are struggling to keep financially afloat. This means all those who were hired during the tech hiring spree or are too expensive to keep have to be let go.

Is My Company Going to Do Layoffs?

If you’re working in the tech industry, you’re probably worried about the security of your job — it’s very normal and I don’t blame you. But you can always protect yourself and this is how.

If your company has announced a hiring freeze across the company, this is normally a good indication that the company is either financially struggling or just about to make ends meet.

Another indication is your workload. If you’ve started to see your workload heavily decrease and projects getting cancelled. This is something that a lot of people don’t initially pick up on at first, but the truth is that C-suite members don’t have the time or finances to start new projects as they are trying to put together all the pieces of the future of the company and how to keep it going.

I urge you to take this with a pinch of salt as these reasons may not highly indicate that the company is going through a rough time and layoffs are pending. It could also mean that the company is dealing with other things that they are prioritizing.

Wrapping it up

It’s a tough time for tech employees at the moment and with that being said, I think every tech employee should remind themselves that these tech layoffs are not personal and unfortunately out of our reach. These tech layoffs have included the most experienced members, and they were still laid off.

If you are feeling really unsure, the advice I would give is always upskill to prepare yourself for the worst and always have a plan B if your company is showing signs of financial difficulty.

Nisha Arya is a Data Scientist and Freelance Technical Writer. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.

More On This Topic

  • What's With All the Layoffs in Tech?
  • How a Polytechnic Helps You Make the Tech-Business Connection
  • DATAcated Expo, Oct 5, Live-streamed,Explore new AI / Data Science Tech
  • Data science SQL interview questions from top tech firms
  • Building Tech Skills in 2021
  • Baidu Research Unveils Top 10 Tech Trends Forecast for 2022

Despite global frenzy, investor enthusiasm in China’s AI startups wanes

Despite global frenzy, investor enthusiasm in China’s AI startups wanes Rita Liao 10 hours

In 2023, the global frenzy around artificial intelligence, fueled by the advent of ChatGPT, swept across the world. In China, where OpenAI’s chatbot is unavailable, startups and tech incumbents scrambled to develop their own AI models and applications, drawing upon the foundational pieces of the American upstart. Individual AI fans accessed ChatGPT through a web of black market vendors, keeping their accounts alive through often unauthorized virtual private networks.

At a glance, generative AI is in full bloom in China, but a closer look suggests otherwise. Despite the hype, venture capitalists haven’t been as enthusiastic about the nascent technology as one might assume.

In 2023, China recorded around 232 investments in the AI space, a 38% decline year-over-year, according to research firm CBInsight. The total amount raised by China’s AI firms amounted to roughly $2 billion, 70% less than the year before.

Another report by a Chinese database shows a greater amount of funding, though it indicates the same downward trend. According to ITJuzi, China recorded 530 funding events in AI during the first 11 months of 2023, a 26% drop year-over-year. Those investments earmarked 63.1 billion yuan ($8.77 billion) in total, 38% less than the previous year and significantly smaller than 2021’s peak of 248.78 billion yuan.

The discrepancy in investment sizes between the two reports might be attributed to their different methodologies in counting funding rounds. ITJuzi might have a better grasp of local funding activities than its foreign counterpart, not least because China’s AI startups have become more discreet about their U.S. dollar financings. Many now fear U.S. regulatory scrutiny over American capital flows into their AI businesses.

Taking a broader view, the slowing in China’s AI funding isn’t entirely unexpected given the ongoing sluggishness of global VC investments. However, China’s AI startups face a unique set of roadblocks. American venture capital, which has historically been the major drive of growth in China’s internet sector, has plummeted since the onset of the U.S.-China decoupling. The prospect of listing Chinese tech firms on U.S. stock markets has also dimmed amid geopolitical tensions, so investors grew more cautious about backing hyped-up businesses with no clear exit channels or monetization plans.

Moreover, the capital-intensive nature of AI startups, which cost significant computing power, coupled with their unproven business models, can deter risk-averse local RMB funds.

A handful of Chinese AI startups with pedigreed founders, such as Wang Xiaochuan’s Baichuan and Kai-Fu Lee’s 01.AI, are still able to pull in handsome funding, but the majority of small players are faced with increasingly conservative investors. The mission of developing China’s equivalent of ChatGPT has fallen upon deep-pocketed tech giants who have been hoarding AI chips, while less resourceful startups explore niche industry applications built on open-source or the country’s homegrown models.

All the while, the technological prowess of China’s large language models remains in question as developers face a prolonged shortage of AI chips. Amid the escalating U.S.-China tech war, Washington placed an export ban on Nvidia’s high-end graphic processing units to China.

Internally, strengthened regulations have led to higher compliance costs for AI startups. Unlike their larger, better-funded counterparts, many startups lack the financial and bureaucratic resources to acquire the required AI license or meet the country’s internet censorship requirements. Some thus turn their focus to the global market, which introduces a different set of challenges. While regulatory and political uncertainty might be less prominent hindrances, these startups have to navigate new user behavior and an internet ecosystem completely cut off from their home market.

The adventurous AI firms might turn to foreign investors, most likely American ones, for financing and eventually help with go-to-market strategies. But before they can engage with American institutions, they need to have the appropriate corporate structure, offshore data storing solutions, and even foreign passports for their founders so the Silicon Valley investors won’t worry about violating U.S. restrictions on China-related investments.

With limited funding available, 2024 might be a year of reckoning for many AI startups in China.

Fate of US venture capital in China teeters on uncertainty

Learn To Leverage AI Art With This $30 Bundle

AI generated image of an abstract art resembling a face.
Image: StackCommerce

TL;DR: Learn how to create your own art and scale your business’s content output with The Complete Generative AI Art & Design Mastery Bundle, now just $29.99.

From marketing campaigns to logos, generative AI has made it easier for businesses and creators to scale their creative output to meet the demands of a faster-paced world. With The Complete Generative AI Art & Design Mastery Bundle, you’ll delve into some of the leading consumer-facing tools that are making it easier for anyone to access AI tools.

While you’re likely already familiar with ChatGPT, this 7-course bundle dives into the uses and benefits of working with AI art creators like Midjourney and DALL-E. You’ll familiarize yourself with these AI image generators and learn how to properly input prompts to get your desired results. You’ll learn how to work with the various settings in both Midjourney and DALL-E and understand how to set parameters to get different kinds of images.

Courses are taught by well-rated instructors from Skill Success, which has been featured by Entrepreneur, Mashable, CNET and other well-respected outlets.

You’ll also learn how to create AI art with Stable Infusion and Text2Image using text prompts and Img2img using existing images.

Through the courses, you’ll learn how to upscale resolution for images and videos and how to make on-the-fly edits to your AI art to make it truly unique and your own. And, of course, you’ll understand how to take all of your creations and put them into usable formats to share with the world. Whether you’re most interested in Midjourney, DALL-E or Stable Infusion, you’ll gain a litany of skills that will help you scale your artistic and design output to share with the masses.

Become an AI art expert with The Complete Generative AI Art & Design Mastery Bundle. For a limited time, you can get this bundle for 77% off $133 at just $29.99.

Prices and availability subject to change.

LatentView Analytics Exceeds Indian IT GenAI Confidence

Unlike Indian IT companies, LatentView Analytics’ approach with generative AI has been notably transparent. During the third quarter, the Chennai-based data analytics firm secured two significant new clients. One of these clients is exclusively leveraging the generative AI solutions offered by LatentView.

“We had two big wins this year. One of them is really on the back of LASER, the generative AI solution that we have. In fact, LASER is going to be the main solution that will be deployed,” said Rajan Sethuraman, CEO, LatentView Analytics, in an exclusive interview with AIM.

Further he said that the project is about $650K worth of work, and is the largest account that they have to date. The new client plans to leverage LASER’s capabilities for analysing unstructured data related to platform engagement. “The project will focus on sentiment analysis,” said Sethuraman, explaining, “It’s essentially about sentiment, user experience, user feedback, and reviews.”

Sethuraman said that being a smaller organisation compared to Indian IT companies, they were able to understand the technology quicker. “I think its an advantage we have over larger organisations that we are a smaller company, focused exclusively on AI and analytics.”

“Larger organisations will have the capability to invest more money in getting things done. However, some of this also calls for nimbleness in terms of how quickly you’re able to understand and appreciate a new technology and how you can quickly connect the dots with a business problem that an organisation is trying to solve,” he added.

LatentView Analytics currently competes with the likes of Mu Sigma, Fractal Analytics, Tiger Analytics, Association of Data Scientists (ADaSci), and Tredence. Sethuraman mentioned that, in some instances, it also sees Indian IT companies as a threat too. “We sometimes compete with the analytics practices of larger organisations, like Accenture, Infosys or TCS. Deloitte, for example, we have seen them as a threat,” he said, sharing the competitive landscape of providing AI and analytics solutions to its customers.

Generative AI Strategy

LatentView told AIM that it is taking a three-pronged approach in terms of how it looks to utilise and leverage generative AI. Firstly, it aims to build products that have generative AI at their core, such as LASER and AI Pen Pal.

Secondly, LatentView plans to add generative AI layers to the existing solutions it offers. “We have our OneCustomerView solution. We have on-shelf availability in the area of supply chain. There is a way we can enhance the acceptance, adoption, and relevance of these solutions by having a generative AI layer,” said Sethuraman.

Lastly, LatentView Analytics aims to increase the productivity of employees within the organisation with the help of generative AI. Sethuraman has expressed the goal of unlocking 20% productivity for all employees, which means that tasks that previously required 10 people can now be accomplished with eight.

Partnerships

The company recently partnered with NVIDIA to develop new innovative solutions for its clients. “We conducted a very interesting proof of concept (POC) that establishes how we can utilise the latest stack that NVIDIA offers on the inference side, as well as some of the other components they have on top. We have built a very interesting solution using that,” said Sethuraman.

LatentView Analytics is currently working with Microsoft as well. “We do a lot of work for Microsoft Azure Stack. In particular, we are having some very good conversations with Microsoft Fabric,” said Sethuraman.

“Fabric integrates many different components. It can serve as a one-stop shop and solution for various things. Organisations can more easily consolidate and accomplish many tasks using the Fabric ecosystem, as opposed to trying to buy different things piecemeal,” he added.

Furthermore, LatentView Analytics has partnered with AWS to build and deliver analytics solutions to financial services, consumer goods, and retail companies. “AWS is a significant player in some of the industries where we are present. So, the plan is just to double down and drive more engagement with them,” said Sethuraman.

Geographical Breakdown

Sethuraman said that the US continues to be the dominant geography, contributing 94.5% of overall revenues, while Europe contributed about 2.2%. However, he believes that Europe and Asia Pacific will gain more traction in the upcoming quarters.

“From a revenue standpoint, it is expected that within that three-year time frame, Europe and APAC could potentially contribute anywhere between 10% of revenue,” he said. Sethuraman told AIM that the Indian market is still in its very early stages, considering bandwidth and other constraints.

“Overall we are anticipating a growth rate of 16 to 18% for the full year, and I believe this is a pretty good number, considering that most other organisations have struggled this year,” concluded Sethuraman.

The post LatentView Analytics Exceeds Indian IT GenAI Confidence appeared first on Analytics India Magazine.

Apple Introduces Cost-Effective Language Models for Limited Domain Use

Why Apple will Build the Best Chatbot

In a recent paper titled “Specialized Language Models with Cheap Inference from Limited Domain Data,” Apple addresses the challenges of applying large language models to tasks with constraints on both inference budgets and in-domain training sets.

The paper outlines four key variables: pre training budget, specialisation budget, inference budget, and in-domain training set size.

Large language models have proven to be versatile tools but face difficulties in scenarios where both large inference budgets and substantial in-domain training sets are lacking. The research formalises these constraints and explores various approaches from the machine learning literature to address the challenges posed by limited resources.

The study reveals that, when constrained by inference cost, alternatives to the conventional practice of training very large vanilla transformer models are more effective.

Specifically, hyper-networks and mixtures of experts exhibit superior perplexity for large pre-training budgets. On the other hand, small models trained on importance sampled datasets emerge as attractive options for scenarios with large specialisation budgets.

Apple’s latest research in language models offers specific recommendations based on the size of the specialization budget. The findings suggest that for scenarios with a large specialization budget, opting for small models pretrained with importance sampling is the most effective approach. This involves pretraining over a generic corpus that has been resampled using importance sampling techniques.

On the other hand, when dealing with a smaller specialisation budget, the research advises investing in the generic pretraining of hyper-networks and mixtures of experts. These asymmetric models exhibit a substantial parameter count during pretraining but can be instantiated as smaller models for specialization purposes.

The research concludes that distillation, a commonly used technique, does not prove to be competitive across the various cost trade-offs considered in the study.

The post Apple Introduces Cost-Effective Language Models for Limited Domain Use appeared first on Analytics India Magazine.

A Data Lake, You Call It? It’s a Data Swamp

A Data Lake, You Call It? It's a Data Swamp.
Image by Author

If you’re a data professional, you’re probably familiar with the data lake architecture. A data lake can store large volumes of raw and unstructured data. So it offers both flexibility and scalability. That said, in the absence of data governance, a data lake can quickly turn into a “data swamp” making it super challenging to derive any value from the massive volume of data.

In this article, we’ll review the features and advantages of data lakes, go over the challenges that lead them to become data swamps, and, more importantly, strategies to mitigate these challenges. Let’s begin!

An Overview of Data Lakes

A data lake is a data repository that allows organizations to store large volumes of raw, unstructured, semi-structured, and structured data at scale. It serves as a flexible and cost-effective solution for managing diverse data types, enabling advanced analytics, machine learning, and other data-driven applications. Now we’ll go over some of the features and advantages of data lakes.

Features of Data Lakes

Let’s review some of the features of data lakes across types of data, data storage, ingestion, and processing:

  • Types of Data: Data lakes can store large volumes of data in the raw unprocessed format.
  • Batch and Real-time Ingestion: Data lakes support both batch and real-time data ingestion, enabling organizations to handle data from various sources, including streaming data.
  • Storage Layer: The storage layer of data lakes is often built on distributed file systems or cloud-based object storage.
  • Processing Frameworks: Data lakes leverage distributed processing frameworks such as Apache Spark, Flink, and Hadoop MapReduce for parallel and scalable data processing.
  • Integration with Analytics Tools: Data lakes integrate with a variety of analytics and business intelligence tools, enabling users to analyze and visualize data using familiar interfaces.

Advantages of Data Lakes

Now let’s go over some advantages of data lakes as a storage abstraction:

  • Flexibility: Data lakes can store a wide range of data types, including text, images, videos, log files, and structured data. This flexibility allows organizations to ingest and process diverse datasets without the need for predefined schemas. Unlike data warehouses, data lakes store raw, unaggregated data in their native format.
  • Scalability: Data lakes are designed to scale horizontally, allowing organizations to store and process massive volumes of data.
  • Cost-Effective Storage: By leveraging cloud-based object storage or distributed file systems, data lakes provide a cost-effective solution for storing large amounts of data. Cloud-based data lakes, in particular, allow organizations to pay for the storage and computing resources they actually use.

To see how data lakes compare to data warehouses and data marts, read Data Warehouses vs. Data Lakes vs. Data Marts: Need Help Deciding?

How and Why a Data Lake Becomes a Data Swamp

A data lake, when managed properly, serves as a centralized repository for storing vast amounts of raw and unstructured data from various sources. However, in the absence of proper governance, a data lake can turn into what is colloquially referred to as a "data swamp."

Governance refers to the set of policies, procedures, and controls that guide the use, access, and management of data within an organization. Here's how the absence of governance can contribute to the transformation of a data lake into a swamp:

  • Data Quality Degradation: Without proper governance, there are no defined standards for data quality, leading to inconsistencies, inaccuracies, and incomplete datasets. The lack of quality control contributes to a decline in the overall reliability of the data.
  • Uncontrolled Data Proliferation: Absence of governance policies results in unregulated data ingestion, leading to a massive influx of data without proper categorization or organization.
  • Inconsistent Data Usage Policies: Without governance, there are no clear guidelines on how data should be accessed, used, and shared. The absence of standardized practices can also hinder collaboration and interoperability among different teams.
  • Security and Compliance Risks: Without proper access controls, unauthorized users may have access to sensitive information. This can lead to data breaches and compliance issues.
  • Limited Metadata and Cataloging: Metadata typically provides information about the source, quality, and lineage of the data. Lack of metadata makes it super challenging to trace the origin and transformations applied to the data. In a data swamp scenario, there is often a lack of a centralized catalog or index, making it difficult for users to discover and understand the available data assets.
  • Lack of Lifecycle Management: Without defined policies for data retention and archiving, the data lake may become cluttered with outdated or irrelevant data, making it harder to find and use valuable information.

Lack of governance, therefore, can turn a data lake into a swamp, diminishing its utility and creating challenges for users and organizations.

Mitigating the Challenges

To prevent a data lake from becoming a swamp, organizations should focus on the following essential strategies:

  • Robust Governance Policies
  • Effective Metadata Management
  • Data Quality Monitoring
  • Access Controls and Security Measures
  • Data Lifecycle Management and Automation

Let's delve deeper into each of the above strategies to understand their importance and how they contribute to maintaining an efficient and useful data lake.

A Data Lake, You Call It? It's a Data Swamp.
Image by Author

Robust Governance Policies

Establishing clear governance policies is foundational for the effective management of a data lake:

  • Defining data ownership ensures accountability and clarity regarding who is responsible for the quality and integrity of specific datasets.
  • Access controls set the boundaries for who can access, modify, or delete data, helping to prevent unauthorized usage.
  • Usage guidelines provide a framework for how data should be utilized, preventing misuse and ensuring compliance with regulatory requirements.

By assigning roles and responsibilities to data stewards, administrators, and users, organizations create a structured and accountable environment for data management.

Effective Metadata Management

A comprehensive metadata management system captures essential information about data assets. Knowing the source of data helps establish its credibility and origin, while details about quality and lineage provide insights into its reliability and processing history.

Understanding transformations applied to the data is also important for data scientists and analysts to interpret and use the data effectively. A well-maintained metadata catalog ensures that users can discover, understand, and use the data within the data lake.

Data Quality Monitoring

Regular data quality checks are essential for maintaining the accuracy and reliability of data within the lake.

  • Conducting these checks involves validating data formats to ensure consistency.
  • Checking for completeness ensures that datasets are not missing critical information.
  • Identifying anomalies helps catch errors or inconsistencies in the data, preventing the propagation of inaccurate insights.

Proactive data quality monitoring ensures that the data lake remains a reliable source for decision-making and analysis.

Access Controls and Security Measures

Enforcing stringent access controls and encryption safeguards the data lake from unauthorized access and potential security threats. Access controls limit who can view, modify, or delete data, ensuring that only authorized personnel have the necessary permissions.

Regularly auditing access logs helps identify and address any suspicious activities, providing a proactive approach to security. Implementing encryption ensures that sensitive data is protected, both in transit and at rest.

These security measures collectively contribute to maintaining the confidentiality and integrity of data in the data lake.

Data Lifecycle Management and Automation

Defining and enforcing data retention policies is necessary for preventing the accumulation of outdated or irrelevant data. Automated data cataloging tools help manage data throughout its lifecycle.

This includes archiving data that is still valuable but not frequently accessed, purging obsolete data, and efficiently organizing data for easy discovery. Automation reduces the manual effort required for managing the vast amount of data in the lake, ensuring that it remains organized, relevant, and easily accessible to users.

In summary, these strategies together help create a well-governed and well-managed data lake—preventing the data lake from turning into a chaotic and unusable data swamp. They contribute to maintaining data integrity, ensuring security, facilitating efficient data discovery, and preserving the overall effectiveness of the data lake environment.

Wrapping Up

In conclusion, data lakes are a powerful solution for managing and extracting value from large and diverse datasets. Their flexibility, scalability, and support for advanced analytics make them valuable for data-driven organizations.

To avoid turning a data lake into a data swamp, however, organizations must invest in robust data governance, implement effective metadata management, enforce security measures, conduct regular data quality assessments, and establish clear policies for data lifecycle management.

Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she's working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.

More On This Topic

  • Cartoon: How Deep Is That Data Lake?
  • Last call: Stefan Krawcyzk’s 'Mastering MLOps' Live Cohort
  • 311 Call Centre Performance: Rating Service Levels
  • If You Can Write Functions, You Can Use Dask
  • 7 Things You Didn't Know You Could do with a Low Code Tool
  • Data Science Minimum: 10 Essential Skills You Need to Know to Start…

Karnataka Govt and Truecaller Collaborate for Cybercrime Awareness

Truecaller and the Department of Electronics and Information Technology (DeIT), Government of Karnataka, have announced a collaboration to combat the growing incidents of cybercrime, impersonation, and fraud and promote user safety in digital communication.

As a part of this collaboration, Truecaller will carry out a series of initiatives to create awareness and build the capacity of citizens across the state to navigate the internet safely, and protect themselves from online harms, especially phone-based frauds and scams.

“Cybercrimes and fraud in digital communication are growing at a rapid pace with advancing technology thus exposing vulnerable citizens to high risk. The Government is taking proactive steps to increase awareness and promote safety in digital communications. Bengaluru is synonymous with innovation and is recognized as a thriving tech hub. We take pride in adopting a collaborative approach to enhancing citizen’s online safety here,” Shri Priyank Kharge, Minister for Information Technology & Bio Technology and Rural Development & Panchayat Raj, said.

To date, Truecaller has conducted cyber safety training for over 1.6 million citizens in collaboration with Delhi Police, Assam Police, and the state governments of Delhi, Jharkhand, Tamil Nadu, Jammu & Kashmir, and Rajasthan.

“Over the past year, our Government Directory Services feature has assisted over 31.2 million users in identifying verified calls from spam and fraud, while also helping them easily access helplines and key government numbers on the platform. At Truecaller, we are consistently working towards improving fraud prevention and are confident that this collaboration will assist citizens of Karnataka in staying one step ahead of fraudsters,” Rishit Jhunjhunwala, Chief Product Officer and Managing Director of India at Truecaller, said.

The post Karnataka Govt and Truecaller Collaborate for Cybercrime Awareness appeared first on Analytics India Magazine.