Рубрика: AI

Image by Author

To make the most out of data, organizations need efficient and scalable solutions that can store, process, and analyze data effectively. From ingesting data from multiple sources through transformation and serving, data storage underpins the data architecture.

So choosing the right data storage solution while factoring in how you’ll access the data and the specific use case is important. In this article, we’ll explore three popular data storage abstractions: data warehouses, data lakes, and data marts.

We’ll go over the basics and compare these data storage abstractions across features like access patterns, schema, data governance, use cases, and more.

Let’s get started!

Data Warehouses

Data warehouses are foundational components of modern data management systems, designed to facilitate efficient storage, organization, and retrieval of structured data for analytical purposes.

Image by Author

What Is a Data Warehouse?

A data warehouse is a specialized database that centralizes, stores, and manages structured and processed data from various sources for the primary purpose of supporting complex analytics and reporting.

The data warehouse is, therefore, a centralized repository for structured data, allowing organizations to:

• Perform complex data analysis
• Generate reports and dashboards
• Support business intelligence (BI) and decision-making processes
• Gain insights into historical and current data trends

Data Type, Access Patterns, and Benefits

Data warehouses primarily store structured data, which is data organized into well-defined tables with rows and columns. This structured format simplifies data retrieval and analysis, making it suitable for reporting and querying.

Data warehouses are optimized for query performance and reporting. They often use indexing and caching mechanisms to accelerate data retrieval, ensuring that analysts and business users can quickly access the information they need.

Data Integration

Data warehouses centralize the integration of data from various source systems. This involves extracting data from source systems, transforming it into a consistent format, and loading it into the warehouse.

ETL processes are commonly employed for data integration in data warehouses. These pipelines extract data from source systems, apply transformations to clean and structure the data, and then load it into the warehouse's database tables. ETL processes ensure data quality and consistency within the data warehouse.

Schema

Data warehouses enforce a schema for data consistency. A schema defines the structure of the data, including the tables, columns, data types, and relationships. This enforced schema ensures that data remains consistent and can be relied upon for analysis.

Data warehouses often use a star or snowflake schema to organize data. In a star schema, a central fact table contains transactional data, surrounded by dimension tables that provide context and attributes. In a snowflake schema, dimension tables are normalized to reduce redundancy. The choice between these schemas depends on the specific data warehousing requirements.

Data Governance and Security

Data warehouses are known for their strong governance and security controls. They are designed for structured data and provide features like data validation, data quality checks, access controls, and auditing capabilities.

Use Cases and Business Units

Data warehouses are primarily used for enterprise-wide analytics and reporting. They consolidate data from various sources into a single repository, making it accessible for analysis and reporting across the entire organization. They support standardized reporting and ad-hoc querying for decision-makers.

Data Lakes

Data lakes represent a flexible and scalable approach to data storage and management, catering to the diverse needs of modern organizations.

Image by Author

What Is a Data Lake?

A data lake is a centralized repository that stores large volumes of raw, structured, semi-structured, and unstructured data, allowing organizations to store and manage vast amounts of information without the constraints of a predefined schema.

The primary purpose of a data lake is to provide a flexible and cost-effective solution for storing and managing diverse data types:

• Data lakes retain data in its raw and native form.
• Data lakes facilitate a wide range of use cases, from traditional analytics to advanced machine learning and AI applications.
• Users can explore and analyze data without predefining its structure or schema.

Data lakes are designed to address the challenges posed by the increasing volume, velocity, and variety of data generated by organizations today.

Data Type, Access Patterns, and Benefits

Data lakes are capable of storing a diverse array of data types, including structured data from relational databases, semi-structured data such as JSON, XML, and unstructured data like text documents, images, and videos. This makes data lakes suitable for handling data in its raw and native form.

Data Integration

Ingesting data into a data lake can be through both batch or real-time data ingestion. Batch processes involve periodically loading large volumes of data, while real-time ingestion enables the continuous flow of data from various sources. This flexibility ensures that data lakes can handle various data velocity requirements.

Data lakes have a schema-on-read approach. Unlike data warehouses, data in a data lake does not have a predefined schema. Instead, the schema is defined at the time of analysis, allowing users to interpret and structure the data based on their specific needs. This schema flexibility is a hallmark feature of data lakes.

Schema

Data lakes offer schema flexibility, allowing data to be ingested without a predefined schema. This flexibility accommodates changes in data structure over time and empowers users to define the schema as needed for their analysis.

Data in a data lake is given structure and meaning at the time of analysis. This approach means that users can interpret and structure the data to suit their analytical requirements.

Data Governance and Security

Data lakes often face governance challenges because they store both structured and unstructured data in their raw form. It can be difficult to manage metadata, enforce data quality, and maintain a unified data catalog, which can lead to issues related to data discovery and compliance.

Use Cases and Business Units

Data lakes are ideal for data exploration and experimentation. They can store vast amounts of raw, unstructured data, making them suitable for data professionals to explore and experiment without predefined schemas.

Data Marts

Data marts are subsets of the enterprise data warehouse catering to specific business units or functions within an organization.

Image by Author

What Is a Data Mart?

A data mart is a specialized subset of a data warehouse or data lake that stores structured data tailored to the needs of a specific business unit, department, or functional area within an organization.

The primary purpose of a data mart is to provide focused and efficient access to data for specific analytical and reporting needs. Key objectives include:

• Supporting specific business units: Data marts are designed to cater to the requirements of individual business units, such as sales, marketing, finance, or operations.
• Simplifying data access: By providing easier access to the relevant data, data marts make it easier for users within a specific domain to access and analyze the information they need.
• Faster time to insights: Data marts can improve query and reporting performance by reducing the volume of data that needs to be processed.

Data marts are, therefore, instrumental in ensuring that relevant data is readily available to decision-makers within various parts of the organization.

Data Types, Access Patterns, and Benefits

Data marts primarily store structured data that is pertinent to the specific business unit or function they serve. This structured format ensures data consistency and relevance to the domain's analytical needs.

Data marts provide a more focused and easier access to data compared to enterprise data warehouses or data lakes. This focused approach enables users to quickly access and analyze data that is directly relevant to their domain.

Data Integration

Data marts typically extract data from central repositories, such as data warehouses. This extraction process involves identifying and selecting data relevant to the specific business unit or function.

Once extracted, data undergoes transformation specific to the mart's needs. This may include data cleansing, aggregation, or customization to ensure that the data aligns with the analytical requirements of the domain it serves.

Schema

Data marts may either adhere to the schema defined in the central data warehouse or employ a custom schema tailored to the specific mart's analytical needs. The choice depends on factors such as data consistency and the mart's autonomy.

Data Governance and Security

Data marts are typically subsets of data warehouses, focusing on specific business domains or units. Governance efforts are concentrated at the mart level, ensuring that the data used by specific business units is compliant with enterprise-wide governance standards set by the data warehouse.

Use Cases and Business Units

Data marts are tailored to the specific needs of business units or domains within an organization. They provide a subset of data from the data warehouse that is relevant to a particular business area. This allows business units to perform specialized analytics and reporting without the complexity of handling the entire enterprise dataset.

Data Warehouses vs. Data Lakes vs. Data Marts: A Comprehensive Comparison

Let’s sum up the key differences between data warehouses, data lakes, and data marts:

Feature	Data Warehouses	Data Lakes	Data Marts
Data Types and Flexibility	Structured data, fixed schema	Various data types, schema flexibility	Structured data, well-defined schema
Data Integration	ETL pipelines	Flexible data ingestion, schema-on-read	Extraction and transformation for domains
Query Performance	Optimized for queries	Performance varies	Optimal performance
Data Governance	Strong data governance and security controls	Data governance challenges	Mart-level governance
Use Cases	Enterprise analytics	Exploration of large volumes of data	Domain-specific analytics

Conclusion

I hope you’ve gained an overview of data warehouses, lakes, and marts. The choice of architecture depends on the organization's specific requirements and the balance between governance and flexibility needed for their data and business needs:

• Data warehouses—with strong governance and security controls—are suitable for enterprise-wide analytics and reporting.
• Data lakes are suitable for data exploration and big data analytics. But can pose governance and security challenges.
• Data marts offer domain-specific analytics aligned with business unit needs while complying with the governance standards of the data warehouse.

You can also explore data lakehouses, a relatively recent and evolving architecture. Data lakehouses aim to bridge the gap between data warehouses and data lakes, offering a unified approach to data storage and analytics.

Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she's working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.

More On This Topic

• Why Organizations Need Data Warehouses
• The Seven Best ELT Tools for Data Warehouses
• Data Lakes and SQL: A Match Made in Data Heaven
• We Don't Need Data Scientists, We Need Data Engineers
• We Don’t Need Data Engineers, We Need Better Tools for Data Scientists
• Data access is severely lacking in most companies, and 71% believe…

Without doubt, public cloud has been transformative for businesses over the years, however, country-specific jurisdiction laws and data protection concerns have brought in a growing emphasis on sovereign cloud. According to a recent survey by HCLTech, approximately 24% of organisations are planning to migrate a portion of their data from the cloud to on-premise data centres within the next three years.

One of the things that the survey did indicate was a very large interest in sovereign cloud, according to Alan Flower who heads the HCL Cloud Native & AI Labs business and is also the HCL CTO for Cloud Native.

“Often what is driving this migration back to on-premise is security or regulatory adherence. There is this kind of awareness from clients that they may benefit from moving into a sovereign cloud, typically to align with their country preferences,” Flower said.

However, another interesting observation from Flower was that many organisations’ migration to the cloud was not transformational, but transitional. If organisations moved their IT from their data centres to someone else’s data centres, the cost savings for them were not very evident.

“In my view, the potential cost savings can have an immensely transformative effect on your business, especially if you’re on a journey of transformation. However, if your ambitions are not aligned with this transformation and you’re merely migrating your IT infrastructure to the cloud, you might find yourself in a situation a couple of years down the road, wondering where the expected impact has gone.”

Demand for sovereign cloud to grow

Sovereign cloud is a cloud infrastructure that is subject to the laws and regulations of a specific country or region designed to ensure data sovereignty, which means that data is subject to the laws and governance of the country in which it is hosted. Over the years, data protection concerns and laws have resulted in a significant rise in the demand for sovereign cloud.

Flower expects the trend around sovereign cloud to continue and a focus on security and regulatory concerns is driving a lot of the activities. “Especially in Europe, we notice a pronounced emphasis on sovereign cloud, which is not a trend seen worldwide but is notably prominent in this continent.” For example, earlier this month, AWS, the largest hyperscaler by market size, announced the release of AWS European Sovereign Cloud.

However going forward, the demand for sovereign cloud may further rise due to the implementation of data protection laws in various countries. For instance, in India, the Digital Personal Data Protection Bill 2023 has been approved by both houses of Parliament and is expected to be enacted soon. This legislation might require certain organisations, especially those in highly regulated industries, to store their data locally or opt for a sovereign cloud solution.

“Moreover, I see lots of indications that clients are increasingly happy to consume what I would call a hybrid-multi-cloud estate. Many clients have adopted a flexible approach that enables them to relocate data workloads to the most suitable destinations. I anticipate that clients will take advantage of this flexibility, making decisions based on efficiency gains or enhanced transformative potential when moving workloads to more appropriate locations.”

Opting for a hybrid cloud strategy

In the future, more and more organisations are expected to adopt a hybrid model strategy recognising that some workloads and data are best suited for on-premises data centres, while others benefit from cloud delivery.

Notably, container-based infrastructure, exemplified by technologies like Kubernetes, has gained significant traction, according to Flower. These container platforms enable centralised management of multi-cloud environments from a single control plane. This approach offers two key advantages. Firstly it offers centralised management meaning organisations can efficiently manage all their workloads, regardless of their cloud environment, as a unified piece of infrastructure. This centralisation simplifies administration and reduces complexity.

Secondly, it also brings in the advantage of workload portability which means organisations can seamlessly migrate workloads between on-premises data centres and various cloud providers. This ease of movement enables organisations to choose cloud providers based on the specific services they offer.

“We are observing a growing trend where clients actively seek the inherent portability facilitated by containerisation. This portability appeals to clients because it allows them to transfer various applications and solutions between different cloud providers seamlessly. Clients appreciate the flexibility to choose providers based on the rich array of services each one offers,” Flower said.

Disconnect between IT and business

The HCLTech report titled ‘Cloud Evolution: Make Innovation a Habit’ also states that many companies plan to increase their investments in cloud technologies going forward. This is mostly driven by a genuine appetite for transformation.

However, Flower also notes that there exists a slight disconnect between IT and the business when it comes to the cloud. While business leaders expressed a clear understanding of how the cloud could address their challenges, such as responding quickly to customer needs, increasing agility, and fostering innovation, IT leaders had a different perspective.

“The IT leaders were telling the survey that they believed that the business may not fully appreciate the cloud’s capabilities. Hence there is a slight disconnect between the expectations of business leaders and IT leaders.

“While the businesses are really sold on the strategic benefits of cloud, the IT leaders just need to get a little bit closer to those business leaders to help them achieve that ambition,” Flower said.

To align with this perspective, IT should focus not only on technology but also on enabling cultural change. Organisations must ensure that IT functions support the business in achieving the expected agility and access to innovation, Flower concluded.

The post Sovereign Cloud to Gain Traction as Security Concerns Loom appeared first on Analytics India Magazine.

Being one of the most watched sporting events in the world, the National Football League’s (NFL) championship game is broadcast in over 130 countries in more than 30 languages. The final game, Super Bowl, is the most watched broadcast every year in the US, with over 115 million viewers tuned in to this year’s edition in February. While the game has been dominating the country, it is being backed by not just the greatest brands in the world pouring in millions of dollars, but also supported by the biggest tech companies in the world.

Amazon Brings AI to NFL

In Amazon’s Q3 earnings, of the many high points in the results, NFL has been a key player. The subscription revenue has increased by 14% to $10.2 billion and is driven by the NFL. The company is in the second season of a 11-year exclusive deal of $11 billion to distribute Thursday Night Football (TNF are NFL games scheduled to be played on Thursday evenings during the NFL season) through Prime Video.

Amazon has brought AI to TNF, making it an interactive experience for their viewers. Features such as ‘X-Ray’, gives fans real-time access to live statistics and data, ‘Rapid Recap’ generates 13 two-minute-long highlights to help viewers catch up on games, and many other features.

‘Prime Vision with Next Gen Stats’ powered by AWS, provides insights by capturing real-time data on player’s location, speed, and acceleration using sensors hidden within their shoulder pads. Amazon collects over 300 million data points per season to train their machine learning models for gaining insights from every game. The insights on pass and position are shown real-time, giving viewers the ability to observe and predict the game strategy, akin to a quarterback (key player).

As per Amazon’s Q3 results, the TNF season opener attracted 15.1 million viewers, and was the Prime Video’s most watched TNF game ever. The first six games brought an average of 12.9 million viewers, which was an increase of 25% from the previous season. Interestingly, Amazon has committed to paying $1 billion annually for the exclusive streaming rights to NFL games.

Tussle for Broadcast Rights

Last year, the National Football League revealed a multi-year deal with Google, giving Youtube TV and Youtube Primetime Channels exclusive rights to distribute NFL Sunday Ticket, that allows viewers to watch Sunday afternoon NFL games that are not typically available on local channels.

The deal is said to be around $2 billion annually for seven years. DirecTV was previously paying $1.5 billion a year for the rights. However, as per a new report, Youtube is said to lose over $8.86 billion from now to 2029 with yearly declines of about $1.27 billion.

Interestingly, Apple was one of the forerunners to bag the deal, however, the agreement did not fall through, as Apple wanted to reportedly pay less for the deal, so as to offer the product at lower prices. Though this deal failed, Apple was not completely left out. The NFL announced Apple Music as the new sponsor of the Super Bowl halftime show from 2023. Taking over Pepsi, who were the sponsors for 10 years, Apple will pay $50 million annually over a five- year span.

Big ‘Technology’ Partners

In 2021, Cisco, an enterprise networking and security company, signed a multi-year deal to become the official technology partner for NFL. The partnership aims to create a unified platform and establish a robust technological foundation for NFL’s operations and communications with improved speed, intelligence and security measures.

Every NFL stadium’s replay control room is built on Cisco technology, and almost all of the league’s official partners and two-thirds of NFL stadiums, including SoFi Stadium in LA and State Farm Stadium in Arizona, that hosted Super Bowls, is powered by Cisco technology.

In Tech Mahindra’s recent Q2 earnings call, CP Gurnani, CEO and MD of the company announced that they are working with NFL. In 2018, the company signed a multi-year deal to be the technology, analytics and strategy partner for Jacksonville Jaguars (NFL team).

The Best ‘Playground’

The first Super Bowl that was held in 1967 had close to 50 million viewers, and was the only year to have two networks broadcast it (CBS and NBC). The viewership has more than doubled with the latest edition (57th) in February, reaching 115.1 million viewers, making it the most watched Super Bowl of all time, and becoming the most popular TV program of all time.

Tech firms understand that the NFL guarantees a large audience, and there’s no better platform than the game’s coverage. By partnering with such a major sporting event, companies leverage their technology and brand power : a probable win for them.

The post Big Tech Eyes NFL’s Gold appeared first on Analytics India Magazine.

Key Takeaways

• Thought Propagation (TP) is a novel method that enhances the complex reasoning abilities of Large Language Models (LLMs).
• TP leverages analogous problems and their solutions to improve reasoning, rather than making LLMs reason from scratch.
• Experiments across various tasks show TP substantially outperforms baseline methods, with improvements ranging from 12% to 15%.

TP first prompts LLMs to propose and solve a set of analogous problems that are related to the input one. Then, TP reuses the results of analogous problems to directly yield a new solution or derive a knowledge-intensive plan for execution to amend the initial solution obtained from scratch.

Introduction

The versatility and computational power of Large Language Models (LLMs) are undeniable, yet they are not without limit. One of the most significant and consistent challenges to LLMs is their general approach to problem-solving, consisting of reasoning from first principles for every new task encountered. This is problematic, as it allows for a high degree of adaptability, but also increases the likelihood of errors, particularly in tasks that require multi-step reasoning.

The challenge of "reasoning from scratch" is especially pronounced in complex tasks that demand multiple steps of logic and inference. For example, if an LLM is asked to find the shortest path in a network of interconnected points, it typically would not leverage prior knowledge or analogous problems to find a solution. Instead, it would attempt to solve the problem in isolation, which can lead to suboptimal results or even outright errors. Enter Thought Propagation (TP), a method designed to augment the reasoning capabilities of LLMs. TP aims to overcome the inherent limitations of LLMs by allowing them to draw from a reservoir of analogous problems and their corresponding solutions. This innovative approach not only improves the accuracy of LLM-generated solutions but also significantly enhances their ability to tackle multi-step, complex reasoning tasks. By leveraging the power of analogy, TP provides a framework that amplifies the innate reasoning capabilities of LLMs, bringing us one step closer to the realization of truly intelligent artificial systems.

Understanding Thought Propagation

Thought Propagation involves two main steps:

1. First, the LLM is prompted to propose and solve a set of analogous problems related to the input problem
2. Next, the solutions to these analogous problems are used to either directly yield a new solution or to amend the initial solution

The process of identifying analogous problems allows the LLM to reuse problem-solving strategies and solutions, thereby improving its reasoning abilities. TP is compatible with existing prompting methods, providing a generalizable solution that can be incorporated into various tasks without significant task-specific engineering.

Figure 1: The Thought Propagation process (Image from paper)

Moreover, the adaptability of TP should not be underestimated. Its compatibility with existing prompting methods makes it a highly versatile tool. This means that TP is not limited to any specific kind of problem-solving domain. This opens up exciting avenues for task-specific fine-tuning and optimization, thereby elevating the utility and efficacy of LLMs in a broad spectrum of applications.

Implementing Thought Propagation

The implementation of Thought Propagation can be integrated into the workflow of existing LLMs. For example, in a Shortest-path Reasoning task, TP could first solve a set of simpler, analogous problems to understand various possible paths. It would then use these insights to solve the complex problem, thereby increasing the likelihood of finding the optimal solution.

Example 1

• Task: Shortest-path Reasoning
• Analogous Problems: Shortest path between point A and B, Shortest path between point B and C
• Final Solution: Optimal path from point A to C considering the solutions of analogous problems

Example 2

• Task: Creative Writing
• Analogous Problems: Write a short story about friendship, Write a short story about trust
• Final Solution: Write a complex short story that integrates themes of friendship and trust

The process involves solving these analogous problems first, and then using the insights gained to tackle the complex task at hand. This method has demonstrated its effectiveness across multiple tasks, showcasing substantial improvements in performance metrics.

Thought Propagation's implications go beyond merely improving existing metrics. This prompting technique has the potential to alter how we understand and deploy LLMs. The methodology underscores a shift from isolated, atomic problem-solving towards a more holistic, interconnected approach. It prompts us to consider how LLMs can learn not just from data, but from the process of problem-solving itself. By continuously updating their understanding through the solutions to analogous problems, LLMs equipped with TP are better prepared to tackle unforeseen challenges, rendering them more resilient and adaptable in rapidly evolving environments.

Conclusion

Thought Propagation is a promising addition to the toolbox of prompting methods aimed at enhancing the capabilities of LLMs. By allowing LLMs to leverage analogous problems and their solutions, TP provides a more nuanced and effective reasoning method. Experiments confirm its efficacy, making it a candidate strategy for improving the performance of LLMs across a variety of tasks. TP may ultimately represent a significant step forward in the search for more capable AI systems.

Matthew Mayo (@mattmayo13) holds a Master's degree in computer science and a graduate diploma in data mining. As Editor-in-Chief of KDnuggets, Matthew aims to make complex data science concepts accessible. His professional interests include natural language processing, machine learning algorithms, and exploring emerging AI. He is driven by a mission to democratize knowledge in the data science community. Matthew has been coding since he was 6 years old.

More On This Topic

• Unraveling the Power of Chain-of-Thought Prompting in Large Language Models
• The secret to analysing large, complex datasets quickly and productively?
• Top Open Source Large Language Models
• More Free Courses on Large Language Models
• Learn About Large Language Models
• Introducing Healthcare-Specific Large Language Models from John Snow Labs

Founded in 2011 by Ashok Soota, a serial entrepreneur and Indian IT veteran Happiest Minds boasts a robust data science team comprising over 300 members, including data engineers, intelligence specialists, and data science experts.

Based in the Silicon Valley of India, Bangalore, and extending its reach across the global landscape, including the US, UK, Canada, Australia, and the Middle East, this IT juggernaut seamlessly blends augmented intelligence with the art of understanding human language, deciphering images, analysing videos, and harnessing cutting-edge technologies such as augmented reality and virtual reality. This dynamic fusion empowers enterprises to craft captivating customer interactions that surpass rivals and set new industry standards.

Happiest Minds distinguishes itself from traditional IT companies by avoiding legacy systems like SAP and ERP, believing that staying entrenched in these technologies limits growth and innovation. “Instead, we have chosen to focus on digital technologies like AI, which is the future of IT,” said Sundar Ramaswamy, SVP, Head of Analytics CoE, in an exclusive interview with AIM.

The team conducts regular market scans to identify the latest technologies and ensures that they are always on the forefront of innovation. This approach allows them to co-create and innovate with clients while building new solutions.

Now Hiring

Happiest Minds is currently on a lookout for a specialist in marketing analytics. The ideal candidate should possess a Master’s or Bachelor’s degree in Computer Science, STEM, or an MBA, demonstrating strong problem-solving skills. They should also have over eight years of experience in the analytics industry, particularly in marketing.

This experience should include a track record of using AI to enhance the customer journey, encompassing areas such as customer acquisition, nurturing, retention, and improving the overall experience.

The technical skills required include proficiency in statistical techniques, ML, text analytics, NLP, and reporting tools. Experience with programming languages such as R, Python, HIVE, SQL, and the ability to handle and summarise large datasets using SQL, Hive-SQL, or Spark are essential. Additionally, knowledge of open-source technologies and experience with Azure or AWS stack is desired.

AI & Analytics Play

This team collaborates closely with domain teams across diverse industry verticals. Their analytics process follows eight key steps. They integrate data from multiple sources, use BI tools for descriptive analytics, perform ad hoc analysis, build data pipelines and auto ML pipelines, retrain models regularly, focus on customer understanding, optimise cloud usage, and ensure data governance. Their key industry verticals are CPG Retail, Healthcare (bioinformatics), FSI, Media Entertainment, and Edtech, with growing interest in manufacturing.

The team works with classical analytics, deep learning, computer vision, NLP, and generative AI. This includes advanced applications like language translation and content generation from 2D to 3D images using generative AI.

Recognising the growing importance of generative AI, they have formed a dedicated task force comprising approximately 50 to 60 members, drawn from diverse domains, under the leadership of their CTO with the primary objective to leverage generative AI in addressing industry-specific challenges.

To achieve this, they’ve identified and categorised 100 to 250 distinct use cases across ten different domains, tailored to the specific requirements of each domain. The team is diligently working on creating demos and proof of concepts (POCs) that are domain-specific.

Some team members come from analytics backgrounds, contributing their technical expertise, while others from domain areas contribute to shaping ideas and ensuring the results align with the industry’s needs. This undertaking is substantial for the organisation, particularly considering they have around 5,500 employees with dedicating 100 to 160 of them solely to generative AI.

In addition to building demos, the company is also focusing on educating its entire workforce about LLMs and their applications to equip all team members with a basic understanding of generative AI’s capabilities and potential applications.

To bring generative AI into action, the company is working with Microsoft’s suite of products. “We are a Microsoft select partner and are also experimenting with different language models,” he added.

The team initially experimented with Google’s BERT and now employ models like GPT-2. They have a strategic inclination towards refining existing models to suit specific applications, rather than developing entirely new foundational models. For example, they collaborate with a healthcare company to craft adaptive translation models with reinforcement learning.

Interview Process

“Data science is not just about technical skills; it also involves an element of art. Candidates are assessed on their ability to communicate their results effectively and their capacity to approach problems with creativity,” said Ramaswamy.

The interview process for data science candidates at Happiest Minds typically involves three to five levels of interviews. The first level is a screening by the HR team based on the job description. This is followed by a written test to assess the candidate’s proficiency in relevant languages and skills. For example, if the position is for a data engineer, the test might evaluate their ability to work with SQL and other database-related tasks.

Technical interviews are conducted, using case studies to evaluate the candidate’s problem-solving ability and approach. The interview process concludes with a leadership interview, especially if the position is a senior one.

In addition to understanding the interview process, candidates often wonder about the common mistakes they should avoid. According to Ramaswamy, there are two main pitfalls that candidates often fall into. First, many candidates focus excessively on specific tools or techniques and become fixated on mastering them.

“While technical proficiency is essential, it’s equally important to explain the problem being solved, the reasons for approaching it a certain way, and considering alternative solutions,” he added.

The second common mistake is becoming too narrowly focused on the solution without understanding the broader context. It’s crucial to see the big picture, why the problem is being solved for the client, and to ask relevant questions about the projects they’ve worked on.

In terms of skills, the company looks for both technical and non-technical abilities. The specific skills depend on the role of the position, such as data engineering, business intelligence, or data science.

However, primary technical skills include proficiency in relevant tools and technologies, certifications, and problem-solving abilities. Non-technical skills are communication and presentation skills, problem-solving skills, and the ability to coach and mentor, as collaboration and teamwork are essential for senior positions.

Work Culture

“As our company name suggests, we aim to cultivate a distinctive work culture based on four fundamental pillars,” Ramaswamy commented. Certified as a Great Place to Work, the company prioritises the well-being of their employees, believing that “a content workforce leads to happy customers.” They monitor and maintain employee happiness closely, offering support to those facing personal or professional challenges.

Collaboration is another key element of their culture, as they encourage a unified approach within and across different units and locations. “As a company born in the digital age, Happiest Minds thrives on agility, adapting swiftly to meet the ever-changing needs of customers and the digital industry,” he added.

Transparency is the fourth pillar, as they openly share key performance indicators and objectives with their employees, investors, and stakeholders. This culture of transparency and goal-oriented approach ensures that their efforts are always aligned with clear objectives and tracked diligently. These four pillars underpin the core values of the organisation, extending throughout all business units, including analytics.

If you think you fit the role, check out their careers’ page now.

Read more: Data Science Hiring Process at PayPal

The post Data Science Hiring Process at Happiest Minds appeared first on Analytics India Magazine.

Skyroot Aerospace, a prominent player in the space launch vehicle design and manufacturing industry, has successfully secured $27.5 million in a pre-Series C funding round led by Temasek. This significant injection of capital brings the total funding for the launch vehicle manufacturer to $95 million, making it the largest funding corpus for any Indian space technology startup. In comparison, Pixxel has raised $71 million in total venture funding, while Agnikul Cosmos stands at $40 million.

This infusion of funds is set to propel Skyroot Aerospace into the global satellite launch services market, with multiple orbital launches planned over the next two years. The startup has outlined a strategic plan for these funds, including increased investments in infrastructure, the strengthening of its technology leadership, attracting top-tier talent, and enhancing its launch frequency and capabilities.

Headquartered in Hyderabad, Skyroot Aerospace was founded in 2018 by former space scientists-turned-entrepreneurs, Pawan Kumar Chandana and Naga Bharath Daka. In November 2022, the company achieved a significant milestone by successfully launching India’s first privately developed rocket, marking the entry of the Indian private space sector into the space launch market.

The startup is positioning itself to offer “on-demand, cost-effective” regular space launch services to the rapidly growing global small and medium-sized satellite market. Its customers will include satellite operators seeking to deploy single satellites or constellations across various orbits.

Pawan Kumar Chandana, Co-founder and CEO of Skyroot, expressed his enthusiasm about the new funding, stating, “As we prepare for the launch of our second mission early next year, this new funding will enable us to accelerate our upcoming launches planned over the next two years. As a key player in India’s private space industry, we are amplifying our capabilities to tap into the expanding global satellite launch market with a goal to emerge as a favourite ‘go-to’ choice in space launch services for small satellites.”

Bharath Daka, Co-founder and COO of Skyroot added, “This fund-raise will help us invest in the enhancement of our production infrastructure, R&D, and team strength, enabling us to achieve a higher launch cadence in the years to come.”

Skyroot Aerospace’s recent funding success highlights its growing prominence in the space-tech sector and paves the way for other Indian space startups to achieve significant capital injections. As the company continues to advance its satellite launch capabilities, it is poised to become a major player in the space industry on a global scale.

The post Skyroot Aerospace Raises $27.5M, Prepares to Launch into Global Satellite Market appeared first on Analytics India Magazine.

Indian IT firm Tech Mahindra, intends to launch ‘Project Indus’ its LLM designed for Hindi and its 37 dialects by the end of December or early January, reported Economic Times. This initiative comes four months after the company introduced ‘Project Indus,’ a strategic effort by the fifth largest software services firm to develop a foundational model for Indian languages.

Over the last two months, the 15-member Project Indus team has gathered 1.2 terabytes of data in Hindi and its related dialects. Currently, they are working on refining this data into web text, which they plan to release as open source by the end of November, stated Nikhil Malhotra, global head of maker’s lab at Tech Mahindra, the ET report added.

“In the meantime, we have started constructing the model… We are looking at probably the end of December or starting of January, we will release the model for at least Hindi and its dialects. And then the other work starts for other dialects in other regions,” Malhotra said.

The team encountered difficulties related to data availability and collection. “In Hindi, the maximum number of tokens available is about 2.8 billion, which doesn’t meet the model’s requirements. For instance, to create a 7 billion parameter model, I would need at least around 100 billion tokens,” explained Malhotra.

At the beginning, a portal was established to gather voice samples in local dialects through crowd-sourcing. Initially, there were 1,500 responses within the first two days, but the response gradually decreased. In total, only 6,000 samples were received, as stated by Malhotra.

To address this, teams were dispatched to regions like Uttar Pradesh, Madhya Pradesh, Haryana, and Jammu to collect data in person. Additionally, the Hyderabad campus of Tech Mahindra organized a camp where employees contributed samples in dialects like Hyderabadi Dakhini.

According to Tech Mahindra’s chief CP Gurnani, the model will be the biggest Indic LLM and could possibly cater to 25% of the world’s population. While Tech Mahindra has not revealed the cost associated with the project or when the model is expected to be launched, the aim is to build a 7-billion parameter LLM to begin with, Malhotra, told AIM in an exclusive interview.

The post Tech Mahindra to Launch OpenAI Rival ‘Project Indus’ Early Next Year appeared first on Analytics India Magazine.

Intel has done better than expected in the third quarter, with an increase in revenue and margins from $12.9 billion in the last quarter to $14.2 billion, driven by a stabilising server chip business and stronger PC market conditions.

The Q3 revenue was down by 8% YoY, indicating a massive recovery from the last quarter which was down by 15% YoY. Gross margins improved, and they’ve secured new customers for chip manufacturing, with more deals on the horizon.

The global PC market is on the rebound, and the company’s forecast for Q4 includes adjusted revenue expectations exceeding Wall Street estimates for both revenue and profit per share, from $14.6 billion to $15.6 billion.

Despite recent challenges, such as heavy manufacturing investments impacting gross margins, Intel is making progress with its turnaround plans.

For the upcoming quarters, Intel is betting big on its foundry services, the company’s nascent chip-manufacturing business, which mustered a meteoric $311 million in revenue, growing 299% from the previous year.

A Slew of Partnerships

The tech giant announced that a major customer committing to Intel 18A and Intel 3 with a prepayment allowed the company to accelerate its plans to build two new leading-edge chip factories at its Ocotillo campus in Chandler, Arizona.

The customer could very well be American electronic design automation company Synopsys, which inked a deal with Intel in August of 2023.

Additionally, IFS and Tower Semiconductor got into an agreement wherein Intel would provide foundry services and 300 mm manufacturing capacity to help Tower serve its customers globally, using Intel’s advanced manufacturing facility in New Mexico. The move came after the merger plans of the two entities failed.

Ambitions and Challenges

“Our ambition is to be the No. 2 foundry in the world by the end of the decade,” Randhir Thakur, president of Intel Foundry Services had reiterated in an interview. CEO Pat Gelsinger’s move to open Intel’s fabrication facility to external clients was an important step in that direction, representing a strategic shift.

To attain this goal, Intel has embarked on a significant investment plan, including allocating $20 billion to build two new facilities in Arizona, with an additional $20 billion earmarked for a site in Ohio.

However, the objective isn’t solely propped on revenue ranking, as there are several key elements to consider.

Firstly, to achieve the status of the second-largest foundry, Intel aims to become a technology leader, with rivals TSMC and Samsung standing tall in the way. These two industry giants have made considerable progress in both R&D and capacity expansion over the past decade, outpacing Intel.

In the fast-paced semiconductor industry, companies must advance to the next process node every two years or faster to stay relevant. Intel, unfortunately, lagged in this aspect and must work to regain its competitive edge.

Advancing in semiconductor technology is just the first step. The ability to turn cutting-edge chip technology into high-volume production with a low rate of defects, known as yield, is equally vital.

The semiconductor industry is highly competitive, with TSMC and Samsung consistently advancing to the next node and achieving rapid revenue growth. In contrast, companies like United Microelectronics Corp (UMC) struggle to keep pace.

This divide has resulted in a market bifurcation where TSMC and Samsung dominate the high-end chip production, which includes chips used in smartphones, AI servers, and crypto miners.

Meanwhile, other companies handle low-end production, including chips for automotive applications, smart speakers, and industrial robots. Approximately 50% of the foundry market comprises products manufactured at 16nm and smaller process nodes, almost exclusively produced by TSMC and Samsung.

Intel’s challenge is not only to advance technologically but also to attract leading clients like Apple, NVIDIA, and AMD. Intel has to prove that it can be trusted with both chip designs and large-scale, on-time, and low-defect chip production.

Apple’s decision to switch from Intel as a Mac processor supplier in favour of in-house processors underscores the challenges of retaining leading clients.

If Intel doesn’t manage to keep up with rivals, it may drop back by several development cycles within half a decade. This could result in a three-tier market: Intel is in the middle, not charging as much as the leaders but lacking the low-cost advantages of smaller players.

Intel might have a profitable niche if it can convince strategic clients and the US government that American-made chips are vital for security-sensitive industries like defence, aerospace, and data management.

Ultimately, Intel’s success in this endeavour depends on both technological breakthroughs and effective salesmanship. It may need to rely on government support and convince customers that being almost leading in technology is sufficient for their needs.

Intel’s journey to become the world’s second-largest foundry involves multiple challenges, and its position within the market is likely to be determined once it can prove itself to clients and stakeholders.

Nonetheless, a multitude of partnerships and a near-300% profit seems more than encouraging and is a nod to the decision to move in this direction. After all, Intel might have found an avenue to scale.

The post Why Intel is Betting Big on Chip Manufacturing appeared first on Analytics India Magazine.

In an era where companies are amassing vast troves of customer data, both online and offline, the challenge lies in effectively harnessing its potential.

As projected by International Data Corporation (IDC), the global data landscape is set to undergo a monumental transformation by 2025, witnessing a tenfold surge from 2012, soaring from 6.5 zettabytes to an astounding 64.5 zettabytes. This vast amount of information represents a considerable repository of untapped value.

Consequently, businesses are exploring novel avenues to engage with consumers, generating first- and zero-party data—information characterised by precision and resilience against data erosion. Amidst this vast landscape of data, encompassing various formats like images, videos, and audio recordings, lies a potential reservoir. Effectively re-imagining this reservoir of data and utilising it to scope necessitates strategic vision, technological infrastructure, and proficient skills for data processing.

Further, we will delve into three specific applications that caption the effective re-imagination of data. These applications showcase innovative and transformative approaches that have the potential to significantly enhance customer experience through personalization.

The scope of Retail Media Networks

The advent of Retail Media Networks (RMNs) introduces promising avenues for both retailers and brands in this pursuit. RMNs offer opportunities to unearth fresh streams of revenue and insights, aligning with the goal of enhancing CX and driving business success.

The scope of RMNs encompasses not only advertising, but also the strategic monetization of first-party data, valuable insights, and innovative out-of-home or in-store strategies that were previously confined to retailer merchant organisations only.

Expected to reach $160 billion in global annual revenue by 2027, (according to Group M’s 2022 TYNY forecast) Retail Media itself is an enticing proposition for businesses the world over. So, through RMNs, retailers and suppliers can gain access to real-time insights that translate to consumer preferences.

These insights empower them to deepen their connection with customers, shape their strategies, and close the loop from advertising impressions to sales.

Another intriguing aspect is that brands are not limited to advertising solely through retailers that sell their products. RMNs have gained significant traction among non-endemic brands, such as fashion brands partnering with beauty retailers or travel agencies collaborating with theme park entities.

This transformative shift in the RMN landscape involves the separation of media and data functionalities, providing brands with unprecedented access to essential first-party data insights, free from the constraints of media transactions.

According to Merkle’s latest Retail Media Research Report, a remarkable 65% of surveyed retailers attest that offering insights independent of media transactions strengthens their vendor relationships, marking a 13% increase from 2021.

Concurrently, brands are eagerly seeking access to profound insights within a privacy-conscious and contextually relevant environment that fosters customer engagement and loyalty.

Elevating brand loyalty through zero-party data

Brand loyalty is often elusive to achieve in today’s competitive landscape. Under these circumstances, the utilisation of zero-party data, willingly shared by customers, is proving to be a game-changer. However, this should be viewed with a lens that helps brands create profound emotional connections that nurture enduring loyalty.

Recognizing that emotional bonds are the driving force behind loyalty, this evolution emphasizes shared values, transparency, trust, and open dialogue, all of which are fundamental to delivering a superior CX.

Zero-party data transcends the limitations of demographics and behaviours.

Consider the case of a tech-savvy 28-year-old professional from San Francisco who displays a keen interest in your offerings. While such insights offer a glimpse into preferences and aspirations, they represent just a fraction of the whole picture.

Continuous engagement through surveys and social media, and website interactions allows customers to express their sentiments. This enables brands to adapt and personalize their offerings.

When customers provide zero-party data, they are entrusting brands and retailers with personal information in exchange for some form of value. Therefore, the most creative and targeted uses of zero-party data are those that provide value based on the specific information customers willingly provide.

Loyalty programs emerge as a natural and effective avenue for brands and retailers to achieve this. Consider an athletic apparel retailer that conducts surveys to understand what sport customers play or the types of fitness activities they engage in.

Armed with this knowledge, the retailer can do more than simply send promotions related to the appropriate attire. They can deliver sport-specific content and even collaborate with local event organisers to inform customers about opportunities to get involved in their interests, all while wearing their favourite fitness apparel.

In essence, zero-party data serves as the foundation for brands and retailers to not only understand their customers on a deeper level but also to offer highly personalized and valuable experiences in an increasingly privacy-conscious milieu. The seamless integration of zero-party data into a comprehensive data strategy can therefore greatly magnify CX enhancement, enriching the customer journey with each data facet. Furthermore, inventive ways of utilizing existing data and insights can complement this approach.

Identity resolution: A synergy of data and insights

Identity resolution platforms use existing data and harmonize it with second and third-party data sources, to weave a cohesive, cookie-less private identity graph. This intricate web empowers businesses with cross-channel targeting precision, catalysing personalised marketing initiatives, and facilitates accurate measurement of outcomes.

In a noteworthy case study with a major US media conglomerate, their effort to achieve seamless personalisation across diverse brands through a unified marketing solution proved extraordinarily successful. This effort merged data from 30 sources, resulting in the synthesis of over 2,000 consumer attributes.

The use of a proprietary identity resolution platform enhanced consumer experiences and provided valuable insights for advertisers as well. Advertisers benefited from the improved accuracy in predicting age and gender, reducing reliance on third-party data by 100%.

For consumers, it led to a high success rate (80%) in anticipating the next optimal action, especially in product recommendations. For this innovative solution, the Merkle team recently received the “Excellence in AI Strategy Consulting Award” at Cypher 2023.

Conclusion: The future is personal

As data burgeons exponentially, reimagining its potential is no longer an option, but a necessity. The vast volume, diversity, and complexity of today’s data offer an extraordinary opportunity and challenge. Enterprises that effectively harness this and apply innovative approaches stand to benefit from the customer engagement landscape.

The future of CX is as such profoundly personalized and insights-driven. An intimate understanding of individual preferences will characterize every interaction. Customers are no longer data points but valued individuals whose experiences demand to be meticulously tailored.

The key to unlocking this lies in strategic data utilisation, not just collecting more but extracting actionable insights. Therefore, fine tuning existing solutions like RMNs, and refocusing on zero-party data insights coupled with identity resolution platforms can produce outcomes that are future-ready and hyper-personal.

The post Reimagine Existing Data & Tech in CXM appeared first on Analytics India Magazine.