Рубрика: AI

OpenAI said that AI tools like ChatGPT today are being used around the world to help farmers in India and Kenya increase crop yields in partnership with Digital Green.

Founded initially as a project within Microsoft Research India’s Technology for Emerging Markets in 2006 by Rikin Gandhi and his colleagues, Digital Green became an independent NGO in 2008.

The organisation focuses on training farmers to produce and share short videos that document their challenges, solutions, and success stories, aiming to facilitate a technology-enabled means of behavior change communication.

Digital Green recently introduced Farmer.chat using GPT-4, covering a wide range of agricultural topics including crop advice, disease identification, weather forecasts, and market information.

Similarly, Jugalbandi, is another AI chatbot powered by OpenAI’s GPT models through Microsoft’s Azure OpenAI Service, which assists farmers and villagers in rural India to access information about various government schemes beneficial to them.

The chatbot, accessible via WhatsApp, retrieves relevant program details typically documented in English and delivers them in the user’s native language from among 10 of the 22 official Indian languages

Meanwhile, Wadhwani AI, a non-profit institute dedicated to developing AI solutions for social good, is exploring the use of generative AI to power Kissan (farmer) call centers. Alpan Raval, the chief scientist for AI/ML at Wadhwani AI, said that they are building a Kissan call center support system using generative AI to assist farmers with their queries.

Their approach involves augmenting human experts’ knowledge by utilising models that provide automated responses based on a knowledge base created from current government reports and documents, with a speech interface enabling conversational AI interactions.

KissanAI is another AI startup that has developed a proprietary AI chatbot that assists farmers with various agricultural tasks such as irrigation, pest control, and crop cultivation.Last year, it released Dhenu, the world’s first agriculture-specific LLM. Dhenu is designed to provide multilingual support, initially available in Hindi and English, making it accessible to a broad range of farmers across India.

The post OpenAI is Helping Farmers in India Increase Crop Yields appeared first on Analytics India Magazine.

Tata Communications recently unveiled a fully automated edge computing platform called CloudLyte, designed to empower future-ready enterprises to thrive in a data-driven world.

Tata Communications CloudLyte caters to the needs of global enterprises through its multi-access, cloud, and infrastructure-agnostic architecture.

Through its unique ‘solution in a box’ approach, Tata Communications CloudLyte provides enterprises with the platform, infrastructure, network, managed services, and the use case as a comprehensive unified offering. This approach enables swift deployment (within minutes) and effortless scaling as required, thus futureproofing investments.

With real-time inferencing and auto scaling, the platform seamlessly extends cloud capabilities to the edge, bringing in the agility and flexibility of the cloud.

Tata Communications CloudLyte also manages edge resources centrally for a seamless experience and has built-in security with features like zero-trust architecture and layered defences — simplifying enterprise operations, maximizing efficiency, and driving business growth.

“By bringing the cloud experience to the edge, Tata Communications CloudLyte allows global enterprises to harness the full potential of the cloud through a well-integrated cloud fabric. With Tata Communications CloudLyte — we’re not just building the future — we’re redefining it. From AI-driven predictive maintenance to retail analytics, the potential is limitless,” Neelakantan Venkataraman, vice president and global head – cloud and edge business, Tata Communications said.

The post Tata Communications Announces CloudLyte, a Fully Automated Edge Computing Platform appeared first on Analytics India Magazine.

Metadata management is critical to data lake architecture, ensuring that data is well-organized, easily discoverable, and effectively utilized. As data lakes store vast amounts of raw data in their native format, managing metadata becomes essential to maintain data quality, improve data governance, and facilitate data analytics and reporting. This article explores the importance of metadata management in data lakes and discusses how ETL processes play a role in capturing, storing, and managing metadata effectively.

What is metadata?

Metadata refers to the data about data. It provides the content, structure, and context of the data stored in a data lake. Metadata includes attributes such as data type, source, creation date, last modified date, and relationships between different data sets.

Importance of metadata management in data lakes

Effective metadata management in data lakes offers several benefits:

Improved data discoverability

Well-managed metadata enables data analysts and scientists to quickly discover and access relevant data sets within the data lake. This accelerates the data discovery process, reduces data silos, and promotes data reuse across the organization.

Enhanced data quality and governance

Metadata management helps maintain data quality by providing information about data lineage, transformations applied, and quality checks performed during the ETL processes. This transparency ensures data integrity and trustworthiness, facilitating better data governance and compliance with regulatory requirements.

Facilitated data analytics and reporting

Metadata provides valuable insights into the structure and content of the data, enabling users to understand the data schema, relationships, and dependencies. This knowledge is crucial for data analytics, reporting, and deriving meaningful insights from the data lake.

ETL and metadata management

The ETL process serves as a linchpin in metadata management within data lakes. It facilitates the seamless movement and transformation of data and acts as a conduit for the acquisition and enrichment of critical metadata. Let’s delve into the multifaceted contributions of ETL at each stage of the data lifecycle in metadata management.

Metadata capture during extraction

The initial stage of the ETL process, extraction, is instrumental in capturing essential metadata about the source data. This metadata encompasses a myriad of details, such as:

• Data Source Information: Identification of the source systems or applications from which the data originates, including database names, table names, and server details.
• Extraction Timestamps: Accurate recording of the date and time when the data was extracted, facilitating traceability and ensuring data lineage can be established.
• Source System Identifiers: Capture of unique identifiers or keys from the source system that allow for the tracing back to the original data source, aiding in data lineage tracking and validation.

By capturing this metadata during the extraction phase, ETL processes provide valuable context and lineage information that is crucial for understanding the data’s origin, quality, and history.

Metadata enrichment during transformation

The transformation phase of the ETL process is where data is cleaned, enriched, and structured to make it suitable for analysis and reporting. This phase also serves as an opportunity to enhance the metadata further by adding:

• Transformation Details: Detailed documentation of the transformations applied to the data, such as data cleansing rules, data type conversions, and calculations, providing insights into the data transformation logic and ensuring repeatability and consistency.
• Quality Metrics: Recording of data quality metrics, such as completeness, accuracy, and consistency checks performed during the transformation process, aiding in assessing data quality and compliance with quality standards.
• Business Rules and Logic: Storage information about any business rules or logic applied to the data is essential for interpreting and analyzing the data correctly and ensuring alignment with business requirements.

By enriching the metadata during the transformation phase, ETL processes contribute to enhanced data governance, transparency, and compliance while facilitating better data analytics and insights generation.

Metadata storage during loading

Once the data has been transformed, it is loaded into the data lake. Alongside the data, the metadata captured and enriched during the extraction. Transformation phases is stored in the data lake or a dedicated metadata repository. This metadata storage includes:

• Loading Timestamps: Accurate recording of the date and time when the data was loaded into the data lake. It facilitates data versioning and ensuring data freshness.
• Data Schema and Structure: Storing information about the data schema, field definitions, relationships, and dependencies. Provides a comprehensive view of the data structure and aiding in data exploration and querying.
• Metadata Cataloging: Organizing and cataloging the metadata to make it easily searchable and accessible for users, analysts, and data scientists. Promoting data discoverability, reuse, and collaboration across the organization.

By storing this metadata alongside the data, organizations can maintain a comprehensive and up-to-date repository of metadata. By providing valuable insights into the data’s structure, lineage, quality, and usage, thereby facilitating data-driven decision-making and innovation.

Benefits of ETL-driven metadata management

The seamless integration of ETL processes with metadata management in data lakes offers a multitude of benefits:

Improved data governance and compliance

ETL-driven metadata management enhances data governance by providing transparency into data lineage, transformations, and quality controls. This transparency ensures that data is managed, accessed, and used in compliance with organizational policies and regulatory requirements. It reduces data inconsistencies and non-compliance risks.

Enhanced data discovery and accessibility

By capturing and storing comprehensive metadata. ETL processes enable users to quickly discover, access, and understand the data within the data lake. This facilitates data reuse, reduces data silos, and promotes collaboration across the organization. Accelerating data-driven initiatives and fostering a culture of data-driven decision-making.

Facilitated data analytics and insights

The rich metadata captured and managed through ETL processes supports data analytics, reporting, and insights generation. It provides the necessary context, lineage, and quality information that analysts and data scientists require to derive meaningful insights, build accurate models, and make informed decisions. Thereby unlocking the full potential of the data lake for advanced analytics and innovation.

Conclusion

Metadata management is an important part of data lake architecture, supporting data discoverability, quality, governance, and analytics. ETL processes play a significant role in capturing, storing, and managing metadata throughout the data lifecycle. By implementing robust metadata management practices and leveraging ETL capabilities effectively. Organizations can maximize the value of their data lakes, enabling data-driven decision-making and fostering innovation across the enterprise.

Last month, US Ambassador to India Eric Garcetti hailed Indian immigrants and said, “…now the joke is, you cannot become a CEO in America if you are not Indian. Whether it is Google, Microsoft, or Starbucks, people have come and made a big difference…”

Responding to Garcetti’s remarks, Rajeev Chandrasekhar, MoS for Electronics and Information Technology, reflected on the transformation witnessed over the decades and said, “India is now home to #AllThingsTech and Indians are occupying leadership positions in almost all top companies of the world.”

Amidst this narrative of success, lies a question: Does every Indian inherently possess the potential to become a CEO, or is it a culmination of skill, struggle, and hard work that propels individuals to new heights of success?

While the success stories of Indian immigrants in the US are aplenty, the route to leadership is far from easy.

Story of every Indian in the US isn’t the same

Indian students studying in the United States, for instance, face significant challenges in securing even internships, attributed to a slowdown in job growth and heightened competition during election years. Factors such as rising inflation, increased cost of living, and sponsorship difficulties further compound their challenges.

As per a recent update, the US Citizenship and Immigration Services (USCIS) reported 780,884 H-1B registrations for FY 2024, an increase of 61% over the 483,927 registrations for FY 2023. But, there has been a significant decrease of nearly 40% in lottery applications for H-1B visas.

The USCIS selected only 14.6% of eligible H-1B registrations for FY 2024, based on a National Foundation for American Policy analysis of government data. That compares to 26.9% for FY 2023 and 43.8% for FY 2022. For FY 2021, nearly half, or 46.1%, were selected in the H-1B lottery process.

When Parag Agrawal was appointed the CEO of Twitter in 2021, it sparked discussions on the factors contributing to the success of Indian-origin individuals in senior positions within tech companies.

Some reports suggested that cultural shifts and strategic innovations spearheaded by Indian-origin CEOs played a pivotal role in the trajectory of companies like Microsoft and Google.

Meanwhile, Microsoft CEO Satya Nadella offered a different perspective in his book ‘Hit Refresh’. “Our industry does not respect tradition. What it respects is innovation,” he emphasised.

Indian Americans contribute to the US economy

Despite constituting only a small fraction of the US population, Indian Americans pose a significant influence through their contributions to the US economy. As highlighted by Tarak Hassan, a BU College of Arts & Sciences professor of economics, “The headline finding is that immigrants are good for local economic growth and, in particular, educated migrants are doing a lot of that.”

A recent study by the Migration Policy Institute (MPI) reveals that workers with immigrant backgrounds are prominently represented in the fields of science, technology, engineering, mathematics (STEM), and social sciences. In 2023, they accounted for 38% of the workforce in STEM-related occupations, where college-educated individuals prevail, with a median salary of $90,900 annually.

It is projected that these occupations will grow by a substantial 11% due to the increased adoption of AI and other technologies, digitalisation of the US economy, and the escalating threat of cyberattacks and data breaches.

As of now, India can’t rule America

Indian-Americans constitute the second-largest immigrant population in the US, estimated at around four million. However, it’s notable that Indians are not widely represented in leadership roles across many of the world’s top companies.

According to the CompTIA report, there are over 5,57,000 software and IT service companies in the US, with approximately 13,400 tech startups launched in 2019 alone. However, only a small fraction of individuals, who were previously Indian citizens but are now US citizens, hold higher positions there.

Despite India’s prominence in the tech industry and the presence of thousands of alumni from the IITs in the US since the 1960s, Indian passport holders occupying top positions in globally significant institutions remains rare.

Hope everyone gets the joke now.

The post Not all Indians are CEOs appeared first on Analytics India Magazine.

You’ve likely heard Elon Musk discuss Grok, the witty open-source AI chatbot created by his company xAI. Have you ever pondered the reason behind his decision to rebrand his AI system from TruthGPT to Grok?

Interestingly, xAI is not the only Silicon Valley company that is fascinated by the name Grok. There is a California-based AI chip company also named Groq (but with a ‘q’), which introduced a language processing unit (LPU), a new end-to-end processing unit system, earlier this year.

That’s not all. Last year, New Relic, the company behind the leading observability platform, also named its AI-powered observability assistant New Relic Grok.

In a recent interaction with AIM, New Relic CEO Ashan Willy joked that Musk stole the name from New Relic. “We did not copyright it,” he remarked.

And that’s not all either. Canadian musician Claire Elise Boucher, popularly known as Grimes released an AI-powered range of toys, one of which is named Grok.

Grimes, who shares three children with Musk, collaborated with Silicon Valley startup Curio to introduce these products. Intriguingly, there’s no connection between Curio’s offerings and Musk’s Grok; in fact, Curio’s AI toys came onto the scene even before Musk’s AI chatbot.

Why is Grok so popular?

So the question does arise, why are so many Silicon Valley products named Grok? The answer could be linked to Robert A Heinlein’s 1961 science fiction novel Stranger in a Strange Land.

Heinlein coined the term ‘grok’. In his novel, Valentine Michael Smith, a Martian, employs the word ‘grok’, which denotes empathising to such a deep extent with others that you merge or blend with them, conveying a profound understanding of someone or something.

In short, Grok means to understand things intuitively and empathetically, and hence, it makes sense to name an AI system Grok in an era when we are chasing superintelligence.

While it’s not certain, Heinlein’s novel is definitely the source of inspiration for Silicon Valley’s fascination with the word grok.

In an interview, Musk mentioned Heinlein’s novel as one of his favourites, along with Isaac Asimov’s works. Musk also tweeted the name of the book last year, but without providing any context.

Musk’s keen interest in colonising Mars might also stem from his deep admiration for Heinlein’s and Asimov’s works, both of which prominently feature Mars in their narratives.

Grokking the code

Even before Musk made the name Grok popular, it was a widely used term, both as a verb and noun, in the programming space.

In 1995, Ric Holt designed a database query engine for manipulating collections of binary relations and named it Grok (now defunct). Jingwei Wu at the University of Waterloo wrote a Java re-implementation of Grok, called JGrok.

In 2006, a number of Zope developers created an open-source web framework based on Zope Toolkit (ZTK) technology called Grok.

Then there is Grok AIOps, which was developed by Grokstream. Over the years, the word grok has been adopted to convey a deep, intuitive understanding of a concept, particularly when working with complex systems or technologies.

In the context of programming, “grokking” refers to not just comprehending code or algorithms superficially but rather internalising them to the point of truly understanding their intricacies and implications.

Silicon Valley’s fascination with science fiction

Interestingly, this is not the first time Silicon Valley got influenced by popular science fiction. Over the years, we have seen many founders being heavily influenced by it.

When OpenAI announced that its professional plan for ChatGPT was priced at $42, many linked it to Douglas Adams’ The Hitchhiker’s Guide to the Galaxy, another popular science fiction.

In Chapter 27 of the novel, two programmers, Lunkwill and Fook, are chosen to ask the Ultimate Question to Deep Thought, a supernatural computer programmed to calculate the answer to the Ultimate Question of Life, the Universe and Everything. The supercomputer pauses for a moment and reveals the answer—42.

OpenAI CEO Sam Altman himself has made references to the number. Last year, when asked about the release date of the upcoming multimodal successor to GPT-3, in a nod to the book, Altman joked, saying that the model will take “a while” to finish and that it kept responding ‘42’ to every prompt.

Moreover, Grok is modelled after Adams’ popular novel, so intended to answer almost anything and, far harder, even suggest what questions to ask, xAI notes.

Similarly, Jeff Bezos, the founder of Amazon, has revealed his enduring love for Star Trek, a passion that dates back to his childhood. He attributes the inspiration for Amazon’s virtual assistant, Alexa, to the omniscient computer featured on the Starship Enterprise in the series.

The post The Fascinating Reason behind Silicon Valley’s Love for the Word Grok appeared first on Analytics India Magazine.

Generative AI is probably the best product from humankind since fire and baked bread.

This analogy stands valid with respect to its comparison with fire because when fire was discovered, people feared it. They saw fire as apocalyptic, capable of causing destruction. It was only when we as humans worked on domesticating fire that evolution fell in place.

Artificial Intelligence (AI), specifically Generative AI stands at a similar juncture. At one side, we have tech enthusiasts, who are supercharged about the possibility of its implications across domains and industries. On the other hand, there are skeptics, who believe AI to be an agent of doom, causing ripples of fear around talent obsolescence by AI tools.

Now, this conversation gains more heat with the rise of a very niche application of Gen AI in the space of audio. After enabling artists and writers with the tools to think, visualize, and create better, Gen AI has carved a unique place in the field of acoustics.

What is generative AI audio?

In the simplest of terms, this technology involves the generation of audio content with text as input or prompts. This audio content could range between anything from the generation of sound effects to an entire music album (more on its applications later).

The anatomy of generative AI audio

Converting texts as simple and vague as cinematic music for a horror short film comprising a string section is a daunting task. It’s a complex magic trick that involves layers of intricate technologies, techniques, and processes.

For the successful generation of audio that is as close to being passable involves techniques such as:

Tokenization – where data is broken down into discrete tokens that are individually analyzed and processed by Machine Learning (ML) algorithms at work. Each token is attributed to a distinct aspect of an audio signal such as pitch, scale, rhythm, and more.

Quantization – where continuous audio signals are represented as discrete values so the same generation technique deployed in LLMs (Large Language Models) can be utilized for audio generation.

Vectorization – that involves the transformation of audio signals into high-dimensional vector spaces to establish the relationship between diverse audio signals. ML models then identify and interpret patterns in signals to generate fresh audio or sound.

Applications of generative AI audio

While it’s easier to classify the applications of Gen AI audio into just music creation or deep fake audios, there are several unique and game-changing use cases of this technology that are highly relevant and required today.

Let’s explore some compelling ones.

Voiceovers and text to speech in EdTech

One of the most novel applications of Generative AI audio lies in the space of EdTech and infotainment, where artificial voice synthesizer technologies to generate voices of tutors and sound effects can be used to uplift storytelling in audiobooks, YouTube videos, course reading materials, eLearning modules, and more.

Sound designing

In the realm of movies and video games, where creativity knows no bounds, technicians are often compelled to invent fresh sounds. Imagine the sounds of Dune that made the auditory experience as immersive as possible; studios are required to push limits of familiarity and blend two or more everyday sounds into something unheard. Gen AI for audio can pull this off seamlessly through prompts that can be later tweaked by experts.

AI music creation

While this is a delicate topic that involves opinions and debates, it’s inevitable to acknowledge the power of Gen AI in creating music from scratch. From gaming to filmmaking, AI tools enable independent creators and artists with limited budget to elevate their content into something epic and cinematic.

Hyper-personalized chatbots

As brands and businesses race relentlessly to deliver the best of personalized experiences to customers, they can amplify this by a notch through Generative AI audio. Based on target audiences and demographics, chatbots can be trained to speak in specific ethnicities involving accents, dictions, and slangs people are familiar with for instant brand connection.

Real-time audio description for accessibility

Smartphones, devices, and even video streaming platforms have real-time audio description of content to simplify complexities for visually impaired people. Such autonomous generation of real-time audio content enables differently abled people to seamlessly achieve their everyday task, which otherwise might be constraining.

Challenges involved in generative AI audio development

Despite its potential being vast and niche, there are several bottlenecks stalling tech enthusiasts and businesses from leveraging the complete potential of this technology. Instead of taking a generic approach, let’s classify them into three unique aspects:

Technical and output-specific challenges

As fascinating as music generation from scratch sounds, the concept is still in its nascent stages. This means it is not devoid of technical concerns such as

• poor audio quality
• missing beats
• robotic delivery of accents and voices
• inconsistencies in real-time generation of audio
• audio results completely in a different tangent from entered prompt
• high latency and more.

Ethical constraints

This is a two-fold challenge that involves:

Deepfakes and misinformation, where synthesized audio morphed over an existing video or plain audio clips can be generated in the voices of targets to extract money or money’s worth or push specific agendas

Ownership and copyright that debates the perpetual question of who owns the music or sound generated by AI? Moreover, is it ethical to train on online data for audio synthesize without fair compensation to original creators?

Sourcing training datasets

The previous challenge is an excellent segue into this topic as black hat techniques prevail in training Generative AI audio models. With that said, there is also a pressing demand for quality audio training datasets that have clean audio customized to distinct requirements.

Bias is another critical concern in training Generative AI audio models, where subconscious input of audio data that is stereotypical, inaccurate or offensive could ruin and rupture outputs generated.

Microsoft is done relying on others for its models. According to reports, though the company had been training smaller models like Orca and Phi all this while (using GPT and incorporating Meta’s Llama on its platform), this time it is training a model large enough to compete with others.

Referred to as MAI-1 (possibly Microsoft AI-1), the model is being developed internally by the company, and is around 500 billion parameters in size.

Its development is being headed by Mustafa Suleyman, formerly a leader in AI at Google and most recently CEO of the AI startup Inflection, who now oversees Microsoft’s AI division. In March, Microsoft acquired a majority of Inflection’s staff and paid $650 million for its intellectual property rights.

However, MAI-1 is a Microsoft-developed model, distinct from those previously developed by Inflection. While it may leverage training data and technology from the startup, it is an independent project.

Though the exact purpose has not been disclosed just yet, it is possible that Microsoft might incorporate its products into all its Copilot products. This would mean that the company will move away from OpenAI’s GPT and Codex models.

This comes in the backdrop of OpenAI partnering with Stack Overflow to improve its products. The big-tech company is also planning to release its search engine powered by Bing possibly as a way to compete with Google, which Microsoft has a history of doing.

No competition?

Microsoft’s strategy has highlighted three trends for 2024 – small language models, multimodal AI, and AI in science.

This time around, it seems like the company has decided to incorporate multimodal AI with large language models. The small language models will continue to be incorporated into the company’s on-edge use cases, such as laptops, while larger ones might be available in other core products.

All of this comes only weeks before the Microsoft Build conference. So it is possible that the model will be introduced at the conference. To train the new model, Microsoft has allocated a significant cluster of servers equipped with NVIDIA GPUs.

To make the case clear, Microsoft’s CTO Kevin Scott went on LinkedIn to explain that this is not in any way a competition with OpenAI. “I’m not sure why this is news, but just to summarise the obvious: we build big supercomputers to train AI models. Our partner OpenAI uses these supercomputers to train frontier-defining models; and then we both make these models available in products and services so that lots of people can benefit from them. We rather like this arrangement,” he said.

He further stated that the company has always been building bigger and better supercomputers for OpenAI to further AI research and wishes to continue this arrangement. “There’s no end in sight to the increasing impact that our work together will have,” he added.

So it seems like Sam Altman’s $50 billion dream of building AGI is going to get funded by Microsoft.

Scott also clarified that all the research that Microsoft does is about building models: “AI models turn out to be interesting things to work on, and our researchers do great work studying and building them.”

He also confirmed that there would be more models coming out soon, including MAI, Phi, and even Turing.

But is it true?

Even though Scott clarified that there is no competition between OpenAI and Microsoft, it is still worth a wonder if a model as big as MAI-1 would actually be used for Microsoft products instead of OpenAI’s GPT.

Moreover, it would be ideal for Microsoft to have a backup plan just in case the deal with OpenAI falls through, as has been the case with several others in the field. Chief Satya Nadella seems to be playing a different AI game. Under him, Microsoft has invested in all kinds of AI companies, from OpenAI and Mistral to Databricks and Figure AI.

Recently an email from 2019 resurfaced, in which Scott told the Microsoft team that they needed to invest in OpenAI as the competition with Google was rising and that their AI model was “scarily good”.

Suleyman recently posted on X saying, “AI is everything at Microsoft”. He also highlighted that the company is building massive products using AI and has a definite vision for Copilot.

Everything about this seems forced. There seems to be no other reason for Microsoft to build such large models and spend so much on compute if they’re not making it for commercial purposes. Moreover, on his hiring, Suleyman was touted as the “new” Sam Altman.

So, is Microsoft possibly becoming the OG of AI, aka OpenAI?

The post Microsoft is Starting to Look a Lot Like OpenAI with ‘MAI-1’ appeared first on Analytics India Magazine.