Granica Launches Industry-first AI Efficiency Platform to Cut Cloud Data Costs up to 80% and Boost ROI from AI Initiatives

Granica Launches Industry-first AI Efficiency Platform to Cut Cloud Data Costs up to 80% and Boost ROI from AI Initiatives June 8, 2023 by Andrew Jolly

MOUNTAIN VIEW, Calif., June 8, 2023 — Granica emerged from stealth today and introduced the industry’s first AI efficiency platform, bringing novel, fundamental research in data-centric AI to the commercial market as an enterprise solution. The company’s cloud-native solution operationalizes a next-generation approach to AI efficiency via data reduction and data privacy, enabling AI teams using Amazon S3 and Google Cloud Storage (GCS) to derive maximum value from their ever-growing volumes of training data. Granica makes it feasible for more AI data to be cost-effectively captured, stored and used to power enterprise AI implementations and thereby improve model performance and business outcomes.

Granica physically reduces the size and cost of petabyte-scale AI training data in cloud object stores by up to 80% using novel compression and deduplication algorithms. Granica also preserves privacy of sensitive information in object data, enabling its safe use in AI and analytics while improving data security posture. At launch, Granica is focused on efficiency for data, with plans to drive efficiency across the end-to-end AI pipeline. For the first time, C-level executives in data intensive industries can drive significant profitability and innovation gains by pursuing AI efficiency, complementing their existing efficiency initiatives.

The company’s revolutionary outcome-based pricing model only charges users a small percentage of the savings generated per month—and only if savings are generated—so that deployment of the solution delivers upside only. Co-founded by CEO Rahul Ponnala and CTO Tarang Vaish, seasoned data experts and former engineers at Pure Storage and Cohesity, Granica’s total funding to date is $45 million from leading venture capital firms New Enterprise Associates (NEA), Bain Capital Ventures (BCV) and others, with participation from several industry luminaries including former Tesla CFO Deepak Ahuja; Eventbrite Chairman and co-founder Kevin Hartz; and Frederic Kerrest, Executive Vice Chairman and co-founder, Okta.

Pete Sonsini, investor at NEA, observed, “The sheer volume of data required to properly train AI models makes responsible and performant AI out of reach for many organizations. Granica democratizes access to AI while keeping data secure — to make AI more accessible, affordable and safe to use. Granica’s fusion of novel research and systems engineering places the company in a strong position to lead the new wave of data-centric AI.”

Enterprises Are Struggling to Achieve Significant ROI on AI Due to Out-of-control Cloud Costs

McKinsey reports that AI adoption has more than doubled since 2017, however Boston Consulting Group determined that a mere 10% of organizations achieve significant financial return on AI.

AI is only as effective as the dataset it is trained on. To power AI models capable of rapidly generating new business insights and efficiencies, enterprises are obligated to store massive volumes of data in cloud object stores. However, for most companies it is too costly to do so at the desired scale, leading to significant volumes of important data being archived or deleted, constraining model performance and thus AI outcomes.

“Our mission is to enable enterprise AI teams to maximize the value of their data and keep much more, if not all, of their AI data ‘hot.’ This is the key to unlocking the transformative potential of artificial intelligence and machine learning,” said Rahul Ponnala, co-founder and CEO of Granica. “Data fuels the AI engines that are quickly becoming essential to modern commerce, science and everyday life. Just look at the sudden explosion in generative AI tools to get a sense of the future reach of this technology.”

Granica Fuses Novel Fundamental Research and Engineering to Close the AI Efficiency Gap for Data-forward Enterprises

The inefficiency in AI is driven largely by the rapid proliferation of training data with low information value. Such data often contains significant redundancy and sensitive information, including personally identifiable information (PII). This information inefficiency increases costs, risks and the time it takes for teams to move data through the AI pipeline.

“There is a huge efficiency gap for AI workloads, especially for training data in cloud object stores like Amazon S3 and Google Cloud Storage. Making data more efficient requires an entirely new layer in the AI stack consumed as an API and directly integrated with AI applications,” said Andrea Montanari, chief scientist at Granica. “At its core, what we’re doing at Granica is fusing fundamental data-centric AI research with large-scale systems engineering expertise to build a platform that drives AI information density and efficiency at cloud scale.”

“Our research team – led by our chief scientist Andrea Montanari, a pioneering expert in information theory and a professor at Stanford University – works in unison with our world class engineering team, pushing the boundaries of data-centric AI to deliver first-to-market solutions that make AI affordable, accessible and safe to use,” Ponnala remarked. “Building a technology company that solves hard problems requires empowering brilliant people as much as commercializing cutting edge research. We take pride in our people-centric approach to our organization.”

Granica Opens New Pathways for Enterprises to Pursue AI Innovation Leveraging the Cloud

Through a cloud service provider-native infrastructure, the Granica AI Efficiency Platform integrates inline with cloud applications, making data as well as downstream pipelines and models more efficient, more performant and privacy preserving. Granica Crunch, the company’s data reduction service, eliminates redundant and low-value data, cutting costs and speeding up downstream processes for hot AI data. Granica Screen, the company’s data privacy service, enables organizations to safely leverage sensitive data for AI and business use cases while improving their data security posture and reducing breach risk. As Granica continues on its mission to build the de facto standard for AI efficiency, data trust and collaboration, additional services and products will be brought to market to provide companies with end-to-end AI efficiency and new ways to maximize ROI on AI.

Benefits of Granica’s AI Efficiency Services:

  • Granica Crunch is the data reduction service for enterprise AI: It provides Byte-granular Data Reduction containing novel compression and deduplication algorithms which losslessly reduce the physical size of enterprise AI training data such as sensor, image and text files, reducing costs to store and transfer objects in Amazon S3 and GCS by up to 80%. Furthermore, it reduces write costs by up to 90% by intelligently batching write requests and optimizing other storage operations.
  • Granica Screen is the data privacy service for enterprise AI: It provides Byte-precise Detection for both high recall and high precision identification and protection of sensitive data, including PII contained in structured, semistructured and unstructured text data. Granica Screen enables enterprises to improve their data security posture and prevent breaches, safely use their data for AI and analytics use cases, and simplify regulatory compliance. It is built to enable privacy-enhanced computing. Granica Screen is available via an Early Access Program.

Everyone Wins with Granica’s Radically New, Customer-oriented Business Model

Granica’s revolutionary business model is focused on delivered outcomes as opposed to mere consumption. The platform is free to deploy with no upfront costs. After deployment, Granica measures how Crunch reduces storage costs relative to the Amazon S3 and GCS baseline. The cloud costs incurred in the user’s environment is then covered by the generated baseline savings, resulting in a savings outcome for the user. Granica customers simply pay Granica a small portion of the savings outcome.

Granica’s pricing model is new to the AI industry and provides the ultimate customer-centric solution. Benefits include:

  • Pay only for value received.
  • No financial risk to try or to expand usage.
  • No need to find or allocate budget.
  • No need for complicated total cost of ownership (TCO) modeling.
  • Easy to forecast future savings for reinvestment into AI data, people and tooling.

Generally available starting today, Granica frees up significant savings to supercharge ROI on AI initiatives. To request a demo, visit https://www.granica.ai/demo.

About Granica

Granica is the world’s first AI efficiency platform. It boosts ROI on AI by increasing the information efficiency of AI training data, freeing up resources that C-level executives and enterprise AI teams can use to reduce costs and improve AI performance and outcomes. Powered by novel compression and deduplication algorithms, its byte-granular inline data reduction of petabyte-scale sensor, image and text AI training data runs transparently in the background, losslessly reducing AI data costs by up to 80%.

Source: Granica

Related

AI: Large Language & Visual Models

AI: Large Language & Visual Models
Image by Editor

Large models, whether they are language models or visual models, are designed to process massive amounts of data using deep learning techniques. These models are trained on vast datasets and can learn to recognize patterns and make predictions with incredible accuracy. Large language models, such as OpenAI's GPT-3 and Google's BERT, are capable of generating natural language text, answering questions, and even translating between languages. Large visual models, such as OpenAI's CLIP and Google's Vision Transformer, can recognize objects and scenes in images and videos with remarkable precision. By combining these language and visual models, researchers hope to create more advanced AI systems that can understand the world in a more human-like way. However, these models also raise concerns about data bias, computational resources, and the potential for misuse, and researchers are actively working to address these issues. Overall, large models are at the forefront of the field of AI and hold great promise for the development of more advanced, intelligent machines.

The Digital Era

The 21st century was marked by a significant increase in the volume, velocity, and variety of data being generated and collected. With the rise of digital technologies and the Internet, data began to be generated at an unprecedented scale and speed, from a wide range of sources including social media, sensors, and transactional systems. Let’s us remind you of some of them:

  • The growth of the Internet: The Internet rapidly grew in size and popularity during the 1990s, creating vast amounts of data that could be analyzed for insights.
  • The proliferation of digital devices: The widespread use of smartphones, tablets, and other connected devices has created a massive amount of data from sensors, location tracking, and user interactions.
  • The growth of social media: Social media platforms such as Facebook and Twitter have created enormous amounts of data through user-generated content, such as posts, comments, and likes.
  • The rise of e-commerce: Online shopping and e-commerce platforms generate large amounts of data on consumer behavior, preferences, and transactions

These and other trends led to a significant increase in the amount of data being generated and collected and created a need for new technologies and approaches to manage and analyze this data. This led to the development of big data technologies such as Hadoop, Spark, and NoSQL databases, as well as new techniques for data processing and analysis, including machine learning and deep learning. Actually, the rise of big data was a key driver of the development of deep learning techniques, as traditional machine learning approaches were often unable to effectively analyze and extract insights from large and complex data sets.

Deep learning algorithms, which use artificial neural networks with multiple layers, were able to overcome these limitations by learning from vast amounts of data and recognizing complex patterns and relationships within that data. This enabled the development of powerful models capable of processing a wide range of data types, including text, images, and audio. As these models became more sophisticated and capable of handling larger and more complex data sets, they gave rise to a new era of AI and machine learning, with applications in fields such as natural language processing, computer vision, and robotics. Overall, the development of deep learning has been a major breakthrough in the field of AI, and it has opened up new possibilities for data analysis, automation, and decision-making across a wide range of industries and applications.

A synergy of Big, Deep, Large

Large languages and visual models, such as GPT3/GTP4 and CLIP, are special because they are capable of processing and understanding large amounts of complex data, including text, images, and other forms of information. These models use deep learning techniques to analyze and learn from vast amounts of data, allowing them to recognize patterns, make predictions, and generate high-quality outputs. One of the key advantages of large language models is their ability to generate natural language text that closely resembles human writing. These models can produce coherent and convincing written passages on a wide range of topics, making them useful for applications such as language translation, content creation, and chatbots. Similarly, large visual models are capable of recognizing and categorizing images with remarkable accuracy. They can identify objects, scenes, and even emotions depicted in images, and can generate detailed descriptions of what they see. The unique capabilities of these models have many practical applications in fields such as natural language processing, computer vision, and artificial intelligence, and they have the potential to revolutionize the way we interact with technology and process information.

The combination of large language and large visual models can provide several synergies that can be leveraged in a variety of applications. These synergies include:

  • Improved multimodal understanding: Large language models are excellent at processing text data, while large visual models are excellent at processing image and video data. When these models are combined, they can create a more comprehensive understanding of the context in which the data is presented. This can lead to more accurate predictions and better decision-making.
  • Improved recommendation systems: By combining large language and visual models, it is possible to create more accurate and personalized recommendation systems. For example, in e-commerce, a model could use image recognition to understand a customer's preferences based on their previous purchases or product views, and then use language processing to recommend products that are most relevant to the customer's preferences.
  • Enhanced chatbots and virtual assistants: Combining large language and visual models can improve the accuracy and naturalness of chatbots and virtual assistants. For example, a virtual assistant could use image recognition to understand the context of a user's request, and then use language processing to provide a more accurate and relevant response.
  • Improved search functionality: By combining large language and visual models, it is possible to create more accurate and comprehensive search functionality. For example, a search engine could use image recognition to understand the content of an image, and then use language processing to provide more relevant search results based on the image's content.
  • Enhanced content creation: Combining large language and visual models can also enhance content creation, such as in video editing or advertising. For example, a video editing tool could use image recognition to identify objects in a video, and then use language processing to generate captions or other text overlays based on the content of the video.
  • More efficient training: Large language and visual models can be trained separately and then combined, which can be more efficient than training a single large model from scratch. This is because training a large model from scratch can be computationally intensive and time-consuming while training smaller models and then combining them can be faster and more efficient.

Overall, the combination of large language and visual models can lead to more accurate, efficient, and comprehensive data processing and analysis, and can be leveraged in a wide range of applications, from natural language processing to computer vision and robotics.

GAI or not GAI

It is difficult to predict whether the development of large models will eventually lead to the creation of general artificial intelligence (GAI), as GAI is a highly complex and theoretical concept that remains the subject of much debate and speculation in the field of artificial intelligence. While large models have made significant advances in areas such as natural language processing, image recognition, and robotics, they are still limited by their training data and programming and are not yet capable of true generalization or autonomous learning. Furthermore, the creation of GAI would require breakthroughs in several areas of AI research, including unsupervised learning, reasoning, and decision-making. While large models are a step in the right direction, they are still far from achieving the level of intelligence and adaptability necessary for GAI. In short, while the development of large models is an important step towards more advanced forms of artificial intelligence, it is still uncertain whether they will ultimately lead to the creation of general artificial intelligence.

Challenges

Data bias is a significant concern in large models, as these models are trained on massive datasets that can contain biased or discriminatory data. Data bias occurs when the data used to train a model does not represent the diversity of the real-world population, resulting in the model producing biased or discriminatory outputs. For example, if a large language model is trained on text data that is biased against a particular gender or ethnicity, the model may produce biased or discriminatory language when generating text or making predictions. Similarly, if a large visual model is trained on image data that is biased against certain groups, the model may produce biased or discriminatory outputs when performing tasks such as object recognition or image captioning. Data bias can have serious consequences, as it can perpetuate and even amplify existing social and economic inequalities. It is therefore crucial to identify and mitigate data bias in large models, both during training and during deployment.

One way to mitigate data bias is to ensure that the datasets used to train large models are diverse and representative of the real-world population. This can be achieved through careful dataset curation and augmentation, as well as through the use of fairness metrics and techniques during model training and evaluation. In addition, it is important to regularly monitor and audit large models for bias and to take corrective action when necessary. This can involve retraining the model on more diverse data or using post-processing techniques to correct biased outputs. Overall, data bias is a significant concern in large models, and it is crucial to take proactive measures to identify and mitigate bias in order to ensure that these models are fair and equitable.

Ethics Side

The decision by OpenAI to give exclusive commercial rights to Microsoft for its large language model GPT-3 has generated some debate within the AI community. On one hand, it can be argued that partnering with a large tech company like Microsoft can provide the resources and funding necessary to further advance AI research and development. Additionally, Microsoft has committed to using GPT-3 in a responsible and ethical way and has pledged to invest in the development of AI that is aligned with OpenAI's mission. On the other hand, some have raised concerns about the potential for Microsoft to monopolize access to GPT-3 and other advanced AI technologies, which could limit innovation and create power imbalances in the tech industry. Additionally, some have argued that OpenAI's decision to grant exclusive commercial rights to Microsoft goes against its stated mission of advancing AI in a safe and beneficial way, as it may prioritize commercial interests over societal benefits. Ultimately, whether OpenAI's decision to give exclusive commercial rights to Microsoft is "ok" or not depends on one's perspective and values. While there are valid concerns about the potential risks and drawbacks of such a partnership, there are also potential benefits and opportunities that could arise from working with a large tech company like Microsoft. It is up to the AI community and society as a whole to closely monitor the impact of this partnership and ensure that AI is developed and deployed in a way that is safe, beneficial, and equitable for all.

Market Share

Each of these models has its own strengths and weaknesses, and they can be used for a variety of natural language processing tasks such as language translation, text generation, question answering, and more. As an AI language model, ChatGPT is considered one of the most advanced and effective language models currently available. However, there are other models that have been developed that can outperform ChatGPT on certain tasks, depending on the specific metrics being used to evaluate performance. For example, some models have achieved higher scores on benchmark natural language processing tasks such as GLUE (General Language Understanding Evaluation) or SuperGLUE, which evaluate a model's ability to understand and reason about natural language text. These models include:

  • GShard-GPT3, a large-scale language model developed by Google that achieved state-of-the-art performance on several NLP benchmarks
  • T5 (Text-to-Text Transfer Transformer), also developed by Google, which has achieved strong performance on a wide range of NLP tasks
  • GPT-Neo, a community-driven project that aims to develop large-scale language models that are similar to GPT-3, but are more accessible and can be trained on a wider range of hardware

It is worth noting, however, that performance on these benchmarks is just one aspect of a language model's overall capabilities, and that ChatGPT and other models may outperform these models on other tasks or in real-world applications. Additionally, the field of AI is constantly evolving, and new models are being developed all the time that may push the boundaries of what is possible.

Useful Links

  1. What Is ChatGPT Doing … and Why Does It Work?
  2. OpenAI's GPT-3: https://openai.com/blog/gpt-3-unleashed/
  3. Google's BERT: https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html
  4. Facebook's RoBERTa: https://ai.facebook.com/blog/roberta-an-optimized-method-for-pretraining-self-supervised-nlp-systems/
  5. Google's T5: https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html
  6. OpenAI's CLIP (Contrastive Language-Image Pre-Training): https://openai.com/blog/clip/
  7. Microsoft's Turing-NLG: https://www.microsoft.com/en-us/research/blog/microsoft-announces-turing-nlg-state-of-the-art-model-for-natural-language-generation/
  8. Hugging Face's Transformer Library: https://huggingface.co/transformers/

Ihar Rubanau is Senior Data Scientist at Sigma Software Group

More On This Topic

  • What are Large Language Models and How Do They Work?
  • Learn About Large Language Models
  • Top Open Source Large Language Models
  • Top Free Courses on Large Language Models
  • Introducing TPU v4: Googles Cutting Edge Supercomputer for Large Language…
  • Introducing Healthcare-Specific Large Language Models from John Snow Labs

EvenUp wants to automate personal injury settlements — to a point

EvenUp wants to automate personal injury settlements — to a point Kyle Wiggers 8 hours

Millions of personal injury cases are settled in the U.S. every year, as few go to trial — but the vast majority are kept under wraps. This leaves lawyers guessing what they should propose as a settlement price, oftentimes resulting in victims being undercompensated.

It’s what led Rami Karabibar to launch EvenUp, a startup that taps AI to generate legal documents to assess injury cases. The platform, aimed at customers in the legal field, attempts to turn raw case files including medical records, police reports and bills to create letters arguing for proposed compensation.

“We’re on a mission to level the playing field in personal injury cases,” Karabibar, who previously worked across private equity, venture capital and venture-backed startups, said.

Karabibar co-founded EvenUp with Ray Mieszaniec, a two-time entrepreneur, whose father was permanently disabled after being hit by a car involved in a police chase. Mieszaniec’s family got just 10% of the average payout for that type of accident — partly because their lawyer didn’t know what the appropriate compensation should be,

EvenUp aims to tackle all categories of personal injury cases, including motor vehicle accidents, police brutality, child abuse and even natural disasters. To do this, Karabibar, Mieszaniec and EvenUp’s third co-founder, Saam Mashhad (a former litigator), built a database of private settlements — including hundreds of thousands of medical records — and trained an AI to estimate fair compensation based on the details of each case.

EvenUp’s platform extracts the relevant info from documents and organizes them into templated “demand packages,” which state the legal and factual basis for a personal injury claim and include a demand for compensation. Designed to be a self-service solution for lawyers, paralegal staff and law firms, EvenUp summarizes notes and copies of raw records into medical digests “optimized for injury law.”

“The more documents and cases we see, the better we are at preparing demand packages, and the better we are at increasing case outcomes and reducing costs,” Karabibar said. “EvenUp reaches deeper in the legal workflow with a higher bar for accuracy than other AI assistants, from extracting data out of raw documents, to valuing what cases are worth, to generating final demand packages that bring that all together.”

As Karabibar alluded to, EvenUp isn’t the only startup applying AI to the tedious — and monotonous — task of drafting legal documents. Lawyaw, which emerged from stealth several years ago, is building software to automate the process of customizing standard documents like NDAs and wills. Elsewhere, Atrium’s software digitizes legal paperwork and builds apps on top to speed up fundraising, commercial contracts, equity distribution and employment issues.

But EvenUp claims that it’s one of the first to tackle personal injury — a law practice area not necessarily held in high regard. So-called “settlement mills,” which charge between 33% to 40% of total awarded compensation, settle a high volume of cases without necessarily focusing on maximizing the value of each claim.

Mieszaniec implies that EvenUp could change this by normalizing the practice of AI-aided personal injury litigation.

“By harnessing the potential of technology, we can create a future where the pursuit of justice is not marred by financial pressure or the representation you have,” Mieszaniec said via email. “It’s time to embrace innovative solutions that streamline the claims process, empower individuals, humanize the process and ensure that no one walks away with a fraction of what they deserve. That’s why we built EvenUp: to level the playing field for personal injury victims.”

EvenUp appears to have won over investors, who recently pledge $50.5 million in the company at a $325 million valuation (according to a source familiar with the matter). Bessemer Venture Partners led the latest round, a Series B, with participation from Bain Capital Ventures, Behance founder Scott Belsky and legal tech firm Clio, bringing EvenUp’s total raised to $65 million.

But can the tech live up to its promises — and address the outstanding legal and ethical implications?

With any AI tech, bias is a major concern. Algorithms trained on biased data can amplify those biases, perpetuating existing inequalities and injustices. For instance, a 2016 ProPublica analysis found that a widely-used algorithm was twice as likely to misclassify Black defendants as presenting a high risk of recidivism than white defendants. One can imagine EvenUp’s AI recommending artificially high or low amounts of personal injury compensation as a result of data set imbalances.

And what about privacy? EvenUp hasn’t disclosed where it sourced the medical and personal injury documents that it used to train its AI — nor whether it took steps to notify the original owners of those records.

That’s assuming once again that the tech works as advertised, even. If there’s any overarching takeaways from the generative AI boom, it’s that even the best AI algorithms today are far from perfect. (See: Microsoft’s Bing chatbot spouting vaccine misinformation and writing a hateful screed from the perspective of Adolf Hitler.)

If EvenUp’s customers share these concerns, it’s not obvious from their rush to adopt the platform. Karabibar claims that EvenUp counts “top trial attorneys” and “America’s largest personal injury law firms” among its customers and that it’s “close to profitable.”

Some, no doubt, are chasing after the chance to reduce filing expenses while maximizing returns. Karabibar doesn’t deny this.

“Injury attorneys are contingency-based, where they make a fixed percentage of the value of the case. Any increase in case outcomes directly impacts their revenue, while also increasing the amount clients receive,” he said.

But Karabibar also makes the case — rather optimistic, in my view — that automating aspects of the filing process could encourage litigators to “concentrate more on the human side of their work.” He‘s also careful to suggest that EvenUp won’t replace lawyers outright. But, reading between the lines a little, it’s hard not to see how some paralegals, most of whom work on a contract basis, might find themselves out of a job if the tech were ever to be adopted on a mass scale.

“They’ll be able to support injury victims through the legal process, and advocating for the equitable outcomes their clients deserve,” he said.

We’ll see if that’s the case. In any event, EvenUp has broad ambitions, with plans to cover document generation in both the pre-litigation and litigation stages customized to each firm, jurisdiction and case type. Karabibar believes that EvenUp will eventually be able to handle 70% of the key documents in the personal injury law workflow.

“We are well-positioned to continue growing despite the turbulent economy, and we believe our products will only become more essential as time goes on,” Karabibar said. “Legal drafting has seen a fundamental step-change given the advent of generative AI. Legal professionals will need to be quick to adapt to this change or be competed out of the profession by more tech-savvy competitors.”

AI in Media: How Is Generative AI Revamping the Media & Entertainment Industry

Featured Blog Image-AI in Media: How Is Generative AI Revamping the Media & Entertainment Industry

In 2021, OpenAI introduced DALL-E, a deep learning model that can generate realistic images from text prompts. This was the first widely recognized and commercial Generative AI tool. Since then, 100s of tools related to Generative AI in media and other domains have been released with applications in all facets, including art, music, marketing, entertainment, etc.

If you are wondering what Generative AI is, let’s discuss it briefly below.

Generative AI – a specialized Artificial Intelligence (AI) domain – uses advanced algorithms to generate realistic text, audio, images, or videos. Most notably, Generative AI dates back to 2014 when GANs (Generative Adversarial Networks) were introduced. GANs were the earliest models that were able to generate lifelike images of humans that never existed. After that, VAE (Variational Autoencoders), Diffusion models, and Transformers models have become the backbone of Generative AI.

Generative AI got a significant spotlight in November 2022 when OpenAI introduced ChatGPT – a Large Language Model (LLM) that can generate human-like text and can have engaging conversations. Just five days after its release, ChatGPT crossed 1 million users.

Let’s discuss how Generative AI models have supercharged the media and entertainment industry.

Generative AI in Media & Entertainment – 4 Major Applications

By 2030, AI is expected to become a 1.5 trillion dollar market, impacting all major industries, including media and entertainment which is at the forefront of this AI impact. For instance, AI in social media, alone, will reach 12 billion dollars in market size by 2031. More and more consumers now prefer to interact with AI-powered chatbots to get instant responses to their queries. And, marketers are more inclined to leverage AI in their advertising campaigns.

Let’s look at some major applications of AI in media.

1. AI-Powered Content Writing

Creative writing is the most celebrated application of Generative AI. LLMs like ChatGPT have entirely removed writer’s block by inspiring new avenues of creativity. Journalists, scriptwriters, social media copywriters, blog writers, and storytellers can use LLMs to create engaging content instantly.

However, AI-based content generation has sparked a debate about whether or not search engines like Google will rank such content. Google recently mentioned its stance that it would rank content according to expertise, experience, authoritativeness, and trustworthiness, irrespective of how it’s generated. But, if someone plans on using AI to manipulate search results, that’s not going to work.

2. AI-Powered Image Generation

Image generation models like Stable Diffusion, DALL-E, and Midjourney can generate high-quality, hyper-realistic images from natural language prompts. In the media and entertainment industry, content creators can use these images in blogs, articles, social media posts, and advertisements. This can not only cut production costs but brings more creativity to an artist’s work. Moreover, these images can be used as input prompts for AI-based video generation.

3. AI-Powered Film Production

Recently, 28 Squared Studios, in collaboration with Moon Ventures, created the short film “The Safe Zone”, written and directed by AI (ChatGPT). ChatGPT provided each character's script, camera positioning, outfits, and facial expressions. The Safe Zone was the first use of AI in filmmaking. In the future, we can expect to see complete films being generated from AI.

The film’s producers say that they created the short film to highlight the collaborative potential between Generative AI and humans, demonstrating how they can work together to produce high-quality content. Additionally, the short film provides a sneak peek into the future of storytelling, offering insight into the harmonious coexistence of humans and AI within the creative domain.

4. AI-Powered Marketing

AI can analyze large volumes of marketing data to validate what works and what doesn’t in the marketing industry. According to a 2023 survey, 73% of marketing executives are using Generative AI tools for B2B and B2C marketing. It can help marketers identify customer behavior and patterns from feedback and generate compelling and engaging text, image, or video content.

Generative AI Videography – A New Era

Many Generative AI models and tools have been released recently that are revolutionizing the field of videography. Some prominent tools are:

  1. Runway Research created a model called “Gen-2” that can create novel videos from text, images, or video clips.
  2. In March 2023, Google Research, in collaboration with The Hebrew University of Jerusalem, presented Dreamix, a diffusion-based model that creators can use for video editing and performing text-based motion.
  3. Nvidia, the American tech company that manufactures and designs GPUs (Graphic Processing Units), introduced its text-to-video model. The model uses Latent Diffusion Models to create high-quality videos.
  4. Artists are using a combination of these AI videography tools for creating music videos.

By leveraging Generative AI’s video production capabilities, creators can tweak visual aesthetics to create an immersive experience for their viewers.

Embracing Generative AI: One Step at a Time

The output created by Generative AI, especially image and video, lacks perfection. For instance, DALL-E and Stable Diffusion sometimes make garbled faces and missing fingers. Therefore, human input is required to refine and create the final content. Although there are some limitations in Generative AI tools, the pace at which these AI systems are evolving is remarkable.

For instance, Coca-Cola has set a new benchmark for how AI can create engaging ads. Their latest advertisement uses AI (Stable Diffusion), Film, and 3D to create an immersive experience.

Coca-Cola® MasterpieceCoca-Cola® Masterpiece
Watch this video on YouTube

Should We Be Worried About Generative AI in Media? – The Risks

Despite Generative AI’s massive advantages like efficiency, personalization, and scalability, there are potential risks and downsides associated with it as well. These are:

1. Job Displacement

According to Goldman Sachs, 300 million jobs will be replaced by AI in 2025. AI has shown the capability for creativity and white-collar jobs previously considered exclusive to humans. If left unchecked, the AI systems can cause a severe economic downturn.

2. Fake Content

In 2018, Filmmaker Jordan Peele and BuzzFeed released a highly realistic deepfake of former U.S. President Barack Obama. The video aimed to raise awareness about the potential misuse of AI. It highlighted how easy it is to use someone’s face and voice to spread false narratives and disinformation.

Recently, in March 2023, AI-generated images of former President Trump restrained by police got viral. The creator Eliot Hughes confirmed that the images were generated using MidJourney V5.

Making pictures of Trump getting arrested while waiting for Trump's arrest. pic.twitter.com/4D2QQfUpLZ

— Eliot Higgins (@EliotHiggins) March 20, 2023

Hence, the use of AI in media can introduce a massive influx of fake content that will be hard to verify and regulate.

3. Intellectual Property & Copyrights

Determining the ownership and rights of AI-generated content is difficult without clear legislation. For instance, due to copyright issues, Getty Images banned AI-generated content from being uploaded on the platform. On top of that, Getty Images sued Stability AI Inc., alleging that they used 12 million images from its database without permission or compensation.

What Lies Ahead?

Generative AI is undoubtedly ground-breaking. It is helping creatives across many industries, especially media, to enhance their creativity and productivity. Though the future of Generative AI is exciting, it is also worrisome.

In March 2023, Elon Musk and 32,000 other technology leaders and AI developers signed a petition aiming to halt the production of AI models stronger than GPT-4 for six months. The idea is to give enough time to governments and policymakers to build robust AI governance frameworks that can mitigate any AI risks without stifling innovation.

Exciting times ahead! For more AI-related content, visit Unite.ai.

Quantum Brilliance Makes Quantum Computing Open for All

Quantum Brilliance Makes Quantum Computing Open for All

After successfully completing the beta testing phase, Quantum Brilliance, a developer of miniaturised, room-temperature quantum computing solutions, has officially launched the Qristal SDK, an open-source software development kit.

The SDK is now available to the public, enabling developers to create and evaluate innovative quantum algorithms tailored specifically for quantum accelerators rather than traditional quantum mainframes.

The Qristal SDK provides developers with fully integrated C++ and Python APIs, along with NVIDIA CUDA features and customizable noise models, to support the creation of quantum-enhanced designs.

The source code, which includes extensive application libraries for various fields such as VQE, QAOA, quantum machine learning, and natural language processing, can be downloaded from the company’s website.

This release aims to facilitate the research and development of applications that integrate Quantum Brilliance’s portable, diamond-based quantum accelerators.

Furthermore, the software incorporates MPI, the globally recognized standard for large-scale parallel computing, allowing users to optimise, simulate, and deploy hybrid applications that leverage parallelized, room-temperature quantum accelerators in high-performance computing (HPC) environments, ranging from supercomputers to edge devices.

The software opens up a wide range of potential use cases, including classical-quantum hybrid applications in data centres, massively parallelized clusters of accelerators for computational chemistry, and embedded accelerators for edge computing applications like robotics, autonomous vehicles, and satellites.

Mark Luo, the CEO and co-founder of Quantum Brilliance, expressed his confidence in the Qristal SDK, stating, “With improvements based on feedback from our beta users, this SDK empowers researchers to leverage quantum-based solutions across various real-world applications. We believe this powerful tool will enable organisations worldwide to understand how quantum accelerators can enhance and facilitate the productization and commercialisation of their offerings.”

Beginning of a new era for quantum computing

One of the distinguishing features of Quantum Brilliance’s quantum systems is their ability to operate at room temperature using synthetic diamonds. Unlike traditional mainframe quantum computers, these compact devices do not require cryogenic cooling, vacuum systems, or precision laser arrays. They consume significantly less power, making on-site or edge deployment feasible in any environment.

At present, Quantum Brilliance’s quantum computing technology is approximately the size of a desktop PC. However, the company is actively working on further miniaturisation, aiming to shrink their technology to the size of a semiconductor chip that can be integrated into any device, effectively making practical quantum computing universally accessible.

In 2019, Quantum Brilliance was established as a company that focuses on quantum products and solutions. Their primary objective is to develop diamond quantum computers accompanied by software and applications. Quantum Brilliance strives to facilitate the widespread adoption of their quantum technology, enabling various industries to leverage edge computing applications and advance into the realm of next-generation supercomputers.

Quantum Brilliance has forged partnerships worldwide, spanning across the Americas, EMEA, and Asia Pacific regions. These partnerships involve collaborations with governments, supercomputing centers, research organisations, and various industries, forming a global network of support for their endeavours.

Another company offering a similar SDK for quantum development is Qiskit. Started in 2017, the company has built its open source SDK in partnership with IBM Research.

The post Quantum Brilliance Makes Quantum Computing Open for All appeared first on Analytics India Magazine.

Hyperion Research Predicts HPC-enabled AI Growth

Hyperion Research Predicts HPC-enabled AI Growth June 8, 2023 by Jaime Hampton

Hyperion Research presented its annual ISC breakfast HPC market update in Hamburg where it shared a forecast that worldwide HPC-enabled AI server revenue will grow by 22.7% (CAGR) over the next 5 years.

Hyperion CEO Earl Joseph predicts that 2023 will be the year of AI regulation: “We expect to see serious AI regulations primarily in Europe and the U.S.," he said. Joseph also shared the prediction that AI will become more pervasive in production tier deployments due to users' higher confidence in its abilities and ease of use.

Slide courtesy of Hyperion Research.

HPC cloud revenue is expected to exceed $11 billion by 2026, Hyperion anticipates, with AI and other data-intensive applications driving this growth as the predominant workload in the cloud.

“ChatGPT put a spotlight on the world of what AI can do. We’re talking to a large number of users, buyers and providers with different ideas of how AI could see use,” said Joseph.

Read more about Hyperion's HPC market update in John Russell's coverage for HPCwire: HPC Market will Reach $33B in 2023 and Pass $50B by 2026 – Hyperion Research.

You can also find the full set of Hyperion's ISC23 breakfast presentation slides at this link.

Related

21 Must-Have Cheat Sheets for Data Science Interviews: Unlocking Your Path to Success

21 Cheat Sheets for Data Science Interviews
Image from Bing Image Creator

With data science being such a broad and constantly developing field, it’s really impossible to have all the knowledge in your head. Especially if some of this knowledge you use only occasionally. Also, if you’re a beginner in a certain field, you’ll have to refresh very often what you learned until it becomes actual knowledge at the crossroads of theory and practice.

Having something that you could look at and get the info you need at a glance would be pretty helpful, right? That ‘something’ is called a cheat sheet. And it has nothing to do with cheating. They are used for learning and revisioning what you already know.

Due to their intention of being (relatively) concise and high-level, having one cheat sheet for the whole data science would beat its (the cheat sheet’s, not data science’s) purpose. Even if creating such a cheat sheet would be possible. Because of that, you’ll have to use different cheat sheets for the various data science fields.

I tried to narrow this down to the cheat sheets covering the concepts a data scientist cannot do without. You can read it as a cheat sheet about cheat sheets talking about:

  • Coding Languages
    • SQL
    • Python
    • R
  • Algorithms and Models
  • Data Structures
  • Data Visualization
  • Probability and Statistics
  • Data Manipulation

Coding Languages

Knowing the coding languages is the basis upon which all other parts of data science are built. Especially popular in the data science community is the holy trinity of coding languages:

  • SQL
  • Python
  • R

SQL

The language specifically designed for querying databases, SQL is a champion when it comes to data extraction and manipulation.

Cheat sheet: SQL Basics Cheat Sheet

Link: https://learnsql.com/blog/sql-basics-cheat-sheet/

What you get: This cheat sheet focuses on enabling you to write functional SQL queries from the start. To do that, you’ll need to be familiar with certain concepts. These are querying a single table, filtering data, and querying multiple tables using the JOINs. There are also the aggregate functions, subqueries, and set operators (UNION, INTERSECT, EXCEPT) covered.

Apart from a short explanation of every concept, the cheat sheet also gives you a query based on the sample data to show you how everything works in practice.

The cheat sheet is also downloadable in PDF or PNG format, making it practical for printing out and having it in handy.

Cheat sheet: The Essential SQL Commands Cheat Sheet for Beginners

Link: https://itechbrand.com/the-essential-sql-commands-cheat-sheet-for-beginners/

What you get: There are no code and data examples like in the previous cheat sheet. This one simply lists the commands in SQL everybody needs. It is great when you want to remind yourself what a certain keyword does. It also covers additional topics, like creating and editing tables, constraints, data, triggers, views, and common table expressions (CTEs).

Cheat sheet: SQL Cheat Sheet – Technical Concepts for the Job Interview

Link: https://www.stratascratch.com/blog/sql-cheat-sheet-technical-concepts-for-the-job-interview/

What you get: Focused on the most critical SQL concepts to do good at a job interview, this cheat sheet covers JOINs, time and date functions, aggregate functions, window functions, and set operators.

Every technical topic and subtopic is explained shortly verbally and using an easy-to-understand graphical representation. Additionally, there’s an interview question and the solution code covering the subject concerned. The code is shown in the widget, so you can play around with it, making it an interactive cheat sheet.

Python

Python is, for a reason, one of the most commonly used programming languages in data science. It excels in all the areas required. It really does everything from data extraction and manipulation or statistical analysis and visualizing data to machine learning, model deployment, and automation.

Cheat sheet: Python Cheat Sheet

Link: https://websitesetup.org/python-cheat-sheet/

What you get: This very comprehensive yet very clear cheat sheet is perfect for anybody wanting to have a basis for starting working in Python. It explains the main data types in Python, including creating and storing strings and doing math operations on data. You’ll also learn about built-in functions, creating functions, lists, tuples, and dictionaries.

The cheat sheet goes on to give you an overview of the conditional statements, Python loops, classes, and even dealing with Python errors.

You can download the cheat sheet in PDF or infographic (PNG) format.

Cheat sheet: Python Cheat Sheet

Link: https://programmingwithmosh.com/wp-content/uploads/2019/02/Python-Cheat-Sheet.pdf

What you get: A rather similar cheat sheet to the one above. It mainly covers the same topics but in less detail. The explanations are excellent and perfect for beginners trying to grasp the basics of Python.

The cheat sheet is downloadable in PDF.

Cheat sheet: Comprehensive Python Cheatsheet

Link: https://github.com/gto76/python-cheatsheet

What you get: While the beginners can use this cheat sheet, too, it covers much more topics than needed on the basic level. There’s not much talking here. The author goes through the topics, lists the keywords, and explains them shortly. It also provides the example code and what it returns.

The topics covered are collections, types, syntax, system, data, advanced, and libraries. Every topic is then divided into subtopics that make this cheat sheet probably the only one needed for most Python users.

R

The R programming language is a little less flexible than Python, so it’s not suitable for model deployment. It is created for statistical analysis and data visualization. It’s not its only purpose because it is also heavily used for data extraction and manipulation, machine learning, and automation.

Cheat sheet: RStudio Cheatsheets

Link: https://www.rstudio.com/resources/cheatsheets/

What you get: These resources is probably the only one you’ll need when it comes to R cheat sheets. There is an extensive number of cheat sheets and topics covered. The users contributed with the cheat sheets covering basic and advanced R.

The Base R Cheat Sheet talks about vectors, programming, data types, mathematical functions, statistics, and other topics.

The Advanced R Cheat Sheet will be useful for those interested in environments, data structures, object-oriented systems, functions, subsetting, debugging, condition handling, and defensive programming.

You can find many more cheat sheets on the source website dedicated to the particular R topic. For example, handling date-times, strings, data transformation, tidying, visualization, deep learning, etc.

Data Structures

21 Cheat Sheets for Data Science Interviews

Data scientists have to be familiar with data structures as a way of organizing and storing data. The chance is you won’t be using all the possible data structures all the time. When the time comes to use a data structure you didn’t use (very often), the cheat sheets can provide you with a general idea about the data structure in question.

Cheat sheet: Data Structures Reference

Link: https://www.interviewcake.com/data-structures-reference

What you get: It lists all the data structures with short definitions and visual representation, which is excellent for a quick reference. If you want more details about each data structure, you can click on it and get detailed information, such as the strength and weaknesses of each, how inserting and deleting works, and an explanation of its specific characteristics.

Cheat sheet: An Executable Data Structures Cheat Sheet for Interviews

Link: https://algodaily.com/lessons/an-executable-data-structures-cheat-sheet

What you get: This one, too, gives you explanations of all the data structures, their pros and cons, and notable uses. The cheat sheet provides additional resources for learning more about each data structure.

On top of that, every data structure has a sample code in JavaScript, Python, and Java that you can run and see what it returns. There’s also a video that leads you through the complete datasheet and helps you understand it better.

Data Manipulation

Data manipulation, munging, or wrangling is when you transform the raw data into a format usable for further analysis and processing. In data science, this is usually done via Python and its library pandas.

Cheat sheet: Pandas Cheat Sheet for Data Science

Link: https://datascientyst.com/pandas-cheat-sheet-for-data-science/

What you get: Perfect for beginners, this cheat sheet shows you the codes for the main commands in pandas and explains what each code will return. The topics covered are pandas setup, data structures, importing and exporting data, inspecting it, and selecting. You’ll also learn how to add and drop rows/columns, sort, filter, group, convert, merge and concatenate data, and apply functions. An easy-to-understand graphical representation accompanies every topic.

Cheat sheet: Pandas Cheat Sheet

Link: https://geekyhumans.com/pandas-cheat-sheet/#Making-changes-to-the-data

What you get: It generally covers all the topics as the previous cheat sheet. The difference is that explaining is done mainly by showing you the code and its output instead of just explaining it.

Cheat sheet: Data Wrangling With pandas Cheat Sheet

Link: https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf

What you get: A detailed cheat sheet dedicated solely to data wrangling. It covers creating DataFrames, method chaining, reshaping data, dealing with rows and columns, using queries, summarizing and grouping data, handling missing data, making new columns, combining data sets, using windows, and plotting. Each topic is visually explained and shortly described, and every pandas keyword is showcased using the code and its output.

Data Visualization

Data Visualization

Visualizing data is an important part of a data scientist’s job. In a way, it’s the point when something understandable only to other data scientists you can also make understandable for the ‘ordinary’ folks. It can be a visualization of data analysis or model insights. Whichever it is, the cheat sheet could come in handy.

Cheat sheet: Data Visualization Cheat Sheet

Link: http://www.biosci.global/customer-stories-en/data-visualization-cheat-sheet/

What you get: It’s a good overview of the graphs used in data visualization. Besides every chart type, there’s a short explanation of what it represents and the image showing it, so you can easily visualize what each graph would look like.

There’s also a visual overview of the criteria for choosing the right graph for your visualization.

Cheat sheet: Data Visualization Cheat Sheet

Link: https://www.kaggle.com/getting-started/160583

What you get: There are no explanations of the charts. But all charts are visually represented here and divided into sections based on their purpose in data visualization. Perfect for beginners and anyone wanting to check quickly if they chose the correct chart and if there are some better options.

Cheat sheet: Data Visualization Cheat Sheets

Link: https://medium.com/responsibleml/data-visualization-cheat-sheets-1c12ba8a7671

What you get: Here are several cheat sheets revolving around the topic of making a good graph. It doesn’t only talk about choosing the right graph. The cheat sheets go more into detail, giving advice, dos and don’ts on presenting data on maps, choosing the right colors (including those for visually impaired people), making the chart more readable, choosing the chart’s axes, and representing the timeline. All cheat sheets are downloadable in PDF.

Statistics & Probability

Having extensive knowledge of statistics and, more specifically, the probability is a must for any data scientist. They use it almost in every part of their job: from data analysis to model building, testing, and evaluation. With statistics being an extensive discipline, the chance is you’ll be using only some of it in your job. For those statistics topics that are new to you or don’t use often, you’ll need a good cheat sheet to help yourself.

Cheat sheet: A Comprehensive Statistics Cheat Sheet for Data Science Interviews

Link: https://www.stratascratch.com/blog/a-comprehensive-statistics-cheat-sheet-for-data-science-interviews/

What you get: This cheat sheet covers all the statistics topics most data scientists will ever need. Those are confidence intervals, hypothesis testing, Z statistics and T statistics, A/B testing, linear regression, probability rules, Bayes theorem, and combinations and permutations. There are detailed explanations of all these concepts, with formulas, graphical representations, and examples.

Cheat sheet: The Most Comprehensive Stats Cheat Sheet

Link: https://terenceshin.medium.com/week-2-52-stats-cheat-sheet-ae38a2e5cdc6

What you get: Generally covers a topic or two as the previous one. However, most of the statistics concepts here are different. They are data types, measures of central tendency (mean, median, mode), measures of variability (range, variance, standard deviation…), measurements of the relationship between variables (covariance and correlation), probability distribution functions, continuous and discrete data distributions, moments, and accuracy.

Cheat sheet: Statistics Cheat Sheet

Link: https://web.mit.edu/~csvoss/Public/usabo/stats_handout.pdf

What you get: This cheat sheet generally doesn’t cover anything that isn’t covered by the previous two cheat sheets. However, apart from theoretical explanations, this one offers very elaborated examples that will for sure make you understand the concept in question.

Algorithms & Models

All the previously mentioned topics usually serve as a basis for the ultimate data scientist’s task: writing algorithms and creating models. This is where the statistics and coding knowledge meet the knowledge of finding a helpful cheat sheet covering algorithms and models.

Cheat sheet: Top Prediction Algorithms

Link: https://blog.dataiku.com/machine-learning-explained-algorithms-are-your-friend

What you get: This cheat sheet explains machine learning in general terms, as well as the most popular algorithms. These are linear and logistic regression, decision tree, random forest, gradient boosting, and neural networks. A very nice feature is an infographic describing each algorithm, its advantages, and disadvantages.

Cheat sheet: Your Ultimate Data Science Statistics & Mathematics Cheat Sheet

Link: https://towardsdatascience.com/your-ultimate-data-science-statistics-mathematics-cheat-sheet-d688a48ad3db

What you get: A detailed explanation of the machine learning metrics. It covers the topics of classifier metrics, regressor metrics, statistical indicators, and types of distribution. The explanations are thorough, with clear graphical representations, formulas, and examples.

Cheat sheet: Cheat Sheet for Machine Learning Models

Link: https://medium.com/analytics-vidhya/machine-learning-models-cheatsheet-7885b33ca44f

What you get: Again, a very thorough cheat sheet focusing on algorithms for machine learning. The explanations are detailed; they contain examples and, most importantly, steps of building each algorithm. The author covers the following topics: multiple linear regression, decision tree regression, logistic regression, naive Bayes classifier, assessing the performances of binary classifiers, ROC curve, support vector machine (SVM), random forest, k-means clustering, k-nearest neighbors, hierarchical clustering, principal component analysis (PCA), linear discriminant analysis (LDA), processing text data, ranking algorithms.

Conclusion

In this article, I covered coding, data structures, data manipulation, data visualization, statistics & probability, and models & algorithms. They are not, of course, the only topics you should cover as a data scientist. But they are the topics most data scientists will need in their careers.

The cheat sheets that I recommended are a narrowed-down list of good cheat sheets that I think best cover the topic in question. They will keep you covered in most cases, and I think they are at least a good starting point.

Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn.

More On This Topic

  • KDnuggets News, June 8: 21 Cheat Sheets for Data Science Interviews; Top 18…
  • 10 Cheat Sheets You Need To Ace Data Science Interview
  • The Complete Collection of Data Science Cheat Sheets — Part 1
  • The Complete Collection of Data Science Cheat Sheets — Part 2
  • KDnuggets™ News 22:n06, Feb 9: Data Science Programming Languages and…
  • 7 Essential Cheat Sheets for Data Engineering

ONDC Has a Data Problem 

The Indian government has projected Open Network for Digital Commerce (ONDC) as a platform to democratise e-commerce in India and break the hold of private players such as Amazon and Flipkart on the Indian market. While addressing the ‘ONDC Elevate’ programme last month, Union Minister Piyush Goyal said that ONDC is an engine of growth and could redefine the e-commerce industry completely.

ONDC has been widely discussed due to its highly competitive price on food delivery, challenging the dominance of established players like Swiggy and Zomato. Many users flocked to the platform to avail the products at such cheap prices. It’s important to note that sellers on ONDC platform were able to sell products at such low prices because there’s no intermediary in between to take a percentage cut on every sale.

According to reports, ONDC will soon start offering mutual funds, insurance, personal loan, and insurance among other things. Users on the platform will only increase, even if not at par with other e-commerce platforms. However, amid this, the concerns regarding the approach of ONDC towards data privacy, particularly in the absence of a comprehensive data protection bill in the country remain.

Data Privacy concerns

“E-commerce platforms have a multitude of data points and they’re used for competitive pricing, recommendations, etc. Our concerns essentially stem from the fact that there is an absence of the data protection bill in the country,” Tejasi Panjiar, associate policy counsel, Internet Freedom Foundation told AIM.

The ONDC Strategy Paper does suggest that data points such as Personal Identifiable Information (PII) as well as seller data critical to trade will remain siloed within the buyer app and seller app respectively and will be protected from third-party access. It also mentions that ONDC may collect personal data about a natural person, living or deceased. However, in the absence of a data protection law, storing consumer data is a privacy concern and further, how ONDC is handling these data points remains unknown.

Moreover, the digital rights advocacy group has also raised concerns that arise regarding the network-wide sharing of ‘anonymised performance metrics.’ These concerns persist as the re-identification of anonymised data remains feasible with just a few data points, such as pin code, date of birth, and age, which can accurately identify users from anonymised datasets.

While once the Digital Personal Data Protection Bill (DPDPB) comes into effect, ONDC has to adhere to the laws of the land, however, Panjiar said that the bill does not do much to address the rights of the citizens either.

Does not come under RTI

ONDC was incorporated on 31 December 2022 with an initial investment from the Quality Council of India (QCI) and Protean eGov Technologies Limited. ONDC is not a government agency or a Public Sector Undertaking (PSU). It is a private non-profit company established by the Department for Promotion of Industry and Internal Trade (DPIIT). This has been done to diminish the motivation for profit maximisation and prioritise ethical and responsible conduct.

However, by doing so, according to the IFF, ONDC does not come under the ambit of Right to Information (RTI). “Given that ONDC is a Section 8 company and not a public authority, it falls outside the ambit of RTI,” IFF said in a blog post.

The exclusion of the ONDC from the purview of the RTI raises a significant issue. By not being subject to RTI, there is a lack of mechanisms for common citizens to obtain crucial information regarding ONDC’s operations, decision-making processes, and adherence to accountability and transparency standards.

“Since our fundamental rights are enforceable against the State, aggrieved citizens won’t be able to avail their right to file a writ petition under Article 32 and 226 in case of violation of their rights,” IFF said.

Additionally, this absence of transparency creates an environment where the public is unable to effectively monitor whether ONDC is upholding best practices and acting in the interest of the citizens it serves.

Grievance redressal

Piyush Goyal once said that it is essential for the ONDC to establish consumer trust by implementing strong mechanisms to address consumer grievances and enforce transparent policies regarding returns, refunds, and cancellations. Other e-commerce has built consumer trust upon a strong consumer grievances redressal channel and it’s a welcoming move for ONDC to have similar channels. However, as things stand, ONDC does n0t have such a mechanism in place. All users can do is write to an email id to share their grievances.

Additionally, the multiplicity of stakeholders within ONDC, encompassing buyers, sellers, and logistics, pose potential obstacles due to the complexity and coordination required among these various parties, potentially impeding the seamless functioning of the network. So far, many users have taken to Twitter to vent their frustrations about the unavailability of a proper grievances redressal mechanism.

Further, there is also a lack of clarity regarding the resolution of privacy-related grievances and breaches of data within the existing framework. Given there is no robust grievances mechanism in place currently, the potential for redressal of such concerns remains uncertain, raising questions about the overall accountability and trustworthiness of the system.

The post ONDC Has a Data Problem appeared first on Analytics India Magazine.

Mastering the Art of Data Storytelling: A Guide for Data Scientists

Mastering the Art of Data Storytelling: A Guide for Data Scientists
Photo by Isaac Smith on Unsplash

If you’re looking into becoming a data scientist, or are already a data scientist — you will have had a read or know the skills required. You will need a programming language, an understanding of mathematical statistics, the ability to create data visualizations, and more.

If you’re looking into becoming a data science professional and need some guidance, have a look at this article: Become a Data Science Professional in Five Steps.

Although the majority of your time will be spent during the data preparation stage trying to find and clean data — there are other important elements to data science.

Once you have found your valuable insights, if it’s trends, patterns, or put into visualizations — you will need to be able to explain these. As a data professional, it can be difficult for non-technical people to understand technical language.

If you are a technical person, it can be challenging to convey your message to non-technical people. Not only will you come across non-technical people, but you may be dealing with someone who prefers explanations through visualizations, or project run-throughs.

Therefore, once you have your findings, you will need to cater to a variety of people — and mastering how to do that can be difficult, but it can be achieved.

Let’s get started…

Non-technical Language

As a data scientist myself, I understand that a lot of stakeholders or managers will not come from a technical background. Therefore, some of the terminology used in your everyday team will be foreign to them. For example, F1 score or cross-validation.

Think about how a teacher explains a topic to a student, and keep that at the front of your mind when you’re explaining to your audience. Translate your data science terminology in a language that everybody can understand. If there is no way you can replace a specific data science term, there is no harm in explaining what it means. You will do more harm by losing your audience's attention to technical words.

Data Visualizations

Different people learn in different ways. Some can read a textbook once and get it. Some need it to be color coded. Some need visualizations. When presenting your findings, don’t limit yourself and put yourself in a rut where you have to answer 1000 questions. Visualizations can answer questions for you.

Data visualizations will allow your audience to have a visual understanding of the steps you took and your findings. Whilst you are talking in the background about the visualizations, their eyes are learning and making sense of what you are saying.

Summary

At the end fo your presentation, ensure to have a summary page of all your important points and data visualizations for your audience to see. During this time, you should be open to questions in which your audience can continuously look at the summary board to sprak new questions.

Your audience asking questions is not a bad thing, it shows that they have been listening, they are interested, and they want to learn and understand more.

Three-act Storytelling

The above points are elements of your storytelling which will make it effective. However, a structure is what will make your data storytelling a success.

The three-act storytelling is a popular model used in narrative fiction that divides a story into three parts:

Setup

Aim: state in the clearest terms the problem you are trying to solve.

This includes an introduction to your project, stating the purpose of the project, what you're trying to solve, etc. During the setup, from a data science perspective you will go further into the problem or issue in more depth to give context to the aim of the project. The aim of your project will equate to your Point 1.

Confrontation

Aim: explain to your audience why it’s important to solve this problem and the different paths you went down to solve the problem.

During the confrontation part, you can continue speaking about the task at hand, and why the company was facing this issue in the first place. You want to keep your audience interested and intrigued, therefore speaking on the problems that the company is facing will always get stakeholders hooked.

Explain to your reader step-by-step the different paths you went through and your outcome for each, in order to complete the task at hand. The different steps you took during the data science project will reflect different points, e.g. Point 2, Point 3,…

Giving your audience context to the failures and obstacles you encountered and why, will help build trust and understanding between you and the audience once you come to a resolution.

Resolution

Aim: Explain the solution you can offer to solve the problem and ensure the audience are satisfied.

This is where the audience goes from being concerned to relieved. Your resolution should state how it overcomes your previous failures and obstacles. Open this section up for questions, as your audience will want to have full trust in your data insights and believe this is the right way to go.

Once the audience is at ease, you can start wrapping up and speaking on the actions that need to be taken, in order for the task to be a success.

The Pyramid Principle

Another structure that is very effective is the pyramid principle. This is an effective communication tool used to clearly communicate complex issues to busy executives. The aim is that ideas in writing should always form a pyramid under a single thought.

So let me explain this a bit more. When dealing with busy executives that want to learn about your data insights, but are short on time or are eager to know the solutions — the pyramid principle is the way to go.

It is broken up into 3 parts:

Your Answer

In this case, your answer will be the solution to the task at hand. This is the main point you want your audience to take away. This is the key message and you want the focus to be surrounded around this main point — the solution.

Supporting Arguments

Once you have stated the solution, your next step is to convince your audience that this is the way to go. In order to do this, you will have to take them through a journey of supporting arguments, with high level insight. During this part, your audience may have a few questions lingering in their mind.

Supporting Facts/Data

During this part, all of the possible questions that your audience may have will be answered here. Each of your supporting arguments needs to be backed up by data and facts to ensure your audience that you have done your homework and that your initial answer/solution did not come out of thin air.

Wrapping it up

Using the skills of using non-technical language and visualizations in either structure: three-act storytelling or the pyramid principle will allow you to master the art of data storytelling.

Your choice on which structure you choose is dependent on how well you know your audience. You can always trial and error both structures to see which one is most effective. A good way to measure how effective the structure is for your audience is by noting which structure had less questions. The less questions your audience has, the more successful your storytelling was.
Nisha Arya is a Data Scientist, Freelance Technical Writer and Community Manager at KDnuggets. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.

More On This Topic

  • The Most Complete Guide to PyTorch for Data Scientists
  • The Ultimate Guide to Acing Coding Interviews for Data Scientists
  • Data storytelling: brains are built for visuals, but hearts turn on stories
  • What will the demand for Data Scientists be in 10 years? Will Data…
  • KDnuggets News, November 9: 7 Tips To Produce Readable Data Science Code •…
  • KDnuggets™ News 20:n42, Nov 4: Top Python Libraries for Data Science,…

Singapore identifies six generative AI risks, sets up foundation to guide adoption

generative AI apps

Singapore has identified six top risks associated with generative artificial intelligence (AI) and proposed a framework on how these issues can be addressed. It also has established a foundation that looks to tap the open-source community to develop test toolkits that mitigate the risks of adopting AI.

Hallucinations, accelerated disinformation, copyright challenges, and embedded biases are among the key risks of generative AI outlined in a report released by Singapore's Infocomm Media Development Authority (IMDA). The discussion paper details the country's framework for "trusted and responsible" adoption of the emerging technology, including disclosure standards and global interoperability. The report was jointly developed with Aicadium, an AI tech company founded by state-owned investment firm Temasek Holdings.

Also: Today's AI boom will amplify social problems if we don't act now, says AI ethicist

The framework offers a look at how policy makers can boost existing AI governance to address "unique characteristic" and immediate concerns of generative AI. It also discusses investment needed to ensure governance outcomes in the longer term, IMDA said.

In identifying hallucinations as a key risk, the report noted that — similar to all AI models — generative AI models make mistakes and these are often vivid and take on anthropomorphization.

"Current and past versions of ChatGPT are known to make factual errors. Such models also have a more challenging time doing tasks like logic, mathematics, and common sense," the discussion paper noted.

"This is because ChatGPT is a model of how people use language. While language often mirrors the world, these systems however do not yet have a deep understanding about how the world works."

These false responses can also be deceptively convincing or authentic, the report added, pointing to how language models have created seemingly legitimate but erroneous responses to medical questions, as well as generating software codes that are susceptible to vulnerabilities.

In addition, dissemination of false content is increasingly difficult to identify due to convincing but misleading text, images, and videos, which can potentially be generated at scale using generative AI.

Also: How to use ChatGPT to write code

Impersonation and reputation attacks have become easier, including social-engineering attacks that use deepfakes to gain access to privileged individuals.

Generative AI also makes it possible to cause other types of harm, where threat actors with little to no technical skills can potentially generate malicious code.

Also: Don't get scammed by fake ChatGPT apps: Here's what to look out for

These emerging risks might require new approaches to the governance of generative AI, according to the discussion paper.

Singapore's Minister for Communications and Information Josephine Teo noted that global leaders are still exploring alternative AI architectures and approaches, with many people issuing caution about the dangers of AI.

AI delivers "human-like intelligence" at a potentially high level and at significantly reduced cost, which is especially valuable for countries such as Singapore where human capital is a key differentiator, said Teo, who was speaking at this week's Asia Tech x Singapore summit.

The improper use of AI, though, can do great harm, she noted. "Guardrails are, therefore, necessary to guide people to use it responsibly and for AI products to be 'safe for all of us' by design," she said.

"We hope [the discussion paper] will spark many conversations and build awareness on the guardrails needed," she added.

Also: 6 harmful ways ChatGPT can be used

During a closed-door dialog at the summit, she revealed that senior government officials also debated the recent advancements in AI, including generative AI models, and considered how these could fuel economic growth and impact societies.

Officials reached a consensus that AI had to be "appropriately" governed and used for the good of humanity, said Teo, who provided a summary as chair of the discussion. Participants at the meeting, which included ministers, represented nations that included Germany, Japan, Thailand, and the Netherlands.

The delegates also concurred that increased collaboration and information exchange on AI governance policies would help identify common grounds and lead to better alignment between approaches. This unity would lead to sustainable and fit-for-purpose AI governance frameworks and technical standards, Teo said.

The officials urged greater interoperability between governance frameworks, which they believe is necessary to facilitate responsible development and adoption of AI technologies globally.

There was also recognition that AI ethics should be infused at early stages of education, while investments in reskilling should be prioritized.

Galvanizing the community

Singapore has launched a not-for-profit foundation to "harness the collective power" and contributions of the global open-source community to develop AI-testing tools. The goal here is to facilitate the adoption of responsible AI, and promote best practices and standards for AI.

Called AI Verify Foundation, it will set the strategic direction and development roadmap of AI Verify, which was introduced last year as a governance-testing framework and toolkit. The test toolkit has been made open source.

Also: This new AI system can read minds accurately about half the time

AI Verify Foundation's current crop of 60 members includes IBM, Salesforce, DBS, Singapore Airlines, Zoom, Hitachi, and Standard Chartered.

The foundation operates as a wholly owned subsidiary under IMDA. With AI-testing technologies still nascent, the Singapore government agency said tapping the open-source and research communities would help further develop the market segment.

Teo said: "We believe AI is the next big shift since the internet and mobile. Amid very real fears and concerns about its development, we will need to actively steer AI toward beneficial uses and away from bad ones. This is core to how Singapore thinks about AI."

In his speech at the summit, Singapore's Deputy Prime Minister and Minister for Finance Lawrence Wong further reiterated the importance of establishing trust in AI, so the technology can have widespread acceptance.

"We are already using machine learning to optimize decision making and generative AI will go beyond that to create potentially new content and generate new ideas," Wong said. "Yet, there remain serious concerns. Used improperly, [AI] can perpetuate dangerous biases in decision making. And with the latest wave of AI, the risks are even higher as AI becomes more intimate and human-like."

Also: AI is more likely to cause world doom than climate change, according to an AI expert

These challenges pose difficult questions for regulators, businesses, and society at large, he said. "What kind of work should AI be allowed to assist with? How much control over decision making should an AI have and what ethical safeguards should we put in place to help guide its development?"

Wong added: "No single person, organization, or even country, will have all the answers. We will all have to come together to engage in critical discussions to determine the appropriate guardrails that will be necessary to build more trustworthy AI systems."

At a panel discussion, Alexandra van Huffelen from the Netherlands' Ministry of Interior and Kingdom Relations, recognized the potential benefits of AI, but expressed worries about its potential impact, especially amid mixed signals from the industry.

The Minister for Digitalisation noted that market players, such as OpenAI, tout the benefits of their products, but at the same time issue caution that AI has the potential to destroy humanity.

"That's a crazy story to tell," van Huffelen quipped, before asking a fellow panelist from Microsoft how he felt as his company is an investor in OpenAI, the brainchild behind ChatGPT.

Also: I used ChatGPT to write the same routine in these 10 languages

OpenAI's co-founders and CEO last month jointly published a note on the company's website, urging the regulation of "superintelligence" AI systems. They talked about the need for a global authority, such as the International Atomic Energy Agency, to oversee the development of AI. This agency should "inspect systems, require audits, test for compliance with safety standards, place restrictions on degrees of deployment and levels of security," among other responsibilities, they proposed.

"It would be important that such an agency focus on reducing existential risk and not issues that should be left to individual countries, such as defining what an AI should be allowed to say," they noted. "In terms of both potential upsides and downsides, superintelligence will be more powerful than other technologies humanity has had to contend with in the past…Given the possibility of existential risk, we can't just be reactive. Nuclear energy is a commonly used historical example of a technology with this property."

In response, Microsoft's Asia President Ahmed Mazhari acknowledged that van Huffelen's pushback was not unwarranted, noting that the same proponents who signed a petition in March to pause AI developments had proceeded the following month to invest in their own AI chatbot.

Also: The best AI chatbots: ChatGPT and alternatives to try

Pointing to the social harm that resulted from a lack of oversight of social media platforms, Mazhari said the tech industry has the responsibility to prevent a repeat of that failure with AI.

He noted that the ongoing discussion and heightened awareness about the need for AI regulations, in particular in the areas of generative AI, was a positive sign for a technology that hit the market just six months ago.

In addition, van Huffelen underscored the need for tech companies to act responsibly, alongside the need for rules to be established, and enforcement to ensure organizations adhered to these regulations. She said it remained "untested" whether this dual approach could be achieved in tandem.

She also stressed the importance of establishing trust, so people want to use the technology, and ensuring users have control over what they do online, as they would in the physical world.

Also: How does ChatGPT work?

Fellow panelist Keith Strier, Nvidia's vice president of worldwide AI initiatives, noted the complexity of governance due to the wide accessibility of AI tools. This general availability means there are more opportunities to build unsafe products.

Strier suggested that regulations should be part of the solution, but not the only answer, as industry standards, social norms, and education are just as crucial in ensuring the safe adoption of AI.

Artificial Intelligence