Synthetic Data for Machine Learning

It’s no secret that supervised machine learning models need to be trained on high-quality labeled datasets. However, collecting enough high-quality labeled data can be a significant challenge, especially in situations where privacy and data availability are major concerns. Fortunately, this problem can be mitigated with synthetic data. Synthetic data is data that is artificially generated rather than collected from real-world events. This data can either augment real data or can be used in place of real data. It can be created in several ways including through the use of statistics, data augmentation/computer-generated imagery (CGI), or generative AI depending on the use case. In this post, we will go over:

  • The Value of Synthetic Data
  • Synthetic Data for Edge Cases
  • How to Generate Synthetic Data

The Value of Synthetic Data

Problems with real data have led to many use cases for synthetic data, which you can check out below.

Privacy issues

Synthetic Data for Machine Learning
Image by Google Research

Healthcare data is widely known to have privacy restrictions. For example, while incorporating electronic health records (EHR) into machine learning applications could enhance patient outcomes, doing so while adhering to patient privacy regulations like HIPAA is difficult. Even techniques to anonymize data aren’t perfect. In response, researchers at Google came up with EHR-Safe which is a framework for generating realistic and privacy-preserving synthetic EHR.

Safety Issues

Collecting real data can be dangerous. One of the core problems with robotic applications like self-driving cars is that they are physical applications of machine learning. An unsafe model can’t be deployed in the real world and causes a crash due to a lack of relevant data. Augmenting a dataset with synthetic data can help models avoid these problems.

Real data collection and labeling are often not scalable

Annotating medical images is critical for training machine learning models. However, each image should be labeled by expert clinicians, which is a time-consuming and expensive process that is often subject to strict privacy regulations. Synthetic data can address this by generating large volumes of labeled images without requiring extensive human annotation or compromising patient privacy.

Manual labeling of real data can sometimes be very hard if not impossible

Synthetic Data for Machine Learning
Optical flow labels of the sparse real-world data KITTI (left) and the synthetic data from Parallel Domain (right). The color indicates the direction and magnitude of flow. Image by author.

In self-driving, estimating per-pixel motion between video frames, also known as optical flow, is challenging with real-world data. Real data labeling can only be done using LiDAR information to estimate object motion, whether dynamic or static, from the autonomous vehicle’s trajectory. Because LiDAR scans are sparse, the very few public optical flow datasets are also sparse. This is one reason why some optical flow synthetic data has been shown to greatly improve performance on optical flow tasks.

Synthetic Data for Edge Cases

A common use case of synthetic data is to deal with a lack of rare classes and edge cases in real datasets. Before generating synthetic data for this use case, please check out the tips below to consider what needs to be generated and how much of it is needed.

Identify your edge cases and rare classes

It is important to understand what edge cases are contained in a dataset. This could be rare diseases in medical images or usual animals and jaywalkers in self-driving. It is also important to consider what edge cases are NOT in a dataset. If a model needs to identify an edge case not present in the dataset, additional data collection or synthetic data generation might be necessary.

Verify the synthetic data is representative of the real-world

Synthetic data should represent real-world scenarios with minimal domain gaps which are differences between two distinct datasets (e.g., real and synthetic data). This can be done by manual inspection or by using a separate model trained on real data.

Make potential synthetic performance improvements quantifiable

A goal of supervised learning is to build a model that performs well on new data. This is why there are model validation procedures like train test split. When augmenting a real dataset with synthetic data, data might need to be balanced based on rare classes. For example, in self-driving applications, a machine learning practitioner might be interested in using synthetic data to focus on specific edge cases like jaywalkers. The original train test split may not have been split by the number of jaywalkers. In this case, it might make sense to move a lot of the existing jaywalker samples over to the test set to ensure that improvement by synthetic data is measurable.

Ensure all of your synthetic data is not just rare classes

A machine learning model should not learn that synthetic data is mostly rare classes and edge cases. Also, when more rare classes and edge cases are discovered, more synthetic data might need to be generated to account for this scenario.

How to Generate Synthetic Data

A major strength of synthetic data is that more can always be generated. It also comes with the benefit of already being labeled. There are many ways to generate synthetic data and which one you choose depends on your use case.

Statistical methods

A common statistical method is to generate new data based on the distribution and variability of the original data set. Statistical methods work best when the dataset is relatively simple and the relationships between variables are well understood and can be defined mathematically. For example, if real data has a normal distribution like human heights, synthetic data can be created using the same mean and standard deviation of the original dataset.

Data augmentation/CGI

A common strategy to increase the diversity and volume of training data is by modifying existing data to create synthetic data. Data augmentation is widely used in image processing. This might mean flipping images, cropping them, or adjusting brightness. Just make sure that the data augmentation strategy makes sense for the project of interest. For example, for self-driving applications, rotating an image by 180 degrees so that the road is at the top of the image and the sky at the bottom doesn’t make sense.

Caption: Multiformer inference on an urban scene from the synthetic SHIFT dataset.

Rather than modifying existing data for self-driving applications, CGI can be used to precisely generate a wide variety of images or videos that might not be easily obtainable in the real-world. This can include rare or dangerous scenarios, specific lighting conditions, or types of vehicles. A couple of the drawbacks of this approach are that creating high-quality CGI requires significant computational resources. specialized software, and a skilled team.

Generative AI

A commonly used generative model to create synthetic data is Generative Adversarial Networks or GANs for short. GANs consist of two networks, a generator, and a discriminator, that are trained simultaneously. The generator creates new examples, and the discriminator attempts to differentiate between real and generated examples. The models learn together, with the generator improving its ability to create realistic data, and the discriminator becoming more skilled at detecting synthetic data. If you would like to try implementing a GAN with PyTorch, check out this TDS blog post.

These methods work well for complex datasets and can generate very realistic, high-quality data, However, as the image above shows, it is not always easy to control specific attributes like the color, text, or size of generated objects.

Conclusion

If a project doesn’t have enough high-quality and diverse real data, synthetic data might be an option. After all, more synthetic data can always be generated. This is a major difference between real and synthetic data as synthetic data is far easier to improve! If you have any questions or thoughts on this blog post, feel free to reach out in the comments below or through Twitter.

Michael Galarnyk is a Data Science Professional, and works in Product Marketing Content Lead at Parallel Domain.

More On This Topic

  • How To Use Synthetic Data To Overcome Data Shortages For Machine…
  • High-Fidelity Synthetic Data for Data Engineers and Data Scientists Alike
  • How to Democratize AI/ML and Data Science with AI-generated Synthetic Data
  • Synthetic Data Platforms: Unlocking the Power of Generative AI for…
  • Data access is severely lacking in most companies, and 71% believe…
  • Build a synthetic data pipeline using Gretel and Apache Airflow

Why Do Big Tech LLM Chatbots Have the Worst Possible Names?

Looks like big tech majors are betting on fresh monikers for LLM chatbots, moving away from giving their chatbots human-like names such as Alexa, Siri, and Cortana, to more non-human (read: boring) names.

“ChatGPT is a lot worse,” quipped OpenAI chief Sam Altman in response to a post on X that labeled Apple’s Vision Pro as the worst name given to tech.

Altman, previously mentioned that they intentionally gave a bland name to ChatGPT to clearly indicate its AI nature, thus preserving the uniqueness of human-to-human interactions. This approach aims to avoid creating an overly humanized AI, like in the movie ‘Her’. In contrast, OpenAI’s GPT Store is filled with AI girlfriends.

“We named it ChatGPT and not a person’s name intentionally. We do a bunch of subtle things in the way you use it to make it clear that you’re not talking to a person,” he revealed in a recent interview.

“I personally have deep misgivings about this vision of the future where everyone is super close to AI friends, not more so with human friends or whatever. I personally don’t want that,” he added.

Why did Google rename Bard to Gemini?

Search giant Google recently made the bold move of rebranding its generative AI chatbot Bard to Gemini. The decision received mixed reactions from the tech community.

“Bard was the most direct way people could interact with our models. It made sense to just evolve it to be Gemini, because you’re actually talking directly to the underlying Gemini model when you use it,” said Google chief Sundar Pichai in a recent interview.

Pichai further added, “I think it will also be the way through which we will keep advancing our models and users can experience it directly. So, we thought the name change made sense,” he added.

Google would have certainly mulled hard over renaming Bard, considering that people recognise it much more than Gemini. However unlike Bard, Gemini is a more gender-neutral name and does not stereotype AI assistants as females, as is the case with Siri, Alexa, and Cortana.

Interestingly, Google chose the name Bard for its AI chatbot to highlight its creative and storytelling capabilities. The term “Bard” traditionally denotes a poet or storyteller, and a notable historical figure associated with this title is William Shakespeare, often referred to as the Bard of Avon.

Some see Bard as adding a more personal touch, while others anticipate that this rebranding will enable Google to shed the negative publicity associated with the previous version, paving the way for a ‘new era of Gemini’.

Why Gemini?

“Gemini sounds like a space programme – futuristic, a leap for mankind. It’s got all the right emotional associations. It’s a constellation, it’s out in space, and it’s made of stars. Plus, the name has a ‘gem’ in it, which feels fancy, valuable, and refined,” wrote a user on Hacker News.

“I’m not saying Gemini is the best name I’ve heard, not even close to it, but it feels 100% appropriate, in a way that Bard does not,” the user added.

“Gemini is such a boring name for an AI (better than GPT, but not much). Bard was a much more interesting jumping off point. They could have continued with character classes (Paladin, Sorcerer), or gone with realistic bardic inspiration (Skald, Poet). Why can’t anyone name AI [well]?” wrote Ethan Mollick, a professor at Wharton, expressing his dissatisfaction with the rebranding of Bard.

Last month, Jeff Dean, chief scientist at Google DeepMind, shed light on the nomenclature. The name ‘Gemini’, derived from Latin for ‘twins’, symbolically represents both Google Brain and DeepMind. In April 2023, Alphabet announced the merger of these two AI units, creating Google DeepMind.

“The Gemini effort came about because we had different teams working on language modeling, and we knew we wanted to work together. The twins are the folks in the legacy Brain team (many from the PaLM/PaLM-2 effort) and the legacy DeepMind team (many from the Chinchilla effort) that started to work together on the ambitious multimodal model project we called Gemini, eventually joined by many people from all across Google,” wrote Dean.

What about Microsoft’s Bing Chat Rebranding?

Similarly, a few months ago, Microsoft, much like Google, renamed Bing Chat to Copilot. Microsoft said it did so to make AI-powered web chat more accessible with a simplified user experience at copilot.microsoft.com.

Microsoft has gone all-in on Copilot. Every AI service that the company is providing is named Copilot. The company even introduced a new Copilot key on the keyboard, a first change in nearly three decades.

“This is clearly the age of the Copilot. The renaming of Bing Chat to Copilot is not merely a cosmetic change, it represents a commitment to creating a more dynamic and responsive search platform, catering to the evolving expectations of users worldwide,” said Microsoft chief Satya Nadella during the 2023 Microsoft Ignite.

The post Why Do Big Tech LLM Chatbots Have the Worst Possible Names? appeared first on Analytics India Magazine.

Google Cloud’s Nick Godfrey Talks Security, Budget and AI for CISOs

Close up of Google Cloud sign displayed in front of their headquarters in Silicon Valley, South San Francisco bay area.
Image: Adobe/Sundry Photography

As senior director and global head of the office of the chief information security officer (CISO) at Google Cloud, Nick Godfrey oversees educating employees on cybersecurity as well as handling threat detection and mitigation. We conducted an interview with Godfrey via video call about how CISOs and other tech-focused business leaders can allocate their finite resources, getting buy-in on security from other stakeholders, and the new challenges and opportunities introduced by generative AI. Since Godfrey is based in the United Kingdom, we asked his perspective on UK-specific considerations as well.

How CISOs can allocate resources according to the most likely cybersecurity threats

Megan Crouse: How can CISOs assess the most likely cybersecurity threats their organization may face, as well as considering budget and resourcing?

Nick Godfrey: One of the most important things to think about when determining how to best allocate the finite resources that any CISO has or any organization has is the balance of buying pure-play security products and security services versus thinking about the kind of underlying technology risks that the organization has. In particular, in the case of the organization having legacy technology, the ability to make legacy technology defendable even with security products on top is becoming increasingly hard.

And so the challenge and the trade off are to think about: Do we buy more security products? Do we invest in more security people? Do we buy more security services? Versus: Do we invest in modern infrastructure, which is inherently more defendable?

Response and recovery are key to responding to cyberthreats

Megan Crouse: In terms of prioritizing spending with an IT budget, ransomware and data theft are often discussed. Would you say that those are good to focus on, or should CISOs focus elsewhere, or is it very much dependent on what you have seen in your own organization?

Nick Godfrey: Data theft and ransomware attacks are very common; therefore, you have to, as a CISO, a security team and a CPO, focus on those sorts of things. Ransomware in particular is an interesting risk to try and manage and actually can be quite helpful in terms of framing the way to think about the end-to-end of the security program. It requires you to think through a comprehensive approach to the response and recovery aspects of the security program, and, in particular, your ability to rebuild critical infrastructure to restore data and ultimately to restore services.

Focusing on those things will not only improve your ability to respond to those things specifically, but actually will also improve your ability to manage your IT and your infrastructure because you move to a place where, instead of not understanding your IT and how you’re going to rebuild it, you have the ability to rebuild it. If you have the ability to rebuild your IT and restore your data on a regular basis, that actually creates a situation where it’s a lot easier for you to aggressively vulnerability manage and patch the underlying infrastructure.

Why? Because if you patch it and it breaks, you don’t have to restore it and get it working. So, focusing on the specific nature of ransomware and what it causes you to have to think about actually has a positive effect beyond your ability to manage ransomware.

SEE: A botnet threat in the U.S. targeted critical infrastructure. (TechRepublic)

CISOs need buy-in from other budget decision-makers

Megan Crouse: How should tech professionals and tech executives educate other budget-decision makers on security priorities?

Nick Godfrey: The first thing is you have to find ways to do it holistically. If there is a disconnected conversation on a security budget versus a technology budget, then you can lose an enormous opportunity to have that join-up conversation. You can create conditions where security is talked about as being a percentage of a technology budget, which I don’t think is necessarily very helpful.

Having the CISO and the CPO working together and presenting together to the board on how the combined portfolio of technology projects and security is ultimately improving the technology risk profile, in addition to achieving other commercial goals and business goals, is the right approach. They shouldn’t just think of security spend as security spend; they should think about quite a lot of technology spend as security spend.

The more that we can embed the conversation around security and cybersecurity and technology risk into the other conversations that are always happening at the board, the more that we can make it a mainstream risk and consideration in the same way that the boards think about financial and operational risks. Yes, the chief financial officer will periodically talk through the overall organization’s financial position and risk management, but you’ll also see the CIO in the context of IT and the CISO in the context of security talking about financial aspects of their business.

Security considerations around generative AI

Megan Crouse: One of those major global tech shifts is generative AI. What security considerations around generative AI specifically should companies keep an eye out for today?

Nick Godfrey: At a high level, the way we think about the intersection of security and AI is to put it into three buckets.

The first is the use of AI to defend. How can we build AI into cybersecurity tools and services that improve the fidelity of the analysis or the speed of the analysis?

The second bucket is the use of AI by the attackers to improve their ability to do things that previously needed a lot of human input or manual processes.

The third bucket is: How do organizations think about the problem of securing AI?

When we talk to our customers, the first bucket is something they perceive that security product providers should be figuring out. We are, and others are as well.

The second bucket, in terms of the use of AI by the threat actors, is something that our customers are keeping an eye on, but it isn’t exactly new territory. We’ve always had to evolve our threat profiles to react to whatever’s going on in cyberspace. This is perhaps a slightly different version of that evolution requirement, but it’s still fundamentally something we’ve had to do. You have to extend and modify your threat intelligence capabilities to understand that type of threat, and particularly, you have to adjust your controls.

It is the third bucket – how to think about the use of generative AI inside your company – that is causing quite a lot of in-depth conversations. This bucket gets into a number of different areas. One, in effect, is shadow IT. The use of consumer-grade generative AI is a shadow IT problem in that it creates a situation where the organization is trying to do things with AI and using consumer-grade technology. We very much advocate that CISOs shouldn’t always block consumer AI; there may be situations where you need to, but it’s better to try and figure out what your organization is trying to achieve and try and enable that in the right ways rather than trying to block it all.

But commercial AI gets into interesting areas around data lineage and the provenance of the data in the organization, how that’s been used to train models and who’s responsible for the quality of the data – not the security of it… the quality of it.

Businesses should also ask questions about the overarching governance of AI projects. Which parts of the business are ultimately responsible for the AI? As an example, red teaming an AI platform is quite different to red teaming a purely technical system in that, in addition to doing the technical red teaming, you also need to think through the red teaming of the actual interactions with the LLM (large language model) and the generative AI and how to break it at that level. Actually securing the use of AI seems to be the thing that’s challenging us most in the industry.

International and UK cyberthreats and trends

Megan Crouse: In terms of the U.K., what are the most likely security threats U.K. organizations are facing? And is there any particular advice you would provide to them in regards to budget and planning around security?

Nick Godfrey: I think it is probably pretty consistent with other similar countries. Obviously, there was a degree of political background to certain types of cyberattacks and certain threat actors, but I think if you were to compare the U.K. to the U.S. and Western European countries, I think they’re all seeing similar threats.

Threats are partially directed on political lines, but also a lot of them are opportunistic and based on the infrastructure that any given organization or country is running. I don’t think that in many situations, commercially- or economically-motivated threat actors are necessarily too worried about which particular country they go after. I think they are motivated primarily by the size of the potential reward and the ease with which they might achieve that outcome.

VC firm HAF Launches INR150 cr Fund to Invest in AI Startups

Hyderabad Angels Fund (HAF) has officially launched its maiden Venture Capital Fund, a Category 1 Alternative Investment Fund (AIF), featuring an investable corpus of INR 100 crore and an additional greenshoe option of INR 50 crore.

This strategic fund is specifically designed to focus on startups placing a significant emphasis on scalability through technology and AI.

The fund aims to deploy up to INR 5 crore across 20 disruptive startups that are positioned to scale with technology and AI in pre-series A/series A with a strong commitment to follow-on investments to support future growth.

“As serial entrepreneurs and angel investors, we are staunch believers in the transformative force of innovation. We are building HAF as a dynamic, collaboration-driven platform where founders not only secure financial backing but also tap into the collective wisdom of accomplished Limited Partners (LPs) who themselves are successful entrepreneurs,” Sri Myneni and Kishore Ganji, Managing Partners of HAF, said.

The post VC firm HAF Launches INR150 cr Fund to Invest in AI Startups appeared first on Analytics India Magazine.

Free Data Engineering Course for Beginners

Free Data Engineering Course for Beginners
Image by storyset on Freepik

It's a great time to break into data engineering. So where do you start?

Learning data engineering can sometimes feel overwhelming because of the number of tools that you need to know, not to mention the super intimidating job descriptions!

So if you are looking for a beginner-friendly introduction to data engineering, this free Data Engineering Course for Beginners, taught by Justin Chau, a developer advocate at Airbyte is a good place to start.

In about three hours you will learn essential data engineering skills: Docker, SQL, analytics engineering, and more. So if you want to explore data engineering and see if it is for you, this course is a great introduction. Now let's go over what the course covers.

Link to the course: Data Engineering Course for Beginners

Why Data Engineering?

This course starts out with an intro on why you should consider becoming a data engineer in the first place. Which I think is super helpful to understand before diving right into the technical topics.

The instructor, Justin Chau, talks about:

  • The need for good quality data and data infrastructure in ensuring the success of big data projects
  • How data engineering roles are growing in demand and pay well
  • The business value you can add to the organization working as a data engineer facilitating the organization’s data infrastructure

Docker

When you’re learning data engineering, Docker is one of the first tools you can add to your toolbox. Docker is a popular containerization tool that lets you package applications—with dependencies and config—in a single artifact called the image. This way Docker lets you create a consistent and reproducible environment to run all of your applications within a container.

The Docker module of this course starts with the basics like:

  • Dockerfiles
  • Docker images
  • Docker containers

The instructor then goes over to cover how to containerize an application with Docker: running through the creation of Dockerfile and the commands to get your container up and running. This section also covers persistent volumes, Docker networking fundamentals, and using Docker-Compose to manage multiple containers.

Overall this module in itself is a good crash course on Docker if you’re new to containerization!

SQL

In the next module on SQL, you’ll learn how to run Postgres in Docker containers and then learn the basics of SQL by creating a sample Postgres database and performing the following operations:

  • CRUD operations
  • Aggregate functions
  • Using aliases
  • Joins
  • Union and union all
  • Subqueries

Building a Data Pipeline From Scratch

With Docker and SQL foundations, you can now learn to build a data pipeline from scratch. You’ll start by building a simple ELT pipeline that you’ll get to improve throughout the rest of the course.

Also, you’ll see how all the SQL, Docker networking, and Docker-compose concepts that you have learned thus far come together in building this pipeline that runs Postgres in Docker for both the source and destination.

dbt

The course then proceeds to the analytics engineering part where you’ll learn about dbt (data build tool) to organize your SQL queries as custom data transformation models.

The instructor works you through getting started with dbt: installing the required adapter and dbt-core and setting up the project. This module specifically focuses on working with dbt models, macros, and jinjas. You'll learn how to:

  • Define custom dbt models and run them on top of the data in the destination database
  • Organize SQL queries as dbt macros for reusability
  • Use dbt jinjas to add control structures to SQL queries

CRON Job

So far, you’ve built an ELT pipeline that runs upon manual triggering. But you certainly need some automation, and the simplest way to do this is to define a cron job that automatically runs at a specific time of the day.

So this super short section covers cron jobs. But data orchestration tools like Airflow (which you’ll learn in the next module) give you more granularity over the pipeline.

Airflow

To orchestrate data pipelines, you’ll use open-source tools such as Airflow, Prefect, Dagster, and the like. In this section you’ll learn how to use the open-source orchestration tool Airflow.

This section is more extensive as compared to the previous sections because it covers everything you need to know to get up to speed to write Airflow DAGs for the current project.

You’ll learn how to set up the Airflow webserver and the scheduler to schedule jobs. Then you’ll learn about Airflow operators: Python and Bash operators. Finally, you’ll define the tasks that go into the DAGs for the example at hand.

Airbyte

In the last module, you’ll learn about Airbyte, an open-source data integration/movement platform that lets you connect more data sources and destinations with ease.

You’ll learn how to set up your environment and see how you can simplify the ELT process using Airbyte. To do so, you’ll modify the existing project’s components: ELT script and DAGs to integrate Airbyte into the workflow.

Wrapping Up

I hope you found this review of the free data engineering course helpful. I enjoyed the course—especially the hands-on approach to building and incrementally improving a data pipeline—instead of focusing on only theory. The code is also available for you to follow along. So, happy data engineering!

Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she's working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.

More On This Topic

  • Free MLOps Crash Course for Beginners
  • Free AI for Beginners Course
  • Free Microsoft Excel for Beginners Course
  • KDnuggets News, October 5: Top Free Git GUI Clients for Beginners •…
  • Top Free Git GUI Clients for Beginners
  • 3 Free Machine Learning Courses for Beginners

Sam Altman has a $7 Tn Pipe Dream

The tech industry is full of AI saviours and self-appointed messiahs who have come and gone or are struggling to stay relevant. The latter is mainly done by making everyone talk about their word and action – at times, even through failed events.

In other news, OpenAI’s much-cherished Sam Altman is looking for someone to invest $5 – $7 trillion (as per sources) to “reshape the business of chips and AI”. Sam Lessin, a Silicon Valley investor and early Facebook executive, compared Altman to Elon Musk, CEO of Tesla and SpaceX.

From a purely tech perspective, Altman feels the world needs better AI infrastructure which OpenAI will help build through the amount asked for. But many, including Lessin, disagree.

Lessin wrote, “I don’t grudge Sam’s showmanship in and of itself—he is just extending the game Elon has played with ‘self-driving cars around the corner’ or ‘Mars by 2024’.” In 2020, Musk said that SpaceX’s first crewed Mars mission could launch as early as 2024. Two years later, he pushed that back to 2029.

Altman and his munks’ chip adventure sounds like a similar pipe dream. With a number that big, the loss of touch with reality appears to be noticeable. It makes one wonder whether it is all a rush to leave a legacy. The news, broken by WSJ, noted that these “fundraising talks are the latest example of ambitious plans from Altman that seek to change the world”.

But the answer to how OpenAI’s ambition to benefit the whole of humanity through projects leading to AGI remains unknown. AI cannot solve the problems en masse, but the amount demanded and thrown around could.

Sasha Lucionni, who works at the intersection of AI and climate, has already done a commendable job of pointing out the project’s environmental impacts. Something significant, yet rarely spoken about in terms of ChatGPT and all of generative AI.

Just How Much is $7 Trillion?

“The amounts Altman has discussed would also be outlandishly large by the standards of corporate fundraising—larger than the national debt of some major global economies and bigger than giant sovereign-wealth funds,” the Journal notes.

  • For starters, it is double the current Indian economy
  • You could stack one dollar bills from Earth to the Moon twice. Now you can imagine it
  • Enough to buy Nvidia, TSMC, Broadcom, ASML, Samsung, AMD, Intel, Qualcomm, and every other chipmaker, designer, intellectual property holder, and equipment vendor of consequence in their entirety – and still have trillions left
  • Current German and French GDPs combined
  • WSJ pointed out that the amount dwarfs the size of today’s global semiconductor industry

In a nutshell, it is an enormous amount to be spent on chips!

The plan sounds herculean, given that just $527 billion worth of chips were sold globally last year. OpenAI has struggled to secure sufficient supply from Nvidia, the seller of most microprocessors fueling the ongoing AI mania. The reason Altman gave for seeking the cash is due to the scarcity of AI chips that OpenAI is facing.

At first, Altman’s request appears to be a joke, but the OpenAI chief has been meeting top members of the UAE government, including the Sheikh. Interestingly, he is also in talks with Masayoshi Son, the CEO of SoftBank. A part of the same group, which had a net loss of ¥931.1 billion, compared with a ¥3 trillion profit last year when the Japanese company cashed in on its stake in Alibaba Group Holding Ltd.

(Source: Google search ‘softbank controversy’)

These tech shenanigans making it to the headlines are not new. But celebrating a figure like Altman has cost the economy in the past. Who can forget the billions Sam Bankman Fried lost? Or what is the amount Musk spent on marketing self-driving cars? Oh, and the collapse of Silicon Valley Bank.

As researchers and analysts continue to decode Altman’s soap opera on their substacks, the OpenAI head has put on the cape to “secure our collective future”. The question arises: To what extent can Altman and leaders alike go to flatter themselves?

Altman’s latest ask to build future tech reminds one of Emily M. Bender’s remark: “Who are they to be speaking for all of humanity? The handful of very wealthy (even by American standards) tech bros are not in a position to understand the needs of humanity at large.”

For the time being, we wait and watch which genie will fulfil Altman’s wish of the $7 trillion magic carpet to help him fly through the Bay Area.

The post Sam Altman has a $7 Tn Pipe Dream appeared first on Analytics India Magazine.

Fulcrum Digital Unveils Ryze, a GenAI Solution for Enterprise AI-driven Content

Fulcrum Digital has launched Ryze, a Generative AI platform, during its annual event, Tandem. Rajesh Sinha, Founder and Chairman of Fulcrum Digital, presented Ryze as a mature GenAI platform aimed at transforming content generation and data processing.

The platform integrates generative models, data analytics, user interaction, and customisation, offering tailored content and data solutions for various industries. Backed by the FulcrumOne digital accelerator, Ryze ensures scalability to meet evolving customer needs.

Positioned as a Software as a Service (SaaS) solution, Ryze enables businesses to enhance content generation for both textual and visual mediums. It features configurable components for LLMs, allowing the accommodation of preferred models like OpenAI’s GPT 4, Llama2, SQL Coder, and GenZ. The platform also facilitates seamless integration of customisable applications and boasts intelligent enterprise assistant capabilities.

Ryze’s launch was accompanied by insights from Sinha, who expressed its importance in Fulcrum Digital’s global mission to harness AI’s transformative potential. He emphasised Ryze’s utilisation of enterprise data, covering text, image, audio, video, and 3D formats from various channels. Sinha highlighted features such as data synthesis, sentiment analysis, and the provision for downloading synthesised data for further analysis and utilisation.

Sachin Panicker, Chief AI Officer at Fulcrum Digital, shared his optimism about Ryze’s potential to catalyse innovation and disruption in the digital age. He outlined key features like conversational interaction, capable comprehension, core business process agents, coding assistance, and sentiment analysis, focusing on empowering businesses with efficient and insightful solutions.

The platform’s use cases across sectors include streamlining claims processing in insurance and automating inventory management in retail. Ryze’s capabilities extend to digitising claims, providing conversational AI for more satisfying customer interactions, and enhancing fraud detection through real-time monitoring and predictive analytics.

The post Fulcrum Digital Unveils Ryze, a GenAI Solution for Enterprise AI-driven Content appeared first on Analytics India Magazine.

Learn Data Science on a Budget

Learn Data Science on a Budget
Image by DALLE

Finding the right platform or course is never easy when you are looking to learn something new. It consists of a lot of trial and error, and the worst thing is when you pay for them. You feel like you’ve been going around in circles trying to find the best course to elevate your career but then you end up back at square one — and you’ve hurt your pockets (sometimes a bit, sometimes a lot)

That’s why I have decided to put this article together, so you can learn from my mistakes and gain all the data science knowledge — on a budget!

Understanding Data Science

Link: Understanding Data Science

Let’s start with DataCamp — an online learning platform that provides a range of different courses. Their ‘Understanding Data Science’ course is catered to beginners who want to get their feet wet in the data science world. And you also don’t have to worry about having coding experience — because it is not needed!

It is a 2-hour course and consists of 15 learning videos and 48 exercises! Broken up into 4 sections, the 1st section Introduction to Data Science is free, and the 3 remaining sections will require you to get a membership. The membership prices are reasonable and with it you will have access to DataCamps full content library, making it useful when you want to take on additional courses.

Data Science from Scratch

Link: Data Science from Scratch

This is for the book worms who prefer this as their learning method. In the ‘Data Science from Scratch’ book you will learn about the discipline of data science, such as the libraries, frameworks, modules, and toolkits. As well as how the algorithms work and how you can implement them from scratch.

If you’re looking for a crash-course in Python and want to learn the basics of data science, dive into the fundamentals of machine learning, and dive deeper — this book is for you.

Introduction to Probability

Link: Introduction to Probability

A 10 week long course that is FREE from Harvard! With a lot of online bootcamp courses, probability is sometimes forgotten about. However, it is essential for your understanding of the foundations of data science. Learning probability can take you from being a ‘good’ data scientist to an ‘expert’ data scientist.

In this course you will learn about uncertainty and randomness, making good predictions, common probability distributions, and more. The course is free but you can add a verified certificate for $209.

Python in a Nutshell

Link: Python in a Nutshell

Python has been ranked as one of the most popular programming languages for a while now — and there’s a reason why. It's simple syntax has made it easy for people to transition into the world of data science with ease. There is a demand for data professionals, and the Python language has definitely paved the way.

This 4th edition ‘Python in a Nutshell’ book consists of 26 chapters which take you from a complete beginner in Python to an absolute expert! From representing data in Python to debugging — this book has it all!

Data Analysis with Python

Link: Data Analysis with Python

Freecodecamp.org started out as an organization that wanted to make learning accessible to everyone — hence the name. If you are looking for a course that takes you through everything, then this Data Analysis with Python course is highly recommended. A lot has changed in the data science world, especially Python, SQL, and more.

In this course you will learn the absolute fundamentals in order to become a data scientist, from reading your own data to using libraries to analyze it and create data visualizations.

Wrapping it up

With these 5 courses/platforms you will be able to take yourself from someone who did not have any experience about data science to fixing up your resume to apply for your first data science role. Data science continues to be a very popular role, and there's no better time to start than now!

Nisha Arya is a Data Scientist and Freelance Technical Writer. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.

More On This Topic

  • How to Set Up Your Data Science Stack on a Budget
  • Stop Learning Data Science to Find Purpose and Find Purpose to…
  • 5 Data Science Projects to Learn 5 Critical Data Science Skills
  • KDnuggets News, May 4: 9 Free Harvard Courses to Learn Data…
  • The Best Data Science Resources, Bootcamp, and Courses to Learn…
  • KDnuggets News March 16, 2022: Learn Data Science Fundamentals & 5…

IBM and Mahindra Finance Collaborate to Develop a Superapp for Finance

IBM and Mahindra & Mahindra Financial Services Limited (Mahindra Finance) today announced a strategic collaboration to build a super app that would serve as a single digital interface for consumers to access multiple businesses within Mahindra Finance.

This app will enable consumers from both metros and non-metros to have 24×7 digital access to avail and manage products & solutions in a secure, streamlined, and simple manner. This would be by housing Mahindra Finance’s various vehicle and non-vehicle lending businesses, including personal loans, enterprise loans, mortgage loans, leasing solutions, payments, fixed deposits, insurance solutions, and other investment & wealth management solutions under one roof.

IBM Consulting will leverage human-centred design methodology to enhance the super app experience, covering every stage of the customer journey. This approach will involve integrating deep understanding of Mahindra Finance’s customer needs with superior application UI design to offer an experience that creates customer value as well as unlocks business value for the company.

At the foundation level, IBM Consulting will bring cloud transformation and cloud management expertise to help design a secure, scalable, and resilient digital platform architecture to be deployed on a hybrid multi-cloud environment that will enable easy integration of new technologies in future, including generative AI.

The app will further provide superior and immersive experiences to the customers of Mahindra Finance and strengthen the company’s position as the financial solutions partner of choice for emerging India.

“We at Mahindra Finance are keen to accelerate the delivery of our digital financial solutions to better serve our customers. The super app is a strategic lever that will expedite the growth of key businesses in an operationally efficient manner that combines omnichannel capabilities. It will also empower us to create hyper-personalised solutions for emerging India,” Raul Rebello, MD & CEO-Designate, Mahindra Finance, said.

The post IBM and Mahindra Finance Collaborate to Develop a Superapp for Finance appeared first on Analytics India Magazine.

HAF Launches INR 150 Cr Fund to Invest in AI Startups

HAF has announced the launch of its inaugural Venture Capital Fund, a Category 1 Alternative Investment Fund (AIF) with an investable corpus of INR 100 crore, along with an additional greenshoe option of INR 50 crore. This fund is strategically tailored to prioritise startups that emphasise scalability through technology and AI.

The fund aims to deploy up to INR 5 crore across 20 disruptive startups positioned to scale with technology and AI in Pre-Series A/Series A, with a strong commitment to follow-on investments to bolster future growth.

The fund encompasses successful entrepreneurs and business leaders in pharma, health sciences tech, education, sustainability, and other sectors.

Sri Myneni and Kishore Ganji, Managing Partners of HAF, express a mission that extends beyond funding, aiming to collectively address individual challenges and connect visionary ideas with strategic capital to innovate and grow globally.

They emphasised, “As serial entrepreneurs and angel investors, we are staunch believers in the transformative force of innovation. We are building HAF as a dynamic, collaboration-driven platform where founders not only secure financial backing but also tap into the collective wisdom of accomplished Limited Partners (LPs) who themselves are successful entrepreneurs.”

Leveraging Hyderabad Angels‘ decade-long investment acumen, the fund strategically taps into the mentorship of Hyderabad’s serial entrepreneurs and investors across diverse sectors, fostering an impactful catalyst for startup founders to drive innovation and achieve rapid scalability.

HAF draws on the established platforms and processes created by Hyderabad Angels.

BVR Mohan Reddy, Founder & Executive Chairman of Cyient, underscores the importance of an active engagement platform between LPs/investors and entrepreneurs, positioning HAF at the forefront of driving collaboration. He stated, “As an active investor in numerous venture funds and startups, I am of the belief that fostering an active engagement platform between LPs investors and entrepreneurs is pivotal for creating successful businesses.”

Mahankali Srinivas Rao (MSR), CEO of T-Hub, a well-accomplished business leader, and an advisor of the fund, acknowledges Hyderabad’s role as the nucleus of the world’s largest innovation hub, stating that HAF is ready to harness the potential inherent in the synergy of funds, direct engagement with industry leaders, and the vibrant startup ecosystem in Hyderabad.

The post HAF Launches INR 150 Cr Fund to Invest in AI Startups appeared first on Analytics India Magazine.