Bedrock Is All About Choices

Amazon Bedrock, now an integral part of Amazon Web Services (AWS), has established itself as a significant player in the generative AI landscape. The recent AWS re:Invent conference unveiled a series of groundbreaking advancements, further solidifying Bedrock’s position.

These updates not only enhance Bedrock’s versatility but also mark its edge over competitors like Hugging Face Transformers, OpenAI, and Google AI. The comparison primarily revolves around foundation models, deployment options, and the balance of security, privacy, and user accessibility.

More choices for users

The recent updates to Amazon Bedrock have introduced several key features that collectively enhance the platform’s capabilities, making it more secure, and providing a range of models for developers to choose from in the generative AI space. These updates have significantly lowered the barrier to entry for AI application development and offer tailored solutions to complex problems.

Guardrails for Amazon Bedrock allows developers to implement custom safeguards that align with their specific use cases and responsible AI policies. This means that developers can now tailor their AI applications to adhere to industry-specific regulations, cultural sensitivities, or organisational policies, thereby broadening the scope of AI’s applicability and ethical alignment.

“Now you can have a consistent level of protection across all of your GenAI development activities. For example, a bank could configure an online assistant to refrain from providing investment advice or to prevent inappropriate content,” Adam Selipsky said at the re:Invent conference.

Another significant update is the development of Agents for Amazon Bedrock. These agents are engineered to streamline the development of generative AI applications by orchestrating multi-step tasks. They use the reasoning capabilities of foundation models to dissect complex user requests into manageable steps and then execute these steps efficiently. This cuts the middle layer of wrappers and functions as a one stop solution to training and deploying models. “Bedrock is shaping up to be AWS’ proprietary alternative to open source tools like Langchain and LlamaIndex,” Sam Charrington, a tech podcaster posted on X.

Enhanced Control for Agents is yet another important enhancement in Bedrock. This enhancement allows developers to fine-tune their applications, ensuring that the end product is not only efficient but also aligned with the specific needs of their project or organisation.

Private Customization of Foundation Models is a critical feature that allows developers to privately and securely customise foundation models with their proprietary data within Bedrock. By enabling the creation of highly tailored applications, this feature offers a competitive edge, particularly for developing AI solutions that are specific to a company’s domain and requirements.

Randall Hunt, VP of Cloud Strategy posted on X, “Think of this as your very own private ChatGPT hosted in YOUR AWS account. APIs and common integrations included.” They’re offering a high level of personalisation for their already large customer base. Daniel Newman posted on X saying he liked AWS’s open model approach. “A Broad LLM and FM strategy makes a lot of sense for most of the AWS customer ecosystem.”

Bedrock vs the Rest

One of the most notable aspects of Bedrock is its extensive range of foundation models sourced from leading AI companies. This diversity in Bedrock is particularly advantageous for developers looking for flexibility and the ability to tailor their applications to a wide range of use cases.

In terms of deployment, Bedrock operates as a fully managed service, which marks a significant departure from the varied deployment options provided by its competitors. It streamlines the development process for users by eliminating the complexities associated with infrastructure management. It presents a more user-friendly approach, especially for those who may not have extensive resources to manage and maintain their AI infrastructure.

This ease of deployment is a key factor in Bedrock’s appeal, as it allows developers to focus more on innovation and less on the operational aspects of their projects. “Bedrock has 10,000 customers worldwide, foundation models are key in generative AI,” Adam Selipsky said at the conference.

The post Bedrock Is All About Choices appeared first on Analytics India Magazine.

5 Free Courses to Master Data Engineering

5 Free Courses to Master Data Engineering
Image by storyset on Freepik

It’s an exciting time for the data field, and data engineers play a big part in it. With preparing and managing the data infrastructure, data engineers bring tools necessary for data collection, storage, processing, and analysis. That’s why data engineers play a significant role in a data-driven company.

With such an important role, the companies would pay quite a sum to hire data engineer to solve their data infrastructure problems. So, how could we try to master the art of data engineer? Here are five free courses you can take to boost your career.

IBM: Data Engineering Basics for Everyone

Let’s start with the fantastic data engineering course from IBM on edX. This course is intended to introduce data engineering basics for beginners to prepare you for more advanced use cases.

I love this course because it gives me a data engineering foundation even though I am a data scientist. I am confident that this course would provide the beginner skills necessary to master data engineer.

The essential concepts you would learn through this course are:

  • Concept of Data Engineer
  • Workflow Management
  • Relational Database
  • Data Lakes
  • Data Stores
  • IBM Cloud Computing

The courses are also self-paced, so you can learn in your allocated time without feeling constrained.

Google Data Engineer Learning Path

Having learned about IBM, you should also look at what Google offers. Within their Cloud Skill Boost is a Data Engineer Learning Path that would upskill your Data Engineer skill with the Google Cloud Platform.

In the learning path, you would learn various data engineering skills through 13 courses designed for any beginner and professional. In the courses, you would understand the following skills:

  • Google Cloud Computing
  • Data Pipelines
  • Data Warehouse
  • Data Lakes
  • Data Stores
  • Serverless Data Processing

Like the previous entry, the course is also self-paced, which you can try to finish at your own pace. With cloud computing in the future, this course will give you an edge for the future employer.

Meta Database Engineer Professional Certificate

Let’s move on to another free data engineering course taught by another famous company, Meta. The Meta Database Engineer Professional Certificate course would focus on the data engineer skill upskilling, which is the database.

Data engineers work for the whole data flow; a big part of this flow is having a robust database. Without a reliable database, the entire data ecosystem would collapse. That’s why Meta provides this course for anyone willing to carve their path in the database.

Within this course, you would learn the following skills:

  • Database Basics
  • Database Structure
  • Database Clients
  • Version Control
  • Advance MySQL

With 9 courses and a flexible schedule, you can leisurely upskill yourself.

UC San Diego Big Data Specialization

A specialized course is essential for your data engineer career as it allows you to stand out. The free course UC San Diego Big Data Specialization would provide a base in the big data world.

In the data engineering field, Big Data is a complex problem that requires a specific approach. Data engineers need to be able to handle designing data flow that enables companies to process and utilize Big Data, which this course is all about. In this course, you will learn:

  • Big Data Basics
  • Big Data Management
  • Big Data Integration and Processing
  • Graph Analytics and Big Data

There are 6 courses in the series with a flexible schedule. There is an estimation that you can finish the course in 3 months if you learn 10 hours per week.

Data Engineering Zoomcamp

The Data Engineering Zoomcamp is a free course taught by Ankush Khanna, Victoria Perez Mola, and Alexey Grigorev for any beginner run by the community.

The courses would provide you with a complete view of data engineering from the expert in the field and prepare you to be ready for the job. With 9 weeks of learning, you would learn the following skills:

  • Data Engineer Basics
  • Data workflow and orchestration
  • Data Warehouse
  • Data Pipeline
  • Analytic Engineering

The course has two ways of learning: Self-paced and Cohort Courses. The cohort learning registration has its opening schedule, so keep an eye on their pages constantly.

Conclusion

To make data-driven decisions company, we require data infrastructure that could support the process. A data engineer takes this task to provide a sophisticated data workflow. To master data engineering skills, we can learn from the five courses discussed in this article. With the structured courses from the renowned instructor, these courses would help you understand everything to get the data engineer job.

Cornellius Yudha Wijaya is a data science assistant manager and data writer. While working full-time at Allianz Indonesia, he loves to share Python and Data tips via social media and writing media.

More On This Topic

  • Free Data Engineering Courses
  • 5 Free Courses to Master Data Science
  • 5 Free Courses to Master Machine Learning
  • 5 Free Courses to Master Linear Algebra
  • 5 Free Courses to Master Calculus
  • 5 Free Courses to Master Generative AI

Top 7 Generative AI Conferences Worldwide 

Top 7 Generative AI Conferences Worldwide

Conferences provide innovation crucibles all around the world, bringing together bright minds who expand the limits of what is possible. They also offer a platform for researchers, practitioners, and enthusiasts to share their latest findings, breakthroughs, and innovations. This knowledge exchange accelerates the dissemination of cutting-edge research across the community.

Generative AI conferences draw attention by promoting global collaboration and are essential for addressing diverse challenges, leveraging a broad range of perspectives, and ensuring that productive technologies are developed responsibly and inclusively.

Here’s a list of influential conferences that bring together researchers, practitioners, and enthusiasts from around the globe:

MLDS (Machine Learning Developers Summit)

Scheduled every year, MLDS is one of the biggest generative AI conferences in India. Mostly hosted in Bengaluru, this conference brings together some of India’s top machine-learning specialists. There will be networking and educational opportunities at the event to help C-suite executives and engineers alike comprehend the implications of technology.

MLDS is a comprehensive summit that covers a broad spectrum of machine learning topics, including generative AI. It includes keynote sessions, hands-on workshops, and networking opportunities, making it a valuable event for beginners and experienced professionals.

NeurIPS (Conference on Neural Information Processing Systems)

Established in 1987, NeurIPS has evolved into a dynamic and multidimensional annual event featuring multiple tracks that span various disciplines. Usually held in the month of December internationally by the Neural Information Processing Systems Foundation, this is one of the largest gatherings in the AI community, featuring a wide range of topics, including generative models.

The conference includes workshops, tutorials, and keynote presentations from leading experts in the field and big companies like Microsoft, Google, IBM, etc, participate as sponsors for specific events and contests of the conference. It has been decided that this annual conference will be held again this year at the New Orleans Ernest N. Morial Convention Center from December 10th to 16th.

ICML (International Conference on Machine Learning)

Recognised as one of the three cornerstone conferences with a significant impact on machine learning and artificial intelligence research, ICML is usually held in July and is hosted by the International Machine Learning Society. This conference serves as a premier machine learning conference that often showcases cutting-edge research on generative models. The conference includes paper presentations, poster sessions, and invited talks.

ICML, first launched in 1980 in Pittsburgh, provides a forum for researchers, practitioners, and educators to present and discuss the most recent advances and challenges in machine learning. The conference invites prominent researchers and industry experts to deliver keynote speeches, sharing insights into the latest trends and challenges in machine learning. ICLM 2024 is scheduled to start from the 21st to the 27th of July at Messe Wien Exhibition Congress Center, Vienna, Austria

CVPR (Conference on Computer Vision and Pattern Recognition)

Hosted by the IEEE Computer Society, the first CVPR was held in 1983 in Washington DC by Takeo Kanade and Dana Ballard and is usually held in June. While primarily focused on computer vision, CVPR frequently features research on generative models for image synthesis. It attracts researchers and professionals from both academia and industry.

It covers a wide range of topics within computer vision, including image and video analysis, deep learning, 3D vision, object recognition, and more. CVPR promotes open access to research findings, allowing for the widespread dissemination of knowledge within the computer vision community. The event is set to be conducted this year from June 17th to 21st at the Seattle Convention Center.

IJCAI (International Joint Conference on Artificial Intelligence)

IJCAI has been the longest-standing premier AI conference series globally since 1969 and typically conducts the conference during the month of August. It is a general AI conference that may include presentations on various topics, including generative AI. It provides a platform for researchers and practitioners to share their latest findings.

Usually, the event used to be held biennially in odd-numbered years, and since 2016, it has been scheduled as an annual event. It focuses on workshops and tutorials on specialised topics, offering participants the opportunity for in-depth learning and discussion while featuring keynote speeches by prominent researchers and thought leaders in artificial intelligence. The conference is a global event, and IJCAI-24 is scheduled for August 3–9 in Jeju Island, South Korea, followed by IJCAI-25 in Montreal, Canada, and IJCAI-ECAI-26 in Bremen, Germany.

AAAI (Association for the Advancement of Artificial Intelligence)

AAAI, which was first held in 1980 at Stanford University, is a major AI conference, usually held in February, that features research on generative AI. It includes technical paper presentations, invited talks, and tutorials. AAAI covers a wide range of topics within artificial intelligence, including but not limited to machine learning, knowledge representation, reasoning, planning, robotics, natural language processing, and applications of AI.

In addition to the leading AAAI conference, the Innovative Applications of Artificial Intelligence (IAAI) conference focuses on applied AI and real-world applications. AAAI includes tutorials and workshops on specialised topics, and the 38th Annual AAAI Conference is set to be held from February 20 to 27 in Vancouver, Canada.

AISTATS (International Conference on Artificial Intelligence and Statistics)

AISTATS, founded in 1985, serves as a crossroads for researchers blending computer science, artificial intelligence, machine learning, statistics, and allied domains. It is held annually in April, focusing on the intersection of artificial intelligence and statistics and featuring research that may include generative models. The conference includes contributed talks, poster sessions, and tutorials.

AISTATS emphasises the integration of statistical methods with artificial intelligence and machine learning techniques. The conference showcases cutting-edge research in statistical modelling, learning algorithms, and the application of statistical techniques to real-world problems. The 27th edition of the International Conference AISTATS is taking place in Valencia, Spain, from May 2 to May 4, 2024.

The post Top 7 Generative AI Conferences Worldwide appeared first on Analytics India Magazine.

Will Stability AI Survive?

Stability AI is a sinking ship.The company’s business model is in crisis and is trying to find ways to stay afloat in the market.

The London-based AI startup, which gained popularity with its text-to-image generation model Stable Diffusion, is contemplating selling the company as its management is grappling with increasing investor pressure regarding its financial standing.

Interestingly, the company approached Cohere and Jasper AI, but Cohere declined to engage in the talks.

Meanwhile, the investors want Stability AI founder Emad Mostaque to step down. Coincidentally, Stability AI’s website is showing an error 404 against Mostaque’s name. The relationship between Stability AI and its investors has soured recently as the company is struggling to generate substantial revenue while offering its models for free to customers.

One of its key investors, Coatue, has claimed that Mostaque’s poor leadership led to the departure of several top employees, placing the startup in a challenging financial position. Representatives from both Coatue and Lightspeed have already stepped down from Stability’s board, expressing disagreements with Mostaque’s management style.

Apparently, Stability AI recently receiving a $50 million investment from Intel didn’t sit well with Coatue, as it has a stake in Intel’s rival, AMD

Competition Galore

Stability AI is finding it difficult to acquire customers as big tech giants have entered the foray with their own image generation models.

Meta has Emu Edit and Video, Google has Imagen, and recently, Amazon also launched its own image generation model- Titan Image Generator, now in preview and available for AWS customers on Bedrock.

To make matters worse, Dall-E-3 on ChatGPT Plus acted as the final nail in the coffin. ChatGPT today has more than 100 million weekly users.

Moreover, Midjourney, a direct competitor to Stable Diffusion, has over 16.4 million active users as of November. On the other hand, Stable Diffusion has more than 10 million daily active users across all channels, according to Mostaque.

It is fascinating that, although Midjourney has been following a subscription model from very early on, it has still been able to retain customers. Midjourney’s monthly subscriptions start from $10 and go up to $120.

Can Stability AI turn it around

As of now, Stability AI offers Stable LM, Stable Audio, and Stable Diffusion XL. The company recently introduced Stable Video, a new free AI research tool that can turn any still image into a short video. Majority of the above models are free to use.

However, Mostaque has recently expressed his intentions to introduce Stability AI Memberships. He believes that in today’s market any AI startup needs to have a business model; otherwise, survival in the market won’t be easy.

“We are doing ok as a business and ramping up nicely” he posted on X.

The company generated $1.2 million in revenue in August and was projected to reach $3 million this month from software and services, according to a post by Mostaque on X on Monday, which he later deleted.

To determine the ideal pricing for his new core Stable models, he ran a poll on X. Ironically, most users voted for $1, indicating that they want the models to be available for free.

What should we price @StabilityAI monthly memberships at allowing commercial use of our new core Stable models (eg SDXL Turbo, Stable Video Diffusion) for those making < $1 million a year in revenue?

— Emad (@EMostaque) November 29, 2023

It appears that Stability AI has already implemented the subscription model on Stable diffusion XL. If you visit the website and attempt to create an image, it asks for a monthly price of $10.

Mostaque mentioned that there has been tension in the organisation lately regarding what to release versus withhold, how to compete on API and other products, including consumer-focused ones.

“We want to release good models, and the line between open-source and releasing in the open, plus how to build a sustainable business, has been something we have spent a lot of time thinking about,” he added.

Mostaque said that for commercial usage you will compulsorily need to have a Stability AI Membership.”For example, we’re considering, for an indie developer, this fee to be $100 a month, but only if you make above a certain amount of revenue, similar to game engines,” he said.

“It’s like Amazon Prime or Netflix for generative AI models,” he added. However, he continued that for non-commercial and academic usage, they will continue to provide the models for free.

It’s commendable that Stability AI is making efforts to avoid the need for a sale. However, if the situation arises where selling becomes the best way out, the company should seriously consider approaching Apple.

The post Will Stability AI Survive? appeared first on Analytics India Magazine.

HPE, NVIDIA Collaborate for Generative AI on Edge and Cloud

HPE, NVIDIA Collaborate for Generative AI on Edge and Cloud

Hewlett Packard Enterprise (HPE) announced an extensive collaboration with NVIDIA to introduce an enterprise computing solution designed for generative AI. This collaboration expands HPE’s suite of purpose-built, AI-native offerings, providing enterprises with an unprecedented opportunity to accelerate AI model training, tuning, and inferencing.

The co-engineered solution simplifies the customisation of foundation models using private data and facilitates the deployment of production applications across diverse environments, from edge to cloud. This offers a seamless full-stack AI tuning and inferencing solution jointly developed by HPE and NVIDIA.

The new enterprise computing solution for generative AI is part of an expanded collaboration between HPE and NVIDIA, providing full-stack, out-of-the-box AI solutions. These solutions integrate HPE Machine Learning Development Environment, HPE Ezmeral Software, HPE ProLiant Compute, and HPE Cray Supercomputers with the NVIDIA AI Enterprise software suite, including the powerful NVIDIA NeMo framework.

Antonio Neri, president and CEO of HPE, emphasised the importance of this collaboration, stating, “Together, HPE and NVIDIA are in a unique position to deliver a comprehensive AI-native solution that will dramatically ease the journey to develop and deploy AI models with a portfolio of pre-configured solutions.”

Jensen Huang, founder and CEO of NVIDIA, added, “Our expanded collaboration with HPE will help enterprises drive unprecedented productivity through AI applications that connect with business data to power accurate assistants, informed chatbots, and semantic search.”

Purpose-built and optimised for AI: The solution features a rack-scale architecture with market-leading HPE ProLiant Compute DL380a pre-configured with NVIDIA L40S GPUs, NVIDIA BlueField-3 DPUs, and the NVIDIA Spectrum-X Ethernet Networking Platform for hyperscale AI.

Read: NVIDIA RTX Brings Alan Wake 2 to Life

This also includes HPE Machine Learning Development Environment with new generative AI studio capabilities for rapid prototyping and testing, and HPE Ezmeral Software with new GPU-aware capabilities to simplify deployment and accelerate data preparation for AI workloads across the hybrid cloud.

By utilising NVIDIA AI Enterprise to accelerate production AI development and deployment with security, stability, manageability, and support. This software suite offers the NVIDIA NeMo framework, guardrailing toolkits, data curation tools, and pretrained models to streamline enterprise solutions.

In addition to the collaboration, HPE Services now offers a broad portfolio of consulting services, workforce training, and deployment solutions to guide enterprises through every step of their AI journey. The comprehensive services, supported by new Global Centers of Excellence for AI and Data, are now open in Spain, the United States, Bulgaria, India, and Tunisia.

The post HPE, NVIDIA Collaborate for Generative AI on Edge and Cloud appeared first on Analytics India Magazine.

OpenAI is Nothing Without Ilya

OpenAI is Nothing Without Ilya

Ilya Sutskever, the chief scientist at OpenAI, will no longer be part of the board, said Sam Altman in the latest OpenAI blog on his return as the CEO of OpenAI.

“I love and respect Ilya, I think he’s a guiding light of the field and a gem of a human being,” “I harbour zero ill will towards him. While Ilya will no longer serve on the board, we hope to continue our working relationship and are discussing how he can continue his work at OpenAI.”

There is possibly some misalignment between Altman and Sutskever’s vision of what the company wants, and is further exaggerated after Sutskever was the one who fired Altman on a Google Meet call.

Now, Altman, Greg Brockman, and Mira Murati are going to lead the company, with Bret Taylor, Larry Summers, and Adam D’Angelo as the initial board, and Microsoft as an observer part of the board.

building this together ❤️ pic.twitter.com/s4PVAywD9e

— Greg Brockman (@gdb) November 30, 2023

More drama from OpenAI?

The plot for the upcoming film, I Love You All (Ilya) is going through a lot of twists and turns. Sutskever seems to be on the grey line now, where he might either quit, or end up getting fired. Though Altman has said that there are no ill intentions against him, Sutskever might be considering leaving OpenAI, though he has not mentioned that.

It is not like he wasn’t guilty with whatever unfolded. Sutskever made a public apology on X that he in no way wanted to harm OpenAI, and deeply regrets his participation in the board’s actions. But now the whole board of the company has changed, even Helen Toner and Tasha McCauley are out of the board. Toner posted on X about resignation from the OpenAI board, but did not say anything concrete about her next plans, just focus on AI safety policy.

Moreover, at the recent talk at NYT Dealbook, Elon Musk said that he has reached out to Sutskever, but he does not want to talk. Musk has been praising Sutskever all this while, and was on his side during all the OpenAI weekend drama. “Ilya has a good moral compass and does not seek power. He would not take such drastic action unless he felt it was absolutely necessary,” Musk posted on X.

Musk had also asked Sutskever on X about why he fired Altman, as there were rumours that OpenAI has built a model that is way too powerful, which allegedly scared Sutskever. This was possibly in reference to Q* that OpenAI is working on, which got leaked unfortunately according to Altman.

Why did you take such a drastic action?
If OpenAI is doing something potentially dangerous to humanity, the world needs to know.

— Elon Musk (@elonmusk) November 21, 2023

If Sutskever leaves, there might be a possible halt to what OpenAI is developing, including Q*. A conspiratorial take can be that Sutskever purposefully leaked Q* to have leverage to not leave OpenAI.

Irrespective of what Sutskever did or did not do, it is clear that he was the most crucial team member of OpenAI since the beginning. Musk, the founder of OpenAI called Sutskever the ‘lynch pin’ of the company. He even fought with Larry Page to poach him from Google and bring to OpenAI.

Everyone wants Ilya

Even if Sutskever has denied Musk about joining xAI at the moment, there might be still a chance for him to come back, given that he is garnering praises and being poached by everyone at the moment, and Musk was the one who hired him first.

The other possibility is that Sutskever might end up joining Google or DeepMind, where he would be able to work on similar projects to OpenAI, and be able to build AGI driven models, while also being aligned with safety and security principles that he believes in.

Interestingly, OpenAI board members, including Sutskever, reportedly approached Anthropic CEO Dario Amodei for a possible merger of the companies during the whole Altman firing fiasco. Though it is not clear if Sutskever was on board with the plan or it was just Toner and McCauley, Sutskever might have possible interests and alignment with Anthropic’s safety driven approach towards AI.

Toner had praised Anthropic and criticised OpenAI for their approach towards generative AI, which Altman had major contentions with. If Toner also ends up at Anthropic, Sutskever might also get poached by the company, where he can join his former OpenAI team members, including the co-founders of Anthropic, who are former OpenAI employees.

As soon as the OpenAI employees started quitting the company following Altman’s departure, almost every tech company in the world was poaching them. A lot of them were ready to join Microsoft, which Altman was also reportedly joining before coming back to OpenAI. At the same time, everyone was taking a shot at Sutskever for firing Altman.

in one tweet: q* spooks board, board fires sam, hires mira, mira hires sam, board replaces her with guy from twitch, satya hires sam, 98% of openai threatens to quit, ilya's like oops j/k nm, new ceo says to the board "pics or gtfo", board has no pics, brings sam back, q* leaks.

— Siqi Chen (@blader) November 24, 2023

All this is if Sutskever leaves OpenAI, which he has not shown any signs of currently. He has even reposted the new announcement which included Altman’s message, which seems like he is fine with whatever has taken place. Being the chief scientist of the company, it is plausible to say that OpenAI would be nothing without Ilya, even Murati would agree.

It is not like he does not have options though.

Maybe it was Altman’s plan all along, as it was highlighted in the announcement – “Greg and I are partners in running this company. We have never quite figured out how to communicate that on the org chart, but we will.”

The post OpenAI is Nothing Without Ilya appeared first on Analytics India Magazine.

Generative AI: Five Major Issues, and How to Fix Them

After playing with GPT for some time, testing GenAI vendor solutions, designing my own, and reading feedback from other users, I uncovered a number of problems. Here I share some of the most common issues, and how to address them. It impacts LLM and synthetic data generation the most, including time series generation.

1. Poor Evaluation Metrics

In computer vision, it is easy to visually assess the quality of an image. Not so much with tabular data. Poor evaluation metrics may result in poor or unrealistic synthetizations. You cannot capture the complex dependencies among features with one- or two-dimensional metrics. In addition, some features may be categorical or text, some numerical, some bin counts. Currently, quality measurements rely on pairwise feature comparisons or blending univariate statistical summaries. A full, true multivariate comparison (real versus synthetic) is difficult to implement. It is now available, with an open-source library for synthetic data: see here.

More to come soon for LLM. There is no more excuse for failure to capture complex multivariate patterns. Poor synthetic data is now easy to detect, and should be a thing of the past.

2. Inability to Sample Outside the Training Set

In one of my recent datasets (health insurance), annual charges per policyholder ranged from $1,000 to $65,000. Of all the vendors that I tested including open-source, none were able to generate synthetic values outside that range. What’s more, the same was true for all the other features, no matter how many observations you synthesize. Worse, it was true for all the datasets. Yet, this is not an issue specific to generative adversarial networks (GANs). In fact, my NoGAN had the same problem. Then, you may ask: what should be the limits, and how can you even synthesize outside the observation range without using bigger training sets?

Now, there is an answer to these questions: see my new article, here. And it is a lot simpler than diffusion models used in computer vision. Using cross-validation, you can even test if the maximum should be $70,000 or $250,000 depending on the number of generated observations. Or incorporate business rules that cap the minimum and maximum, for specific features. All this fine-tuned with a simple hyperparameter. Now, you can generate much richer and realistic data, even naturally occurring outliers! Also great with small datasets.

3. New Datasets Require New Hyperparameters

And by the same token, a lot of retraining and preprocessing, involving both human and computing time. In short, it is expensive. What if there was an algorithm that works a lot faster, with auto-tuning and explainable AI? That is, a robust algorithm that you can rely on without significant (if any) onboarding for each new dataset? In the context of synthetic data, there is one: NoGAN. Actually more than one, but NoGAN now has its own Python library. Since the ideas behind NoGAN originate from NLP, you can expect a version for LLM in the near future. If you want to start light with synthetic data, with a free implementation that outperforms everything else and runs 1000x faster than deep neural networks, and comes with the best evaluation metric, see my recent presentation, here.

I am currently developing a Web API where you can upload your dataset and have it synthesized. Since it’s free, my incentives are aligned with those of the user (cost optimization), not with those of cloud companies charging by the bandwidth. Yet, to facilitate adoption, I also focused on high quality and ease of use. It is under construction, on GenAItechLab.

4. Poor but Expensive Training

These days, the tendency is towards bigger and bigger training sets. For AI engineers, it is the easiest solution to overcome many problems. It is also the most expensive. Yet, I showed instances where randomly erasing 50% of your training data had no impact on performance. For simple algorithms such as linear regression, a 90% reduction resulted in improved predictions, thanks to overfitting reduction: see here. Then, as discussed in section 2, it is possible to sample outside the observation range even with small training sets. And for LLMs, customized solutions usually outperform generic versions. The idea is to only use carefully selected inputs most relevant to the output. Content taxonomy should be part of the equation.

One of the reasons behind inflated training sets is the instability of deep neural networks such as GAN. Using a fixed number of epochs, however large, does not fix the problem. Using different seeds can lead to very different results. The quality of the output depends on how close your initial configuration is, to a decent local minimum of the loss function. This is determined by the seed. However, alternatives such as NoGAN and especially NoGAN2, start with a very good approximation, a better loss function, and much less computing time. In short, reducing convergence issues, and cost.

5. Lack of Replicability

In the previous section, I mentioned the concept of seed. In short, a seed is an integer that initializes all the random number generators used in your algorithm, whether a deep neural network or anything else relying on random numbers. Because most implementations are created by engineers rather than scientists, seeds are not part of the hyperparameters. Thus, running the same algorithm twice leads to different results, making it impossible to replicate a great synthetization. It is a fact a life with current GenAI techniques, not even discussed anywhere. By contrast, all my GenAI algorithms are replicable. Yet, it would be rather easy to make GAN models replicable: I did it (just use the same seed, assuming you have one to begin with). It is something that was overlooked by developers and vendors alike, in the design stage. Note that it is a lot more difficult to achieve replicability in GPU implementations.

Author

Generative AI: Five Major Issues, and How to Fix Them

Vincent Granville is a pioneering GenAI scientist and machine learning expert, co-founder of Data Science Central (acquired by a publicly traded company in 2020), Chief AI Scientist at MLTechniques.com, former VC-funded executive, author and patent owner — one related to LLM. Vincent’s past corporate experience includes Visa, Wells Fargo, eBay, NBC, Microsoft, and CNET.

Vincent is also a former post-doc at Cambridge University, and the National Institute of Statistical Sciences (NISS). He published in Journal of Number Theory, Journal of the Royal Statistical Society (Series B), and IEEE Transactions on Pattern Analysis and Machine Intelligence. He is the author of multiple books, including “Synthetic Data and Generative AI” (Elsevier, 2024). Vincent lives in Washington state, and enjoys doing research on stochastic processes, dynamical systems, experimental math and probabilistic number theory. He recently launched a GenAI certification program, offering state-of-the-art, enterprise grade projects to participants.

Nagarro’s Approach to Generative AI: Tailoring Tools for Enterprise Needs

About 95% of Indian IT leaders believe that generative AI will soon have a prominent role in their organisations, according to a Salesforce report released earlier this year. Indeed, the Indian IT sector has been quick to leverage the technology and is upskilling its workforce for a generative AI-powered future.

At Nagarro, generative AI has opened new opportunities allowing it to improve its offerings and deliver more value to its clients, according to Anurag Sahay, managing director – AI and data science, Nagarro. Founded in 1996, the global technology consulting and digital product engineering company made €856.3 million in 2022, up from €546.0 million in 2021 — a growth of 56.8%.

“We tailor generative AI tools to meet our clients’ specific needs, enhancing their data value and optimising their use of this technology,” Sahay told AIM.

With a workforce of around 19,000 and operations in 36 countries, the company’s expertise and services span areas such as digital product engineering, technology consulting, AI/ML, IoT, API management, and cloud services, among others.

Making a difference with generative AI

“We’ve integrated large language models (LLM) with enterprise knowledge bases. This makes LLMs more useful, their answers more grounded and the experience is more consistent in general. Grounding LLMs with specific information has resulted in a wide variety of contextual use cases getting enabled in the enterprise,” Sahay said.

Nagarro also has enhanced its AI-based chatbots, utilising extensive organisational knowledge for more profound and precise interactions and solutions. The integration of an LLM’s conversational skills with specific knowledge and context contributes to a more natural and enjoyable chatbot experience.

Moreover, Sahay said the company is developing interfaces that understand natural language, making our software easier and more intuitive to use, streamlining tasks and improving efficiency. “We believe that natural language is going to be one of the more powerful interfaces for all enterprise applications.”

Integration with Genome and Forcastra

The company integrates various LLMs such as the GPT models, DALL-E and Whisper, among others into its Genome and Forcastra AI platforms. While Nagarro developed Genome AI to revolutionise customer experience, Forcastra AI focuses on intelligent forecasting and informed decision-making.

The Genome AI platform serves as a transformative tool for brands and enterprises, leveraging LLMs, knowledge graphs, and foundational AI models to facilitate playbook-based automation across the ecosystem of suppliers, partners, and customers.

“The platform’s capabilities extend to generating advanced AI-based recommendations for products, customers, offers, promotions, and AI-based demand planning,” he said.

Moreover, Genome ensures interoperability with a diverse array of foundational AI models and effortlessly integrates with managed AI services specific to hyperscalers. “The foundational LLM models, integrated across hyperscalers like Azure, and Google further amplify its versatility to enhance forecasting accuracy.”

Forcastra AI, on the other hand, employs intelligent segmentation, feature engineering, and a funnel-based approach, encompassing a variety of models such as multi-SKU neural networks, machine learning models, hierarchical, zero-inflated, etc, Sahay said.

“Notably, Forcastra AI goes beyond traditional structured data inputs by harnessing the power of LLMs. This integration allows it to incorporate real-world signals from unstructured datasets, including product descriptions, reviews, social mentions, images, and more.

In addition, Forcastra AI utilises LLMs to craft a natural language conversation interface and conversational nudges enhancing the user experience, making the forecasting process more approachable and user-friendly.

“The combined utilisation of LLMs, and other models in both Genome and Forcastra AI reflects Nagarro’s commitment to delivering sophisticated and comprehensive solutions in the realm of AI and data science,” Sahay added.

Ginger now provides more sophisticated and context-aware responses

Nagarro developed Ginger as the de facto interface for its employees to interact with the organisation and its systems. Today, Ginger AI is an integral part of Nagarro’s Fluidic Enterprise AI suite, which helps redefine workplace dynamics by enhancing responsiveness, efficiency, intimacy, creativity, and sustainability.

For instance, platform users can directly retrieve services and data, expediting information access. Additionally, personalised data nudges guide employees toward data-centric decisions, surpassing conventional decision-making approaches.

Now, generative AI integration has resulted in enhanced conversational abilities of chatbots and other AI interfaces. “As the friendly face of the company for every employee, Ginger serves as an all-encompassing workplace assistant, streamlining information access and daily task management. These AI systems can now provide more sophisticated and context-aware responses, resulting in improved customer interactions and support.”

Challenges with generative AI adoption

The integration of generative AI also comes with its own set of challenges from data availability, hardware cost and compute power. Sahay believes for companies, particularly in the field of AI services, integrating responsible AI becomes a key factor in setting themselves apart.

“At Nagarro, this is particularly crucial as we aim to distinguish ourselves. The primary challenge encountered by Nagarro in generative AI is protecting sensitive enterprise data when using generative AI models. Proper security measures must be in place to safeguard confidential information,” he said.

Compliance with copyright, privacy laws, and licensing agreements related to the use of LLMs is also essential for Nagarro. “We must ensure that it operates within legal boundaries. Moreover, ensuring responsible and ethical AI use is a challenge. This includes addressing issues like hallucinations, misinformation, and bias in AI-generated content.

“Nagarro is actively addressing these challenges by implementing appropriate security measures, complying with legal requirements, and adopting responsible AI practices to mitigate the risks associated with generative AI adoption.

“While we strive for cutting-edge AI technology, incorporating responsible AI design principles is not just a trend but a long-term strategy for building robust and resilient AI solutions. It’s a nuanced approach that goes beyond technical excellence and considers the ethical, interpretative, and human aspects of AI, providing a comprehensive framework for responsible AI development,” he concluded.

The post Nagarro’s Approach to Generative AI: Tailoring Tools for Enterprise Needs appeared first on Analytics India Magazine.

NVIDIA Unveils CUDA Quantum 0.5 to Accelerate Quantum Workflows with GPUs

Is IBM the NVIDIA of Quantum Computing?

NVIDIA announced the launch of CUDA Quantum 0.5, the latest iteration of its CUDA Quantum platform, which is tailored explicitly for developing quantum-classical computing applications. It boasts an open-source programming model that seamlessly integrates quantum processor units (QPUs), GPUs, and CPUs. By accelerating workflows spanning quantum simulation, quantum machine learning, and quantum chemistry, CUDA Quantum optimises these intricate processes through its compiler toolchain, harnessing the immense power of GPUs.

At its core, CUDA Quantum 0.5 brings forth a suite of innovations. A significant addition is the support for adaptive quantum kernels, a development spearheaded by the QIR alliance. This advancement enables the platform to navigate complex quantum error correction and hybrid quantum-classical computations, crucial for intricate control flow and intertwined primitives.

Further augmenting its capabilities, CUDA Quantum 0.5 introduces Fermionic and Givens rotation kernels, catering specifically to quantum chemistry simulations. These kernels streamline operations on fermionic systems, empowering researchers to develop novel quantum algorithms tailored for applications in chemistry, thereby accelerating research in this domain.

In a significant stride towards quantum mechanics integration, the platform now supports exponentials of Pauli matrices. This enhancement proves invaluable for researchers engaged in quantum simulations of physical systems such as molecules, paving the way for the development of quantum algorithms tailored for optimization problems, thereby broadening the practical applications of quantum computing.

The integration of IQM and Oxford Quantum Circuits’ (OQC) QPU backends stands as a monumental achievement for CUDA Quantum 0.5. This integration expands its compatibility across a diverse range of quantum computing technologies, complementing the existing support for platforms from Quantinuum and IonQ. Developers and researchers now gain the flexibility to execute CUDA Quantum code seamlessly across multiple quantum platforms, opening doors to a myriad of possibilities.

A notable addition to this iteration is the advancement in tensor network-based simulators. These simulators prove invaluable for large-scale simulations of quantum circuits involving numerous qubits, surpassing the memory constraints of traditional state vector-based simulators. Moreover, the inclusion of a matrix product state (MPS) simulator, leveraging tensor decomposition techniques, facilitates handling a vast number of qubits and deeper gate depths within a relatively confined memory space, redefining the boundaries of quantum circuit simulations.

For those eager to explore the capabilities of CUDA Quantum 0.5, a comprehensive Getting Started guide lays out the steps for delving into Python and C++ examples. Advanced users can further explore the tutorials gallery to unleash the full potential of quantum-classical applications. To engage with the CUDA Quantum community, the open-source repository serves as a central hub for feedback, issue reporting, and collaborative feature suggestions.

The post NVIDIA Unveils CUDA Quantum 0.5 to Accelerate Quantum Workflows with GPUs appeared first on Analytics India Magazine.

KorrAI’s Mission Against Urban Subsidence

The world is sinking, and some prominent cities of the world are facing the threat of extinction. India has witnessed land subsidence in Joshimath and other Himalayan regions, which have been gradually sinking. The threat of subsidence has also been affecting cities like Jakarta and Venice, amongst others.

A study recently identified 200 urban locations in 34 countries that experienced land subsidence during the past century. Calculations suggest that nearly 2.2 million square kilometres of land, which is nearly 8% of the global land surface, is exposed to a high to very high probability for potential land subsidence, involving 1.2 billion urban inhabitants and threatening nearly US$ 8.2 trillion in GDP.

At this crucial stage, scientists have come together and are applying various methods like artificial recharge and deep soil mixing to control the situation. Moreover, satellites and imagery extracted from them could be effectively used to study the regions’ ground motion and geological build and preemptively anticipate and act on potential land subsidence (PLS).

KorrAI—an Indian-Canadian geospatial tech company, has been directly involved in providing the solution. It employs satellite imagery and machine learning to address the challenges arising from ground motion— identifying and mitigating risks associated with subsidence, landslides, and settlement phenomena.

The Y Combinator-funded company has clientele spanning urban planners for mine sites, airports, and construction projects, property insurers, and private and public infrastructure entities that construct highways, railways, subways, and utility lines.

The services provided by the company become increasingly prevalent as an estimated fifth of the world’s population is said to be affected by land subsidence by 2040, intricately linked to flooding and rising sea levels.

With a mission to harness technology to ensure safer, sustainable urban futures, KorrAI’s CEO Rahul Anand, in an exclusive interview with AIM, defined their goal, vision, and challenges. “In the US alone, land sinking causes an annual economic damage of approximately $4 billion,” Anand stated, drawing a concerning parallel to escalating wildfire damages, which have spiked to nearly $230 billion annually.

Anand co-founded KorrAI in 2020 along with Rob McEwan, a data scientist skilled in remote sensing with satellites. However, there’s an exciting story to KorrAI’s focus shifting towards this aspect from being a project funded by the Canadian Space Agency to constructing cloud-based data processing pipelines for the RadarSat constellation mission, specifically in SAR technology.

Back in 2020, Anand’s computer science and engineering background led him to stumble upon an intriguing computer vision problem. “A mining company manually searching for mineral deposits like gold, sparked my curiosity. I created a model to identify patterns, aiding them in locating these deposits in the wild.”

This chance encounter gave birth to myriad opportunities, steering Anand into uncharted territories. “It was a colossal market waiting to be tapped, especially with the escalating capabilities of satellite imagery this decade.”

UrbanSAR and GlobalSAR

Anand detailed the company’s proprietary data pipelines: “In the past two years, we’ve honed our focus on ground motion monitoring, culminating in the development of two pivotal algorithms: UrbanSAR and GlobalSAR.” He added that UrbanSAR is meticulously crafted for urban environments, while GlobalSAR is adept at handling diverse settings, spanning both urban and non-urban areas.

He also detailed that their preferred use of radar satellites stems from the uninterrupted data flow, which empowers them to analyse and interpret ground movements over time with remarkable precision.

Radar satellites, equipped with Synthetic Aperture Radar (SAR) excel in detecting minuscule changes on Earth’s surface, reaching down to the millimetre level. Unlike optical satellites reliant on light, SAR systems can penetrate through cloud cover, ensuring data collection irrespective of weather conditions or time of day.

Anand highlighted the technological prowess embedded in their infrastructure and detailed an essential aspect of their pipeline. “Our system integrates a crucial calibration mechanism using a global network of GNSS stations,” he added. Explaining that the calibration process serves as a cornerstone, correcting the atmospheric noise and ensuring enhanced data accuracy.

Crucially, Anand underscored the backbone of their cutting-edge framework—an adaptive, scalable compute environment. “Our entire data processing framework operates within a highly adaptable compute infrastructure.” It adjusts seamlessly based on demand, empowering us to offer insights with unparalleled flexibility and competitive rates in the market.

Looking to Launch Their Own Satellites

KorrAI employs multispectral and hyperspectral imagery from various space agencies and companies in their operations. “However, we have encountered limitations with this model, particularly due to the competitive nature of satellite tasking by governments and the cost structures set by private operators, which restrict our ability to develop new use cases,” he said.

“In light of these challenges, our strategy is to find a balance: we aim to leverage the broad array of existing data while also launching our own satellites to enable tailored data collection for our unique requirements, particularly for scenarios demanding extremely high spatial resolution.”

However, to encounter problems arising from hosting a vast amount of data and to streamline the process, the company has inked a partnership with AWS. ”These datasets can be enormous, reaching petabytes, so one of our key strategies for optimising processing time is to organise the raw data into an efficient structure. We’re collaborating with the AWS open data program to host the European Space Agency’s Sentinel 1 data coverage in North America,” said Anand, adding that they plan on expanding this initiative to India as well.

Y Combinator’s Influence

Being a Y Combinator (YC) company has also had a profound impact on their business. The company has clientele from its partnership with YC and is looking to scale its operations globally which is already spread across regions like Hong Kong, the Maldives, the U.S., Canada, and Australia, tailored to regional needs.

“Through Y Combinator, we’ve connected with diverse companies venturing into satellite launches,” he highlighted. “This partnership potential has been instrumental. Collaborating with satellite companies to develop custom payloads allows us to target high-value monitoring in the near future.”

Speaking to the holistic impact of YC beyond immediate business gains, Anand praised its ecosystem-building prowess. “YC’s contribution transcends mere business dealings,” he reflected. “It’s about investments, networking, and accessing a wealth of entrepreneurial expertise. It’s among the finest accelerators, fostering a community where founders can contribute and glean insights from thousands of experienced peers.”

The post KorrAI’s Mission Against Urban Subsidence appeared first on Analytics India Magazine.