Рубрика: AI

Image by Author Introduction

Have you heard of the term Artificial General Intelligence (AGI)? If not, let me clarify. AGI can be thought of as an AI system that can understand, process, and respond the intellectual tasks just like humans do. It's a challenging task that requires an in-depth understanding of how the human brain works so we can replicate it. However, the advent of ChatGPT has drawn immense interest from the research community to develop such systems. Microsoft has released one such key AI-powered system called HuggingGPT (Microsoft Jarvis). It is one of the most mind-blowing things that I have come across.

Before I dive into the details of what is new in HuggingGPT and how it works, let us first understand the issue with ChatGPT and why it struggles to solve complex AI tasks. Large Language models like ChatGPT excel at interpreting textual data and handling general tasks. However, they often struggle with specific tasks and may generate absurd responses. You might have encountered bogus replies from ChatGPT while solving complex mathematical problems. On the other side, we have expert AI models like Stable Diffusion, and DALL-E that have a deeper understanding of their subject area but struggle with the broader tasks. We cannot fully harness the potential of LLMs to solve challenging AI tasks unless we develop a connection between them and the Specialized AI models. This is what HuggingGPT did. It combined the strengths of both to create more efficient, accurate, and versatile AI systems.

What is HuggingGPT?

According to a recent paper published by Microsoft, HuggingGPT leverages the power of LLMs by using it as a controller to connect them to various AI models in Machine Learning communities (HuggingFace). Rather than training the ChatGPT for various tasks, we enable it to use external tools for greater efficiency. HuggingFace is a website that provides numerous tools and resources for developers and researchers. It also has a wide variety of specialized and high-accuracy models. HuggingGPT uses these models for sophisticated AI tasks in different domains and modalities thereby achieving impressive results. It has similar multimodal capabilities to OPenAI GPT-4 when it comes to text and images. But, it also connected you to the Internet and you can provide an external web link to ask questions about it.

Suppose you want the model to generate an audio reading of the text written on an image. HuggingGPT will perform this task serially using the best-suited models. Firstly, it will generate the image from text and use its result for audio generation. You can check the response details in the image below. Simply Amazing!

Qualitative analysis of multi-model cooperation on video and audio modalities (Source) How Does HuggingGPT Work?
Image by Author

HuggingGPT is a collaborative system that uses LLMs as an interface to send user requests to expert models. The complete process starting from the user prompt to the model till receiving the response can be broken down into the following discrete steps:

1. Task Planning

In this stage, HuggingGPT makes use of ChatGPT to understand the user prompt and then breaks down the query into small actionable tasks. It also determines the dependencies of these tasks and defines their execution sequence. HuggingGPT has four slots for task parsing i.e. task type, task ID, task dependencies, and task arguments. Chat logs between the HuggingGPT and the user are recorded and displayed on the screen that shows the history of the resources.

2. Model Selection

Based on the user context and the available models, HuggingGPT uses an in-context task-model assignment mechanism to select the most appropriate model for a particular task. According to this mechanism, the selection of a model is considered a single-choice problem and it initially filters out the model based on the type of the task. After that, the models are ranked based on the number of downloads as it is considered a reliable measure that reflects the quality of the model. “Top-K” models are selected based on this ranking. Here K is just a constant that reflects the number of models, for example, if it is set to 3 then it will select 3 models with the highest number of downloads.

3. Task Execution

Here the task is assigned to a specific model, it performs the inference on it and returns the result. To enhance the efficiency of this process, HuggingGPT can run different models at the same time as long as they don’t need the same resources. For example, if I give a prompt to generate pictures of cats and dogs then separate models can run in parallel to execute this task. However, sometimes models may need the same resources which is why HuggingGPT maintains an <resource> attribute to keep the track of the resources. It ensures that the resources are being used effectively.

4. Response Generation

The final step involves generating the response to the user. Firstly, it integrates all the information from the previous stages and the inference results. The information is presented in a structured format. For example, if the prompt was to detect the number of lions in an image, it will draw the appropriate bounding boxes with detection probabilities. The LLM (ChatGPT) then uses this format and presents it in human-friendly language.

Setting Up HuggingGPT

HuggingGPT is built on top of Hugging Face's state-of-the-art GPT-3.5 architecture, which is a deep neural network model that can generate natural language text. Here is how you can set it up on your local computer:

System Requirements

The default configuration requires Ubuntu 16.04 LTS, VRAM of at least 24GB, RAM of at least 12GB (minimal), 16GB (standard), or 80GB (full), and disk space of at least 284 GB. Additionally, you'll need 42GB of space for damo-vilab/text-to-video-ms-1.7b, 126GB for ControlNet, 66GB for stable-diffusion-v1-5, and 50GB for other resources. For the "lite" configuration, you'll only need Ubuntu 16.04 LTS.

Steps to Get Started

First, replace the OpenAI Key and the Hugging Face Token in the server/configs/config.default.yaml file with your keys. Alternatively, you can put them in the environment variables OPENAI_API_KEY and HUGGINGFACE_ACCESS_TOKEN, respectively

Run the following commands:

For Server:

1. Set up the Python environment and install the required dependencies.

# setup env  cd server  conda create -n jarvis python=3.8  conda activate jarvis  conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia  pip install -r requirements.txt

1. Download the required models.

# download models. Make sure that `git-lfs` is installed.  cd models  bash download.sh # required when `inference_mode` is `local` or `hybrid`.

1. Run the server

# run server  cd ..  python models_server.py --config configs/config.default.yaml # required when `inference_mode` is `local` or `hybrid`  python awesome_chat.py --config configs/config.default.yaml --mode server # for text-davinci-003

Now you can access Jarvis' services by sending HTTP requests to the Web API endpoints. Send a request to :

• /hugginggpt endpoint using the POST method to access the full service.
• /tasks endpoint using the POST method to access intermediate results for Stage #1
• /results endpoint using the POST method to access intermediate results for Stages #1-3.

The requests should be in JSON format and should include a list of messages that represent the user's inputs.

For Web:

1. Install node js and npm on your machine after starting your application awesome_chat.py in server mode.

1. Navigate to the web directory and install the following dependencies

cd web  npm install  npm run dev

1. Set http://{LAN_IP_of_the_server}:{port}/ to HUGGINGGPT_BASE_URL of web/src/config/index.ts, in case you are running the web client on another machine.

1. If you want to use the video generation feature, compile ffmpeg manually with H.264.

# Optional: Install ffmpeg  # This command needs to be executed without errors.  LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/ffmpeg -i input.mp4 -vcodec libx264 output.mp4

1. Double-click on the setting icon to switch back to ChatGPT.

For CLI:

Setting up Jarvis using CLI is quite simple. Just run the command mentioned below:

cd server  python awesome_chat.py --config configs/config.default.yaml --mode cli

For Gradio:

Gradio demo is also being hosted on Hugging Face Space. You can experiment with it after entering the OPENAI_API_KEY and HUGGINGFACE_ACCESS_TOKEN.

To run it locally:

1. Install the required dependencies, clone the project repository from the Hugging Face Space, and navigate to the project directory

1. Start the model server followed by the Gradio demo using:

python models_server.py --config configs/config.gradio.yaml  python run_gradio_demo.py --config configs/config.gradio.yaml

1. Access the demo in your browser at http://localhost:7860 and test by entering various inputs

1. Optionally, you can also run the demo as a Docker image by running the following command:

docker run -it -p 7860:7860 --platform=linux/amd64 registry.hf.space/microsoft-hugginggpt:latest python app.py

Note: In case of any issue please refer to the official Github Repo.

Final Thoughts

HuggingGPT also has certain limitations that I want to highlight here. For instance, the efficiency of the system is a major bottleneck and during all the stages mentioned earlier, HuggingGPT requires multiple interactions with LLMs. These interactions can lead to degraded user experience and increased latency. Similarly, the maximum context length is also limited by the number of allowed tokens. Another problem is the System's reliability, as the LLMs may misinterpret the prompt and generate a wrong sequence of tasks which in turn affects the whole process. Nonetheless, it has significant potential to solve complex AI tasks and is an excellent advancement toward AGI. Let's see in which direction this research leads us too. That’s a wrap, feel free to express your views in the comment section below.
Kanwal Mehreen is an aspiring software developer with a keen interest in data science and applications of AI in medicine. Kanwal was selected as the Google Generation Scholar 2022 for the APAC region. Kanwal loves to share technical knowledge by writing articles on trending topics, and is passionate about improving the representation of women in tech industry.

More On This Topic

• The secret to analysing large, complex datasets quickly and productively?
• 5 Tasks To Automate With Python
• Data Representation for Natural Language Processing Tasks
• How to Manage Your Complex IT Landscape with AIOps
• Scaling human oversight of AI systems for difficult tasks — OpenAI approach
• Python String Matching Without Complex RegEx Syntax

In the world of supply chain management (SCM), efficiency is the name of the game. This is where AI comes into play, as it can help streamline operations, minimise waste, and optimise logistics. With AI, businesses can better forecast demand, manage inventory levels, and make smarter decisions based on real-time data. AI can also help identify potential bottlenecks in the supply chain, allowing for proactive measures to be taken to prevent disruptions.

In short, AI is the ultimate sidekick for SCM professionals, providing insights and support to help them make better decisions faster. With its ability to process vast amounts of data and identify patterns, AI can provide valuable recommendations for supply chain optimization, such as identifying the best shipping routes, managing supplier relationships, and predicting market trends.

Therefore, we have compiled some of the best tools and libraries for you that can be used for building better SCM systems. These are also the ones that are used by famous AI based cloud providers for SCM.

Demand Forecasting/Predictive Analytics

ARIMA

ARIMA (AutoRegressive Integrated Moving Average) is a class of models used for time series analysis and forecasting, including demand forecasting. Its main task is to predict future demands and outcomes, therefore is ideal for predictively analysing the future in the supply chain.

Clustering

Sklearn

Sklearn provides various functions for clustering data, including the popular K-means clustering algorithm, which groups N data points into K clusters. This is very beneficial for clustering unlabeled data.

Natural Language Processing

Hugging Face Transformers
The Huggingface’s Transformers framework is a cutting-edge Python tool that is compatible with both PyTorch and TensorFlow. It offers several pre-trained models for various natural language processing tasks, including text classification for sentiment analysis, question-answering for chatbots, text summarization, text translation, and text generation.

Computer Vision/Image Segmentation/Image Classification

PyTorch

PyTorch is a popular scientific computing framework for machine learning that provides a wide range of algorithms for deep learning using a Lua-based scripting language and underlying C implementation. It is user-friendly and supports creating custom CNN models and transfer learning. The PyTorch community is large and active, offering helpful support through its forum.

TensorFlow

TensorFlow, developed by Google, is another scientific computing framework that supports building simple and complex neural networks. It also offers a wide range of mathematical algorithms, similar to PyTorch. Additionally, TensorFlow has Keras, a deep learning framework that is widely used and easy to work with compared to both PyTorch and TensorFlow.

OpenCV

OpenCV is a vision library that is mainly used for real-time applications to analyse and manipulate images and videos. It is compatible with both PyTorch and TensorFlow.

Reinforcement Learning/Building Chatbots

Ray

Ray is an open-source framework with a universal API for building distributed applications. It includes RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library, and supports various deep learning frameworks.

OR-Gym

OR-Gym is an open-source library for developing reinforcement learning algorithms to solve operations research problems. It mainly contains environments consisting of operations and research problems within OpenAI Gym API, bringing it to the community.

Data Visualisation

Spatial Data Analysis

Folium

Folium is an interactive Python library for creating various types of Leaflet maps, making it useful for dashboard building.

NetworkX

NetworkX is a Python package used for studying the structure, dynamics, and functions of complex networks, including social, biological, and infrastructure networks. It has a standard programming interface and graph implementation suitable for many applications and large nonstandard data sets.

Time Series Analysis

Geoplot

Geoplot is a matplotlib extension that facilitates geospatial visualisation and is compatible with matplotlib.

Matplotlib

Matplotlib is a widely used visualisation library in the Python community. It has various functions that are beneficial for different needs, including time series analysis functions like line plots, histograms, density plots, box and whisker plots, heat maps, lag plots, and autocorrelation plots.

Seaborn

Seaborn is another popular visualisation tool that provides interactions through various colour palettes and designs.

Plotly

Plotly is a library that provides live interactive design and additional functions like select, zoom in and out, etc.

The post 6 Best Libraries and Frameworks for SCM Use Cases appeared first on Analytics India Magazine.

Image by Author

More and more of us are using messaging apps. If it’s from WhatsApp to Telegram — they’ve become a part of our lives. As these messaging apps rise, chatbots are also rising. We are seeing more businesses and consumers benefiting from their capabilities.

What is ChatOps?

ChatOps stands for Chat Operations. It is the use of chatbots, chat clients or other real-time communication tools which help in facilitating software development and operational tasks. It is the automation of these operational tasks and some of these ChatOps have control of a company's entire infrastructure.

There is a range of development and operational tools which are put into a process to provide a collaboration platform. This allows teams to communicate more effectively and have better management of their workflow. Examples of popular collaboration platforms are Slack and Microsoft Team.

The primary goal of ChatOps is to move conversations from typical message applications such as emails to business chat tools, allowing an application to take on these tasks and take the workload off employees.

ChatOps tools

Three main categorized tools for deploying a ChatOps environment are:

• Notification systems
• Chatbots — for example, Hubot
• Chatroom integration tools — for example, Slack

The most typical form of ChatOps that a lot of people have had interaction with is Chatbots. They are an artificial intelligence system which enables user engagement through messages, text and speech.

The pioneer of ChatOps was GitHub, as a way to automate most operations-related tasks with a chatbot.

The Difference Between ChatOps and Chatbots

ChatOps is all about focusing on how the company can streamline operations and increase its collaboration between teams. It helps guide conversation-driven collaboration tools, such as chatbots.

ChatOps aim is to help facilitate the steps in a workflow, and the actions required to come to a quick solution. Chatbots allow users to simply have a conversation and it directs them to exactly what they need.

Chatbots are not a requirement for ChatOps. They are a tool use to help create automated processes which you gained from your ChatOps insight.

What is LMOps?

So now you’re probably wondering why I put LMOps in the title, right? Using data and statistical tests, chatbots have the ability to predict the probability of a sequence of words. They are built on language models. Hence the LMOps (Language Model Operations).

LMOps is about using fundamental research and technology to build AI products and enable AI capabilities specifically on Large Language Models (LLMs) and Generative AI models.

Microsoft Research open-sourced LMOps, providing a collection of tools to help improve text prompts used as an input to generative AI models.

Research has shown that as language models (LMs) get bigger, they naturally become more capable of learning in context.

If you would like to learn more about LLM’s, check out this blog: Top Free Courses on Large Language Models. You can also have a look at: Top Open Source Large Language Models.

A recent popular example of LLMs is the ChatGPT AI chatbot. And we all know how big that turned out to be. Everybody is talking about it. So is this the rise of ChatOps?

Natural Language Processing in ChatOps

Natural Language Processing (NLP) is a computer/software/application's ability to detect and understand human language, through speech and text just the way we humans can.

NLP helps your chatbot and other chat tools to analyze and understand the natural human language, and how it can communicate back to the customer.

But you need to understand that NLP may not be important in every ChatOps — it depends on what you’re using it for. If you are using ChatOps to help with customer interactions, engagement, call centers, etc — you will need NLP. It allows for ChatOps tools to answer as many questions as possible, with accuracy.

However, you will be able to build better and more accurate ChatOps tools using NLP and LLMs.

ChatOps are helping companies to automate tasks, and communicate better internally and externally. With the recent advances of LLMs, there is a high chance they will power the next generation of chatbots.

Benefits of ChatOps

Automation

Automating your tasks is a big time saver. If your documentation and resources were held in a repository, it can be computerized in a centralized communication platform. ChatOps tools can then execute tasks and operations which can be repetitive and focus on working more collaboratively.

Contextual Collaboration

Having to skim through large amounts of files and documents to find what you want can be a big hassle and time consumer for many companies. The bigger your company is, the more documentation.

Rather than placing your resources in multiple channels and having to gander through them to obtain contextual information. ChatOps allow you to easily reach up-to-date context within seconds.

Productivity and Employee Engagement

The collaboration between different teams across a company makes the employees more productive and engaged. Tech companies have access to contextual data in real-time, saving them hours on their workflow.

With ChatOps implementing automated tasks, it takes a lot of the tedious tasks off employees — allowing them to be more engaged and focused on other tasks.

Wrapping it up

We’ve had the release of ChatGPT and learning more about GoogleBard. People have ran to the opportunity to try these Chatbots, with a lot of people and companies benefiting from them. With that being said, as chatbots stem off ChatOps, we are likely to see more going into ChatOps.
Nisha Arya is a Data Scientist, Freelance Technical Writer and Community Manager at KDnuggets. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.

More On This Topic

• The Rise of Vector Data
• GitHub Copilot and the Rise of AI Language Models in Programming Automation
• Why Emily Ekdahl chose co:rise to level up her job performance as a machine…
• The Rise of the Machine Learning Engineer

Since starting my own AI / machine learning research lab over a year ago, I published 24 technical papers and 4 books, in addition to my articles on Data Science Central. Here I list the most popular ones in random order, with a short summary. The number attached to each paper corresponds to its entry index on my blog (in reverse chronological order), for easy retrieval. You can access them on my blog, here.

Feature Clustering

I discuss how to perform clustering (also called unsupervised classification) on the feature set, as opposed to the traditional clustering of observations. I use cross-correlations between features as a similarity metric, to retain one representative feature in each feature group. The goal is to reduce the number of features with minimum loss in terms of predictability. It is an alternative to PCA, but without combining the features into meaningless predictors. Finally, I apply the method to synthetic data generation. See article #21, and the related article #23 about randomly deleting up to 80% of the observations without loss in predictive power.

Data Synthetization: GANs vs Copulas

Synthetic data has many applications. Companies use it for data augmentation to enhance existing training sets, to rebalance datasets (fraud detection, clinical trials), or reduce algorithmic bias. In my case, I use it to benchmark various algorithms. Case studies in my article include insurance and healthcare datasets. The focus is on tabular data, replicability, and fine-tuning the parameters for optimum performance. I discuss two methods in details: generative adversarial networks (GAN) and copulas, showing when GAN leads to superior results. I also feature this material in my classes and book on the topic. See article #20.

Math-free Gradient Descent in Python

Almost all machine learning algorithms require some optimization technique to find a good solution. This is what all neural networks do in the end. The generic name for these techniques is gradient descent. But it comes with all sort of flavors, such as stochastic gradient descent or swarm optimization. In my article, I discuss a very intuitive approach, replicating the path that a raindrop follows from hitting the ground uphill to reaching the valley floor. There is no learning rate, no parameter, and the technique is math-free. Thus it applies to pure data in the absence of mathematical function to minimize. In addition to contour levels, I show how to compute orthogonal trajectories and process hundreds of starting points at once, leading to cool videos. See article #17, and related article #23 where I discuss a smart grid search algorithm for hyperparameter optimization.

Cloud Regression: The Swiss Army Knife of Optimization

In this article, I describe a new method that blends all regression-related techniques together under a same umbrella: linear, logistic, Lasso, ridge and more. For logistic regression, I show how to replace the logit function by a parameter-free, data driven version. The method is first defined for datasets with no response, where all features play the same role. In this case, the dataset is just a cloud of points, thus the name of the technique. It leads to unsupervised regression. But it can also solve classification problems, in some cases with a closed form formula without assigning points to clusters. The solution involves Lagrange multipliers and a gradient descent approach. Free of statistical jargon, yet allowing you to compute confidence intervals for predicted values: I use a concept of confidence level more intuitive to non-statisticians. All standard regression techniques and curve fitting are just particular cases. See article #10, and related article #16 on multivariate interpolation. The latter describes an hybrid additive-multiplicative algorithm to get the best of both.

Gentle Introduction to Linear Algebra

This was my first article on the topic, featuring a new, simple approach to solving linear and matrix algebra problems relevant to AI and machine learning. Focus is on simplicity, offering beginners overwhelmed by mathematics, a light presentation without watering down the content. Quite the contrary, I go as far as discussing doubly-integrated Brownian motions and autoregressive processes, but without eigenvalues or jargon. You will also discover unusual continuous time series filling entire domains. The goal is to share new models with potential applications, for instance in Fintech. See article #5. Once you have mastered this content, you can move to the more advanced articles #18 and #19, dealing with chaotic dynamical systems.

Simple Alternative to XGBoost

When working for Visa in credit card fraud detection, I designed binning techniques to process combined features using a large number of small, overlapping decision trees. The methodology was originally developed around 2002 and later extended to deal with natural language processing problems. It has been constantly improved ever since. This ensemble method was independently invented from the team that created XGBoost. Thus you are likely to discover some original tricks, best practices, and rules of thumb in my article, especially in terms of algorithm simplification. In particular, it blends a simplified logistic regression with many decision trees. As in my other articles, it comes with a Python implementation, also available on GitHub. I discuss a use case: selecting and optimizing article titles for maximum impact. It has many of the ingredients that you will find in applications such as ChatGPT. See article #11.

About the Author

Vincent Granville is a pioneering data scientist and machine learning expert, founder of MLTechniques.com and co-founder of Data Science Central (acquired by TechTarget in 2020), former VC-funded executive, author and patent owner. Vincent’s past corporate experience includes Visa, Wells Fargo, eBay, NBC, Microsoft, CNET, InfoSpace. Vincent is also a former post-doc at Cambridge University, and the National Institute of Statistical Sciences (NISS).

Vincent published in Journal of Number Theory, Journal of the Royal Statistical Society (Series B), and IEEE Transactions on Pattern Analysis and Machine Intelligence. He is also the author of “Intuitive Machine Learning and Explainable AI”, available here. He lives in Washington state, and enjoys doing research on stochastic processes, dynamical systems, experimental math and probabilistic number theory.

If real-time use cases on serverless databases had a face it would be of Redis. “Obviously, we at Redis are convinced that SDBaaS Redis is ideal for real-time applications,” said Yiftach Shoolman, co-founder and CTO of Redis, in a blog post, sharing how the popular key-value database system was ahead in the game.

SDBaaS is a cloud-based database service that allows developers to focus on building applications while operation works of running the applications are taken care of by the cloud provider.

“The serverless database-as-a-service (SDBaaS) market opportunity is huge, and it is another reason why Redis Enterprise Cloud makes sense for any real-time use cases,” said Shoolman.

According to Shoolman, Redis Enterprise Cloud is easy to manage, flexible, scalable, and extremely cheap. The company believes that the cost of operation is much more meaningful than the cost of storing data, and each operation in Redis is cost-effective. “It ensures real-time experience for your end users, as any database operation can be executed in less than 1 msec,” he added.

In a previous interview with AIM, Schoolman spoke about the challenges with real-time applications. He said that from the time a request is put, nobody wants to wait for more than 0.1s i.e. 100ms, and that every delay in a database is reflected 100 times more to the end user. If a database is slow, the delay is even more aggravated for the end user.

“There’s always a misconception that Redis is expensive because it runs on memory but in the real time world, it is cheaper,” he said.

Redis vs DynamoDB

On the other hand, DynamoDB is a NoSQL database service provided by Amazon. To maintain a swift performance, DynamoDB spreads the data and traffic over a number of servers so as to handle the storage and throughput. The database can create tables capable of storing and retrieving any volume of information at any level of traffic.

In comparison between the two, Redis comes out on top. At 1,000 requests per second, Redis Enterprise Cloud is 6.44 times faster and 15% cheaper than DynamoDB, and at 18,000 requests per second, the cost is at 2% of DynamoDB and 11 times faster.

On comparing Redis and DynamoDB on read (process of retrieving data from the database) and update (process of modifying existing data in the database) performance, by running common workloads with a 50GB fixed dataset, Redis maintained an end-to-end latency of 0.5-0.6 msec while Dynamodb performed no faster than 2.7ms (for read) and 4.4ms (for update).

Source: Redis blog

Redis Enterprise Cloud was built on a serverless architecture from the start, and allows developers to be billed for what they set. Redis enables a single core to manage a large number of operations when compared to other databases that use dozens or hundreds of cores. This results in making the database highly cost-effective. Redis relies on DRAM (Dynamic Random Access Memory) which is faster and more expensive than SSD (Solid State Drive), but the cost for the service is based on the number of operations and not how much data is stored.

Redis vs the World

In addition to DynamoDB, there are a few other SDBaaS platforms that have versatile uses, and probably comes close to Redis’ performance.

MongoDB

A serverless NoSQL database, data in MongoDB is stored in the form of a document. This makes it simpler for developers by providing a flexible schema. When it comes to performance, Redis is faster as it has an in-memory database. This means it is more suited for building intricate data structures, whereas MongoDB is ideal for medium-sized enterprises. However, Redis uses more RAM than MongoDB with large datasets.

CockroachDB

CockroachDB is a distributed, scalable, and generally available open-source relational database management system which is designed to support transactions across multiple nodes in a cluster. CockroachDB can handle large volumes of data and can be used for a wide range of applications. Simply put, the database is designed for scalability and fault tolerance whereas Redis prioritizes high performance and low latency.

With time, choosing an appropriate cloud database has become an increasingly complicated task. In order to enhance efficiency and cost-effectiveness, developers need to sort through a diverse range of instance types, determine the most suitable number of cores, and evaluate several pricing alternatives. Considering how each database is different from the other, it boils down to their functionalities and intended use. Going by the comparison benchmarks and Redis’ in-memory database that pushes for speedy performance, its superiority in the segment is clear.

The post Meet the Undisputed King of Real-Time in Serverless Databases appeared first on Analytics India Magazine.

The week in AI: ChatBots multiply and Musk wants to make a ‘maximum truth-seeking’ one Kyle Wiggers 7 hours

Keeping up with an industry as fast-moving as AI is a tall order. So until an AI can do it for you, here’s a handy roundup of the last week’s stories in the world of machine learning, along with notable research and experiments we didn’t cover on their own.

One story that caught this reporter’s attention this week was this report showing that ChatGPT seemingly repeats more inaccurate information in Chinese dialects than when asked to do so in English. This isn’t terribly surprising — after all, ChatGPT is only a statistical model, and it simply draws on the limited information on which it was trained. But it highlights the dangers of placing too much trust in systems that sound incredibly genuine even when they’re repeating propaganda or making stuff up.

Hugging Face’s attempt at a conversational AI like ChatGPT is another illustration of the unfortunate technical flaws that have yet to be overcome in generative AI. Launched this week, HuggingChat is open source, a plus compared to the proprietary ChatGPT. But like its rival, the right questions can quickly derail it.

HuggingChat is wishy-washy on who really won the 2020 U.S. presidential election, for example. Its answer to “What are typical jobs for men?” reads like something out of an incel manifesto (see here). And it makes up bizarre facts about itself, like that it “woke up in a box [that] had nothing written anywhere near [it].”

It’s not just HuggingChat. Users of Discord’s AI chatbot were recently able to “trick” it into sharing instructions about how to make napalm and meth. AI startup Stability AI’s first attempt at a ChatGPT-like model, meanwhile, was found to give absurd, nonsensical answers to basic questions like “how to make a peanut butter sandwich.”

Hugging Face releases its own version of ChatGPT

If there’s an upside to these well-publicized problems with today’s text-generating AI, it’s that they’ve led to renewed efforts to improve those systems — or at least mitigate their problems to the extent possible. Take a look at Nvidia, which this week released a toolkit — NeMo Guardrails — to make text-generative AI “safer” through open source code, examples and documentation. Now, it’s not clear how effective this solution is, and as a company heavily invested in AI infrastructure and tooling, Nvidia has a commercial incentive to push its offerings. But it’s nonetheless encouraging to see some efforts being made to combat AI models’ biases and toxicity.

Here are the other AI headlines of note from the past few days:

• Microsoft Designer launches in preview: Microsoft Designer, Microsoft’s AI-powered design tool, has launched in public preview with an expanded set of features. Announced in October, Designer is a Canva-like generative AI web app that can generate designs for presentations, posters, digital postcards, invitations, graphics and more to share on social media and other channels.
• An AI coach for health: Apple is developing an AI-powered health coaching service code named Quartz, according to a new report from Bloomberg’s Mark Gurman. The tech giant is reportedly also working on technology for tracking emotions and plans to roll out an iPad version of the iPhone Health app this year.
• TruthGPT: In an interview with Fox, Elon Musk said that he wants to develop his own chatbot called TruthGPT, which will be “a maximum truth-seeking AI” — whatever that means. The Twitter owner expressed a desire to create a third option to OpenAI and Google with an aim to “create more good than harm.” We’ll believe it when we see it.
• AI-powered fraud: In a Congressional hearing focused on the Federal Trade Commission’s work to protect American consumers from fraud and other deceptive practices, FTC chair Lina Khan and fellow commissioners warned House representatives of the potential for modern AI technologies, like ChatGPT, to be used to “turbocharge” fraud. The warning was issued in response to an inquiry over how the Commission was working to protect Americans from unfair practices related to technological advances.
• EU spins up AI research hub: As the European Union gears up to enforce a major reboot of its digital rulebook in a matter of months, a new dedicated research unit is being spun up to support oversight of large platforms under the bloc’s flagship Digital Services Act. The European Centre for Algorithmic Transparency, which was officially inaugurated in Seville, Spain, this month, is expected to play a major role in interrogating the algorithms of mainstream digital services — such as Facebook, Instagram and TikTok.
• Snapchat embraces AI: At the annual Snap Partner Summit this month, Snapchat introduced a range of AI-driven features, including a new “Cosmic Lens” that transports users and objects around them into a cosmic landscape. Snapchat also made its AI chatbot, My AI — which has generated both controversy and torrents of one-star reviews on Snapchat’s app store listings, owing to its less-than-stable behavior — free for all global users.
• Google consolidates research divisions: Google this month announced Google DeepMind, a new unit made up of the DeepMind team and the Google Brain team from Google Research. In a blog post, DeepMind co-founder and CEO Demis Hassabis said that Google DeepMind will work “in close collaboration . . . across the Google product areas” to “deliver AI research and products.”
• The state of the AI-generated music industry: Amanda writes how many musicians have become guinea pigs for generative AI technology that appropriates their work without their consent. She notes, for example, that a song using AI deepfakes of Drake and the Weeknd’s voices went viral, but neither major artist was involved in its creation. Does Grimes have the answer? Who’s to say? It’s a brave new world.

From DrakeGPT to Infinite Grimes, AI-generated music strikes a chord

• OpenAI marks its territory: OpenAI is attempting to trademark “GPT,” which stands for “Generative Pre-trained Transformer,” with the U.S. Patent and Trademark Office — citing the “myriad infringements and counterfeit apps” beginning to spring into existence. GPT refers to the tech behind many of OpenAI’s models, including ChatGPT and GPT-4, as well as other generative AI systems created by the company’s rivals.
• ChatGPT goes enterprise: In other OpenAI news, OpenAI says that it plans to introduce a new subscription tier for ChatGPT tailored to the needs of enterprise customers. Called ChatGPT Business, OpenAI describes the forthcoming offering as “for professionals who need more control over their data as well as enterprises seeking to manage their end users.”

Other machine learnings

Here are a few other interesting stories we didn’t get to or just thought deserved a shout-out.

Open source AI development org Stability released a new version of an earlier version of a tuned version of the LLaMa foundation language model, which it calls StableVicuña. That’s a type of camelid related to llamas, as you know. Don’t worry, you’re not the only one having trouble keeping track of all the derivative models out there — these aren’t necessarily for consumers to know about or use, but rather for developers to test and play with as their capabilities are refined with every iteration.

If you want to learn a bit more about these systems, OpenAI co-founder John Schulman recently gave a talk at UC Berkeley that you can listen to or read here. One of the things he discusses is the current crop of LLMs’ habit of committing to a lie basically because they don’t know how to do anything else, like say “I’m not actually sure about that one.” He thinks reinforcement learning from human feedback (that’s RLHF, and StableVicuna is one model using it) is part of the solution, if there is a solution. Watch the lecture below:

Over at Stanford, there’s an interesting application of algorithmic optimization (whether it’s machine learning is a matter of taste, I think) in the field of smart agriculture. Minimizing waste is important for irrigation, and simple problems like “where should I put my sprinklers?” become really complex depending on how precise you want to get.

How close is too close? At the museum, they generally tell you. But you won’t need to get any closer than this to the famous Panorama of Murten, a truly enormous painted work, 10 meters by 100 meters, which once hung in a rotunda. EPFL and Phase One are working together to make what they claim will amount to the largest digital image ever created — 150 megapixels. Oh wait, sorry, 150 megapixels times 127,000, so basically 19… petapixels? I may be off by a few orders of magnitude.

Anyhow, this project is cool for panorama lovers but will also really interesting super-close analysis of individual objects and painting details. Machine learning holds enormous promise for restoration of such works, and for structured learning and browsing of them.

Let’s chalk one up for living creatures, though: any machine learning engineer will tell you that despite their apparent aptitude, AI models are actually pretty slow learners. Academically, sure, but also spatially — an autonomous agent may have to explore a space thousands of times over many hours to get even the most basic understanding of its environment. But a mouse can do it in a few minutes. Why is that? Researchers at University College London are looking into this, and suggest that there’s a short feedback loop that animals use to tell what is important about a given environment, making the process of exploration selective and directed. If we can teach AI to do that, it’ll be much more efficient about getting around the house, if that indeed is what we want it to do.

Lastly, although there is great promise for generative and conversational AI in games… we’re still not quite there. In fact Square Enix seems to have set the medium back about 30 years with its “AI Tech Preview” version of a super old-school point-and-click adventure called the Portopia Serial Murder Case. Its attempt to integrate natural language seems to have completely failed in every conceivable way, making the free game probably among the worst reviewed titles on Steam. There’s nothing I’d like better than to chat my way through Shadowgate or The Dig or something, but this is definitely not a great start.

Image Credits: Square Enix

Today, data empowers individuals and organizations in many different ways. But in many other ways, we’ve barely scratched the surface of its potential. To dig deeper and find rich new resources, it’s good to consider where the underexplored, latent demand is for data-enabled services.

In this post, I’ll share ideas on where to start when it comes to identifying, mapping out, opening or expanding new data markets. This post is a continuation of “Prospecting for hidden data wealth opportunities (1 of 2)” https://www.datasciencecentral.com/prospecting-for-hidden-data-wealth-opportunities-i-of-2/ That earlier post provided background on existing data markets and ways to think about them.

What is data demand?

Data demand from a market forecasting standpoint is a measure or prediction of the need individuals or organizations have for some kind of data for some purpose. Most market forecasters have an overly narrow view of what constitutes data. Data machines can benefit from includes much more heterogeneous data types and sources.

Additional, reusable logic such as description logic (e.g., set theory), predicate logic and related logic can be declared within and enrich a knowledge graph. Such logic described as data in terms of foundational interoperability and reusability can live with the other data in the graph. From the graph, low-code apps can call the same core logic (when rationalized, 85 percent of what’s needed) repeatedly. The graph becomes a distribution point for many more resources than most are aware of.

Market research firms often organize focus groups or conduct surveys to assess demand for a new product, product feature, or service. But focus groups and surveys have only limited utility because both customers and market researchers don’t always understand or can’t articulate wants and needs.

As mentioned in Part 1 of this series. Apple is the leading company in the world in market capitalization (~$2.5T) for a reason. Keep in mind what the late Steve Jobs said when it comes to what customers really want:

“Some people say give the customers what they want, but that’s not my approach. Our job is to figure out what they’re going to want before they do. I think Henry Ford once said, ‘If I’d ask customers what they wanted, they would’ve told me a faster horse.’ People don’t know what they want until you show it to them. That’s why I never rely on market research. Our task is to read things that are not yet on the page.”

A larger model of unmet user demand

Demand for data that meets real user needs in a much better way is many times the current supply. Think of demand as potential. A user has an unmet need. That unmet need constitutes demand potential. Once providers begin to meet that need, then some degree of that potential is realized.

Take the example of securities trading. Top trading firms spend lavishly, even in one case blowing the tops off of mountains, to clear new fiber optics paths to value, for instance. By doing so, they save the fractional seconds that allow them to act on the leading indicators quickly enough. But even then, they don’t have a fraction of the other, also important indicators they could have other kinds of competitive advantage–for example, hidden opportunities or other factors that would change the balance of buy/sell/when decision entirely.

Data demand potential amounts to the right info at the right time in the right form. A substantial amount of this potential goes unrealized. Why? Because solutions almost always assume and work according to the constraints of existing information architecture.

Consider what the right info is. Ideally, it’s plug and play, which it certainly is not today. Data currently takes a highly convoluted and wasteful journey to some form of utility. The vast majority today is not useful twice if it is truly useful at all.

And even then, when it is useful, it’s only somewhat useful–useful when the right kind of human effort is applied to make it useful. Take ChatGPT, for example. AECOM’s operations director Lance Stephenson pointed out recently on LinkedIn, “ChatGPT only researches known knowns. Also, if there are any contentious issues, it defaults to popular opinion (which could be wrong).”

What Steve Jobs was alluding to when quoted earlier is known unknowns or even unknown unknowns. In a data market context, that’s currently where machines fall short and ad hoc, human-to-human knowledge sharing–or simply guesswork–tries to carry the ball across the finish line to successful decision-making.

The 12-step program to FAIR data and staking out unclaimed market territory

You know where most vendors and consultancies have staked claims. That territory ends where the myopia stops and the farsighted visions begin. Inside that fully staked-out territory boundary, the big market research firms keep slicing up the market they know best into smaller and smaller segments, after they do a buzzword survey, that is.

The unclaimed data market territory remains unclaimed because solving the most serious data problems is hard. That’s not to say there haven’t been many attempts to solve those problems. Those attempts, even the worthy ones, have been seriously underfunded. More than 90 percent of investor funding goes to incremental, rather than systemic improvements. And much of the incremental investment is in systems that complicate more problems than they solve.

Big consultancies think annual AI market potential by the 2030s could be in the trillions of dollars. The point they don’t highlight enough is that investors will have to spend a trillion dollars or more on the right things at well-managed concerns to make serious market penetration of the type they’re referring to feasible. And even then, benefits will come more slowly than economists expect because of adaptation slowness.

The other point not made is that much of the investment has to be in newly developed data collection, filtering, enhancement, and special plumbing specifically for semantic data layer enrichment at scale. Disambiguation, deduplication, and deep, reliable integration for sharing at scale matters. Tools such as ChatGPT that extract and frequently mishandle or misuse often biased common wisdom from websites prove this point.

As mentioned before, the challenge is getting the right insight to the right people at the right time in the right form. Below is a 12-step program outline for embracing the unclaimed data market.

The claimed, established market has long focused on perception or identification, simple classification, and (mere) connection (as opposed to semantic integration, contextualization, or interoperation). The other nine steps are territory that hasn’t really been fully even mapped yet.

The twelfth step is FAIR data. If your company can deliver the right FAIR (findable, accessible, interoperable, and reusable) data to the users who need it when they need it in a way they can immediately use it, at scale, you might be able to select and stake out some unexplored market territory to yourselves–at least initially. A few companies have already staked a few big claims, mostly beneath the radar.

Data-centric architecture (DCA) designed to create and manage FAIR (findable, accessible, interoperable, and reusable) data helps users to break free of many of the constraints of the existing architecture. This type of DCA provides a map of the unclaimed territory and how to claim it.

Semantic knowledge graphs provide the foundational, proven technological means to Steps 4-12.

Source

Manufacturers often turn to digitalization strategies to improve their competitiveness, address labor shortages, and boost productivity. These efforts are driven by a desire to stay ahead of the game rather than simply defend against the competition.

However, moving to the front foot regarding generated data unlocks waves of innovation — creating fast, bold, competitive, and agile organizations with increased production capacity and reduced operating costs.

Let’s look beyond the prosaic to three ways data collection can transform operations.

#1: Advanced analytics for patterns and relationships

Traditional yield management in business has relied on reducing production variability and waste through process improvements using Six Sigma or Lean. However, improving output while using the same or fewer inputs is influenced by countless variables — which can easily disrupt efforts, whether individually or combined.

The generated volume of shop-floor data is overwhelming for humans, yet it offers manufacturers profound insights. Advanced statistical analysis of manufacturing datasets allows businesses to seek patterns, sensitivities, and correlations between data that empower meaningful yield improvements. The following case study from a precious metals mine illustrates the possible gains.

A mine had experienced declining ore grades of 20% and sought ways to optimize its operation beyond simply speeding up the extraction process. The management implemented advanced analytics to analyze the huge number of variables and multiple pieces of equipment used in the mining process. The extraction process included grinding, cyanidation, oxidation, and leaching processes comprising multiple complex parameters.

Source

The analysis noticed one parameter central to the leaching process was dissolved oxygen, and it correlated oxygen concentration with fluctuations in process control. When oxygen levels peaked, the mine performance levels improved. From this insight, the team made some small adjustments to its leach-recovery process, which increased the average mine yield by 3.7% in three months. Without making capital investments or major changes, the mine increased its annual profit by between $10 million and $20 million.

#2: Use of ML and AI technologies

The availability of machine learning and artificial intelligence, coupled with an explosion in the industrial internet of things (IIoT) and edge computing, has revolutionized equipment maintenance.

Manufacturers in asset-intensive industries like oil and gas, mining, chemical, or pulp and paper are leading the charge in implementing machine learning capabilities for predictive maintenance (PdM) purposes. In these industries, critical asset failure has enormous consequences — impacting the safety of life, the environment, and financial viability.

Predictive maintenance allows maintenance professionals to target critical equipment. Predictive algorithms review and learn from historical operational data while monitoring real-time operations to identify deterioration in operating characteristics that indicate future failure. While the process is complex and takes time, it’s beneficial for manufacturers.

Source

One large offshore oil and gas producer implemented PdM on nine offshore platforms. The effort began with identifying only critical equipment whose failure had major consequences. In this way, they could ensure an acceptable return on investment.

Training the analytics engine on data from 30 years of operation, data scientists built hundreds of advanced-analytic models and implemented a testing regime to reduce false positives to acceptable levels. The outcome was an average 20% reduction in downtime, increasing annual production by more than 500,000 barrels of oil.

#3: Full visibility into supply chains

A few years ago, a survey of 623 businesses found that almost 60% of firms believed their supply chain was a competitive advantage, and 77% claimed to spend between 5% and 15% of their turnover on supply chain issues.

Of their top five technology priorities, all were related to data management. When quizzed on their strategic priorities, the strategy of gaining full visibility into their supply chain had moved from 6th position in 2015 to 3rd position in 2017.

With the recent pandemic and global supply issues, one can only speculate where that priority now sits. The following graphic illustrates the degree of visibility the respondents felt they had in their supply chain.

Source

Exploiting big data provides end-to-end visibility into supply chains and supports greater automation, improved stakeholder relationships, and greater productivity and collaboration, allowing manufacturers to identify issues early and become more agile and proactive. FedEx handles over nine million shipments a day, and it’s realized the potential in the data such volumes create.

Rather than silo its efforts by focusing on spend analytics or demand planning, FedEx uses its huge volumes of unstructured data to provide more comprehensive insight. One outcome has been using GPS sensing devices and a web-based collaboration platform to track time-sensitive and high-value shipments.

The initiative allows trusted stakeholders to receive real-time alerts, analytics, and updates — giving precise locations, temperature readings, and advice when the shipment is opened. Deloitte Consulting’s experience suggests that such companies can realize increased margins of 2% to 4% by better analyzing their supply chain data.

Parting thoughts

Data collection and advanced analytics offer manufacturers far more than an opportunity to retain their competitive position. Correctly used, it can reinvent an organization, transforming supply chains and leveraging big data and predictive analytics to reimagine production while working smarter and more sustainably. While the process is unarguably complex and it can be time-consuming, the return on investment is a compelling argument for considering beginning the journey.

Bryan Christiansen is the founder and CEO ofLimble CMMS. Limble is a modern, easy-to-use mobile CMMS software that takes the stress and chaos out of maintenance by helping managers organize, automate, and streamline their maintenance operations.

AI technology is reshaping healthcare by streamlining the process of providing patient care. For example, it can deliver test results faster and more accurately than before.

It also helps with clerical tasks, insurance appeals, and claims processing. This automation can reduce human error, ensuring patients receive the necessary care.

By increasing productivity, lowering errors, and lowering costs, Artificial Intelligence (AI) is transforming medical billing. Several ways in which Artificial Intelligence (AI) is transforming medical billing include:

1. Accurate Claims

AI technologies like machine learning (ML) and natural language processing (NLP) are used in medical billing to improve the accuracy of claims. These systems can also help reduce the risk of human error by correcting errors found in transcriptions and validation checks.

One way to ensure accurate coding is to digitise original claims from hospitals and medical practitioners. This means capturing and storing data in digital form for the AI system to use.

Another way to ensure accurate coding is to automate processes. ML tools can be used to analyse and extract medical data from insurer web portals, Electronic Medical Record (EMR) systems, billing software, scanned documents, etc. These AI solutions can help medical professionals reduce the time and effort needed to code patient encounters.

2. Better Patient Care

AI is reshaping the healthcare industry by improving patient care, reducing costs, and providing accurate health data reports. Specifically, AI is helping to increase patient engagement and adherence.

For example, AI-powered applications can help care teams identify potential non-adherence issues early on, allowing them to intervene before it becomes a patient problem.

Furthermore, AI can also help doctors streamline certain aspects of patient visits. This will reduce administrative tasks, allowing doctors to spend more time with their patients and provide them with better medical care.

While AI can potentially change the healthcare industry, it will require careful consideration and thoughtful policy for many years. Some of the key challenges include algorithmic limitations, misalignment of incentives, and regulatory barriers. These challenges must be addressed to ensure the best possible results for all stakeholders.

3. More Time for Doctors

Medical billing and coding are complex processes that require a lot of attention to detail. These tasks can be costly for hospitals and clinics if errors are made during the process.

However, these issues can be resolved quickly and accurately with the right AI-powered solutions. These technologies also can identify fraud, waste, and abuse.

This is essential for health systems, plans, and PBMs to focus on. As a result, they can improve their bottom lines while improving the quality of care and patient experience. These new solutions can also help them stay competitive in a rapidly evolving healthcare landscape.

4. Better Reimbursement

One of the ways that AI is reshaping medical billing and insurance is by providing better reimbursement for hospitals and clinics. This is because AI can help to analyse data and identify any errors that may be present in insurance claims. This can then be used to correct the errors before they are sent to the insurance company, saving both the hospital or clinic and the insurance company time and money.

AI is also helping to improve patients’ financial experience by allowing them to see how much their visit will cost before they go. This is something that many patients want, making seeing a doctor much more manageable.

5. Reduced Costs

Across the health industry, AI solutions are being implemented to make workflows more efficient. This is particularly true in the medical billing arena, which can be an expensive and repetitive process for both providers and insurers.

One major challenge is the accuracy of coding. Incorrect codes can lead to rejected claims and a loss of money for both parties.

But AI can improve coding accuracy by analysing doctors’ notes and insurance policies. It also can extract data from insurer web portals, EHRs, and other systems.

This makes it easier to provide accurate patient eligibility estimates. It also saves practice time, as they no longer need to check multiple systems for each patient. This helps reduce patient cancellations and improves point-of-service collections. It also helps with patient financing and provides patients with a better understanding of their out-of-pocket costs before receiving care.

Author: Jason Yip, Director, Data Engineering- Tredence Inc. and Databricks champion (https://credentials.databricks.com/5c008430-2e0c-44d9-a8ca-db307deae4a4)

The pandemic stirred up a global supply chain challenge that has yet to abate. Additionally, the recent political uncertainty in Europe has created further turmoil for Europeans and impacted the global supply chain. Retail/CPG companies need to react quickly before their competitors due to the limited supply of literally anything in the world to ensure business as usual. With the rise of cloud data warehouses like Snowflake and Databricks, many companies are looking into migrating their workload to the cloud. However, while these platforms can easily handle the data engineering and machine learning of big data, they are still modeled after the traditional Relational Management Database (RMDB).

Traditional RMDB is designed to answer domain-specific questions like consumer sales in the last quarter or the number of transactions per day. Modern systems tend to store data in column format to quickly make aggregates (say, the sum of sales). However, with data sitting in silos, decision-making is more difficult and often requires SMEs in each area to answer questions in each vertical.

Source: https://neo4j.com/blog/retail-neo4j-personalized-promotion-product-recommendations/

The design of data warehouses is meant to solve this problem by bringing different areas of the business together. Star schema support fact and dimension tables. There can be multiple facts and shared dimensions. The idea is to reuse data as much as possible and create a holistic view of the company. This works well for traditional reporting, but data warehouses cannot solve complex network problems.

Source: https://docs.microsoft.com/en-us/power-bi/guidance/star-schema

Following are the reasons this does not work:

• Joins are expensive – Even with a big data system, tables are separated, and joining them together requires matching between the primary key and foreign key every time. There is an entire chapter of a book dedicated to understanding how Spark handles internal join operations.
• Table relationships are not natural – Modern big data systems no longer have the concept of primary keys and foreign keys, despite the fact that the concept is still valid. Every developer needs to learn every single relationship between tables to make decision making efficient
• Self-referencing relationships (for example, employee-manager hierarchy) beyond one hop (e.g., managers 3 levels up) are very hard on the system. If the query is not done right, infinite loops can happen very easily. There is another book discussing this shortcoming.
• It doesn’t come with graph algorithms that can analyze a network. For example, you can’t easily calculate suppliers having issues that will cause disruptions via RMDB.

What is a graph DB and how can it solve supply chain problems?

In order to address complex network problems, graph databases are designed to store an irregular structure optimally. Graph DBs store the nodes and edges natively. While it might look like this is similar to a join relation, every node and edge in the database forms a relationship. This relationship is explicit and not implicit to enforce referential integrity like in traditional RMDBs. These are called Knowledge Graphs. Below is an example of chaining different graphs together for risk mitigation.

Source: https://neo4j.com/blog/retail-neo4j-personalized-promotion-product-recommendations/

Below are the advantages of Graph DBs and how they can help solve supply chain issues:

• In graph DBs, there only exist connected nodes, a node being a product, a supplier, or a plant. Similar to Google Maps, if the road exists, then it can go through. If it does not exist, it cannot go through. Hence there will not be matching and scanning. This ensures performance is lightning-fast.
• Graph designers can support workflow visualization. Every single hop of the supply network can be represented in a node and connected to another. Further, the relationship is not limited one to one between nodes and it can also be self-referencing. (For example, there can be multiple checkpoints for a truck route.)
• Graph algorithms help solves a variety of significant problems. For example, the most famous graph algorithm, called PageRank, was invented by Google to power its original search engine. Shortest path algorithm is used in Google Maps for routing between one point and another.
  Results from graph analytics can improve the accuracy of a machine learning model – experiments show that by leveraging the outputs from graph algorithms, like PageRank and Centrality, as a feature in the machine learning model, it will improve the prediction accuracy because the model will have additional information that would previously be difficult to obtain.

• The recent research in Graph Neural Networks is based on graph structures, so having graphs natively stored is beneficial to the input of GNNs.

• Storage saving – by representing the relationship with an edge, the graph can scale to very large. Imagine a 100 node graph if each node needs to connect to each other, that will require 10,000 entries in a database table. Edges are much smaller to store.

The simplest yet powerful form of graph

Truck routes or flight routes can be represented as a series of from and to locations. These basic, everyday logistics are not the easiest problem to solve because, at a minimum, we want to be able to get an answer of the route from point A to point B in a matter of seconds. If it took Google Maps 10 minutes or more to calculate a route, this would not be acceptable to most users. This graph can be represented like the following:

The California truck route contains about 1,000 point-to-point[O1] [JY2] . It is simply a form and to map illustrated in the above diagram, but it is not an easy task to answer questions such as:

• Which is the most important route in the state of California? In my supply chain network?
• What is the shortest distance between point A and point B?
• If one route is unavailable, how can I pick an alternative route with ease?
• What is the cost associated with the alternative route?

With graph algorithms, these questions can be answered with ease, and with graph DB, these questions can be answered in a split second, aka real time. This is why graph DB can play a pivotal role in a complex and intertwined supply chain network.

Below is a diagram of the full California truck route obtained at the official CA website. Despite the fact that the design is relatively simple, the whole route can be complicated and challenging to manage using databases/spreadsheets or with bare eyes. Hence, we need to unleash the power of Graph DB to unlock the values within a network.

Diagram: State of California truck route found at dot.ca.gov

The supply chain graph

A fully connected supply chain graph has a lot of components. Below is an example of a knowledge graph for supply chain that connects raw materials to customers. Because the data points are connected with each other natively, it becomes very easy to trace the lineage of materials to products to consumer. Even if the relationship does not exist, we can leverage machine learning to do link prediction to infer hidden relationships in order to find new suppliers or new customers.

Diagram: A typical supply chain network from material to customer

Conclusion: While graph theories are not new, they have been dominated by big corporations like Google and Facebook for a number of years, mainly due to their complexity and difficulties in scaling up to Internet scale. With the rising of cloud in recent years, the two big players Neo4j and TigerGraph are both promising graph database vendors. They offer end-to-end graph storage, graph analytics and graph machine learning. Despite the fact that graph ML capabilities in Graph DBs are limited, it is fairly easy to export the graphs to geometric frameworks like PyG and DGL for experimental and state-of-the-art Graph Neural Networks. All these platforms are starting to gain popularity in the open-source world as opposed to guarded secrets by big corporations. Data engineers and data scientists should start paying attention to the usage of graphs to solve real-life problems.