Рубрика: AI

It’s often said in jest that most software engineers know nothing about hardware. But could there be any truth to this harmless jibe? Well, it turns out that a lot of people agree with it. In an online discussion, a surprising number of developers, programmers, and software engineers chimed in not just to agree with it but also ascribe reasons to why that may be the case.

A software engineer said, “It’s easier to count on someone than to count on yourself.” This might encapsulate a fundamental reason why many software engineers don’t delve into the hardware side of things. It’s simpler to rely on hardware experts rather than spreading oneself thin by trying to master both the fields.

However, not everyone fits this stereotype.

Another user proudly declared, “Imma Software engineer, Knows about hardware, Including networking (sic).” Clearly, there are software engineers who bridge the gap to understand the intricacies of hardware. Yet, they might be the exception rather than the rule.

Is There a Need?

“Highly accurate, when someone speaks about hardware or how software interacts with hardware that’s where I get silent,” said a software engineer. “I only know about hardware things or gadgets I use.” This indicates that many software engineers have a practical, user-oriented knowledge of hardware rather than a deep, technical understanding.

While most software engineers would want to believe that there is not a need for them to know the intricacies of hardware, as long as what they are using offers support for the software they want to use and build. But on the contrary, a user offered a thought-provoking take, suggesting that understanding hardware could bolster several fields, such as cybersecurity.

“I think it would help in programming to know how the chip and memory think only to secure the program from hackers,” he said. This highlights a practical benefit of hardware knowledge that goes beyond mere academic interest.

Moreover, software engineers who know a thing or two about hardware can create better softwares and build good software capability on the hardware. This perspective suggests that a deeper understanding of hardware can lead to more efficient and innovative software solutions.

The roles of software engineers are also changing with the advent of AI tools. For over a decade, a popular belief has been that a computer science degree is all you need to tread the path to wealth, especially in a country like India. “Winter is coming for software engineering,” said Deedy Das from Menlo Ventures, sharing a graph about how by 2040, software engineering roles will almost become a distant memory.

Besides, educational background also factors in this debate. “Unless you were a computer engineering major…,” quipped a user, implying that those with a more hardware-focused education are better equipped in this area. This distinction highlights the difference in curricula between software engineering and computer engineering programs.

What’s the Future?

Another one gave a very basic analogy to highlight the importance of understanding hardware for people in developer jobs. “Have you ever dealt with keyboards with wrong wiring?” Practical issues like these can starkly expose a software engineer’s hardware limitations, turning theoretical gaps into tangible challenges.

However, not everyone sees this disconnect as a negative. This is both good and bad. And it’s supposed to be that way. “Specialisation allows professionals to excel in their chosen fields, fostering expertise rather than spreading knowledge too thinly,” said a user.

“Most hardware engineers know nothing about software,” another turned the tables. This two-way street suggests that both domains have their own complexities and require dedicated focus. Keeping up with hardware innovations is a formidable challenge in itself, demanding constant learning and adaptation.

Ultimately, the consensus suggests that while many software engineers may lack a deep understanding of hardware, this isn’t necessarily a problem. The specialisation in their respective fields allows for more focused expertise, though cross-knowledge can certainly offer significant benefits.

While it’s often true that most software engineers know little about hardware, this division of knowledge is not inherently detrimental. As technology evolves, perhaps we will see more integrated roles bridging this gap.

For now, we can appreciate the humorous takes and real-life insights that illustrate this dynamic. Every time a program goes “brrrr”, a software engineer is happy, perhaps unaware of the hardware magic at work.

The post Most Software Engineers Know Nothing About Hardware appeared first on AIM.

The media (and movies) have made AI conversations confusing and too emotional. I propose a simple exercise for middle and high school students (and others who might be interested) to provide hands-on training to understand how an AI model works. That exercise will focus on defining, designing, and implementing the AI Utility Function that drives the AI model’s recommendation and decision-making processes.

The exercise will be centered around something we do nearly every day: deciding where to dine. We will follow this four-step process in defining, creating, and then executing our “Dining Recommendation” AI Utility Function:

• Step 1: Define AI Utility Function Variables and Metrics. In a collaborative setting, students will brainstorm and identify the variables that “might” impact their dining decisions. Examples could include affordability, food quality, and ambiance. This step establishes what factors are considered in the AI model and introduces students to AI model variable selection based on intent, relevance, and impact. We will narrow down that list to the top 10 variables for the rest of the exercise.
• Step 2: Create Entity Models (Analytic Profiles). Utilizing the prioritized variables, we will employ ChatGPT to construct entity models or analytic profiles for each selected restaurant. We will rate each variable on a scale from 1 to 100 for each restaurant. A high rating (80 – 100) indicates that the restaurant strongly represents the variable, while a low ranking (1 – 20) suggests that the restaurant does not represent that variable well. For example, if ‘ambiance’ is a variable, a fine dining restaurant might score 90 on this scale, signifying a solid correlation with the variable ‘ambiance.’
• Step 3: Personalize Variable Weights. Students will assign weights to each variable based on their dining intent and preferences. For example, if a student is looking for a quick and affordable meal, they might assign higher weights to ‘convenience’ and ‘affordability.’ This refinement teaches them the importance of weighing the different variables that might impact the decision-making processes and tailoring the model to individual intent and desired outcomes.
• Step 4: Calculate AI Model Restaurant Recommendations. Finally, using the weighted variables, the AI Utility Function calculates and ranks each restaurant by generating a score that aligns with the student’s preferences. This step synthesizes the variables and their associated weights into a practical outcome, demonstrating how AI models can provide personalized recommendations based on user-defined criteria.

Hopefully, this exercise offers an excellent way to introduce middle and high school students (as well as curious seniors) to concepts of decision-making processes using AI methodologies and the importance of defining a healthy, representative AI Utility Function.

Refresher: What is an AI Utility Function

The AI Utility Function is a mathematical model that quantifies an AI system’s preferences, guiding it to make decisions that maximize the desired outcomes based on specified criteria and constraints.

The AI utility function is crucial in guiding AI models to generate more relevant, meaningful, responsible, and ethical outcomes. It defines what success looks like by quantifying preferences and trade-offs among different potential outcomes. For instance, an AI model might use an AI utility function that considers treatment effectiveness, side effects, and patient quality of life in a healthcare setting. This allows the AI model to recommend treatments that balance these factors in a way that aligns with the defined healthcare goals, ensuring the relevance and meaningfulness of the outcomes (Figure 1).

Figure 1: The AI Utility Function

Furthermore, by integrating ethical guidelines and responsibilities into the AI Utility Function, AI models can be directed to uphold human values and societal norms. For example, an AI model assigned to credit scoring can be programmed with an AI Utility Function that “penalizes” decisions, leading to unfair biases against certain groups. Moreover, the AI Utility Function can be adjusted in real time to reflect evolving human insights and regulatory requirements, ensuring that AI systems remain relevant and aligned with current ethical and societal expectations. As a result, AI utility functions not only provide a technical framework for decision-making but also serve as a means to ensure that AI technologies contribute positively and ethically to society.

Let’s go through an exercise to help every high school student (and hopefully everyone else) understand our role in defining and crafting an AI Utility Function to ensure more relevant, meaningful, responsible, and ethical outcomes.

Step 1: Define AI Utility Function Variables and Metrics

First, we will ask students to identify and brainstorm the different variables that might impact their decision on where to eat. These variables could include cost, distance, food type, and more. Encourage the students to think critically about why each factor is essential and how it might impact the recommendation process of the AI model. This will introduce them to the concept of variables in programming and data science and teach them to recognize how various factors can affect outcomes in everyday decisions.

In step 1, the students will work in small teams to brainstorm the variables and factors that could impact their choice of where to eat. This exercise aims to gather as many factors as possible without getting stuck in over-analysis. Below is a partial list of the factors the students might have identified (Figure 2).

Figure 2: Variables That Might Impact Deciding Where to Go Eat

I have selected ten variables to simplify the rest of the exercise. Although we could have used more variables to allow the AI model to provide even more accurate recommendations, I decided to stick with 10 to make it more manageable for our human minds, which can easily get overwhelmed.

The ten variables we will be using for the rest of the exercise are:

• Affordability
• Convenience
• Menu
• Ambiance
• Food quality
• Organic food
• Social Media reviews
• Value for Money
• Parking / Accessibility
• Hygiene / Cleanliness

Step 2: Create Entity Models (Analytic Profiles)

Analytic profiles (or entity models) are structured data frameworks that define and quantify the characteristics and behaviors of specific entities, such as customers or products, to support targeted analyses and predictive modeling.

Step 2 involves creating entity models, or analytic profiles, for a select number of restaurants. We can use ChatGPT to generate the variables’ weights that comprise the entity models and then tweak the variables’ weights based on our experience and further research.

Using the list of 10 variables, ChatGPT can generate an entity model for each selected restaurant, scoring the variables that comprise the entity models on a scale from 1 to 100 on how well they meet each variable. For example, a budget-friendly restaurant like McDonald’s might score high on affordability (80 – 90) but lower on organic food options (1 – 10). This step illustrates how data can be quantified and standardized to enable comparisons, reflecting how the AI Utility Function evaluates options based on predefined criteria.

We will create analytic profiles for each of these restaurants:

• McDonald’s is a global fast-food chain known for its affordable burgers, fries, and quick service.
• Chipotle is a fast-casual restaurant specializing in Mexican-inspired burritos and bowls. Its focus is on fresh, organic, and locally sourced ingredients.
• Applebee’s is a popular chain that offers a casual dining experience. The menu includes burgers, chicken, pasta, and bar-style appetizers.
• Flemings Steak House is a high-end steakhouse known for its prime meats, fine wines, and elegant dining atmosphere.
• Legal Seafood is a reputable seafood chain renowned for its commitment to freshness and quality. It serves various fish and shellfish dishes.
• LeTour is a sophisticated dining establishment that offers gourmet cuisine emphasizing artistic presentation and exceptional service.

Figure 3 contains the Analytic Profiles that ChatGPT built for our six selected restaurants, with my tweaks based on my personal experiences.

Figure 3: Restaurant Analytic Profiles or Entity Models

AI Utility Function Exercise Summary – Part 1

The “Becoming an AI Utility Function” exercise provides a practical and enlightening way for students to explore the inner workings of AI decision-making processes using a familiar daily activity—choosing where to dine. This exercise consists of four critical steps designed to improve understanding of how AI models incorporate different data points to make informed, personalized recommendations.

In Part 1, we covered the first two steps:

• In Step 1, students were introduced to the foundational concept of defining relevant variables and metrics, such as affordability, food quality, and ambiance. This stage is essential for setting up the criteria the AI will use later to evaluate options. Students learn to think critically about the factors that most influence their decisions, mirroring how AI systems must be programmed to consider various elements in their algorithms.
• In Step 2, we create entity models for each restaurant. Each establishment is scored on a scale of 1 to 100 based on defined variables in this step. This quantification process demonstrates to students how raw data and subjective assessments are standardized for fair comparisons across different options. It clearly illustrates how AI systems process and evaluate diverse inputs.

In Part 2, we will complete the exercise with steps 3 and 4. Hopefully, this hands-on exercise demystifies AI for students and highlights the importance of defining AI Utility Functions to ensure ethical and responsible AI outcomes. By understanding these processes, students can appreciate how AI models are designed to reflect human values and societal norms.

LLMs hallucinate—generate incorrect, misleading, or nonsensical information. Some, like OpenAI CEO Sam Altman, consider AI hallucinations creativity, and others believe hallucinations might be helpful in making new scientific discoveries. However, they aren’t a feature but a bug in most cases where providing a correct response is important.

So, what’s the way to reduce LLM hallucinations? Long-context? RAG? Fine-tuning?

Well, long-context LLMs are not foolproof, vector search RAG is bad, and fine-tuning comes with its own challenges and limitations.

Here are some advanced techniques that you can use to reduce LLM hallucinations.

Using Advanced Prompts

There’s a lot of debate on whether using better or more advanced prompts can solve LLM hallucinations.

Source: X

While some believe that writing more detailed prompts doesn’t help the case, others like Google Brain co-founder Andrew Ng see a potential there.

Ng believes that the reasoning capability of GPT-4 and other advanced models makes them quite good at interpreting complex prompts with detailed instructions.

“With many-shot learning, developers can give dozens, even hundreds of examples in the prompt, and this works better than few-shot learning,” he wrote.

Source: X

Many new developments are also being done to make prompts better like Anthropic releasing a new ‘Prompt Generator’ tool that can turn simple descriptions into advanced prompts optimised for LLMs.

You can now generate production-ready prompts in the Anthropic Console.
Describe what you want to achieve, and Claude will use prompt engineering techniques like chain-of-thought reasoning to create more effective, precise and reliable prompts. pic.twitter.com/TqylVRkfP5

— Anthropic (@AnthropicAI) May 10, 2024

Recently, Marc Andreessen also said that with the right prompting, we can unlock the latent super genius in AI models. “Prompting crafts in many different domains such that you’re kind of unlocking the latent super genius,” he added.

CoVe by Meta AI

Chain-of-Verification (CoVe) by Meta AI is another technique. This method reduces hallucination in LLMs by breaking down fact-checking into manageable steps, enhancing response accuracy, and aligning with human-driven fact-checking processes.

CoVe involves generating an initial response, planning verification questions, answering these questions independently, and producing a final verified response. This method significantly improves the accuracy of the model by systematically verifying and correcting its own outputs.

It enhances performance across various tasks, such as list-based questions, closed-book QA, and long-form text generation, by reducing hallucinations and increasing factual correctness.

Chain-of-Verification (CoVe) reduces hallucinations in LLMs.
It does this by using verification questions to fact-check the answers of an LLM.
In experiments, CoVe decreases hallucinations across a variety of tasks. pic.twitter.com/ri2DRNFoL2

— TuringPost (@TheTuringPost) September 29, 2023

Knowledge Graphs

RAG is not limited to vector database matching anymore, there are many advanced RAG techniques being introduced that improve retrieval significantly.

For example, the integration of Knowledge Graphs (KGs) into RAG. By leveraging the structured and interlinked data from KGs, the reasoning capabilities of current RAG systems can be greatly enhanced.

Source: X

RAPTOR

Another technique is Raptor, a method to address questions that span multiple documents by creating a higher level of abstraction. It is particularly useful in answering queries that involve concepts from multiple documents.

Methods like Raptor go really well with long context LLMs because you can just embed full documents without any chunking.

This method reduces hallucinations by integrating external retrieval mechanisms with a transformer model. When a query is received, Raptor first retrieves relevant and verified information from external knowledge bases.

This retrieved data is then embedded into the model’s context alongside the original query. By grounding the model’s responses in factual and pertinent information, Raptor ensures that the generated content is accurate and contextually appropriate.

Mitigating LLM Hallucinations via Conformal Abstention

The paper ‘Mitigating LLM Hallucinations via Conformal Abstention’ introduces a method to reduce hallucinations in LLMs by employing conformal prediction techniques to determine when the model should abstain from providing a response.

By using self-consistency to evaluate response similarity and leveraging conformal prediction for rigorous guarantees, the method ensures that the model only responds when confident in its accuracy.

This approach effectively bounds the hallucination rate while maintaining a balanced abstention rate, particularly benefiting tasks requiring long-form answers. It significantly improves the reliability of model outputs by avoiding incorrect or nonsensical responses.

Reducing Hallucination in Structured Outputs via RAG

Recently, ServiceNow reduced hallucinations in structured outputs through RAG, enhancing LLM performance and enabling out-of-domain generalisation while minimising resource usage.

The technique involves a RAG system, which retrieves relevant JSON objects from external knowledge bases before generating text. This ensures the generation process is grounded in accurate and relevant data.

By incorporating this pre-retrieval step, the model is less likely to produce incorrect or fabricated information, thereby reducing hallucinations. Additionally, this approach allows the use of smaller models without compromising performance, making it efficient and effective.

Researchers at ServiceNow Propose a Machine Learning Approach to Deploy a Retrieval Augmented LLM to Reduce Hallucination and Allow Generalization in a Structured Output Task
Quick read: https://t.co/7TRlPqSqFS
Paper: https://t.co/J2kX5XTDFQ@ServiceNow…

— Marktechpost AI Research News (@Marktechpost) April 26, 2024

All these methods and more can help prevent hallucinations and create more robust LLM systems.

The post 6 Techniques to Reduce Hallucinations in LLMs appeared first on AIM.

Perplexity AI co-founder Aravind Srinivas is super relaxed and busy with media engagements as the company onboards new gentlemen to the board. Meanwhile, OpenAI experiments with a generative AI search experience, and Google announces several upgrades to Google Search.

“Startups have to be aggressive in terms of competing against incumbents who already have, like, a billion users (Google Search). OpenAI has 100 million users. We don’t have that today, so it’s on us to achieve that,” said Srinivas, in a recent interview with Bloomberg.

Perplexity AI recently onboarded three industry leaders as new advisors to boost search, mobile and distribution: Emil Michael (ex-Uber), Rich Miner (Android co-founder), and Mikhail Parakhin (ex-Bing CEO).

“Parakhin is the world’s leading technical expert on search infrastructure and integrating it with LLMs,” said Srinivas, adding that he has been at the forefront of AI and search, and his expertise will enable them to evolve their core answer engine, build in-house search infrastructure, anticipate new and unexpected challenges, and integrate new AI capabilities.

Perplexity has not given up yet: Over the last few weeks, Srinivas and his team have constantly been iterating and shipping products and features faster in an attempt to stay ahead of its competitors, alongside new players emerging in the space who are building similar solutions.

Recently, the company partnered with SoundHound AI to Bring LLMs-powered voice assistants to cars and IoT devices. This was announced ahead of the OpenAI Spring Update, which gave the world a glimpse of their—or rather ‘Her’ like—characteristics.

At Google I/O, the tech giant also introduced ‘AI Overviews,’ which generates summaries for the queries provided by the user on the go— again similar to Perplexity, but on steroids.

But, Srinivas, was quick to point out the drawbacks and limitations of its competitors and said— “When people go to Google, they hate the bad latency, they expect instant rendering of the 10 blue links,” explained Srinivas.

He said that Google’s UI is quite cluttered, making people wonder whether they will get token streams as answers, ad-serving links, or some random panel.

A Wake-Up Call for Perplexity AI

Recently, OpenAI partnered with Reddit to bring enhanced content to ChatGPT and new products, helping users discover and engage with Reddit communities.

Interestingly, the company mentioned in the blog post that it will also become a Reddit advertising partner.

Surprisingly, OpenAI isn’t planning to charge much from its users and is likely on the path to building its own search engine and monetising it like Google Search.

In the latest quarter, Google earned $46.156 billion from Search and related revenues, a 14% increase.

To do this, OpenAI will use Reddit’s Data API, which provides real-time, structured, and unique content from Reddit. This will enable OpenAI’s AI tools to better understand and showcase Reddit content, especially on recent topics.

Coincidently, this year Google also partnered with Reddit to use its content available for training Gemini.

Not to forget, OpenAI has also entered into content licensing partnerships with publishers including the Associated Press, Axel Springer (Business Insider, Politico, Bild, and Welt), The Financial Times, Le Monde, Prisa Media (El País, Cinco Días, As, and El Huffpost), and Dotdash Meredith (People, Better Homes & Gardens, Investopedia, Food & Wine, and InStyle).

In a recent podcast with Lex Fridman, OpenAI chief, Sam Altman said that, “The intersection of LLMs plus search, I don’t think anyone has cracked the code on yet. I would love to go do that. I think that would be cool.”

The company also hired Shivakumar Venkataraman,a 21-year Google veteran who previously led the company’s search ads business, as its new vice president. Venkataraman’s extensive experience in search advertising at Google makes him a valuable asset for OpenAI as it plans to build ‘Google Search Alternative’.

“We are a business and will find plenty of things to charge for, and that will help us provide free, outstanding AI service to (hopefully) billions of people,” said OpenAI chief Sam Altman in his recent blog post.

OpenAI hit the $2 billion revenue milestone in December and the company believes it can more than double this figure in 2025, based on strong interest from business customers seeking to use its technology to adopt generative AI tools in the workplace.

The post Bad Times for Perplexity AI Begins appeared first on AIM.

Vision Pro to Make a Worldwide Release Soon?

The Apple Vision Pro is likely to make a worldwide release next month. In a recent report, Bloomberg highlighted that employees from several international stores have been flown into the US to undergo training in the usage of the headset.

This move is speculated to be a step towards a worldwide release for the Vision Pro, which was released in the US in February last year. Now, Apple is expected to release the spatial computer during the upcoming Worldwide Developers Conference to be held between June 10 and 14.

Ahead of its release, let’s look at how the Vision Pro and similar spatial computers could change industries for the better in India, especially when it comes to tasks that are dangerous or extensive.

Top Apple Vision Pro Use Cases for Users

1. Disaster Management and Emergency Response
2. Healthcare
3. Aerospace and Aviation
4. Aerospace and Aviation
5. Oil and Gas
6. Agriculture and Food Production
7. Cultural Preservation
8. Urban Planning and Infrastructure
9. Manufacturing
10. Scientific Research

1. Disaster Management and Emergency Response

As the seventh largest country in the world, India is disaster-prone in several areas, be it in terms of floods, cyclones, and earthquakes, or even smaller challenges of dealing with rain-inundated areas or fighting a fire.

Issues with disaster management, according to the National Disaster Management Authority, include the lack of local data and coordination. These can be rectified through the use of the device, which helps in assessing, compiling, and providing real-time information, and improving coordination efforts between departments and on-ground responders.

Additionally, it can double up as a training device, allowing responders to simulate potential disasters in order to devise effective strategies.

2. Healthcare

The Vision Pro has already seen several uses in healthcare, both in the UK as well as in India. Recently, doctors in Chennai made use of the mixed-reality headset during several surgeries, to aid them in real-time monitoring.

Additionally, in areas lacking in continuous training capabilities, a one-time investment in the MR device could help train several batches of students in particularly sensitive areas, like surgery or even diagnosis, without the potential risks associated with it.

3. Aerospace and Aviation

Similarly, with real-time monitoring a major factor in the aviation and aerospace industries, the headset could help in improving these capabilities, while lowering the costs in providing a portable real-time monitoring device.

It could also help lower the risk of accidents, particularly in low visibility areas and situations where pilots are cut-off from ATC towers. It can also aid repair work, particularly in spotting anomalies, as well as in training both flight crew and technicians in simulated settings.

4. Oil and Gas

Safety being a high priority in the oil and gas industry, the Vision Pro could serve as a potential remedy for issues in real-time assessments for risk and emergency responses.

Like disaster management, this could also help companies draft effective emergency response strategies through the use of MR simulations. MR headsets could also lower risks with fewer workers needed to deal with situations in hazardous environments, with the use of real-time monitoring by remote experts.

5. Agriculture and Food Production

With drought and food shortages a major point of contention in India, particularly recently, the Vision Pro could help assess on-ground situations on a case-by-case basis. Though this would be an expensive investment, it could help government officials aid farmers in understanding the reasons behind crop failure.

6. Cultural Preservation

An interesting use case for the Vision Pro could be in preserving cultural monuments within the country. In an era of rapid urbanisation and looming threat of losing out on history, the tool could help in mapping out and providing a timeless experience for tourists, even in the event of the loss of a historical site.

Additionally, MR could help tourists and historians understand heritage sites better with simulated environments created out of ruins.

7. Urban Planning and Infrastructure

As mentioned, rapid urbanisation has taken a grip over India, however, concerns on effective urbanisation have long been a point of contention. In addition to architectural capabilities, the Vision Pro could help in aiding urban planning, even going so far as to simulate developments in villages, towns and cities, to understand how they will fare in the future.

This would also help in ensuring that long-term proposals for urban planning remain effective well into the future.

8. Manufacturing

Large-scale manufacturers could lower the costs of training or even bridge the training gap by investing in MR capabilities. The potential for workers to train using the Vision Pro, without the need for an instructor or without the necessary training materials, could help in lowering the risk of accidents caused due to the lack of training.

This would be particularly helpful in the case of India, where there is a lack of qualified candidates for highly specialised roles like machine operating.

9. Scientific Research

A lot of the use cases mentioned above, such as healthcare, agriculture, aviation, engineering, could all use the Vision Pro for research purposes too. Despite a more hefty price, compared to current simulation tech, scientific visualisation could become much cheaper with devices like the Vision Pro.

Especially with tech like Google DeepMind’s AlphaFold making rapid advancements.

The post Top 9 Apple Vision Pro Use Cases in India appeared first on AIM.

Edge computing is quickly gaining popularity because it enables low-latency, high-efficiency, real-time operations by moving storage and processing to the network’s boundary. If you are like many people, going from a centralized data center to a solution like this is promising. Unfortunately, significant cybersecurity concerns exist.

The Major Cybersecurity Risks of Edge Computing

While this technology is promising, its deployment comes with these five cybersecurity risks.

1. IoT-specific vulnerabilities

Internet-connected devices are notoriously vulnerable to man-in-the-middle attacks and botnets because they have few built-in security controls. The number of IoT attacks surpassed 112 million in 2022, up from 32 million in 2018. They pose the most significant cybersecurity risk since they are fundamental to most edge computing solutions.

2. Overabundance of logs

When you manage hundreds or thousands of endpoints, staying on top of security logs can be challenging. Considering over 50% of chief information security officers believe the number of daily alerts their teams receive is overwhelming, the additional responsibility of monitoring a decentralized frameworkwould undoubtedly pose a cybersecurity risk.

Fifty-six percent of security professionals already spend at least 20% of the workday reviewing and addressing security alerts. Moving your storage and processing to the network’s boundary would likely inundate you with dozens extra daily, making you more likely to overlook critical risks and waste time on false positives.

3. Data compromises

You cannot secure every IoT device in the same way as a centralized data center because a decentralized framework is not built for it. The data collected, stored and shared at the edge is more easily compromised by man-in-the-middle and ransomware attacks.

Take sniffing attacks, for example, where attackers can intercept and steal unencrypted data during transmission using a packet sniffer. Edge devices are particularly vulnerable because encryption is resource-intensive and often lacks enough processing power. Moreover, turning plaintext into ciphertext is slow, whereas speed is the whole point of this technology.

4. Expansive attack surface

If you are like most people, you use edge computing to reduce latency, increase bandwidth and improve performance — meaning you have to place devices as close to the network’s boundary as possible. Consequently, you have an expansive, distributed attack surface, with each machine a potential entry point for attackers.

5. New budget limitations

Edge computing is technically complex, requiring extensive telecommunications and IT infrastructure investments. Even if you can afford such a significant upfront investment, device maintenance and labor expenses leave less room in the budget for failures, recovery or the deployment of other defenses.

Mitigation strategies for edge computing risks

You can overcome numerous cybersecurity risks with strategic planning and investments.

1. Utilize authentication controls

Authentication controls like multi-factor authentication, one-time passcodes and biometrics prevent unauthorized access, preventing attackers from manipulating devices or stealing information. Since human error accounts for 27% of data breaches, you should leverage this technology even if you trust your team.

2. Automate log monitoring with AI

Automating log monitoring with artificial intelligence (AI) can help you identify indicators of compromise (IOCs) before they develop into full-blown threats. Common examples of suspicious activity include unusual network activity and failed login attempts. Once you train your algorithm to detect them, you can let it work without human intervention.

Research shows AI consistently outperforms humans in this area. One study reported their algorithm had a 99.6% recall rate for high-priority notifications, meaning it missed almost no critical alerts. Moreover, it had a 0.001% false positive rate — an impressive figure, considering even 1% translates into 10 extra alerts if you review 10,000 daily.

3. Authenticate devices and users

Edge device authentication verifies every endpoint before it can access networks or systems. With this tool, you can prevent people from connecting vulnerable, potentially compromised machines, stopping infiltration. It also helps you identify IOCs, as you can trace unusual activity to specific machines.

4. Encrypt network traffic

While encryption is an essential cybersecurity best practice, it is too resource-intensive for widespread deployment in most edge computing applications. To get around this, leverage data classification to decide which endpoints and information to prioritize. Then, you should encrypt at rest and in transit — internally and externally — using minimum key sizes.

5. Deploy an intrusion detection AI

You need a purpose-built intrusion detection mechanism since limitations like energy, processing power and memory hold back this computing technology. Consider using a deep learning algorithm, which you can tailor to your applications and autonomously adapt over time.

A deep-learning AI can learn to recognize and classify previously unknown attack patterns and cyber threats. Since it can process tremendous amounts of information, it can manage most — if not all — endpoints without having to be integrated into each one. Its scalability and ease of deployment make it an ideal solution for these computing environments.

Managing edge-related cybersecurity risks is possible

Shrugging off edge computing because of its security weaknesses could put you at a competitive disadvantage and block you from highly efficient operations. That would be unacceptable for use cases like self-driving vehicle development, remote monitoring and service delivery. If you want the benefits without the risks, consider leveraging mitigation strategies.

Image inpainting is one of the classic problems in computer vision, and it aims to restore masked regions in an image with plausible and natural content. Existing work employing traditional image inpainting techniques like Generative Adversarial Networks or GANS, and Variational Auto-Encoders or VAEs often require auxiliary hand-engineered features but at the same time, do not deliver satisfactory results. Over the past few years, diffusion-based methods have gained popularity within the computer vision community owing to their remarkable high-quality image generation capabilities, output diversity, and fine-grained control. Initial attempts at employing diffusion models for text-guided image inpainting modified the standard denoising strategy by sampling the masked regions from a pre-trained diffusion model, and the unmasked areas from the given image. Although these methods resulted in satisfactory performance across simple image inpainting tasks, they struggled with complex mask shapes, text prompts, and image contents that resulted in an overall lack of coherence. The lack of coherence observed in these methods can be owed primarily to their limited perceptual knowledge of mask boundaries, and unmasked image region context.

Despite the advancements, research, and development of these models over the past few years, image inpainting is still a major hurdle for computer vision developers. Current adaptations of diffusion models for image inpainting tasks involve modifying the sampling strategy, or the development of inpainting-specific diffusion models often suffer from reduced image quality, and inconsistent semantics. To tackle these challenges, and pave the way forward for image inpainting models, in this article, we will be talking about BrushNet, a novel plug and play dual-branch engineered framework that embeds pixel-level masked image features into any pre-trained diffusion model, thus guaranteeing coherence, and enhanced outcome on image inpainting tasks. The BrushNet framework introduces a novel paradigm under which the framework divides the image features and noisy latent into separate branches. The division of image features and noisy latents diminishes the learning load for the model drastically, and facilitates a nuanced incorporation of essential masked image information in a hierarchical fashion. In addition to the BrushNet framework, we will also be talking about BrushBench, and BrushData that facilitate segmentation-based performance assessment and image inpainting training respectively.

This article aims to cover the BrushNet framework in depth, and we explore the mechanism, the methodology, the architecture of the framework along with its comparison with state of the art frameworks. So let’s get started.

BrushNet: Image Inpainting with Dual-Branch Diffusion

Image inpainting, a method that attempts to restore the mission regions of an image while maintaining overall coherence has been a long standing problem in the computer vision field, and it has troubled developers and researchers for a few years now. Image inpainting finds its applications across a wide variety of computer vision tasks including image editing, and virtual try-ons. Recently, diffusion models like Stable Diffusion, and Stable Diffusion 1.5 have demonstrated remarkable ability to generate high-quality images, and they provide users the flexibility to control the semantic and structural controls. The remarkable potential of diffusion models is what has prompted researchers to resort to diffusion models for high-quality image inpainting tasks that align with the input text prompts.

The methods employed by traditional diffusion-based text guided inpainting frameworks can be split into two categories, Sampling Strategy Modification and Dedicated Inpainting Models. The Sampling strategy modification method modifies the standard denoising process by sampling the masked regions from a pre-trained diffusion model, and copy-pastes the unmasked regions from the given image in each denoising step. Although sampling strategy modification approaches can be implemented in arbitrary diffusion models, they often result in incoherent inpainting results since they have limited perceptual knowledge of mask boundaries, and unmasked image region context. On the other hand, dedicated inpainting models fine-tune a image inpainting model designed specifically by expanding the dimensions of the input channel of the base diffusion model to incorporate corrupted image and masks. While dedicated inpainting models enable the diffusion model to generate more satisfactory results with specialized shape-aware and content aware models, it might or might not be the best architectural design for image inpainting models.

As demonstrated in the following image, dedicated inpainting models fuse masked image latent, noisy latent, text, and mask at an early stage. The architectural design of such dedicated inpainting models easily influences the masked image features, and prevents the subsequent layers in the UNet architecture from obtaining pure masked image features due to the text influence. Furthermore, handling the generation and condition in a single branch imposes further burden on the UNet architecture, and since these approaches also require fine-tuning in different variations of the diffusion backbone, these approaches are often time-exhaustive with limited transferability.

It might appear that adding an additional branch dedicated to extract masked image features might be an adequate solution to the problems mentioned above, however, existing frameworks often result in extracting and inserting inadequate information when applied directly to inpainting. As a result, existing frameworks like ControlNet yield unsatisfactory results when compared against dedicated inpainting models. To tackle this issue in the most effective manner possible, the BrushNet framework introduces an additional branch to the original diffusion network, and thus creates a more suitable architecture for image inpainting tasks. The design and architecture of the BrushNet framework can be summed up in three points.

1. Instead of initializing convolution layers randomly, the BrushNet framework implements a VAE encoder to process the masked image. As a result, the BrushNet framework is able to extract the image features for adaptation to the UNet distribution more effectively.
2. The BrishNet framework gradually incorporates the full UNet feature layer by layer into the pre-trained UNet architecture, a hierarchical approach that enables dense per-pixel control.
3. The BrushNet framework removes text cross-attention from the UNet component to ensure pure image information is considered in the additional branch. Furthermore, the BrushNet model also proposes to implement a blurred blending strategy to attain better consistency along with a higher range of controllability in unmasked regions of the image.

BrushNet : Method and Architecture

The following figure gives us a brief overview of the BrushNet framework.

As it can be observed, the framework employs a dual-branch strategy for masked image guidance insertion, and uses blending operations with a blurred mask to ensure better preservation of unmasked regions. It is worth noting that the BrushNet framework is capable of adjusting the added scale to achieve flexible control. For a given masked image input, and the mask, the BrushNet model outputs an inpainted image. The model first downsample the mask to accommodate the size of the latent, and the masked image is fed as an input to the VAE encoder to align the distribution of the latent space. The model then concatenates the masked image latent, the noisy latent, and the downsampled mask, and uses it as the input. The features that the model extracts are then added to the pre-trained UNet layer after a zero convolution block. After denoising, the model blends the masked image and the generated image with a blurred mask.

Masked Image Guidance

The BrushNet framework inserts the masked image feature into the pre-trained diffusion network using an additional branch, that separates the feature extraction of masked images from the process of image generation explicitly. The input is formed by concatenating the masked image latent, noisy latent, and the downsampled mask. To be more specific, the noisy latent provides information for image generation during the current generation process, and helps the framework enhance the semantic coherence of the masked image feature. The BrushNet framework then extracts the masked image latent from the masked image using a Variational AutoEncoder. Furthermore, the framework employs cubic interpolation to downsample the mask in an attempt to ensure the mask size aligns with the masked image latent, and the noisy latent. To process the masked image features, the BrushNet framework implements a clone of the pre-trained diffusion model, and excludes the cross-attention layers of the diffusion model. The reason is the pre-trained weights of the diffusion model serve as a strong prior for extracting the features of the masked image, and excluding the cross-attention layers ensure that the model only considers pure image information within the additional branch. The BrushNet framework inserts the features into the frozen diffusion model layer by layer, thus enabling hierarchical dense per-pixel control, and also employs zero convolution layers to establish a connection between the trainable BrushNet model, and the locked model, ensuring the harmful noise have no influence over the hidden states in the trainable copy during the initial training stages.

Blending Operation

As mentioned earlier, conducting the blending operation in latent space resizes the masks that often results in several inaccuracies, and the BrushNet framework encounters a similar issue when it resizes the mask to match the size of the latent space. Furthermore, it is worth noting that encoding and decoding operations in Variational AutoEncoders have inherent limited operations, and may not ensure complete image reconstruction. To ensure the framework reconstructs a fully consistent image of the unmasked region, existing works have implemented different techniques like copying the unmasked regions from the original image. Although the approach works, it often results in a lack of semantic coherence in the generation of the final results. On the other hand, other methods like adopting latent blending operations face difficulty in preserving the desired information in the unmasked regions.

Flexible Control

The architectural design of the BrushNet framework makes it a suitable choice for plug and play integrations inherently to various pre-trained diffusion models, and enables flexible preservation scale. Since the BrishNet framework does not alter the weights of the pre-trained diffusion model, developers have the flexibility to integrate it as a plug and play component with a fine-tuned diffusion model, allowing easy adoption and experimentation with pre-trained models. Furthermore, developers also have the option to control the preservation scale of the unmasked regions by incorporating the features of the BrushNet model into the frozen diffusion model with the given weight w that determines the influence of the BrushNet framework on the preservation scale, offering developers the ability to adjust the desired levels of preservation. Finally, the BrushNet framework allows users to adjust the blurring scale, and decide whether or not to implement the blurring operation, therefore easily customizing the preservation scale of the unmasked regions, making room for flexible adjustments and fine-grained control over the image inpainting process.

BrushNet : Implementation and Results

To analyze its results, the BrushNet framework proposes BrushBench, a segmentation-based image inpainting dataset with over 600 images, with each image accompanied by a human-annotated mask, and caption annotation. The images in the benchmark dataset are distributed evenly between natural and artificial images, and also ensures even distribution among different categories, enabling a fair evaluation across different categories. To enhance the analysis of the inpainting tasks even further, the BrushNet framework categorizes the dataset into two distinct parts on the basis of the methods used: segmentation-based, and brush masks.

Quantitative Comparison

The following table compares the BrushNet framework against existing diffusion-based image inpainting models on the BrushBench dataset with the Stable Diffusion as the base model.

As it can be observed, the BrushNet framework demonstrates remarkable efficiency across masked region preservation, text alignment, and image quality. Furthermore, models like Stable Diffusion Inpainting, HD-Painter, PowerPaint, and others demonstrate strong performance on image inside-inpainting tasks, although they fail to replicate their performance on outside-inpainting tasks especially in terms of text alignment and image quality. Overall, the BrushNet framework delivers the strongest results.

Furthermore, the following table compares the BrushNet framework against existing diffusion-based image inpainting models on the EditBench dataset, and the performance is comparable to the one observed on the BrushBench dataset. The results indicate the BrushNet framework delivers strong performance across a wide range of image inpainting tasks with different mask types.

Qualitative Comparison

The following figure qualitatively compares the BrushNet framework against existing image inpainting methods, with results covering artificial intelligence and natural images across different inpainting tasks including random mask inpainting, segmentation mask inside inpainting, and segmentation mask outside-inpainting.

As it can be observed, the BrushNet framework delivers remarkable results in the coherence of the unmasked region, and the coherent regions, and successfully realizes the awareness of the background information owing to the implementation of the dual-branch decoupling approach. Furthermore, the untouched branch of the pre-trained diffusion model also provides the advantage of covering different data domains like anime and painting better, resulting in better performance across different scenarios.

Final Thoughts

In this article we have talked about BrushNet, a novel plug and play dual-branch engineered framework that embeds pixel-level masked image features into any pre-trained diffusion model, thus guaranteeing coherence, and enhanced outcome on image inpainting tasks. The BrushNet framework introduces a novel paradigm under which the framework divides the image features and noisy latent into separate branches. The division of image features and noisy latents diminishes the learning load for the model drastically, and facilitates a nuanced incorporation of essential masked image information in a hierarchical fashion. In addition to the BrushNet framework, we will also be talking about BrushBench, and BrushData that facilitate segmentation-based performance assessment and image inpainting training respectively.

By 2025, the world will generate 181 zettabytes of data (that’s like streaming Netflix continuously for 36 million years!) This data boom isn’t confined to business sectors—it permeates every facet of our lives. From healthcare and banking to financial services, businesses are bombarded with data yet unaware of how to extract consumer insights, manage data, and make informed decisions. This is where AI steps up, giving back the charge to businesses with automated data management tools. From data collection to analysis, AI simplifies modern-age data management complexities, making the process much faster. In the following section, we’ll expand on the notion, delving into the fundamentals of data management, how AI impacts and augments your approach, and the best practices guiding your way to transform into an AI-enhanced data powerhouse.

How has data management evolved?

Data management is the practice of taking care of your business information. It’s about ingesting data you generate, storing it securely and efficiently, and ensuring it’s readily accessible to authorized users.

Over the years, data management has undergone profound changes. Traditionally, it relied on manual efforts: defining data formats upfront and allocating considerable resources to cleaning and fixing inconsistencies. However, this reactive approach often hampered the value extraction, leading to slow progress.

In contrast, modern data management operates with a proactive stance. Harnessing AI technology, it automates data collection, stores it in the database of choice (NoSQL, cloud, or relational), and cleans and transforms it using various techniques. Unlike conventional practices, AI-enhanced data management upholds data quality through rigorous validation checks, expediting the extraction of valuable insights while keeping them secure from unauthorized access.

Furthermore, modern data management has revolutionized how information is stored and accessed by making it easier for businesses to break down data silos. Techniques like ETL (Extract, Transform, and Load) and ELT (Load, Transform, and Load), coupled with AI, have streamlined data integration processes, which enables users to scale up or down as per business needs and make informed decisions based on a single source of truth.

How does AI augment data management effectiveness?

AI-enhanced data management tools efficiently scan metadata, analyze data relationships leveraging ML, and automate data lineage mapping. These functionalities help modern businesses keep up with the ever-evolving data deluge, guiding decision-making with actionable insights, reducing costs and errors, and streamlining workflows with automated data integration. To break it down, AI enhances how you manage data in the following ways:

Data cleansing and standardization

Using machine learning algorithms, AI identifies patterns in your data and flags any anomalies. Once identifying bad data entries (missing, duplicate, invalid data, or inconsistent formatting), it eliminates poor quality results, automatically replaces them with good data from the source, and refreshes the AI model. The more data you process through ML algorithms, the higher the speed and better its performance, thanks to deep learning that can learn and take decisions like humans.

Data noise reduction

Noisy data can affect your entire data pipeline if not addressed. AI removes this noise, an inevitable human error, from your datasets and enhances data quality using a combination of advanced techniques, such as:

• Machine Learning: AI algorithms trained on vast amounts of data learn to distinguish between noise and desired signals. This allows for targeted noise removal and signal preservation.
• Deep Learning: Deep neural networks mimic the human brain to analyze data and predict noise presence. They can suppress noise while maintaining the clarity of important information.
• Spectral Subtraction: Estimates and removes the noise “fingerprint” from the data. Simply put, it analyzes the background noise pattern in your data, similar to taking a fingerprint of it, and cancels it out, leaving you with a clearer signal.
• Adaptive Filtering: Continuously monitors the data and adjusts noise reduction based on real-time changes, effectively suppressing noise in dynamic environments.
• Beamforming: AI focuses on the desired signal’s direction, filtering out noise and interference from other directions.

Many AI noise reduction techniques operate in real-time, enabling immediate noise suppression during communication or data acquisition. This eliminates the need for separate noise reduction steps, streamlining workflows and saving time.

Addressing missing data

There are various approaches to address missing data when it comes to AI and choosing the optimal technique depends on its characteristics. Here are some of these techniques:

1. Deletion

This approach removes entire rows or columns containing the missing values. It’s tempting in the case of small datasets. However, deleting too much data can reduce the statistical power of your analysis, leading to misleading results. Consider this technique only if:

• A column has an unrealistically high percentage of missing values
• Your dataset is massive, and losing a few rows won’t make a difference
• The missing data doesn’t affect your analysis goals (e.g., highly skewed data)

1. Imputation

Imputation replaces missing values with educated estimates. It is a good option when:

• The missing data percentage is low

• Deleting rows would result in a substantial loss of information

1. Arbitrary Value Replacement

This involves replacing missing values with a pre-determined constant (e.g., -999). It can be an option if deletion or standard imputation methods introduce bias, especially when missingness isn’t random.

With these techniques, AI helps create a more complete and trustworthy data foundation, reducing the potential bias or misleading conclusions that missing data can present.

Analyzing trends

Staying ahead in today’s dynamic B2B environment requires anticipating customer needs and capitalizing on emerging trends. AI combines techniques to identify and capitalize on emerging trends proactively through various methods, including:

• Data Mining: Unearths hidden patterns in complex datasets, revealing valuable insights you might miss.
• Natural Language Processing (NLP): Makes sense of customer conversations, social media chatter, and news articles—aiding in understanding the sentiments and opinions of your target audience.
• Deep Learning: Predicts future trends in areas like sales or market shifts, giving you a roadmap for strategic decision-making.

These empower organizations to leverage the first-mover advantage by identifying trends before competitors, making data-centric decisions based on concrete AI insights, and enhancing customer centricity by better understanding consumer needs and preferences.

Closing remarks

Data deluge will only grow over time. It is critical to assess your data management practices and unclog workflow bottlenecks using intelligent technologies. This raises fundamental questions for businesses: do you have a solid data foundation? Do you have the right strategy to manage data? Is your current approach incorporating AI?

Addressing these questions can steer you toward enhanced data management. AI data management tools, when harnessed effectively, can help you achieve that, turning these enhancements into your competitive advantage.

These solutions provide an array of functionalities that can help you extract insights from data without necessitating extensive programming expertise.