Рубрика: AI

Image by Ahmad Ardity from Pixabay

The good news is that the data science community is taking more of an interest in knowledge graphs lately. But unsurprisingly, some data science folks exploring graphs themselves are barely scratching the surface of knowledge graph potential.

Until data scientists view the root problem to be solved through the lens of architectural transformation and grasp the scope of the problem to be solved, they won’t be designing and building knowledge graphs in a way with system-wide benefits. Labeling data scraped off the public web won’t solve fundamental architectural problems.

Data scientists will need to support the creation of shared understanding–knowledge articulated in a findable, accessible, interoperable and reusable (FAIR) fashion in a knowledge graph–by allying with knowledge architects and engineers who understand how to build and connect interoperable knowledge domains at scale. Otherwise, knowledge graph efforts will fail the same way most master data management efforts have failed.

Leadership can play a critical role here in supporting the nurturing and growth of a knowledge building and sharing human resource that has clout when it comes to broadening and deepening company AI efforts. Leadership in this case starts with granting real authority and long-term commitment to knowledge architects and engineers, professionals who are by definition well versed in ontologies, the key to logically consistent, scalable and reusable knowledge graphs.

Unless these professionals have authority and budget to proceed with building out a consistent and scalable graph infrastructure and leadership mandates the use of this infrastructure, knowledge graph efforts won’t succeed in enterprises, and data quality will continue to suffer. Tribalism will undermine those efforts. Machine learning folks, for example, who advocate monolithic, statistical-only AI over hybrid AI and who are seeking more power than they already have will continue to trample on the few ontologists who have the skills and experience to back serious knowledge graph initiatives.

Knowledge graphs in demand for question-answering accuracy

In November 2023, benchmarking reports from data.world and Meta confirmed the need for semantic technologies such as knowledge graphs in order to improve question-answering accuracy.

Data.world reported in its results report that zero-shot GPT-4 prompting of SQL databases had 16 percent accuracy. The same GPT-4 prompting of a KG representation of those SQL DBs, meanwhile, resulted in 54 percent accuracy. (Sequeda, Allemang, Jacob, “A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases,” Nov. 14, 2023.)

Meta, for its part, reported that humans could answer questions with 92 percent accuracy, versus 15 percent for GPT-4 with plugins. (Mialon, Fourrier, Swift, Wolf, LeCun, and Scialom, “GAIA: A Benchmark for General AI Assistants,” November 23, 2023.)

Data science, data theory…and the issue of poor data quality

On November 16, 2023 statistics professor Mark Handcock moderated the third seminar in the UCLA series on Data Theory in the World, on the subject of Knowledge Graphs in Data Science. It was a helpful seminar but didn’t address what knowledge graphs are capable of beyond single, presumably understaffed, ad-hoc projects. Two PhDs who hold data science roles at Snap and Microsoft told stories about their public/private entity matching and COVID research knowledge graph-building efforts.

The University of California at Los Angeles (UCLA) is unusual in that it offers a major in Data Theory. Data doesn’t often get this level of foundational attention. It clearly deserves to be a research field in its own right, given that machine learning’s accuracy is entirely dependent on the quality and comprehensiveness of data inputs. (See “Data management implications of the AI act” at https://www.datasciencecentral.com/data-management-implications-of-the-ai-act/ for more detail.)

Handcock introduced the session by defining data theory as the mathematical and statistical foundation of data science. He pointed out that UCLA’s Data Theory degree program is a joint effort between the mathematics department and the statistics and data science department.

Handcock defined data science as the science of how to learn from data. His definition focuses on the machine learning side of things and assumes that the data is already worthy of a machine learning effort. The truth is that consistent and pervasive data quality is mostly lacking or absent when it comes to many machine learning efforts.

Just to name one example of how low data quality can be for machine learning purposes, recent research from Amazon and UC Santa Barbara indicates that much of the web-scraped content that ends up in ML training sets is machine-translated into different languages, which implies the incremental degradation of the input data with each translation.

Moreover, the authors discovered selection bias in translated content: “We also find evidence of a selection bias in the type of content which is translated into many languages…. A limited investigation suggests this selection bias is the result of low-quality content generated in English (likely produced to generate ad revenue) and translated en masse into many lower resource languages via machine translation (again, likely to generate ad revenue).” (Thompson, Dhaliwal, Frisch, Domhan and Federico, “A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism,” January 11, 2024.)

The power of verbs and relationship richness in graphs

Knowledge graphs when built well and used properly counter the trend toward lower data quality and improve question-answering accuracy across domains. One of the fundamental characteristics of knowledge graphs is that they are designed to be relationship-rich. The reason for the relationship richness is that the right relationships (the verbs or predicates in RDF triples) contextualize the data properly so that it can be integrated logically and efficiently, domain by domain. Mere labeling efforts overlook the necessity of the consistent use of these predicates.

Relationship logic is key to building context. Ian knows Mary because they ride the bus to work together. Google suggests that they work at the same place when Maps sees the routes they take.

In this simple example, the probabilistic logic of machine learning needs the deterministic logic of relationships and connections–determined facts. The two are symbiotic. Otherwise, generative AI tends to hallucinate, and we can’t get to general intelligence or high confidence levels. Thus a need for “neurosymbolic AI”–neural nets and symbolic logic working together–in which case, the logic is conveyed via webs of relationships via RDF triples in knowledge graphs.

Knowledge graphs can have a huge impact on hundreds of different use cases, but only if they are logically consistent (in accordance with FAIR principles) and operational at the enterprise scale. Data theory degree programs seem to be missing the point of knowledge graphs if they’re not advancing this larger vision.

Introduction

In the realm of modern software development, Application Programming Interfaces (APIs) stand as the backbone of data engineering, facilitating seamless data exchange and integration. As an expert in data engineering, big data, and file formats, I understand the pivotal role APIs play in today’s technological landscape. APIs serve as the conduits through which applications communicate, allowing disparate systems to work in harmony.

In this comprehensive guide, we will delve into the intricacies of building robust APIs, an essential skill for any data engineer or developer. You will gain practical insights into the art and science of crafting APIs that not only enable data flow but do so with reliability, efficiency, and scalability in mind. Our focus here is clear: to provide step-by-step guidance that empowers you to master the craft of API development, ensuring your applications are agile and equipped to thrive in the data-driven era. So, let’s embark on this journey of API excellence, where technical precision meets the real-world demands of modern software development.

Understanding the basics of APIs

What is an API?

In the realm of data engineering, an API, or Application Programming Interface, is a fundamental component that acts as a bridge between different software systems. APIs serve as a set of rules and protocols, defining how various software components should interact and communicate. Think of an API as a language that allows different applications to understand and exchange data seamlessly. These interfaces are pivotal in enabling the integration of disparate systems, ensuring they work cohesively to deliver the functionalities users require.

The role of robust APIs in data engineering

In data engineering, APIs play a paramount role in facilitating data exchange and integration. They act as the glue that binds together various data sources, databases, and analytics tools. Data engineers leverage APIs to fetch data from diverse platforms, transform it into the required format, and make it available for analysis or further processing. APIs serve as the conduits through which data flows in a controlled and structured manner, ensuring the reliability and consistency of data operations. Without APIs, the intricate web of data interactions in the modern data ecosystem would be far more complex to manage.

Common use cases

APIs find applications across a wide array of industries and domains. In the realm of big data and data engineering, some common use cases include real-time data streaming, data extraction from web services, integration of data from different data stores (SQL and NoSQL databases) and connecting analytics platforms to data sources. APIs also play a critical role in enabling the Internet of Things (IoT), allowing devices to communicate and share data seamlessly. In essence, APIs are the cornerstone of modern data engineering, powering the integration and flow of data across systems, applications, and industries.

Planning Your API

Define objectives

When learning how to build an API, it’s imperative to begin with a clear understanding of your objectives. Define the specific goals your API will achieve. Are you looking to provide access to a database of financial data for reporting purposes? Or perhaps you intend to create an interface for IoT devices to send sensor data to a centralized system. Identifying these objectives upfront will guide the design and development process, ensuring that your API aligns precisely with your intended outcomes.

Choosing the right data format

In the realm of data engineering, selecting the appropriate data format for your API is akin to choosing the right tool for the job. JSON (JavaScript Object Notation) and XML (eXtensible Markup Language) are two prevalent data formats used for API communication. JSON, known for its simplicity and ease of parsing, is widely favored for its efficiency in transmitting structured data. XML, on the other hand, offers extensibility and strong schema support, making it suitable for more complex data structures.

The choice between JSON and XML depends on the specific needs of your API. JSON is often preferred for its lightweight nature in scenarios where simplicity and speed are paramount, while XML shines in situations that require strict data validation and complex document structures.

Security considerations

Security is a paramount concern when designing and building APIs. To protect your API and the data it handles, you must consider authentication and authorization mechanisms. Authentication ensures that only authorized users or applications can access your API, preventing unauthorized access. Authorization, on the other hand, defines what actions or resources users are allowed to access once authenticated.

Common security protocols like OAuth and API keys play crucial roles in API security. OAuth, for example, is commonly used for user authentication, while API keys are often employed for application-level authentication. By incorporating robust security measures into your API, you not only protect sensitive data but also build trust with your users, a cornerstone of API development in the modern data engineering landscape.

Designing your robust API: REST vs. GraphQL

REST vs. GraphQL

In the realm of API design, two prominent paradigms exist: REST (Representational State Transfer) and GraphQL. RESTful APIs rely on a predefined set of endpoints, each representing a resource, and utilize HTTP methods (GET, POST, PUT, DELETE) for interaction. They are known for their simplicity, statelessness, and cacheability, making them suitable for a wide range of applications.

In contrast, GraphQL offers a more flexible approach. It allows clients to request precisely the data they need, reducing over-fetching and under-fetching of data. GraphQL provides a single endpoint for all interactions, and clients can specify their data requirements in the query. This flexibility can be advantageous in scenarios where optimizing data retrieval is critical, but it may require more careful planning and validation.

Choosing between REST and GraphQL depends on your project’s requirements. REST is well-suited for straightforward interactions, while GraphQL shines in scenarios where customization and efficiency in data retrieval are paramount.

Endpoint design

Effective endpoint design is pivotal to the usability of your API. Clear and intuitive endpoint naming conventions make it easier for developers to understand and use your API. Use descriptive names that reflect the resource being accessed, and structure endpoints logically. Avoid overly complex or nested endpoints that can lead to confusion. Well-designed endpoints enhance the overall developer experience, fostering adoption and reducing the learning curve.

Data modeling

Data modeling involves structuring your data in a way that aligns with your API’s objectives. Proper data modeling ensures that your API provides meaningful and consistent data representations. Consider the relationships between different data entities, define data types, and establish clear data schemas. Effective data modeling simplifies the integration of your API into various applications and reduces data inconsistencies.

Versioning

As your API evolves, maintaining backward compatibility with existing clients becomes crucial. Versioning allows you to introduce changes and enhancements to your Robust API without breaking existing integrations. By specifying a version in the API endpoint (e.g., /v1/resource), you ensure that clients can continue to access the original version while newer versions are introduced. Versioning is essential for API stability and long-term maintenance, particularly in dynamic data engineering environments where requirements can change rapidly.

Building your robust API

Tools and frameworks

When it comes to building robust APIs, selecting the right tools and frameworks is paramount. In the data engineering and big data domain, some popular choices include Express.js for Node.js applications, Django Rest Framework for Python, and Ruby on Rails for Ruby enthusiasts. These frameworks provide a solid foundation for API development, offering features like routing, serialization, and authentication out of the box. Additionally, tools like Postman and Swagger can assist in API testing and documentation, streamlining the development process.

Coding best practices

Efficiency and maintainability are cornerstones of API development. Adhering to coding best practices ensures that your codebase remains clean and easy to work with. Utilize consistent naming conventions, practice modularization to separate concerns, and employ version control systems like Git to track changes systematically. Document your code comprehensively, both for your benefit and for other developers who may collaborate on the project. Following these practices not only enhances code readability but also accelerates the development and debugging process.

Handling errors and exceptions

Error handling is a critical aspect of API development. Robust error handling ensures that your API provides meaningful feedback to clients in case of unexpected situations. Utilize HTTP status codes to convey the outcome of requests clearly. Implement detailed error messages that assist developers in identifying and addressing issues efficiently. Additionally, consider logging errors to facilitate debugging and monitoring, contributing to the overall reliability of your API.

Documentation

Thorough API documentation is non-negotiable. Document every aspect of your API, including endpoints, request and response structures, authentication procedures, and usage examples. Effective documentation empowers developers to understand and integrate with your API effortlessly. Tools like Swagger and API Blueprint can automate documentation generation, saving time and ensuring accuracy.

Testing

Testing is the bedrock of API reliability. Rigorously test your API endpoints to verify their correctness and resilience. Implement unit tests, integration tests, and end-to-end tests to cover various aspects of functionality. Automation tools like Postman or Newman can streamline the testing process, enabling you to catch issues early in the development cycle. Testing not only enhances the reliability of your API but also instills confidence in both developers and users.

Optimizing and scaling your robust API

Performance optimization

To ensure your API delivers optimal performance, consider techniques such as caching, compression, and asynchronous processing. Caching frequently requested data can significantly reduce response times. Implement compression algorithms like GZIP to minimize the data transferred over the network. Additionally, use asynchronous processing for time-consuming tasks to free up resources and enhance responsiveness.

Scalability

Scaling an API to handle increased traffic is a vital consideration. Horizontal scaling, which involves adding more server instances, is a common strategy. Load balancing distributes incoming requests evenly across these instances to prevent overload. Employing a microservices architecture can also facilitate scalability by breaking down the application into smaller, independently deployable components. This modular approach allows you to scale specific parts of your API as needed.

Monitoring and analytics

Monitoring tools are essential for tracking the health and usage of your API. Implement tools like Prometheus, Grafana, or specialized API monitoring solutions to collect metrics on request rates, response times, error rates, and resource utilization. Analytics enable you to gain insights into user behavior, identifying trends and potential performance bottlenecks. By proactively monitoring and analyzing your API, you can make informed decisions about optimizations and scaling strategies, ensuring it continues to meet the demands of your users and data engineering processes.

Conclusion: Mastering the craft of robust API development

In closing, building robust APIs is the foundation of successful data engineering in today’s digital landscape. We’ve explored the essential steps, from planning and design to optimization and scalability. These APIs are the lifeblood of data flow, enabling applications to exchange information seamlessly. As you embark on your API development journey, remember that knowledge is power. Apply the insights gained here to your projects, and you’ll be well-equipped to create APIs that drive efficiency, reliability, and innovation in your data engineering endeavors.

Avoid the AI siren song[1]. Avoid the advice that leads you to believe an artificial intelligence (AI) project is just like any other IT project and that the approach you used for your ERP / MRP / BFA / CRM implementations will work here. Be cautious of the “start small” advice. Instead, think:

Start small, but start small and strategic, not small and random.

AI projects require significant data, technology, people skills, and culture investments to succeed. That means you will need the senior management support and fortitude to support those investments and stay the course as your organization learns to apply AI to derive and drive new customer, product, service, and operational value sources.

However, building senior management support (and the necessary budget) will NOT happen by targeting random use cases. Senior management wants to prioritize their investments in initiatives that deliver substantial, quantifiable business and operational value. Senior management wants to focus on something strategically crucial to the business. Senior management wants to focus on strategic business initiatives (Figure 1).

Figure 1: Strategic Business Initiatives

Understanding Your AI Investment Requirements

Yes, the future of AI is bright (grab them sunglasses!). However, organizations must be prepared to invest in their data, technology, people, and cultural capabilities to create an AI strategy that drives business and operational success.

1) Data. Investing in data is crucial for the success of your organization. This includes managing, governing, and ensuring your data’s quality, granularity, latency, and enrichment. The accuracy, reliability, and timeliness of your AI models depend entirely on your data’s accuracy, reliability, and timeliness. However, it’s essential not to apply data management and governance indiscriminately. Instead, take the time to understand which data sets are most relevant and valuable for achieving your organization’s business and operational goals. Then, focus your data management and governance investments on the data sets that support the achievement of those strategic business initiatives.

2) Technology. Organizations must be prepared for investments in new data management, data engineering, and analytics processing technologies. Technology is not my space, but check out this AI design guide from Dell Technologies for more information about the technology requirements to support your AI journey.

However, I do understand analytics and how to apply analytics to create new sources of business and operational value. The emergence of autonomous analytics – analytics that can learn and adapt with minimal human intervention – will transform every aspect of society, including enhancing the quality of healthcare, tackling environmental challenges, addressing transportation safety and bottlenecks, accelerating manufacturing excellence, fostering social and economic equity, and much more.

Autonomous Analytics is often based on reinforcement learning (RL) that learns from experience and feedback. For example, Reinforcement Learning (RLHF – Reinforcement Learning with Human Feedback) enables Generative AI products such as ChatGPT, Bing AI, and Google Bard to learn and adapt from human interactions.

3) People Skills. Data science is a team sport comprised of Data Engineers, Data Scientists, and Business Stakeholders. The first key to developing your people skills is applying a common framework, like the “Thinking Like a Data Scientist” framework, around which people’s skills can be evaluated and developed (Figure 2).

Figure 2: The Art of Thinking Like a Data Scientist Methodology

The other key to developing your people skills is clearly articulating the roles, responsibilities, and expectations of the Data Engineers, Data Scientists, and Business Stakeholders to deliver more relevant, meaningful, responsible, and ethical business and operational outcomes.

4) Cultural Transformation. Even if we have a solid data management strategy and capability, modern and extensible technology capabilities, and appropriately skilled people…, we still have nothing unless we have cultural empowerment.

Cultural empowerment means embracing ambiguity, diversity, collaboration, experimentation, and learning from failures.

The force multiplier for AI success is cultural empowerment, which includes:

• Personalize the Organization’s Mission by ensuring everyone is connected to the organization’s mission and understands how their role contributes to the success of that mission.
• Speak the Language of Your Customers by using their language to describe their aspirations, desired outcomes, needs, and challenges.
• Foster Organizational Improvisation by embracing a culture of experimentation and agility, where employees are encouraged to try new things, learn from mistakes, and adapt quickly to changing circumstances.
• Embrace an “AND” Mentality where different perspectives and approaches are complementary rather than conflicting, fueling the drive toward innovation.
• Ensure Everyone Has a Voice by amplifying voices that might otherwise be unheard, creating safe spaces for open dialogue, and encouraging dissent.
• Unleash the Curiosity-Creativity-Innovation Pyramid that fosters a culture of learning, exploration, and invention, encouraging experimentation that fuels creativity, allowing individuals to connect seemingly disparate ideas, and forge new paths leading to breakthrough advancements.

To embark on a successful AI journey, your organization needs to invest in four key areas: data, technology, people skills, and cultural transformation. These areas are interdependent and mutually reinforcing and need to be aligned in lockstep with your strategic vision and goals. Data is the fuel for AI, technology is the engine, people skills are the drivers, and cultural transformation is the road map. By investing in these areas, you can unleash the power of AI to create value, innovate, and transform your organization (Figure 3). 🚀

Figure 3: Why Data Management is Today’s Most Important Business Discipline

Start Small and Strategic – Part 1: Summary

These areas of data, technology, people skills, and culture require substantial investments in time, money, and patience to nurture and grow AI’s transformational power. Something with that sort of investment and potential should be targeted at something important to your organization – a strategic business initiative. In Part 2, we will deep-dive into strategic business initiatives and their supporting (tethered) use cases.

[1] A siren song is an idiom that refers to something alluring and tempting but is ultimately dangerous, deceptive, or destructive.

Let’s dive into the cloud, but not just any cloud—the cloud of the future, specifically the realm of cloud security in 2024. We’re not just talking about your everyday, run-of-the-mill updates here.

We’re looking at the big players, the game changers, the trends that are going to set the stage for how we protect our digital skies.

So, grab your virtual umbrella (or shield, for that matter), and let’s explore these five trends set to redefine cloud security in 2024.

1. AI & ML: The dynamic duo of defense

First up, we have AI and machine learning. Yeah, yeah, we’ve all heard the buzzwords, but here’s the kicker: in 2024, they’re not just buzzwords anymore, they’re the backbone of cloud security.

Imagine having a bouncer at the door of your cloud data, but instead of checking IDs, they’re analyzing patterns, behaviors, and anomalies faster than you can say “unauthorized access.”

This dynamic duo is getting smarter, more intuitive, and more proactive in sniffing out threats before they even become threats. It’s like having a security guard who can predict the future—creepy, but incredibly cool.

2. Zero Trust architecture

Moving on to a concept that sounds like it came straight out of a spy movie: Zero Trust Architecture. The name pretty much gives it away. In the world of cloud security, it’s all about trust issues. And by issues, I mean, don’t give it away freely.

Zero Trust operates on the principle that trust is a vulnerability. By 2024, this isn’t just a good practice, it’s the standard. Access is only given on a strictly need-to-know basis, and every request is treated with a healthy dose of skepticism as if everyone is wearing a “potential threat” sign around their neck.

3. Enhanced encryption techniques

Here’s where things get a bit James Bond. Encryption isn’t new, but by 2024, the techniques are so advanced, they might as well be. We’re talking quantum encryption, homomorphic encryption, and other types that sound like they belong in a sci-fi movie.

The gist is, that as hackers get smarter, so does encryption. The goal is to create a world where data is so securely locked away that even if it’s stolen, it’s as good as gibberish without the key.

Think of it as writing your diary in a language only you understand—because sometimes, keeping secrets secret is the best policy.

4. Regulatory compliance as a baseline, not a ceiling

With great power comes great responsibility, and with great data comes great regulation. GDPR was just the beginning.

By 2024, compliance isn’t just about avoiding fines, it’s about building trust. Companies are using regulatory compliance as the baseline, the foundation upon which they build even more robust security measures.

It’s like saying, “Sure, I can do ten push-ups because the coach told me to, but watch me do twenty just to show off.” It’s about going above and beyond, making data security a competitive advantage rather than a checkbox.
This means conducting a cloud security assessment on a regular basis, to identify and fill gaps, even when not mandated by the law, or relevant regulatory authorities.

5. The rise of cloud security platforms

Army knife of cloud security. Instead of juggling a dozen different tools, each with its own login and interface, everything is consolidated into one platform.

These platforms are becoming smarter, more integrated, and more customizable. They’re like the command center of cloud security, giving you a bird’s eye view of everything happening in your digital realm.

By 2024, these platforms are not just nice to have, they’re essential for keeping your cloud castle safe from invaders.

Conclusion

So, there you have it, folks—the cloud security landscape of 2024 is shaping up to be quite the spectacle. From AI-enhanced defenses to uncrackable encryption and everything in between, it’s clear that the future of cloud security is all about being smarter, more integrated, and more proactive.

As we continue to migrate our digital lives to the cloud, these advancements aren’t just comforting, they’re crucial. And while the threats may evolve, so too do our defenses, ensuring that our data remains safe and sound in the digital ether.

So here’s to a future where the only thing we have to worry about in the cloud is…well, actual weather.

Image by Cathrin2014 from Pixabay

In July 2023, Teresa Tung, managing director and cloud-first chief technologist at Accenture, gave a Factory of the Future talk at the Databricks Data + AI Summit on digital twins, knowledge graphs, and generative AI for warehouse automation. Two points she made that resonated with me: 1) Digital twins are for end-to-end automation, and 2) a knowledge graph in the mix can be “a twin of twins” that provides the semantic layer, the meaning and the means of interoperability between all the twins so that the system as a whole can deliver cohesive insight for overall optimization and other decision making. Stardog was the semantic graph database management system mentioned in Tung’s slide deck.

In the case of warehouse automation, as Tung pointed out, there are many interacting parts, including:

• Physical products being warehoused or transported
• Twins of different systems components in data product form, and then
• The “twin of twins” knowledge graph that enables these twins to interoperate.

All these elements need to be modeled individually before they’re brought together to interoperate and perform their roles in the system. That’s not to mention the role of generative AI (GAI) modeling, which makes it possible for factory managers to pose questions of the system via the graph in natural language, rather than using a database query language such as SQL or SPARQL. The GAI also enables its own take on predictive analytics.

Levels of interoperability

Wired co-founder Kevin Kelly’s Mirrorworld vision articulated in his 2019 article implies a world of interoperable digital twins, with each twin “mirroring” its physical world counterpart in terms of its behavior, including interactions with other twins. In the case of warehouse automation, for example, there’s the need for a twin that serves as a model of a plant to interoperate with a twin of an asset and a twin of an operational process.

At its most basic, one-to-one level, interoperability involves simple information sharing between components using the same software protocols, such as between a web page and a web browser. That level of sharing has existed for many decades and is the least scalable form of sharing.

Structural interoperability mandates the same data structure and syntax for the sharing to happen, allowing different kinds of software to share. Some limited scaling is possible.

Semantic interoperability requires a shared, organically growing resource of contextualized meaning–as in a well-designed knowledge graph–to enable enterprise- or even supply chain-wide interoperability scaling. Effective and scalable semantics is essential to a world of interoperable digital twins, and standards-based knowledge graphs make sense to serve as the semantic layer for broad, supply chain-scale sharing.

In this sense, the relationship and rule logic previously in applications becomes findable, accessible, interoperable, and reusable (FAIR) when it resides in the graph. What’s suggested here is the ability to use FAIR knowledge graphs (which are by definition standards-based) as a shared logic foundation for model-driven development. More on this later.

Because of the many sharable, combined contexts that enable scalable interoperation, FAIR knowledge graphs move us closer to a fuller form of machine-readable understanding and therefore more generalizable AI. Also implied here is the need to model an effective form of understanding that’s both machine and human comprehensible and manageable.

State-of-the-art digital twins in medical research

The modeling and simulation capabilities evident in digital twins are leading more organizations to fund digital twin initiatives as an aid to understanding the most complex interactions between systems. In 2022, The University of Nebraska-Lincoln awarded a $5 million grant to Nokia to build a digital twin of the human immune system over a five-year period.

The hope is that the twin Nokia builds will allow them to predict responses of the immune system to diseases and how disease behavior in the body changes over time. Eventually, the company hopes to build personalized immune system twins of patients.

Nokia obtained an additional infrastructure grant from the National Institutes of Health that will help them develop tooling for the immune system project. The company also partnered with the Johnny Carson Center for Emerging Media Arts for a medical artist to help design a user interface for the project.

Among other things, “The immune system keeps a record of every germ (microbe) it has ever defeated so it can recognize and destroy the microbe quickly if it enters the body again,” according to Australia’s Victoria State Government Department of Health on its Health Matters site. Nokia began its research by mathematically modeling T and helper T cells and how they connect. T cells are a type of white blood cells that fights infections.

Model-driven development for interoperating digital twins

Once relationship and rule logic such as Web Ontology Language (OWL), Shapes and Constraints Language (SHACL), and Datalog (a subset of the logic programming language Prolog) resides in a knowledge graph, that graph can serve as a driver of low-code development for a digital twin environment.

Semantic Arts estimates that 85 percent of code traditionally trapped in applications can be reused via knowledge graphs. One thing is certain: Lots of work is necessary to model all the twins and the twin of twins semantic layer, but the investment can pay off in a big way in terms of scalability and code writing reduction.

In the data-driven world of modern business, the quality of data flowing through your pipelines is just as critical as the data itself. High-quality data is the lifeblood of insightful analytics and informed decision-making. However, ensuring this level of quality within a data pipeline presents a complex challenge, often overlooked in the rush to harness big data’s potential. Data quality is not just about having accurate data; it encompasses completeness, consistency, reliability, and timeliness – all crucial for deriving meaningful insights. This article offers an in-depth guide to improving data quality throughout your pipeline. We will delve into where strategies and best practices are designed to ensure that your data is abundant and of high quality but also pristine, accurate, and ultimately reliable for making critical business decisions. Navigating the nuances of data quality can transform your data pipeline into a powerful asset for your organization.

Understanding data quality in the context of a data pipeline

Data quality within a data pipeline is a multifaceted concept beyond mere accuracy. It involves ensuring that data is complete, consistent, reliable, and timely as it moves through various collection, processing, and analysis stages. Quality is crucial in a data pipeline because it directly impacts the validity of business insights derived from the data. Poor data quality can cause wrong decisions, slow work, and missed chances. For instance, complete data sets may result in biased analytics, while consistent data can cause confusion and mistrust among users. Furthermore, the speed at which data is processed and made available for decision-making, referred to as timeliness, is essential in fast-paced business environments where real-time data is increasingly becoming the norm. Understanding these dimensions of data quality is the first step in ensuring that your data pipeline is not just a conduit for data but a reliable source of actionable business intelligence.

Assessing your current data quality

Before enhancing the data quality in your pipeline, it’s crucial to assess its current state. Begin by examining the data for accuracy, completeness, and consistency. Utilize tools to analyze data patterns, identify anomalies, and flag data deviating from established norms. This assessment should also review how current data management practices impact data quality, including data entry processes and automated data collection methods. Identifying the primary sources of poor data quality – be it through human error, system glitches, or inadequate data processing methods – is essential. This initial evaluation forms the foundation for targeted improvement strategies, ensuring a focused approach to enhancing data quality.

Implementing data cleansing techniques

Data cleansing is a critical step in improving data quality within your pipeline. This process involves identifying and rectifying data inaccuracies, inconsistencies, and redundancies. Start by removing duplicate entries to avoid skewed analysis. Address inaccuracies by cross-verifying data against reliable sources and correcting any errors found. Standardize data formats to ensure consistency across the dataset. For instance, unify date formats, categorize similar data types, and align disparate data sets. Employ feasible automation tools to streamline this process, ensuring continuous and efficient data cleansing. Regularly implementing these techniques is critical to maintaining the integrity and reliability of data throughout its lifecycle in the pipeline.

Data validation and verification strategies

Implementing robust validation and verification strategies is essential to ensure data integrity in your pipeline. Data validation involves checking incoming data against predefined rules and criteria to ensure it meets quality standards. This could include verifying data formats, ensuring mandatory fields are not empty, and checking for logical consistency in data sets. Verification is cross-referencing data with external authoritative sources or historical data to confirm accuracy. Automating these processes can significantly enhance efficiency. Regularly applying these strategies as data flows through the pipeline ensures that only valid, accurate data is used for decision-making and analysis.

Regular data quality audits

Conducting regular data quality audits is crucial for maintaining the integrity of your data pipeline. These audits entail a detailed examination of the data within your system to ensure it adheres to quality standards and business requirements. Set periodic intervals for these audits to examine the data’s accuracy, completeness, and consistency. Use a mix of manual checks and automated tools for a thorough evaluation. Audits help identify emerging issues, such as new types of data errors or gradual changes in data quality. The knowledge acquired from these audits empowers you to make well-informed modifications to your data processing methods, continually enhancing the overall quality of data in your pipeline.

Leveraging automation for continuous data quality improvement

Automating data quality processes is critical for continuous improvement. Utilize software that routinely scans, cleans, and validates data, identifying and rectifying real-time issues. This increases efficiency and ensures consistent data quality standards are maintained throughout the data pipeline, reducing the risk of human error.

Best practices for ongoing data quality management

For effective data quality management, establish a continuous monitoring and improvement culture. Regularly update and refine data quality metrics and standards—train staff on data accuracy and consistency. Encourage proactive reporting of data issues and foster collaboration between teams to maintain high data quality standards throughout the pipeline.

Final words

Enhancing data quality in your pipeline is not a one-time effort but an ongoing commitment. By assessing the current data quality, implementing cleansing techniques, validating and verifying data, conducting regular audits, and leveraging automation, you can significantly improve the reliability and usefulness of your data. Adopting best practices for data quality management ensures that these efforts are ingrained in your organization’s culture. High-quality data is a valuable asset in today’s data-driven world, and investing in these processes will pay dividends in the form of more accurate insights, better decision-making, and, ultimately, a more decisive competitive edge in your business operations.

Announcements

• Developing high-quality software can quickly devolve into a chaotic, unproductive mess without the proper processes and frameworks in place. Organizations require not only the right coding tools but also effective testing and deployment solutions to ensure the highest quality software is what hits the market. The proliferation of generative AI, open-source technologies and low-code/no-code tools can streamline and enhance developer tasks but only if they are applied as part of a bigger strategy that empowers developers to innovate more efficiently. Tune into the Software Development Methodologies summit to hear leading experts share the latest development tools and trends to smooth over and enhance what quickly becomes a complicated process.
• Sustainable IT is the backbone of the green economy. As businesses become increasingly reliant on artificial intelligence and other resource-intensive technologies, actionable measures need to be taken to reduce the environmental and social impact of technological innovation. Tune into the upcoming Achieving IT Sustainability Goals summit to hear leading experts discuss strategies for tracking carbon emissions metrics, enhancing direct current energy efficiency, optimizing data usage, and other methods to help design a sustainable IT strategy that delivers tangible results.

Top Stories

• Digital twins, interoperability and FAIR model-driven development
  February 5, 2024
  by Alan Morrison
  In July 2023, Teresa Tung, managing director and cloud-first chief technologist at Accenture, gave a Factory of the Future talk at the Databricks Data + AI Summit on digital twins, knowledge graphs, and generative AI for warehouse automation. Two points she made that resonated with me: 1) Digital twins are for end-to-end automation, and 2) a knowledge graph in the mix can be “a twin of twins” that provides the semantic layer, the meaning and the means of interoperability between all the twins so that the system as a whole can deliver cohesive insight for overall optimization and other decision making.
• Your AI Journey: Start Small AND Strategic – Part 1
  February 3, 2024
  by Bill Schmarzo
  Avoid the AI siren song. Avoid the advice that leads you to believe an artificial intelligence (AI) project is just like any other IT project and that the approach you used for your ERP / MRP / BFA / CRM implementations will work here. Be cautious of the “start small” advice.
• Better LLMs with Shorter Embeddings: Part 3
  January 31, 2024
  by Vincent Granville
  In this article, I discuss some secret sauce to further reduce the size of the token database by several orders of magnitude. This makes scalability a lot easier and much less costly. Finally, a key to success is fast search, for instance ANN vector search. Here, ANN stands for approximate nearest neighbors. I explain how to adapt fast search to my slightly unusual architecture.

In-Depth

• How to Enhance Data Quality in Your Data Pipeline
  February 6, 2024
  by Ovais Naseem
  In the data-driven world of modern business, the quality of data flowing through your pipelines is just as critical as the data itself. High-quality data is the lifeblood of insightful analytics and informed decision-making.
• 5 trends & advances that are set to define cloud security in 2024
  February 5, 2024
  by Rob Turner
  Let’s dive into the cloud, but not just any cloud—the cloud of the future, specifically the realm of cloud security in 2024. We’re not just talking about your everyday, run-of-the-mill updates here. We’re looking at the big players, the game changers, the trends that are going to set the stage for how we protect our digital skies. So, grab your virtual umbrella (or shield, for that matter), and let’s explore these five trends set to redefine cloud security in 2024.
• Building robust API: step-by-step guide
  February 3, 2024
  by Ovais Naseem
  In the realm of modern software development, Application Programming Interfaces (APIs) stand as the backbone of data engineering, facilitating seamless data exchange and integration. As an expert in data engineering, big data, and file formats, I understand the pivotal role APIs play in today’s technological landscape.
• What data scientists overlook when it comes to knowledge graphs
  February 1, 2024
  by Alan Morrison
  The good news is that the data science community is taking more of an interest in knowledge graphs lately. But unsurprisingly, some data science folks exploring graphs themselves are barely scratching the surface of knowledge graph potential.
• Mastering E-commerce data governance: Best practices, challenges, and future trends for quality, compliance, and growth
  January 31, 2024
  by John Lee
  Data governance is more important than ever in e-commerce, where massive amounts of data are generated and processed daily. Big Data presents opportunities and challenges for e-commerce businesses, requiring a strategic approach to data quality, security, and compliance.
• DSC Weekly 30 January 2024
  January 30, 2024
  by Scott Thompson
  Read more of the top articles from the Data Science Central community.

Image by 10302144 from Pixabay

Eric Baum in his book What Is Thinking? defines understanding as “a compressed representation of the world.” Another word for a representation is a model.

Understanding in Baum’s sense is a form of distillation and abstraction. Humans refine their level of understanding of a topic by reviewing examples of people, geographic locations, things, and ideas interacting. They capture the essence of those examples with an overarching model that captures traits those examples share.

What’s common among a group of examples? Computer scientists would call a set of those shared characteristics a class. A class hierarchy consists of layers of abstraction.

Conceptualization, persistence, and cross-domain generalization

Deep learning and other kinds of machine learning are frequently used for classification tasks. But Gadi Singer, VP & Director, Emergent AI Research at Intel Labs, points out in an August 2022 blog post that conceptualization is beyond the reach of deep learning on its own.

What’s the difference between classification as machines do it and conceptualization? Machine classification, according to Singer, predicts a class label output in response to input data.

A concept, according to Oxford Languages, is by contrast an abstract idea, a notion. The adult human mind is naturally able to conceptualize. A concept, Singer says, isn’t tied to particular data sets. The ability to conceptualize is pivotal to the ability to understand.

Key aspects of a concept include these:

• A concept can be unbounded in terms of its dimensions, “a sponge that absorbs relevant knowledge over time and experiences.”
• A concept inherently stays the same while continuing to acquire more and more properties. Singer points to the example of a lawyer who sells his Ferrari and decides to become a monk. That lawyer is still the same person.
• A concept can be applied across different, unrelated domains. Singer provides the example of the concept of traversability. A human might first conceptualize traversability as it applies to rock climbing. But then the concept soaks up more knowledge through examples in unrelated domains, such as the term’s unpredictable utility when playing a game of Risk or locating someone in a company who can fix your laptop.

Ultimately, it’s the ability to abstract at higher levels and more broadly generate new knowledge that distinguishes human conceptualization from what statistical machine learning currently offers. No wonder concepts such as “dog,” “democracy” or “uncle” can be elusive when trying to work with them in a machine learning scenario. And no wonder the statistical machine learning variety of AI alone lacks generalizability.

Neurosymbolic AI: Blending learning and reasoning abilities for better machine understanding

One stumbling block with machine understanding is that neural networks haven’t been known for their ability to reason logically. Years ago, ex-Google and University of Toronto emeritus and deep learning luminary Geoffrey Hinton even asserted that deep learning networks did not have the cognitive understanding necessary for logical reasoning or identifying causal relationships.

More recently, researchers have found ways to build logical reasoning capabilities into neural networks. Artur d’Avila Garcez is a professor of computer science at the City University of London and the author of two books on neurosymbolic learning and reasoning, a topic he’s been researching for over 20 years.

In 2023, Garcez and co-author Son Tran of the University of Tasmania published their research on a system that interprets propositional logic formulae and enables reasoning with the help of restricted Boltzmann Machines (RBMs), a visible + hidden two-layer neural network. (Tran and Garcez, “Neurosymbolic Reasoning and Learning with Restricted Boltzmann Machines,” February 2023.)

These researchers tested the system and confirmed the ability to represent these logical formulae and reason with them on data and knowledge in the RBM network. The authors also confirmed that the network can learn from the data and knowledge.

The implication in this case is that the reasoning capabilities that have been used by coders using common programming languages for decades can now be brought into neural network environments. In the process, the massive parallelism of neural nets can allow for more complex problem solving … and question-answering.

A data quality gap remains

In a previous post, I pointed out that pervasive data quality is mostly lacking or absent when it comes to many machine learning efforts. Garcez and Tran in their paper described above mention both “data and knowledge” in their neurosymbolic training sets, which implies to me that the predicate logic inherent in relationship-rich, logically consistent knowledge of knowledge graphs is, to them, an essential part of effective neurosymbolic AI as well.

Others in the knowledge graph community make the point that knowledge graphs deliver the context necessary for contextual computing. Without context, there can be no generalizability. Rich context becomes a key data quality advantage of knowledge graph approaches that should not be overlooked in machine learning environments.

Nonprofit fundraising tools can be excellent resources for assisting organizations in maintaining compliance. However, anyone considering these platforms should know a few things to stay on the right track and avoid issues.

Organizations must protect donors’ privacy

When a nonprofit’s staff members know details about donors’ sexual orientation, income, race, age and ethnicity, it’s easier to target messages, develop impactful campaigns and follow up with parties about future contributions.

However, asking for those details is a relatively new practice for many organizations. Some donors feel awkward being asked about those details, especially if they don’t know why nonprofits request it. It’s also critical that nonprofit workers never assume specifics about donors based on the relatively few details available.

For example, providing ongoing support to LGBT+ organizations doesn’t necessarily mean a donor is queer themselves. Various factors could compel them to support these nonprofits without being part of the marginalized group the organization serves.

Set the foundation for better results by carefully studying what your state or country requires regarding storing donor data and keeping it safe. Then, tell your donors what information you’d like and how your organization will use it. Be as specific as possible about how additional donor information will make it easier for the nonprofit to operate more effectively and achieve better results for service users and stakeholders.

Consider creating a website section to explain the applicable data compliance requirements, whether related to stored information or content in transit. Then, stipulate how your nonprofit meets or exceeds what regulators require. Doing those things will build donors’ trust, increasing the chances they’ll provide more specific details instead of choosing “prefer not to say” when discussing their demographics.

Learning about location-based specifics prevents pitfalls

What constitutes compliance can vary significantly depending on where a nonprofit operates and the source of its donations. For example, some states require nonprofits to periodically file paperwork to retain a tax-exempt status, whereas others don’t mandate further action after initially filing for it.

Then, no matter whether a nonprofit raises funds in the U.S. or abroad, there will almost always be at least two sets of requirements to consider. For example, the U.S. has both federal and state requirements.

Then, the European Union enforces stipulations applying to all member states. However, each member state can also have compliance requirements only relating to that country — as long as the member state’s mandates do not conflict with EU laws.

Many fundraising platforms operate in the cloud, making it easy to use them around the world and see the same database information from any location. However, before nonprofit representatives buy fundraising tools, they should verify with sales representatives that the chosen solutions comply with all applicable compliance requirements.

Compliance supports smooth, above-board operations

Many donors understandably want to know where their money goes if they decide to part with it. It’s hard to be enthusiastic about a charity’s work if the people who donated funds learn that more resources go to administrative and overhead costs than the service users who need assistance. When they ask how the funds get spent, fundraising software that supports compliance can aid nonprofit workers in providing prompt, accurate answers.

Many compliance-centered fundraising software products allow users to run reports, track statistics and see how trends change over time. Such details can help people identify which factors make donors more or less likely to give.

They’ll likely be more generous if they know the nonprofit stays in compliance by doing everything according to regulations. However, as a 2022 study showed, nonprofit giving extends beyond money. The results indicated donors would rather give their time to organizations instead of donating funds.

A closer look revealed it was because such situations increase donors’ perception of control. Consider if someone spends two hours per week serving meals at a soup kitchen. Then, they can see the direct impact of those efforts. Conversely, the outcomes are harder to quantify if someone chooses to give $20 per month to the organization and never receives concrete details about how the nonprofit used the money.

The ideal scenario is one where donors trust that nonprofits operate according to compliance requirements and are transparent about the usage of funds. Fundraising software can help those things happen.

Are you ready to purchase fundraising software?

Fundraising software can save nonprofits time, increasing the effective use of human resources and boosting service user outcomes. Choosing a platform that simultaneously supports compliance will help nonprofit workers make the most of whichever tool they choose.

2024 is the year of great data science predictions targeting big business churn. It is the time to yield benefits from the popular data science frameworks that are streamed to do wonders for industries far and wide. Data science is not just a spoof on the big number game that guides businesses’ growth. It is the elemental factor that impacts the same. The burgeoning expansion of data science can be seen across industries and worldwide sectors.

The global data science platform market is assessed to reach USD 133.70 billion by 2024 (Mordor Intelligence Report). The year 2025 also predicts the data analytics landscape shall expect significant transformations. With data-driven decision-making bringing enough boost to business expansion, industries are paved to reshape as per recent evolution.

Role of data scientists in the future:

The revolutionary role of certified Data scientists cannot be denied in the wake of business tech advancements. Data scientist is rated among the top 20 fastest-growing occupations in the US with 36% projected growth by 2031 (US Bureau of Labor Statistics). The following features shall reveal the role of data scientists in the future:

• Mitigating risk and fraud
• Delivering relevant products
• Personalized customer experiences
• Evolutionary skillsets
• Collaboration with domain experts

What makes the data science industry relevant in the future?

• Empowering management and officers to make better decisions
• Making direct actions based on trends which in turn help in defining goals
• Identifying opportunities
• Identification and refining audiences
• Recruiting the right talent for the organizations
• Testing these decisions
• Challenging the staff to adopt the best practices and focus on the issues that matter
• Decision-making with quantifiable, data-driven evidence

Top 10 Data Science Trends Outlook by Global Industry Experts:

• Rahul Pradhan, VP, Product and Strategy, Couchbase

Prediction 1:

Real-time data shall become the standard for businesses to power generative experiences with artificial intelligence. Data layers should support both transactional and real-time analytics. He further goes on to say that, the world of data science should expect a paradigm shift from model-centric to data-centric Artificial intelligence. Multimodal LLMs and databases will enable a new frontier of AI applications across industries.

• Nima Negahban, CEO & Co-Founder, Kinetica

Prediction 2:

Generative AI turns its focus towards structured enterprise data. Businesses shall embrace the use of generative AI for extracting insights from structured numeric data, enhancing generative AI’s conventional applications in producing original content. She further predicts that English will replace SQL as the lingua franca of business analysts.

• Vasu Sattenapalli, CEO, RightData

Prediction 3:

Natural Language Processing powered analytics will be the next wave of self-service. Voice and GenAI will evolve with data analytics; where you can ask questions of your data verbally and get a prompt response.

• Shawn Rogers, CEO & Fellow, BARC

Prediction 4:

Artificial intelligence will drive innovation in data management, especially through automation and speed. NLP and conversational UI will pave the way toward data democratization; making exciting data insights.

• Andy Oliver, Director of Marketing, CelerData

Prediction 5:

Java will continue to be used for building a massive legacy and even current systems and applications. More unified analytical database technologies for functions and insights will become commonplace that demand real-time analysis.

• Casey Ciniello, Product Owner & Marketing Manager, Infragistics

Prediction 6:

More businesses will rely on predictive analytics to make decisions in 2024. Organizations will realize the 360-degree impact of data analytics to make data-driven decisions. Predictive analytics will drive businesses forward by helping them make informed, data-driven decisions, improve productivity, and increase sales and revenue.

• Justin Borgman, Co-Founder & CEO, Starburst

Prediction 7:

Data products and data sharing will converge in 2024. Data products will emerge as the method of data mesh. Incorporating GenAI features to streamline data product creation and enable seamless sharing will mark the critical move in data value realization.

• Brian Peterson, Co-Founder & CTO, DialPad

Prediction 8:

Massive Influx of data talent and artificial skills will be on the rise. Evolving AI advancements are surely going to assist in bringing in massive human insight and sifting through data faster. Commoditization of data will train AI is another revelation for the data science future.

• Nick Elprin, Co-Founder & CEO, Domino Data Lab

Prediction 9:

Smaller specialized large language models will triumph over giant general processes. Generative AI will unlock the value and risks hidden in unstructured enterprise data. GenAI-based chatbots and fine-tuned foundation models will open an array of data applications.

• Arina Curtis, CEO & Co-Founder, DataGPT

Prediction 10:

Data and business teams will lock horns onboarding AI products. Businesses of all sizes, big as well as small, will prioritize clean datasets.

Final Word:

Data science is the norm ahead! Data-driven decision-making is the fuel that shall target big business innovation and evolution in the future to follow. Not only that, data is the driving force behind every mainstream business landscape empowerment. Harnessing the power of the data science industry is all that is required to move forward with finesse. Bring the maximum boost by comprehending the upcoming data science trends and predictions for elevated business grounds like a pro!