Рубрика: AI

In this blog, you will learn to visualize live data streams in real time, all within the comfort of your favorite tool, the Jupyter Notebook.

In most projects, dynamic charts within Jupyter Notebooks need manual updates; for example, it may require you to hit reload to fetch new data to update the charts. This doesn’t work well for any fast-paced industry, including finance. Consider missing out on crucial buy indications or fraud alerts because your user did not hit reload at that instance.

Here, we'll show you how to move from manual updates to a streaming or real-time method in Jupyter Notebook, making your projects more efficient and reactive.

What’s covered:

• Real-Time Visualization: You'll learn how to bring data to life, watching it evolve second by second, right before your eyes.
• Jupyter Notebook Mastery: Harness the full power of Jupyter Notebook, not just for static data analysis but for dynamic, streaming data.
• Python in Quant Finance Use Case: Dive into a practical financial application, implementing a widely used in finance with real-world data.
• Stream Data Processing: Understand the foundations and benefits of processing data in real-time, a skill becoming increasingly crucial in today's fast-paced data world.

By the end of this blog, you'll know how to build similar real-time visualizations like the one below within your Jupyter Notebook.

Quick recap of real time data processing

At the heart of our project lies the concept of stream processing.

Simply put, stream processing is about handling and analyzing data in real-time as it's generated. Think of it like Google Maps during a rush hour drive, where you see traffic updates live, enabling immediate and efficient route changes.

Interestingly, according to the CIO of Goldman Sachs in this Forbes podcast, moving towards stream or real-time data processing is one of the significant trends we’re headed toward.

Jupyter Notebooks as a Real-time Data Analytics Tool

It’s about combining the power of real-time data processing with an interactive and familiar environment of Jupyter Notebooks.

Besides that, Jupyter Notebooks play well with containerized environments. Therefore, our projects aren't just stuck on local machines; we can take them anywhere – running them smoothly on anything from a colleague's laptop to a cloud server.

Our use-case in Python in finance: Bollinger Bands

In finance, every second counts, whether for fraud detection or trading, and that’s why stream data processing has become essential. The spotlight here is on Bollinger Bands, a tool helpful for financial trading. This tool comprises:

• The Middle Band: This is a 20-period moving average, which calculates the average stock price over the past 20 periods (such as 20 minutes for high-frequency analysis), giving a snapshot of recent price trends.
• Outer Bands: Located 2 standard deviations above and below the middle band, they indicate market volatility — wider bands suggest more volatility, and narrower bands, less.

In Bollinger Bands, potentially overbought conditions are signaled when the moving average price touches or exceeds the upper band (a cue to sell, often marked in red), and oversold conditions are indicated when the price dips below the lower band (a cue to buy, typically marked in green).

Algo traders usually pair Bollinger Bands with other technical indicators.

Here, we made an essential tweak while generating our Bollinger Bands by integrating trading volumes. Traditionally, Bollinger Bands do not consider trading volume and are calculated solely based on price data.

Thus, we have indicated Bollinger Bands at a distance of VWAP ± 2 × VWSTD where:

• VWAP: A 1-minute volume-weighted average price for a more volume-sensitive perspective.
• VWSTD: Represents a focused, 20-minute standard deviation, i.e., a measure of market volatility.

Technical implementation:

• We use temporal sliding windows (‘pw.temporal.sliding’) to analyze data in 20-minute segments, akin to moving a magnifying glass over the data in real time.
• We employ reducers (‘pw.reducers’), which process data within each window to yield a particular outcome for each window, i.e., the standard deviations in this case.

Glance at tools used for enabling real time streaming data within our Jupyter Notebook

• Polygon.io: Provider of real-time and historical market data. While you can certainly use its API for live data, we've pre-saved some data into a CSV file for this demo, making it easy to follow without needing an API key.
• Pathway: An open-source Pythonic framework for fast data processing. It handles both batch (static) and streaming (real-time) data.
• Bokeh: Ideal for creating dynamic visualizations, Bokeh brings our streaming data to life with engaging, interactive charts.
• Panel: Enhances our project with real-time dashboard capabilities, working alongside Bokeh to update our visualizations as new data streams come in.

Step-by-Step Tutorial: Visualizing Real-Time Data within Jupyter Notebook.

This involves six steps:

1. Doing pip install for relevant frameworks and importing relevant libraries.
2. Fetching sample data
3. Setting up the data source for computation
4. Calculating the stats essential for Bollinger Bands
5. Dashboard Creation using Bokeh and Panel
6. Hitting the run command

1. Imports and Setup

First, let’s quickly install the necessary components.

%%capture --no-display  !pip install pathway

Start by importing the necessary libraries. These libraries will help in data processing, visualization, and building interactive dashboards.

# Importing libraries for data processing, visualization, and dashboard creation    import datetime  import bokeh.models  import bokeh.plotting  import panel  import pathway as pw

2. Fetching Sample Data

Next, download the sample data from GitHub. This step is crucial for accessing our data for visualization. Here, we have fetched Apple Inc (AAPL) stock prices.

# Command to download the sample APPLE INC stock prices extracted via Polygon API and stored in a CSV for ease of review of this notebook.    %%capture --no-display  !wget -nc https://gist.githubusercontent.com/janchorowski/e351af72ecd8d206a34763a428826ab7/raw/ticker.csv

Note: This tutorial leverages a showcase published here

3. Data Source Setup

Create a streaming data source using the CSV file. This simulates a live data stream, offering a practical way to work with real-time data without necessitating an API key while building the project for the first time.

# Creating a streaming data source from a CSV file    fname = "ticker.csv"  schema = pw.schema_from_csv(fname)  data = pw.demo.replay_csv(fname, schema=schema, input_rate=1000)    # Uncommenting the line below will override the data table defined above and switch the data source to static mode, which is helpful for initial testing  # data = pw.io.csv.read(fname, schema=schema, mode="static")    # Parsing the timestamps in the data    data = data.with_columns(t=data.t.dt.utc_from_timestamp(unit="ms"))

Note: No data processing occurs immediately, but at the end when we hit the run command.

4. Calculating the stats essential for Bollinger Bands

Here, we will briefly build the trading algorithm we discussed above. We have a dummy stream of Apple Inc. stock prices. Now, to make Bollinger Bands,

1. We’ll calculate the weighted 20-minute standard deviation (VWSTD)
2. The 1-minute weighted running average of prices (VWAP)
3. Join the two above.

# Calculating the 20-minute rolling statistics for Bollinger Bands      minute_20_stats = (      data.windowby(          pw.this.t,          window=pw.temporal.sliding(              hop=datetime.timedelta(minutes=1),              duration=datetime.timedelta(minutes=20),          ),          behavior=pw.temporal.exactly_once_behavior(),          instance=pw.this.ticker,      )      .reduce(          ticker=pw.this._pw_instance,          t=pw.this._pw_window_end,          volume=pw.reducers.sum(pw.this.volume),          transact_total=pw.reducers.sum(pw.this.volume * pw.this.vwap),          transact_total2=pw.reducers.sum(pw.this.volume * pw.this.vwap**2),      )      .with_columns(vwap=pw.this.transact_total / pw.this.volume)      .with_columns(          vwstd=(pw.this.transact_total2 / pw.this.volume - pw.this.vwap**2)          ** 0.5      )      .with_columns(          bollinger_upper=pw.this.vwap + 2 * pw.this.vwstd,          bollinger_lower=pw.this.vwap - 2 * pw.this.vwstd,      )  )

# Computing the 1-minute rolling statistics    minute_1_stats = (      data.windowby(          pw.this.t,          window=pw.temporal.tumbling(datetime.timedelta(minutes=1)),          behavior=pw.temporal.exactly_once_behavior(),          instance=pw.this.ticker,      )      .reduce(          ticker=pw.this._pw_instance,          t=pw.this._pw_window_end,          volume=pw.reducers.sum(pw.this.volume),          transact_total=pw.reducers.sum(pw.this.volume * pw.this.vwap),      )      .with_columns(vwap=pw.this.transact_total / pw.this.volume)  )

# Joining the 1-minute and 20-minute statistics for comprehensive analysis    joint_stats = (      minute_1_stats.join(          minute_20_stats,          pw.left.t == pw.right.t,          pw.left.ticker == pw.right.ticker,      )      .select(          *pw.left,          bollinger_lower=pw.right.bollinger_lower,          bollinger_upper=pw.right.bollinger_upper      )      .with_columns(          is_alert=(pw.this.volume > 10000)          & (              (pw.this.vwap > pw.this.bollinger_upper)              | (pw.this.vwap < pw.this.bollinger_lower)          )      )      .with_columns(          action=pw.if_else(              pw.this.is_alert,              pw.if_else(                  pw.this.vwap > pw.this.bollinger_upper, "sell", "buy"              ),              "hold",          )      )  )  alerts = joint_stats.filter(pw.this.is_alert)

You can check out the notebook here for a deeper understanding of the computations.

5. Dashboard Creation

It's time to bring our analysis to life with a Bokeh plot and Panel table visualization.

# Function to create the statistics plot      def stats_plotter(src):      actions = ["buy", "sell", "hold"]      color_map = bokeh.models.CategoricalColorMapper(          factors=actions, palette=("#00ff00", "#ff0000", "#00000000")      )        fig = bokeh.plotting.figure(          height=400,          width=600,          title="20 minutes Bollinger bands with last 1 minute average",          x_axis_type="datetime",          y_range=(188.5, 191),      )      fig.line("t", "vwap", source=src)      band = bokeh.models.Band(          base="t",          lower="bollinger_lower",          upper="bollinger_upper",          source=src,          fill_alpha=0.3,          fill_color="gray",          line_color="black",      )      fig.scatter(          "t",          "vwap",          color={"field": "action", "transform": color_map},          size=10,          marker="circle",          source=src,      )      fig.add_layout(band)      return fig      # Combining the plot and table in a Panel Row    viz = panel.Row(      joint_stats.plot(stats_plotter, sorting_col="t"),      alerts.select(          pw.this.ticker, pw.this.t, pw.this.vwap, pw.this.action      ).show(include_id=False, sorters=[{"field": "t", "dir": "desc"}]),  )  viz

When you run this cell, placeholder containers are created in your notebook for the plot and table. They'll be filled with live data once the computation starts.

6. Running the Computation

All the preparations are complete, and it's time to run the data processing engine.

# Command to start the Pathway data processing engine  %%capture --no-display  pw.run()  

As the dashboard updates in real-time, you'll see how the Bollinger Bands trigger actions — green markers for buying and red for selling, often at a slightly higher price.

Note: You should manually run pw.run() after the widget is initialized and visible. You can find more details in this GitHub issue here.

TL;DR

In this blog, we understand Bollinger Bands and take you through a journey of visualizing real-time financial data in Jupyter Notebook. We showed how to transform live data streams into actionable insights using Bollinger Bands and a blend of open-source Pythonic tools.

The tutorial provides a practical example of real-time financial data analysis, leveraging open source for an end-to-end solution from data fetching to interactive dashboarding. You can create similar projects by:

• Doing this for a stock of your choice by fetching live stock prices from APIs like Yahoo Finance, Polygon, Kraken, etc.
• Doing this for a group of your favorite stocks, ETFs, etc.
• Leveraging some other trading tool apart from Bollinger Bands.

By integrating these instruments with real-time data within a Jupyter Notebook, you’re not just analyzing the market but experiencing it as it unfolds.

Happy streaming!

Mudit Srivastava works at Pathway. Prior to this, he was a founding member of AI Planet and is an active community builder in the domain of LLMs and Real-time ML.

More On This Topic

• 10 Jupyter Notebook Tips and Tricks for Data Scientists
• How to Setup Julia on Jupyter Notebook
• Jupyter Notebook Magic Methods Cheat Sheet
• Cutting Down Implementation Time by Integrating Jupyter and KNIME
• How to Digest 15 Billion Logs Per Day and Keep Big Queries Within 1 Second
• How to Use Kafka Connect to Create an Open Source Data Pipeline for…

Those working in technology, security and data know protecting critical infrastructure from cybersecurity threats is a non-negotiable aspect of a functioning, modern society. However, the same mentality must apply to households. They are just as vulnerable and deserve protection. What are advanced strategies to deter threat actors and maintain privacy and data integrity?

Why cybersecurity is essential to the home

More people work from home and use an increasing amount of devices in their personal and professional lives. Tech stacks are more complex as apps and smart technologies run lives. Various providers with a mixture of compliance adherence and security strategies control them. These inconsistencies open backdoors and vulnerabilities for cybercriminals.

Protecting the home from hackers guards identities, bank accounts and data. When criminals compromise these spaces, people struggle to stay positive, maintain jobs or keep obligations in order.

Cybersecurity literacy is a requirement for everyone in a tech-laden world producing 2.5 quintillion bytes of data daily. These techniques expand upon basic recommendations, like using VPNs and two-factor authentication.

1. Use a router cascade

A router cascade is a series of connected routers with different address ranges, where each device may have customized security. The benefit of having more than one router is overseeing traffic management and creating layered defenses, like complex firewall structures. If someone breaches one router, devices connected to the protected second one can disconnect while preserving some assets and isolating the problem.

2. Review connected devices

Most people connect their phones, laptops, tablets and watches to their Wi-Fi and forget about it. Everything automatically connects to make lives easier, so they typically do not consider reviewing what devices and accounts can access the network.

Take inventory of what tech items are necessary to remain connected and provide unique nicknames to identify when something is amiss quickly. Users may also set up notification systems when irregular connection requests occur.

3. Have multiple backups

Regardless of efforts, breaches and data exfiltration may still happen. End users never want to pay ransoms or lose years of work, so having disconnected, immutable storage devices in varied formats is crucial for integrity and resilience. For example, a robust strategy may include multiple external hard drives, a cloud storage solution and a solid-state drive.

Each has its benefits. Stealing external devices requires specific techniques, especially when they are usually offline. In this way, defending the home protects tech assets, Wi-Fi connections and data. Cloud providers may only maintain customer bases with high security standards, so they seek the advice of third-party professionals and internal experts to employ the industry’s best defenses. The only way these remain relevant is by maintaining a backup schedule.

4. Change to a WPA3 Router

Wi-Fi Protected Access (WPA) is a security framework, and WPA3 is the most recent rendition as of 2024. Figure out what kind of router is in the home, and ensure it is not a version before this. WPA3 routers improve from older models by:

• Making passwords harder to decipher.
• Protecting old data from decryption.
• Safeguarding systems when connecting smart home devices.
• Securing open Wi-Fi.

5. Segment networks

Setting up network segments is separate from a router cascade. Network segmentation isolates particular activities and communications to unique connections. Many experience this when connecting to business Wi-Fi, noticing different guest and employee networks. Households may set up individual networks for questionably secure items like IoT devices or direct visitors to a guest connection to maintain integrity over the leading home network.

A well-defended home network

Cyber defenses encompass more than strong passwords and integrated encryption. There are more actions homes must take to deter the increase and severity of digital crimes. The most valuable thing on the planet right now is data, so staying educated and proactive is the top-recommended strategy to keep safe against compromises and breaches.

Open source development has been the reason for the rapid growth of tech. Yann LeCun, the famous proponent of open source projects, takes every opportunity to elaborate on how vital open source development is.

The sustainability of open source projects, however, is dependent on the financial returns it sees over time. “There is a lot of unnecessary friction today to sponsor specific features, issues or milestones for open source projects,” said Birk Jernström, the founder of Polar.

The company, founded in 2022, is a platform that manages the subscriptions and payments for people who create and support open-source software. It also offers tools for working with data.

Many open source projects start with being freely available and eventually seek funding. Red Hat for example, known for its Linux distribution, monetised by selling subscriptions for technical support, updates, and training to businesses. This model helped fund continuous open-source development while providing enterprise-level services.

Alternatively, Blender, a 3D creation suite, supports its development through the Blender Development Fund, donations, and paid services like professional training and Blender Cloud subscriptions.

For smaller projects, platforms like Patreon or Open Collective let supporters donate monthly or per project. GitHub Sponsors allows direct donations to developers. These models rely on voluntary support, which may not match the actual effort needed for development.

However, Polar takes it a step further and allows funding for specific features, issues, or milestones. This drives the project in the direction that is valued by the customers. It motivates the developer who would know they’ll get paid for hitting clear goals.

Jernström clarified the difference from existing funding platforms, pointing out, “There is no one-size-fits-all solution to this and that’s what we want to build, one platform for multiple solutions.”

Polar is changing open-source funding

GitHub is keen on giving developers the options to choose how they want to monetise their work. In 2019, they launched GitHub sponsors, but as one user pointed out, it is nothing more than ‘coffee money’ between persons. Polar, according to Jernström, gives maintainers the option to be ‘entrepreneurs’.

There have been donations in the past with platforms like Open Collective, and Stack Aid, among others, allowing individuals and companies to pledge financial support directly towards specific issues or feature requests in open-source projects. Polar intends to go beyond this ‘coffee money’ funding and provide a steady stream of income.

Jernström explained, “Donations and sponsorships are great when they happen. Problem is, they rarely do. In order to drive meaningful (full-time work) capital to OSS initiatives, I believe it has to charge for add-on value and that such services and subscriptions are mutually beneficial.”

Polar facilitates the sale of add-on services, subscriptions, or premium features, and inturn maintainers can craft offerings that align with their project’s goals and community’s needs. The platform takes a 10% commission including the 5% charges for Stripe transactions.

“As an ecosystem, we should be focused on how we can get 10x, 100x and then 1000x funding. Five percent of nothing is nothing. That’s the real problem in OSS today. Let’s fix that first,” he said.

This could include anything from offering paid support, consulting, custom development work, access to premium features, or early access to new releases.

This is an upgrade from voluntary support to making it easy for backers to financially support the issues and features they care about. By handling the financial transactions, tax considerations, and potentially even compliance issues, Polar lets developers focus on working on the projects itself.

Ease and transparency

Transparency has always been very important to open source funding. For example, the open collective for example is designed around transparency, with all financial transactions visible to the public by default. Expenses, income, and budgets are tracked and displayed on the platform, and contributors can see how funds are used and allocated.

Polar goes the same route and has complete control over which issues or features they want to highlight for funding through the platform. This ensures that they can align any external funding with their project’s roadmap and priorities.

Maintainers can set goals for funding specific initiatives within their projects, providing clarity to potential backers about what their contributions will support.

Andreas Kling, a key contributor to the SerenityOS and Ladybird who uses Polar for funding, said, “We’ve been using Polar for funding GitHub issues for a couple of months now, and it always makes me super happy when I see someone collect a reward!”

SerenityOS is a Unix-like OS with a classic desktop interface and user-friendly design, supported by an active developer community. Ladybird is its companion lightweight web browser, offering fast and secure browsing seamlessly integrated with the OS.

Kling added, “I’m super happy to see Polar take on the task of becoming a Merchant of Record and abstracting away much of the complexity for all developers.”

By addressing these critical and often overlooked aspects of open-source project maintenance, Polar is setting a precedent for how platforms can support the sustainable development of open-source software.

The post Now Open Source Projects Can Make Money appeared first on Analytics India Magazine.

Pixxel, a Bengaluru-based space technology company, has announced a partnership with the Indian Institute of Astrophysics (IIA) through a Memorandum of Understanding (MOU).

This collaboration will allow Pixxel to use IIA’s Laboratory for Space Sciences for testing its hyperspectral imagers and to benefit from IIA’s expertise and resources, including telescopes and observatories.

Pixxel is a private space tech company that specialises in launching commercial satellites. Expected to launch this year, it is preparing to deploy Firefly, a constellation of six high-resolution hyperspectral satellites aimed at geospatial analysis. Pixxel has already launched three satellites, enhancing data analysis capabilities across various sectors.

The partnership grants Pixxel access to IIA’s technical expertise and resources, supporting its research and development efforts. This MOU reflects a mutual goal to advance space technology and hyperspectral imaging, aiming to improve industry operations through detailed electromagnetic spectrum analysis.

In a similar intent of collaboration,Indian Space Research Organization (ISRO) and IIA signed an MOU. Signed at the end of last year, it focused on Space Situational Awareness (SSA) and Astrophysics, highlighting the importance of collaborations in space technology development.

Pixxel aims to revolutionise how industries utilise data through its hyperspectral imaging system, overcoming challenges related to data volume management, processing, and storage.

The company, which has raised $76.7M over eight rounds of funding, is developing its Aurora platform to make hyperspectral analysis more accessible and useful for various applications.

The post Pixxel and IIA Join Forces for Advanced Satellite Imaging appeared first on Analytics India Magazine.

Image generated with DALLE-3

In the ever-evolving landscape of technology, the data revolution emerges as a formidable force, reshaping the fabric of industries, economies, and societal norms. Data science and machine learning are at the heart of this transformative surge, serving as crucial catalysts for innovation. They propel us into an era where problem-solving transcends mere human cognition, evolving into a collaborative dance between human intellect and intelligent machines. This article embarks on a comprehensive journey, delving into the emerging trends within data science and machine learning, uncovering the pivotal developments steering us toward a future powered by data.

AI-Powered Automation: Transforming Industries with Intelligent Systems

A significant trend in data science and machine learning revolves around incorporating artificial intelligence (AI) to drive automation. Industries across the spectrum are harnessing the potential of machine learning algorithms to streamline everyday tasks, fine-tune processes, and boost efficiency. Whether in manufacturing, healthcare, finance, or logistics, the wave of AI-powered automation is fundamentally transforming the operational landscape of businesses. This shift trims costs and elevates overall productivity, marking a revolutionary stride in how enterprises navigate their day-to-day functions.

Use Cases

1. Finance:

In finance, automated trading systems have taken center stage, employing the power of machine learning to dissect market trends and seamlessly execute trades in real time. It's a sophisticated technology integration into the dynamic realm of financial markets, ushering in a new era of efficiency and data-driven decision-making.

Image from AISmartz

1. Healthcare:

In healthcare, the incredible capabilities of machine learning algorithms are stepping into pivotal roles. These algorithms are lending a helping hand in diagnostics, offering insights into predictive analytics for patient outcomes, and even contributing to the precision of robotic surgeries. It's a remarkable fusion of technology and medicine that's reshaping the landscape of patient care.

Exponential Growth in Natural Language Processing (NLP)

Natural Language Processing (NLP) has taken center stage in the expansive realm of machine learning. Thanks to strides in deep learning models such as GPT-3, machines are rapidly evolving, displaying a remarkable proficiency in deciphering and generating language that mimics human expression. This transformative trend is reshaping how we engage with technology, from the intuitive responses of chatbots and virtual assistants to the seamless intricacies of language translation and content creation. The newfound ability of machines to grasp and respond to natural language not only redefines our communication landscape but also opens up novel avenues for enhanced accessibility across various domains.

Use Cases

1. Content Generation:

Models like GPT-3 have transformed the landscape of content creation and writing industries by producing text resembling human language. Their influence is palpable, ushering in a new era where artificial intelligence collaborates with writers to craft compelling and coherent content.

Image from AnalyticsVidhya

1. Chatbots and Virtual Assistants:

Natural Language Processing (NLP) plays a pivotal role in the functionality of chatbots such as Siri and virtual assistants like Alexa. It's the magic behind their knack for comprehending and responding to our everyday language queries, making interactions more human and intuitive.

1. Language Translation:

In language translation, Google Translate relies on the finesse of Natural Language Processing (NLP) to deliver precise and accurate translations across various languages. This sophisticated use of technology makes seamless communication possible across linguistic boundaries.

Ethical AI and Responsible Data Science Practices

In the ever-evolving decision-making landscape, the pivotal role of data cannot be overstated. What's increasingly taking the spotlight is the imperative need for ethical considerations in AI and data science. There's a noticeable surge in the recognition of ethical principles as integral elements in the development and deployment phases of machine learning models. Issues such as bias, fairness, transparency, and accountability have risen to the forefront of discussions, shaping the narrative around responsible data science practices. Organizations are actively embracing this ethical shift, adopting frameworks and guidelines that seek to strike a delicate balance between innovation and ethical considerations, steering the course toward a more conscientious era in the world of data.

Use Cases

1. Facial Recognition:

The ethical landscape surrounding facial recognition technology is complex, primarily because of the potential biases inherent in the system. This has prompted a pressing need for conscientious and responsible deployment, as the consequences of biased facial recognition can have profound implications on privacy, security, and social justice.

1. Credit Scoring:

Navigating the terrain of credit scoring with machine learning demands meticulous consideration, as the models involved must be crafted with precision to mitigate any potential discriminatory practices. This conscientious approach is crucial to ensure fairness and equity in lending practices, acknowledging these models' significant impact on individuals' financial opportunities.

Edge Computing and Decentralized Machine Learning

The widespread adoption of Internet of Things (IoT) devices has triggered a notable upswing in data generation right at the edge of networks. A trend gaining significant traction is the fusion of edge computing with decentralized machine learning geared towards processing data near its source. This strategic move holds the promise of curbing latency and optimizing bandwidth usage. Its relevance is especially pronounced in sectors like autonomous vehicles, smart cities, and industrial IoT, where split-second decision-making is paramount. Integrating machine learning models into edge devices is instrumental in fostering systems that are intelligent and highly responsive to real-time demands.

Use Cases

1. Autonomous Vehicles:

In the realm of autonomous vehicles, edge computing has proven transformative. Enabling the swift processing of data directly from sensors empowers these vehicles to make rapid decisions, enhancing their ability to navigate the road with agility and ensuring a level of responsiveness critical to their safe and efficient operation

1. Smart Cities:

Incorporating decentralized machine learning into smart city applications marks a significant stride forward. This innovation facilitates real-time data analysis from various sensors, contributing to the city's overall efficiency by providing timely insights for better decision-making and resource allocation. It exemplifies the seamless technology integration to create more intelligent, responsive urban environments.

Image from TowardsDataScience Interdisciplinary Collaboration and Hybrid Skill Sets

The landscape of data science and machine learning is expanding beyond traditional boundaries, evolving into an interdisciplinary domain. There's a noticeable trend wherein professionals from diverse backgrounds collaborate seamlessly to tackle intricate problems. The demand for hybrid skill sets, amalgamating proficiency in data science, domain-specific knowledge, and effective communication, is steadily increasing. In this interconnected data ecosystem, professionals adept at bridging the gap between technical intricacies and understanding non-technical stakeholders are emerging as increasingly invaluable assets.

Use Cases

1. Healthcare Analytics:

In the intricate realm of healthcare, a dynamic collaboration unfolds as data scientists and healthcare professionals join forces. Together, they sift through vast troves of patient data, applying their combined expertise to glean valuable insights to enhance treatment outcomes and usher in a new era of personalized and effective healthcare solutions.

1. Finance and Data Analysis:

Collaboration emerges at the intersection of finance and data science as professionals with dual expertise unite forces. Together, they channel their knowledge to craft predictive models that delve into the intricate tapestry of market trends, exemplifying a harmonious blend of financial acumen and data-driven insights.

Wrapping it Up

Fueled by data science and machine learning, the ongoing data revolution fundamentally reshapes our daily lives and professional landscapes. Whether it's the advent of AI-powered automation, the increasing emphasis on ethical considerations, or the collaborative synergy of interdisciplinary approaches, the discussed trends provide a nuanced glimpse into these fields' dynamic and ever-evolving nature. Successfully navigating this revolution necessitates a steadfast commitment to staying abreast of developments, embracing responsible practices, and cultivating a culture of perpetual learning. Looking ahead, the convergence of data science and machine learning promises to unravel new possibilities, continuously propelling innovation across diverse industries.

Aryan Garg is a B.Tech. Electrical Engineering student, currently in the final year of his undergrad. His interest lies in the field of Web Development and Machine Learning. He have pursued this interest and am eager to work more in these directions.

More On This Topic

• Exploring the Latest Trends in AI/DL: From Metaverse to Quantum Computing
• The AIoT Revolution: How AI and IoT Are Transforming Our World
• KDnuggets News, July 27: The AIoT Revolution: How AI and IoT Are…
• 5 Key Data Science Trends & Analytics Trends
• Navigating Today’s Data and AI Market Uncertainty
• Navigating Data Science Job Titles: Data Analyst vs. Data Scientist…

Image by atul prajapati from Pixabay

February’s Enterprise Data Transformation Symposium, hosted by Semantic Arts, featured talks from two prominent members of pharma’s Pistoia Alliance: Martin Romacker of Roche and Ben Gardner of AstraZeneca.

It’s been evident for years now that the Pistoia Alliance, organized originally in 2008 by Pfizer, GlaxoSmithKline and Novartis for industry data sharing purposes, has been seeing benefits from its members’ efforts, particularly when it comes to learning from each others’ new best practices.

As Gardner pointed out, science in general has been nudging pharma in the direction of knowledge graph adoption, for good reason. Scientists need to share their findings and scale up their collaborations.

Pharma, considering that it must bring together comprehensive knowledge of the body, mind, spirit and chemistry, may be the most knowledge-intensive industry around. No wonder that pharma pioneered the notion of findable, accessible, interoperable and reusable (FAIR) data in 2016.

FAIR was a necessary step for the industry to be able to make progress on the scientific data sharing front. Data sharing in the most reusable way requires contextualization, so that compatible contexts can complement and contrast effectively with one another.

In that sense, scientists need a contiguous, logically connected, and ordered landscape to explore together with the help of machines, so they can conduct their research in a fully collaborative, unimpeded, expedited fashion. That landscape – contextualized via knowledge graphs designed to snap, grow and evolve together– is the polar opposite of disconnected, opaque, difficult to work with data in silos.

Use case: Food pairing

The Symposium also highlighted other knowledge graph use cases, some of which wouldn’t immediately jump to mind, at least not for me.

Take the notion of food pairing, for example. Foods that provide a novel multisensory experience are a big deal for consumer products companies facing their own challenges on how to innovate and stay relevant. Those companies need to decide which new products to offer, as well as which products to discontinue when ingredient blends fall out of favor.

What’s not as obvious is how big the gap is that relational databases alone are leaving for graph databases to fill when it comes to food pairing trend intelligence. Or how much opportunity lies in a straightforward, methodical knowledge graph-based approach to a food pairing software as a service. Or how other, not-yet-explored opportunities lie in adjacent spaces such as fragrance pairing.

Stratos Kontopoulos, a knowledge graph engineer at FoodPairing AI, described during his presentation the ways in which the company’s in-house data collecting, sifting, contextualizing and analyzing process helped Unilever Knorr, who had partnered with the World Wildlife Fund to uncover innovative ways to establish a more diverse diet.

For the project, Foodpairing collected and evaluated 3.8 million recipes, selecting ten percent of these that met the project’s vegetable-only criteria. Then they brought this data from the selected recipes together with a unifying knowledge model in a knowledge graph.

With the graph and its in-house sensory/food preparation and experience model of taste, texture and smell, etc., Foodpairing was able to effectively disambiguate and characterize the sensory experience of different ingredient blendings and uncover trends that were difficult or impossible to discover at scale using older methods. As the company expands its reach, the articulation and precision of these consumer taste graphs will surely open up new avenues of product innovation exploration.

Use case: Residential real estate

During his presentation, chief architect and co-founder Tavi Truman of RocketUrBiz said real estate agents deal with a lot of manual or near-manual processes. Much of their activity involves capturing accurately and appropriately responding to what was said by buyers, sellers or intermediaries during all parts of the marketing and sales process.

Ironically, those processes are still manual when it comes to the Multiple Listing Service (MLS), email, real-time conversations and 61 different categories of other software agents might be using. The problem is that the data in those applications is siloed and disconnected, so the user experience is also fragmented and wasteful. Sound familiar? That user experience fragmentation is rooted in neglected legacy data architecture.

That’s why Truman and his co-founder, CEO Debra Schwartz, focused their company’s solution on a transformed, knowledge graph-based architecture to be able to unify the user experience.

Key to unifying the user experience has been linguistically clear and logically consistent modeling of marketing, sales and transactions using common logic and Basic Formal Ontology (BFO), an ISO standard, in conjunction with Yet Another Workflow Language (YAWL). Interoperability at the data layer in this sense unifies the user experience.

The company also supports application development in C#, Java or Python so that customers can harness the power of its TrueSpark interoperability platform for the full unified experience.

The importance of modular architecture design

When wrapping up the Symposium, Semantic Arts President Dave McComb noted that many of the talks alluded to how critical a modular approach to modernization is. Foodpairing AI started small but incrementally added novel ingredient trends, combinations and product insights. RocketUrBiz, similarly, has modeled its interoperability platform and workflows a step at a time over years.

The strategy of adoption, as SA’s Mike Atkin pointed out, is an incremental one too. What’s implied here is that the projects the Symposium highlighted require consistent support of those who are committed to long-term transformation.

While security practitioners will reap the benefits of AI, it’s equally likely that cybercriminals will explore ways to wield it as yet another weapon in their arsenal. AI will no doubt expand organisations’ attack surfaces as bad actors push its uses to new extremes, according to Splunk’s Digital Resilience Report.

“Generative AI is poised to enhance the portfolios and tactics of malicious actors. In 2024, we foresee the emergence of novel attack methods, where AI will not be the sole instrument introducing new threats as the robust adoption of 5G in India will also broaden the attack surface in ways that currently lack adequate protection, therefore presenting more opportunities for cybercriminals,” said Robert Pizzari, group vice president, strategic advisor, Asia Pacific, Splunk.

Here are some of the key trends in security and observability that have been outlined by Splunk for 2024:

CISOs will have more at stake: In 2024, CISOs will also have more at stake as the regulatory environment becomes more stringent, more complex and harder to navigate. 79% of line-of-business stakeholders see the security team as either a trusted source of information or a key enabler of the organisation’s mission. (from State of Security 2023).

AI will take on security tasks: Recent research from Splunk’s CISO Report revealed that 86% of security leaders believe generative AI will alleviate skills gaps and talent shortages. AI will be more like that assistant you can’t function without, taking on repetitive, mundane and labour-intensive tasks.

CIOs and CTOs will cut back on their architecture and infrastructure spending, making this the year of mindful budgets and massive disruption: Though people are excited about AI, they are also nervous – CIOs and CTOs will feel the demand to get more from less.

AI will change the way we detect and identify anomalies — it won’t replace manual troubleshooting’: AI will bring a more concise understanding of what’s going on in an environment. First AI will tackle anomaly detection, next up will be investigation and automated response. We will see automated remediation in the near future.

Observability becomes a meaningful signal to security operations: For many vendors, observability products are completely separate from security products. Customers are often frustrated by their lack of interoperability. Whether your servers live in the cloud or a back corner of your garage, a DevSecOps mindset will lead your organisation — big or small — toward digital resilience.

The post AI Will Open a Pandora’s Box of Escalating Privacy and Security Woes: Splunk Report appeared first on Analytics India Magazine.

Image by Editor

Wanting to learn a new skill or sector can be daunting, especially when paying a hefty cost. This is why KDnuggets are here to help you. Rather than going down the constant rabbit hole of trying to find the right course for you to learn about AI, we have compiled a list of YouTube channels that can kickstart your learning, without any cost.

Want it to be fun and interactive — we have YouTube channels for that. Want it to be highly educational and offer courses, we have that too. Want to turn into an AI expert for free?

Check out these YouTube channels:

Matt Wolfe

Link: Matt Wolfe YouTube Channel

With currently 214 videos, Matt Wolfe is a tech fanatic, and all he does is talk about tech. If you want to learn about AI, what’s happening in the sector, what to look out for and the future of AI — check out Matt Wolfe's YouTube channel. His videos will discuss news in the AI sector, whilst also reviewing tools and products — saving you a lot of the hard work. Learn about ChatGPT, AI Music, Generative Art and more tutorials with a no-code and futurism concept.

AI Explained

Link: AI Explained YouTube Channel

Another YouTube channel that goes into simplifying complex AI processes, products, and mechanisms for the everyday person. You can be a beginner in the AI world or a completely experienced tech enthusiast — this channel caters to all. You will get to sit back and watch ultimate reviews on new AI products, as well as what these new products and services mean for the future.

Two Minute Papers

Link: Two Minute Papers YouTube Channel

Are you trying to learn about AI but are unfortunately too busy to find the time? Have a look at the Two Minute Papers YouTube channel which goes into the latest AI and machine learning research projects based on papers — and explained in 2 minutes. With over 1.5 million subscribers, with Two Minute Papers, you will learn about AI research papers in a fun and interactive way.

DeepLearning.AI

Link: DeepLearning.AI YouTube Channel

Founded in 2017 by Andrew Ng, the DeepLearning.AI platform has become one of the most fast-growing and popular learning platforms for AI. The platform aims to fill the need for world-class AI education by creating high-quality AI programs and fostering a tight-knit community.

The AI Advantage

Link: The AI Advantage YouTube Channel

Ever looked at AI tools and services and thought how could I implement this into my day-to-day life or make use of it? You can learn now with the AI Advantage YouTube channel which goes into how you can leverage AI to have a competitive advantage in the business landscape as well as how you can make your tasks more productive for you.

MattVidPro AI

Link: MattVidPro AI YouTube Channel

If you want to learn about AI so that you can keep up with the times and not fall behind, I would highly recommend the MattVidPro AI YouTube channel. This channel will provide you with an in-depth coverage of all AI technologies, especially the newest ones and their capabilities. He also goes into practical guides on how you can make use of these AI tools — so you don’t have to figure it out for yourself.

Siraj Raval

Link: Siraj Raval YouTube Channel

Maybe you want to see the fun side of these AI tools. Well, now you can with Siraj Raval’s YouTube channel which blends the educational side of AI with entertainment. He makes learning about AI and its concepts fun and interactive. Find out the fun and engaging things you can create with AI tools such as ChatGPT.

Wrapping it up

Learning does not have to be highly technical and always through processes. Learning through interactive videos, opinionated pieces and tutorials are also a good way to gauge the sector you’re interested in, as well as keeping up to date with what’s going on in the world of AI.

If you know of any other great YouTube channels that our readers can benefit from, drop them in the comments!

Nisha Arya is a Data Scientist and Freelance Technical Writer. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.

More On This Topic

• Top YouTube Channels for Learning Data Science
• KDnuggets News 22:n16, Apr 20: Top YouTube Channels for Learning…
• Top 15 YouTube Channels to Level Up Your Machine Learning Skills
• Top 7 YouTube Courses on Data Analytics
• The Best Courses for AI from Universities with YouTube Playlists
• Simplifying Decision Tree Interpretability with Python & Scikit-learn