Рубрика: AI

An integral part of ML Engineering is building reliable and scalable procedures for extracting data, transforming it, enriching it and loading it in a specific file store or database. This is one of the components in which the data scientist and the ML engineer collaborate the most. Typically, the data scientist comes up with a rough version of what the data set should look like. Ideally, not on a Jupyter notebook. Then, the ML engineer joins this task to support making the code more readable, efficient and reliable.

ML ETLs can be composed of several sub-ETLs or tasks. And they can be materialized in very different forms. Some common examples:

• Scala-based Spark job reading and processing event log data stored in S3 as Parquet files and scheduled through Airflow on a weekly basis.
• Python process executing a Redshift SQL query through a scheduled AWS Lambda function.
• Complex pandas-heavy processing executed through a Sagemaker Processing Job using EventBridge triggers.

Entities in ETLs

We can identify different entities in these types of ETLs, we have Sources (where the raw data lives), Destinations (where the final data artifact gets stored), Data Processes (how the data gets read, processed and loaded) and Triggers (how the ETLs get initiated).

• Under the Sources, we can have stores such as AWS Redshift, AWS S3, Cassandra, Redis or external APIs. Destinations are the same.
• The Data Processes are typically run under ephemeral Docker containers. We could add another level of abstraction using Kubernetes or any other AWS managed service such as AWS ECS or AWS Fargate. Or even SageMaker Pipelines or Processing Jobs.You can run these processes in a cluster by leveraging specific data processing engines such as Spark, Dask, Hive, Redshift SQL engine. Also, you can use simple single-instance processes using Python processes and Pandas for data processing. Apart from that, there are some other interesting frameworks such as Polars, Vaex, Ray or Modin which can be useful to tackle intermediate solutions.
• The most popular Trigger tool is Airflow. Others that can be used are Prefect, Dagster, Argo Workflows or Mage.

Should I use a Framework?

A framework is a set of abstractions, conventions and out-of-the-box utilities that can be used to create a more uniform codebase when applied to concrete problems. Frameworks are very convenient for ETLs. As we’ve previously described, there are very generic entities that could potentially be abstracted or encapsulated to generate comprehensive workflows.

The progression that I would take to build an internal data processing framework is the following:

• Start by building a library of connectors to the different Sources and Destinations. Implement them as you need them throughout the different projects you work on. That’s the best way to avoid YAGNI.
• Create simple and automated development workflow that allows you to iterate quickly the codebase. For example, configure CI/CD workflows to automatically test, lint and publish your package.
• Create utilities such as reading SQL scripts, spinning up Spark sessions, dates formatting functions, metadata generators, logging utilities, functions for fetching credentials and connection parameters and alerting utilities among others.
• Choose between building an internal framework for writing workflows or use an existing one. The complexity scope is wide when considering this in-house development. You can start with some simple conventions when building workflows and end up building some DAG-based library with generic classes such as Luigi or Metaflow. These are popular frameworks that you can use.

Building a “Utils” Libraries

This is a critical and central part of your data codebase. All your processes will use this library to move data around from one source into another destination. A solid and well-though initial software design is key.

But why would we want to do this? Well, the main reasons are:

• Reusability: Using the same software components in different software projects allows for higher productivity. The piece of software has to be developed only once. Then, it can be integrated into other software projects. But this idea is not new. We can find references back in 1968 on a conference whose aim was to solve the so-called software crisis.
• Encapsulation: Not all the internals of the different connectors used through the library need to be shown to end-users. Hence, by providing an understandable interface, that’s enough. For example, if we had a connector to a database, we wouldn’t like that the connection string got exposed as a public attribute of the connector class. By using a library we can ensure that secure access to data sources is guaranteed. Review this bit
• Higher-quality codebase: We have to develop tests only once. Hence, developers can rely on the library because it contains a test suite (Ideally, with a very high test coverage). When debugging for errors or issues we can ignore, at least at first pass, that the issue is within the library if we’re confident on our test suite.
• Standardisation / “Opinionation”: Having a library of connectors determines, in certain way, the way you develop ETLs. That is good, because ETLs in the organization will have the same ways of extracting or writing data into the different data sources. Standardisation leads to better communication, more productivity and better forecasting and planning.

When building this type of library, teams commit to maintain it over time and assume the risk of having to implement complex refactors when needed. Some causes of having to do these refactors might be:

• The organisation migrates to a different public cloud.
• The data warehouse engine changes.
• New dependency version breaks interfaces.
• More security permission checks need to be put in place.
• A new team comes in with different opinions about the library design.a

Interface classes

If you want to make your ETLs agnostic of the Sources or Destinations, it is a good decision to create interface classes for base entities. Interfaces serve as template definitions.

For example, you can have abstract classes for defining required methods and attributes of a DatabaseConnector. Let’s show a simplified example of how this class could look like:

from abc import ABC        class DatabaseConnector(ABC):            def __init__(self, connection_string: str):          self.connection_string = connection_string        @abc.abstractmethod      def connect(self):          pass              @abc.abstractmethod      def execute(self, sql: str):          pass

Other developers would subclass from the DatabaseConnector and create new concrete implementations. For instance, a MySqlConnector or CassandraDbConnector could be implemented in this fashion. This would help end-users to quickly understand how to use any connector subclassed from the DatabaseConnector as all of them will have the same interface (same methods).

mysql = MySqlConnector(connection_string)  mysql.connect()  mysql.execute("SELECT * FROM public.table")    cassandra = CassandraDbConnector(connection_string)  cassandra.connect()  cassandra.execute("SELECT * FROM public.table")

Simples interfaces with well-named methods are very powerful and allow for better productivity. So my advice is to spend quality time thinking about it.

The right documentation

Documentation not only refers to docstrings and inline comments in the code. It also refers to the surrounding explanations you give about the library. Writing a bold statement about what’s the end goal of the package and a sharp explanation of the requirements and guidelines to contribute is essential.

For example:

"This utils library will be used across all the ML data pipelines and feature engineering jobs to provide simple and reliable connectors to the different systems in the organization".

Or

"This library contains a set of feature engineering methods, transformations and algorithms that can be used out-of-the-box with a simple interface that can be chained in a scikit-learn-type of pipeline".

Having a clear mission of the library paves the way for a correct interpretation from contributors. This is why open source libraries (E.g: pandas, scikit-learn, etc) have gained such a great popularity these last years. Contributors have embraced the goal of the library and they are committed to follow the outlined standards. We should be doing something pretty similar at organizations.

Right after the mission is stated, we should develop the foundational software architecture. How do we want our interfaces to look like? Should we cover functionality through more flexibility in the interface methods (e.g: more arguments that lead to different behaviours) or more granular methods (e.g: each method has a very specific function)?

After having that, the styleguide. Outline the preferred modules hierarchy, the documentation depth required, how to publish PRs, coverage requirements, etc.

With respect to documentation in the code, docstrings need to be sufficiently descriptive of the function behaviour but we shouldn’t fall into just copying the function name. Sometimes, the function name is sufficiently expressive that a docstring explaining its behaviour is just redundant. Be concise and accurate. Let’s provide a dumb example:

❌No!

class NeptuneDbConnector:  	...  	def close():  	    """This function checks if the connection to the database               is opened. If it is, it closes it and if it doesn’t,               it does nothing.            """  

✅Yes!

class NeptuneDbConnector:  	...  	def close():  	    """Closes connection to the database."""

Coming to the topic of inline comments, I always like to use them to explain certain things that might seem weird or irregular. Also, if I have to use a complex logic or fancy syntax, it is always better if you write a clear explanation on top of that snippet.

# Getting the maximum integer of the list  l = [23, 49, 6, 32]  reduce((lambda x, y: x if x > y else y), l)

Apart from that, you can also include links to Github issues or Stackoverflow answers. This is really useful, specially if you had to code a weird logic just to overcome a known dependency issue. It is also really convenient when you had to implement an optimisation trick that you got from Stackoverflow.

These two, interface classes and clear documentation are, in my opinion, the best ways to keep a shared library alive for a long time. It will resist lazy and conservative new developers and also fully-energized, radical and highly opinionated ones. Changes, improvements or revolutionary refactors will be smooth.

Applying Software Design Patterns to ETLs

From a code perspective, ETLs should have 3 clearly differentiated high-level functions. Each one related to one of the following steps: Extract, Transform, Load. This is one of the simplest requirements for ETL code.

def extract(source: str) -> pd.DataFrame:      ...    def transform(data: pd.DataFrame) -> pd.DataFrame:      ...      def load(transformed_data: pd.DataFrame):      ...

Obviously, it is not mandatory to name these functions like this, but it will give you a plus on readability as they are widely accepted terms.

DRY (Don’t Repeat Yourself)

This is one of the great design patterns which justifies a connectors library. You write it once and reuse it across diferent steps or projects.

Functional Programming

This is a programming style that aims at making functions “pure” or without side-effects. Inputs must be immutable and outputs are always the same given those inputs. These functions are easier to test and debug in isolation. Therefore, provides a better degree of reproducibility to data pipelines.

With functional programming applied to ETLs, we should be able to provide idempotency. This means that every time we run (or re-run) the pipeline, it should return the same outputs. With this characteristic, we are able to confidently operate ETLs and be sure that double runs won’t generate duplicate data. How many times you had to create a weird SQL query to remove inserted rows from a wrong ETL run? Ensuring idempotency helps avoiding those situations. Maxime Beauchemin, creator of Apache Airflow and Superset, is one known advocate for Functional Data Engineering.

SOLID

We will use references to classes definitions, but this section can also be applied to first-class functions. We will be using heavy object-oriented programming to explain these concepts, but it doesn’t mean this is the best way of developing an ETL. There’s not a specific consensus and each company does it on its own way.

Regarding the Single Responsibility Principle, you must create entities that have only one reason to change. For example, segregating responsibilities among two objects such as a SalesAggregator and a SalesDataCleaner class. The latter is susceptible to contain specific business rules to “clean” data from sales, and the former is focused on extracting sales from disparate systems. Both classes code can change because of different reasons.

For the Open-Close Principle, entities should be expandable to add new features but not opened to be modified. Imagine that the SalesAggregator received as components a StoresSalesCollector which is used to extract sales from physical stores. If the company started selling online and we wanted to get that data, we would state that SalesCollector is open for extension if it can receive also another OnlineSalesCollector with a compatible interface.

from abc import ABC, abstractmethod        class BaseCollector(ABC):        @abstractmethod        def extract_sales() -> List[Sale]:              pass    class SalesAggregator:  	        def __init__(self, collectors: List[BaseCollector]):  		self.collectors = collectors  	        def get_sales(self) -> List[Sale]:   		sales = []  		for collector in self.collectors:  			sales.extend(collector.extract_sales())  		return sales    class StoreSalesCollector:  	def extract_sales() -> List[Sale]:  		# Extract sales data from physical stores    class OnlineSalesCollector:  	def extract_sales() -> List[Sale]:  		# Extract online sales data    if __name__ == "__main__":       sales_aggregator = SalesAggregator(              collectors = [                  StoreSalesCollector(),                  OnlineSalesCollector()              ]       sales = sales_aggregator.get_sales()

The Liskov substitution principle, or behavioural subtyping is not so straightforward to apply to ETL design, but it is for the utilities library we mentioned before. This principle tries to set a rule for subtypes. In a given program that uses the supertype, one could potential substitute it with one subtype without altering the behaviour of the program.

from abc import ABC, abstractmethod      class DatabaseConnector(ABC):  	def __init__(self, connection_string: str):  		self.connection_string = connection_string    	@abstractmethod  	def connect():  		pass    	@abstractmethod  	def execute_(query: str) -> pd.DataFrame:  		pass      class RedshiftConnector(DatabaseConnector):  	def connect():  	# Redshift Connection implementation    	def execute(query: str) -> pd.DataFrame:  	# Redshift Connection implementation      class BigQueryConnector(DatabaseConnector):  	def connect():  	# BigQuery Connection implementation    	def execute(query: str) -> pd.DataFrame:  	# BigQuery Connection implementation      class ETLQueryManager:  	def __init__(self, connector: DatabaseConnector, connection_string: str):  		self.connector = connector(connection_string=connection_string).connect()    	def run(self, sql_queries: List[str]):  		for query in sql_queries:  			self.connector.execute(query=query)

We see in the example below that any of the DatabaseConnector subtypes conform to the Liskov substitution principle as any of its subtypes could be used within the ETLManager class.

Now, let’s talk about the Interface Segregation Principle. It states that clients shouldn’t depend on interfaces they don’t use. This one comes very handy for the DatabaseConnector design. If you’re implementing a DatabaseConnector, don’t overload the interface class with methods that won’t be used in the context of an ETL. For example, you won’t need methods such as grant_permissions, or check_log_errors. Those are related to an administrative usage of the database, which is not the case.

The one but not least, the Dependency Inversion principle. This one says that high-level modules shouldn’t depend on lower-level modules, but instead on abstractions. This behaviour is clearly exemplified with the SalesAggregator above. Notice that its __init__ method doesn’t depend on concrete implementations of either StoreSalesCollector or OnlineSalesCollector. It basically depends on a BaseCollector interface.

How does a great ML ETL look like?

We’ve heavily rely on object-oriented classes in the examples above to show ways in which we can apply SOLID principles to ETL jobs. Nevertheless, there is no general consensus of what’s the best code format and standard to follow when building an ETL. It can take very different forms and it tends to be more a problem of having an internal well-documented opinionated framework, as discussed previously, rather than trying to come up with a global standard across the industry.

Hence, in this section, I will try to focus on explaining some characteristics that make ETL code more legible, secure and reliable.

Command Line Applications

All Data Processes that you can think of are basically command line applications. When developing your ETL in Python, always provide a parametrized CLI interface so that you can execute it from any place (E.g, a Docker container that can run under a Kubernetes cluster). There are a variety of tools for building command-line arguments parsing such as argparse, click, typer, yaspin or docopt. Typer is possibly the most flexible, easy to use an non-invasive to your existing codebase. It was built by the creator of the famous Python web services library FastApi, and its Github starts keep growing. The documentation is great and is becoming more and more industry-standard.

from typer import Typer    app = Typer()      @app.command()  def run_etl(      environment: str,      start_date: str,      end_date: str,      threshold: int  ):      ...

To run the above command, you’d only have to do:

python {file_name}.py run-etl --environment dev --start-date 2023/01/01 --end-date 2023/01/31 --threshold 10

Process vs Database Engine Compute Trade Off

The typical recommendation when building ETLs on top of a Data Warehouse is to push as much compute processing to the Data Warehouse as possible. That’s all right if you have a data warehouse engine that autoscales based on demand. But that’s not the case for every company, situation or team. Some ML queries can be very long and overload the cluster easily. It’s typical to aggregate data from very disparate tables, lookback for years of data, perform point-in-time clauses, etc. Hence, pushing everything to the cluster is not always the best option. Isolating the compute into the memory of the process instance can be safer in some cases. It is risk-free as you won’t hit the cluster and potentially break or delay business-critical queries. This is an obvious situation for Spark users, as all the compute & data gets distributed across the executors because of the massive scale they need. But if you’re working over Redshift or BigQuery clusters always keep an eye into how much compute you can delegate to them.

Track Outputs

ML ETLs generate different types of output artifacts. Some are Parquet files in HDFS, CSV files in S3, tables in the data warehouse, mapping files, reports, etc. Those files can later be used to train models, enrich data in production, fetch features online and many more options.

This is quite helpful as you can link dataset building jobs with training jobs using the identifier of the artifacts. For example, when using Neptune track_files() method, you can track these kind of files. There’s a very clear example here that you can use.

Implement Automatic Backfilling

Imagine you have a daily ETL that gets last day’s data to compute a feature used to train a model If for any reason your ETL fails to run for a day, the next day it runs you would have lost the previous day data computed.

To resolve this, it’s a good practice to look at what’s the last registered timestamp in the destination table or file. Then, the ETL can be executed for those lagging two days.

Develop Loosely Coupled Components

Code is very susceptible to change, and processes that depend on data even more. Events that build up tables can evolve, columns can change, sizes can increase, etc. When you have ETLs that depend on different sources of information is always good to isolate them in the code. This is because if at any time you have to separate both components as two different tasks (E.g: One needs a bigger instance type to run the processing because the data has grown), it is much easier to do if the code is not spaghetti!

Make Your ETLs Idempotent

It’s typical to run the same process more than once in case there was an issue on the source tables or within the process itself. To avoid generating duplicate data outputs or half-filled tables, ETLs should be idempotent. That is, if you accidentally run the same ETL twice with the same conditions that the first time, the output or side-effects from the first run shouldn’t be affected (ref). You can ensure this is imposed in your ETL by applying the delete-write pattern, the pipeline will first delete the existing data before writing new data.

Keep Your ETLs Code Succinct

I always like to have a clear separation between the actual implementation code from the business/logical layer. When I’m building an ETL, the first layer should be read as a sequence of steps (functions or methods) that clearly state what is happening to the data. Having several layers of abstraction is not bad. It’s very helpful if you have have to maintain the ETL for years.

Always isolate high-level and low-level functions from each other. It is very weird to find something like:

from config import CONVERSION_FACTORS    def transform_data(data: pd.DataFrame) -> pd.DataFrame:      data = remove_duplicates(data=data)      data = encode_categorical_columns(data=data)      data["price_dollars"] = data["price_euros"] * CONVERSION_FACTORS["dollar-euro"]      data["price_pounds"] = data["price_euros"] * CONVERSION_FACTORS["pound-euro"]      return data

In the example above we are using high-level functions such as the “remove_duplicates” and “encode_categorical_columns” but at the same time we’re explicitly showing an implementation operation to convert the price with a conversion factor. Wouldn’t it be nicer to remove those 2 lines of code and replace them with a “convert_prices” function?

from config import CONVERSION_FACTOR    def transform_data(data: pd.DataFrame) -> pd.DataFrame:      data = remove_duplicates(data=data)      data = encode_categorical_columns(data=data)      data = convert_prices(data=data)      return data

In this example, readability wasn’t a problem, but imagine that instead, you embed a 5 lines long groupby operation in the “transform_data” along with the “remove_duplicates” and “encode_categorical_columns”. In both cases, you’re mixing high-level and low-level functions. It is highly recommended to keep a cohesive layered code. Sometimes is inevitable and over-engineered to keep a function or module 100% cohesively layered, but it’s a very beneficial goal to pursue.

Use Pure Functions

Don’t let side-effects or global states complicate your ETLs. Pure functions return the same results if the same arguments are passed.

❌The function below is not pure. You’re passing a dataframe that is joined with another functions that is read from an outside source. This means that the table can change, hence, returning a different dataframe, potentially, each time the function is called with the same arguments.

def transform_data(data: pd.DataFrame) -> pd.DataFrame:      reference_data = read_reference_data(table="public.references")      data = data.join(reference_data, on="ref_id")      return data

To make this function pure, you would have to do the following:

def transform_data(data: pd.DataFrame, reference_data: pd.DataFrame) -> pd.DataFrame:      data = data.join(reference_data, on="ref_id")      return data

Now, when passing the same “data” and “reference_data” arguments, the function will yield the same results.

This is a simple example, but we all have witnessed worse situations. Functions that rely on global state variables, methods that change the state of class attributes based on certain conditions, potentially changing the behaviour of other upcoming methods in the ETL, etc.

Maximising the use of pure functions leads to more functional ETLs. As we have already discussed in points above, it comes with great benefits.

Paremetrize As Much As You Can

ETLs change. That’s something that we have to assume. Source table definitions change, business rules change, desired outcomes evolve, experiments are refined, ML models require more sophisticated features, etc.

In order to have some degree of flexibility in our ETLs, we need to thoroughly assess where to put most of the effort to provide parametrised executions of the ETLs. Parametrisation is a characteristic in which, just by changing parameters through a simple interface, we can alter the behaviour of the process. The interface can be a YAML file, a class initialisation method, function arguments or even CLI arguments.

A simple straightforward parametrisation is to define the “environment”, or “stage” of the ETL. Before running the ETL into production, where it can affect downstream processes and systems, it’s good to have a “test”, “integration” or “dev” isolated environments so that we can test our ETLs. That environment might involve different levels of isolation. It can go from the execution infrastructure (dev instances isolated from production instances), object storage, data warehouse, data sources, etc.

That’s an obvious parameter and probably the most important. But we can expand the parametrisation also to business-related arguments. We can parametrise window dates to run the ETL, columns names that can change or be refined, data types, filtering values, etc.

Just The Right Amount Of Logging

This is one of the most underestimated properties of an ETL. Logs are useful to detect production executions anomalies or implicit bugs or explain data sets. It’s always useful to log properties about extracted data. Apart from in-code validations to ensure the different ETL steps run successfully, we can also log:

• References to source tables, APIs or destination paths (E.g: “Getting data from `item_clicks` table”)
• Changes in expected schemas (E.g: “There is a new column in `promotion` table”)
• The number of rows fetched (E.g: “Fetched 145234093 rows from `item_clicks` table”)
• The number of null values in critical columns (E.g: “Found 125 null values in Source column”)
• Simple statistics of data (e.g: mean, standard deviation, etc). (E.g: “CTR mean: 0.13, CTR std: 0.40)
• Unique values for categorical columns (E.g: “Country column includes: ‘Spain’, ‘France’ and ‘Italy’”)
• Number of rows deduplicated (E.g: “Removed 1400 duplicated rows”)
• Execution times for compute-intensive operations (E.g: “Aggregation took 560s”)
• Completion checkpoints for different stages of the ETL (e.g: “Enrichment process finished successfully”)

Manuel Martín is an Engineering Manager with more than 6 years of expertise in data science. He have previously worked as a data scientist and a machine learning engineer and now I lead the ML/AI practice at Busuu.

Manuel Martín is an Engineering Manager with more than 6 years of expertise in data science. He have previously worked as a data scientist and a machine learning engineer and now I lead the ML/AI practice at Busuu.

More On This Topic

• Schedule & Run ETLs with Jupysql and GitHub Actions
• Software Engineering Best Practices for Data Scientists
• Awesome Tricks And Best Practices From Kaggle
• Can Data Science Be Agile? Implementing Best Agile Practices to Your Data…
• MLOps Best Practices
• MLOps: The Best Practices and How To Apply Them

The use of AI in the energy sector is being used through the entire process. Data driven solutions are already being used in seismic processing, predictive maintenance, operational optimization, high-frequency trading, reservoir simulation, and computational chemistry just to name a few.

Chiranjib Sur who is the Head of Engineering and Scientific Software at Shell explains just how they are doing this at scale. He has been working with the company for over eleven years as a computational and data scientist. With a PhD in Physics, he brings a profound understanding in multidisciplinary fields. He spoke about what and who is responsible for scaling the software in the energy sector.

Data driven decision making in the energy sector

“What needs to be done isn’t really rocket science.” explains Chiranjib Sur. “We apply some heuristic principles, some rules and human intelligence. Even Elon Musk, the actual rocket scientist does the same,” he jokes.

To gain insight first it is important to set the problem statement. “The problem with data is that we can represent the same data in multiple ways, both the insights might be accurate and yet very different.”

He then asked,“The next question is, If we want to make the right decision for the next business model, based on some data, how do we do that?” In the context of Shell, Gur gave the example of trying to figure out how many barrels of oil will be produced in the near future based on the history of oil production of the specific region.

“Now there are two parts – data and science,” he says. With the energy sector the ‘science’ part of it requires interdisciplinary expertise. “To find out where the oil pockets are, we need to understand geology,” he said.

This is the first step. Understanding that data doesn’t work independently but in conjunction with larger context. “Oil extraction isn’t an easy process as they stick to rocks, it is imperative that our probabilities align to give us the maximum accuracy,” he explains.

From millions of parameters in the database a few hundred observations are made leading to the final decision that leads to a prediction that the net amount of oil from this well would be in a range.

Without getting into very technical details he tried to convey that different kinds of data driven techniques start with a stochastic optimisation. Solving this, leads to structured data from which you can make observations by predictive modeling.

How this becomes an engineering problem is not to do with the data itself but how the data is hosted and communicated. When we deal with data that’s as large as 100 terabytes, we have to figure out a way that data scientists and AI engineers have to work with that finally has to benefit the business model.

“With all this accuracy is of utmost importance. We cannot afford to say that this message is up for predictive maintenance. We need to be deterministic. And to be deterministic, we need to be right,” he explains.

To do this at scale

In the case of Shell, we collaborate and work with experts in each field. “This is an interdisciplinary field and we get different teams to collaborate,” Gur explains. Recently it was announced that Shell is collaborating with C3 AI. Scale is a critical challenge in the oil and gas industry, and this collaboration is addressing it effectively is a common objective.

“AI contributes to the evolution of predictive analytics, enabling the system to teach and learn from itself, ultimately resulting in better and more accurate predictions.” Gur concluded.

The post Cypher 2023: Scaling Data Science in the Energy Sector appeared first on Analytics India Magazine.

Earth Observation (EO) datasets have emerged as invaluable tools for understanding and monitoring our planet’s dynamic processes. Radha Krishna Kavuluru Scientist – ‘SD’ ( Project Manager: NISAR ) at ISRO, at Cypher 2023, India’s biggest AI conference, delves into the Earth Observation Datasets created by ISRO.

Addressing a packed audience mostly from the AI community, Kavuluru said, “Your data is different from our data and probably some of you may already use drone data or satellite imagery.”

Use case of EO datasets.

The first use case of EO datasets, according to Kavuluru, is around climate change which is critical in today’s age. Another critical application of satellite data in agriculture is the monitoring of vegetation health. By analysing near-infrared and other spectral data, remote sensing can provide valuable information about the vigour and overall health of crops.

Besides, the datasets are also used for disaster management. Earth observation data, particularly from weather and climate satellites, are essential for early warning systems. They help in monitoring the development of natural disasters like hurricanes, typhoons, cyclones, floods, and droughts.

This early detection allows authorities to issue timely alerts and take preventive measures to protect lives and property. Moreover, these datasets are also being used by the government for defence, Kavuluru said.

Bhoonidhi

Bhoonidhi is an initiative by ISRO that aims to provide open access to EO satellite data. It is an online platform that enables users to access and download remote sensing data from various satellites, including Indian and foreign sensors.

“What happens is the data that is collected by ISRO’s ground stations comes into this portal in real-time. There is some analytics built into it.”

However, given there are thousands of datasets and each image could be of the size of 203 gigabytes. Downloading and making sense of the datasets is a challenge, Kavuluru said, “For this companies often talk to cloud service providers or build their own infrastructure.”

Codelab

To tackle this very problem, ISRO is coming up with something called Codelab, which is going to be launched early next year.

“This is almost synonymous with Google Colab, but the additional benefit is the Bhoonidhi portal will be integrated into this and you can leverage the infrastructure of ISRO directly on your browser.”

One advantage it offers is the ability to scale computing and storage horizontally, eliminating the need to download the data.

Kavuluru further adds that the IO datasets have multiple commercial uses and can be leveraged by organisations for their benefits. Satellite data is already reshaping numerous sectors by providing valuable insights and optimising various processes.

“This technology has not only improved efficiency but also empowered businesses and governments to make data-driven decisions with far-reaching implications.”

The post Cypher2023: How Earth Observation Datasets Can Benefit Enterprises? appeared first on Analytics India Magazine.

“Python!”
“No, R.”
“Fools, it’s obviously Rust.”

Many data science learners and experts alike are keen to pin down the very best language for data science. In my opinion, most people are wrong. Amidst the hunt for the newest, the sexiest, the most container-able data science language, people are looking for the wrong thing.

Image from Reddit

It’s easy to overlook. It’s easy to even discount it as a language. But the humble Structured Query Language, or SQL, is my pick for the language to learn for data science. All those other languages certainly have their place, but SQL is the one non-negotiable language that I consider a base requirement for anyone working in data science. Here’s why.

A Universal Language for Databases

Look, databases come hand in hand with data science. It’s in the name. If you’re working with data science, you’re working with databases. And if you’re working with databases, you’re probably working with SQL.

Why? Because SQL is the universal database query language. There is no other. Imagine someone told you that if you just learned a specific language, you’d be able to speak to and understand every single person on Earth. How valuable would that be? SQL is that language in data science, the language that everyone uses to manage and access databases.

Image from X

Every data scientist needs to access and retrieve data, to explore data and build hypotheses, to filter, aggregate, and sort data. And hence, every data scientist will need SQL. As long as you know how to write a SQL query, you’ll go far.

Someone, reading this article right now, is piping up about the NoSQL movement. Indeed, certain data is now more commonly stored in non-relational databases, such as by key-value pairs or graph data. It’s true that there are benefits to storing data like that – you gain more scalability and flexibility. But there’s no standard NoSQL query language. You might learn one for one job, and then need to learn an entirely new one for a new job.

Plus, you will very rarely find a business that works entirely with NoSQL databases, while many companies don’t need non-relational databases.

Cleaning and Processing

There’s that famous (and debunked) stat about how data scientists spend 80% of their time cleaning. While it’s not true, I think if you ask any data scientist what they spend time on, data cleaning will rank in the top five tasks. That’s why this section is the longest.

You can clean and process data with other languages, but SQL in particular offers unique advantages for certain aspects of data cleaning and processing.

SQL's expressive query language allows data scientists to efficiently filter, sort, and aggregate data using concise statements. This level of flexibility is especially useful when dealing with large datasets where manual data manipulation would be time-consuming and error-prone. Compare that to a language like Python, where achieving similar data manipulation tasks might require writing more lines of code and dealing with loops, conditions, and external libraries. While Python is renowned for its versatility and rich ecosystem of data science libraries, SQL's focused syntax can expedite routine data cleaning operations, enabling data scientists to swiftly prepare data for analysis.

Plus, any data scientist will complain about the bane of their existence: missing values. SQL's functions and capabilities for handling missing values—such as using COALESCE, CASE, and NULL handling—provide straightforward approaches to address gaps in data without the need for complex programming logic.

The other bane of a data scientist’s existence is duplicates. Happily, SQL offers efficient methods to identify and eliminate duplicate records from datasets, like the `DISTINCT` keyword and the `GROUP BY` clause.

You’ve probably heard of ETL pipelines. Well, SQL can be used to create data transformation pipelines, which take raw or semi-processed data and convert it into a format suitable for analysis. This is particularly beneficial for automating and standardizing that repetitive data-cleaning processes we all know and hate.

SQL's ability to join tables from different databases or files streamlines the process of merging data for analysis is essential for projects involving data integration or aggregating data from diverse origins. Which, for a data scientist, comprises a majority of projects.

Finally, I like to remind people that data science does not happen in a vacuum. SQL queries are self-contained and can be easily shared with colleagues. This fosters collaboration and ensures that others can reproduce data cleaning steps without manual intervention.

Plays Well with Others

Now, you won’t get far in data science if you only know SQL. But happily, SQL integrates perfectly well with any other of the top data science languages like R, Python, Julia, or Rust. You get all the benefits of analysis, data viz, and machine learning while still retaining SQL’s strength for data manipulation.

Image from LinkedIn

This is especially powerful when you think about all that data cleaning and processing I talked about earlier. You can use SQL to preprocess and clean data directly within databases, and then lean on Python, R, Julia, or Rust to perform more advanced data transformations or feature engineering, leveraging the extensive libraries available.

Many organizations rely on SQL – or, more accurately, rely on data scientists who know how to use SQL – to generate reports, dashboards, and visualizations that inform decision-making. Familiarity with SQL enables data scientists to produce meaningful reports directly from databases. And because SQL is so widespread, these reports are usually compatible and interoperable across almost any system.

Because of how interoperable it is with reporting tools and scripting languages like Python, R, and JavaScript, data scientists can actually automate the reporting processes, seamlessly combining SQL's data extraction and manipulation capabilities with the visualization and reporting features of these languages. The upshot is you get comprehensive and insightful reports that effectively communicate data-driven insights to stakeholders, all inside one place.

Jobs, Jobs, Jobs

There’s a reason you’ll get asked a bunch of SQL interview questions at any data science interview. Almost every data science job requires at least a basic familiarity with SQL.

Here’s an example of what I mean: the job listing says, “Expertise in SQL, and R or Python for data analysis and platform development.” In other words, SQL is a must. And then either R or Python, but one is as good as another to most employers. But thanks to SQL domination, there’s no alternative to SQL. Every data science job will require you to work with SQL.

The really cool thing about it is that it makes SQL the ultimate transferable tool. One job may prefer Python, while a startup might require Rust due to personal preference or legacy infrastructure. But no matter where you go, or what you do, it’s SQL or bust. Take the time to learn it, and you’ll always be able to tick off a job requirement.

Ultimately, if you find a job as a data scientist that doesn’t require SQL, you’re probably not going to be doing a whole lot of data science.

Why Is SQL So Necessary for Data Science?

It really comes down to the database. Data science requires the storage, manipulation, retrieval, and management of a lot of data. That data lives somewhere. It can only be accessed with one tool, normally, and that tool is SQL. SQL is the language to learn for data science and will be for as long as we rely on databases to do data science.
Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn.

Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn.

More On This Topic

• Why You Need To Learn More Than One Programming Language!
• N-gram Language Modeling in Natural Language Processing
• Why and how should you learn "Productive Data Science"?
• Leveraging GPT Models to Transform Natural Language to SQL Queries
• Best Resources to Learn Natural Language Processing in 2021
• Learn Neural Networks for Natural Language Processing Now

In a world where organisations are looking at short term goals when adapting or implementing generative AI solutions, Genpact believes that it’s time they started focusing on long term returns. Katie Stein, the chief strategy officer at Genpact said that organisations are getting on the edge in implementing these solutions.

She also shared Genpact’s generative AI strategy on how they are scaling and deploying them responsibly. “I would argue that generative AI is our moment of fire, our moment to change and transform how businesses run and to change society.” she said.

Drawing a comparison of how society changed with how humankind evolved after the discovery of fire she explained, “Something fundamental changed with the introduction of ChatGPT. It truly democratised and consumerised AI. It changed how the average person thinks about AI and engages with it.”

She acknowledged that this is just the beginning, and also the hard phase of the development of AI, but all of this would eventually determine how we progress with AI.

On the contrary to the popular obsession with productivity and AI, Stein warned about the dangers of hyperfocus on just AI productivity, and immediate result. “I hear a lot about productivity. And I will say the hyper focus on productivity is dangerous, limiting and will leave AI in the same station,” she added.

She reasons that so much digital transformation should be thought of as end-to-end solutions and not just immediate outcomes. And as a result, companies are rushing to optimise and put in point solutions.

She also said that the companies need to step back and reimagine the kinds of outcomes, the economic values that AI can create for them and maybe even completely change the way they collaborate or work with different models – “not just for the immediate future but in the long run.”

Towards Inclusive and Responsible Generative AI

As the fear continues to grow around AI replacing people, Stein touched upon certain points during her talk suggesting an inclusive way for corporations to move towards AI advancements.

She urged for organisation, whether it’s through external parties, or through in-house platforms to think about how they decide to skilling at scale for the emerging roles.

“So when I think about the skills that are going to go into this next phase, it’s a new set of skills, you have prompt engineers. But we also need more experienced designers to make better user interfaces and extend experience. We need architects, we need strategists. There’s many roles we need to train and create” she advised.

Stein who has been with Genpact for almost a decade now, further shed light upon ways organisations should pick AI models as per their need instead of jumping on the bandwagon as others.

“Many organisations haven’t yet understood how fundamental this change is for them,” she said. “At Genpact we’re working with our responsibility AI framework, which has the foundational elements around security and privacy,” Stein added.

She raised several significant questions — how do we make sure there isn’t bias in models as we train them? How do we make sure that bias does not creep in as we use AI for HR or recruiting? How do we monitor that going on?

Further, she suggests for organisations to build responsible AI they need to put in place terms, strategies, metrics and guardrails that can be used for the base.

Lastly, she spoke about the role of globalisation in the regulatory framework. In the last one year government bodies around the world have begun the dialogues to come up with regulations for AI. “It is very unsure at this stage, and will continue to evolve and fit,” Stein concluded.

The post Cypher 2023: Genpact Sets the Stage on Fire with Long Term Vision for Generative AI 🔥 appeared first on Analytics India Magazine.

The rise of generative AI has sparked extensive debates regarding the potential displacement of jobs. Whether it’s ChatGPT affecting roles in coding and human resource management, or DALLE and Midjourney’s influence on the field of graphic design, this discourse has been ongoing for some time.

Raghav Gupta, co-founder & CEO at Futurense Technologies, in his talk titled, ‘Future-Proofing Careers in the Age of Automation’, during the ongoing Cypher2023 event, which is India’s largest AI conference, delves into the same discourse and talks mentions that AI should be seen as an opportunity rather than a threat.

“Today, I’d like to discuss a topic that’s been on many of our minds and a subject we’ve been hearing and talking about extensively. I won’t offer a highly technical perspective; instead, I’ll provide a pragmatic and emotional approach to navigating job displacement in the age of AI,” Gupta said.

Impact on Futurense

Futurense Technologies is a company that offers recruitment, talent transformation, and career acceleration services. They provide a platform for talent to unlock their true potential. Gupta says his customer service team would take nearly 1800 calls a day with clients as the first process of screening. “However, now we have added a layer of AI bot that makes sure the first level of screening happens through the bot. So now the average calls that the team does has decreased from 1900 to roughly 750 to 800,” Gupta said.

Similarly, Futurense’s marketing team previously hired content writers and graphic designers to do different jobs. “Now, we have one specialist who knows how to use ChatGPT and Midjourney. The same individual can be a content writer and a graphic designer in an organisation.”

“While we had 18 members in our marketing team, now we have 21, but the amount of content we are dispensing has quadrupled. We are also increasing our output, which in turn, is on average increasing the number of jobs.”

AI will create newer jobs

Gupta states eventually everyone will have to adapt to AI, but those early adopters will have an advantage. While there might be some job displacements initially, AI will also create newer jobs. For insurance, an AI trainer and operator.

“AI is as good as the model it is trained on and this training of that model is a continuous process. Till now only very big multibillion-dollar organisations were training the model, but we will see small businesses start having these jobs where they will train the data, the model.”

“AI’s predictive abilities are remarkable, but they occasionally offer broad predictions that may not precisely fit your specific business needs. This calls for individuals or a team who could be likened to modern-age McKinsey consultants. So, Gupta said we will have an AI prediction auditor, an individual or a group of individuals who will possess a unique skill set, enabling them to grasp the intricacies of your business context while also comprehending the subtleties of macro socio-economic developments.”

In conclusion, Gupta encourages everybody to embrace AI as part of their professional journey, and soon they will realise that it’s not a threat but a tremendous opportunity. While AI adoption will eventually become widespread, those who adapt quickly and effectively will reap the greatest rewards.

The post Cypher2023: Future-Proofing Careers in the Age of Automation appeared first on Analytics India Magazine.

In aerospace, the transformation of aircraft production is an intricate dance between human ingenuity and cutting-edge technology. Established on 23 December 1940, Hindustan Aeronautics Limited (HAL) is one of the oldest and largest aerospace and defence manufacturers in the world.

Himagiri Gedela, head of projects at HAL, during the ongoing Cypher 2023 event, which is the largest AI conference in India, delves into how integrating the power of simulation into HAL’s strategic capacity planning, the PSU has unlocked a realm of possibilities that can revolutionise the way it approaches production capacity.

“Over the past two decades, I have served in various roles within Hindustan Aeronautics Limited (HAL). My journey has taken me from planning and process management to overseeing production projects and optimising assembly lines,” Gedala said.

HAL’s contribution to India’s aerospace and defence capabilities has been nothing short of remarkable. From manufacturing iconic aircraft to the current production of the indigenous Tejas series, we’ve come a long way, he said.

Managing supply chain through simulation

Gedala emphasises HAL’s unwavering commitment to exploring diverse avenues, such as AI, to harness technology’s potential in enhancing the PSU’s productivity.

HAL has bases in Bangalore, Nasik, Lucknow, Kanpur and Korwa. But given their divisions are scattered across the country, supply chain bottlenecks have been an issue for his team.

“In Nasik, we build the Russian origin aircraft, whereas, in Bangalore, we build the western origin and the indigenous flights. These are the two locations where we build the aircraft, whereas the other locations are supplier units,” Gedala continued, “Now, the challenge with HAL is to manage the supply chain.”

While earlier HAL developed all the components that go into an aircraft, now, HAL has started developing an ecosystem with vendors who supply some of these components to HAL. But these only further diversifies the supply chain, adding to the constraint.

To solve the problem, HAL has developed a simulated model that offers a unique perspective by accounting for production variability and supply chain uncertainties. By accurately pinpointing bottlenecks and inefficiencies, Gedala says that HAL has been able to make strategic decisions.

For instance, increasing the number of production lines or reducing variability in both production and supply chains can lead to significant improvements, he mentioned.

“These simulations enable us to make informed investment decisions and identify the most cost-effective path to expanding our production capabilities.”

However, the next step for HAL is to make the model learn on its own and this is where the PSU is turning its attention to AI.

The post Cypher2023: Elevating Aircraft Production Through Strategic Planning appeared first on Analytics India Magazine.

Cypher 2023 was jam packed. The day one of the events witnessed more footfall than we anticipated, close to 1500+ participants and 600+ companies took part in India’s biggest AI conference.

The event kicked off with Karthik Ranganath, general manager of Business IT at Shell R&D, talking about unleashing AI innovations for the better.

In his keynote discussion, Ranganath spoke about how Shell is harnessing the power of AI to make the energy sector more efficient, alongside sharing its partnerships with multiple AI startups to help tackle some of the pressing challenges in the energy sector.

This was followed by a talk on “Digital Minds” by Jacy Reese Anthis, cofounder at Sentience Institute, who emphasised on the relationship between humans and machines, in the backdrop of generative AI advancements.

He explained that there is a profound shift in not only how we interact with computers but also how we as a society interact with each other. “We talk to the computer like we talk to a friend with natural language instead of writing commands in code,” he said.

With the increase in chatbots, there is a danger in forming attachments to the computer which take on human-like characteristics with its language. His philosophical questions were thought provoking and left the audience reflecting on his ideas.

George Kuruvilla, the chief data platforms evangelist at SingleStore, shared how the company has built one of the best real time data management systems, eliminating the need to run to multiple vendors to store, manage and harness data.

He explained the real time ease of working with a single platform. He also spoke of the issues that startups face in data management while also explaining how their larger customers work with extensive data.

Biren Ghosh with his jovial personality really connected with the audience.

Ghosh spoke about how his company Technicolor Creative Studios has employed all the AI tools that create innovative animations and visualisations. He also showcased this with stunning videos that he played while talking about how and the why of the entire creative process.

Moving on to a more instructive session, Abhishek Nandy chief data scientist at PrediQt Business Solutions Pvt. Ltd and Intel Corporation certified instructor, gave a workshop on the AI kit tailored for Intel® architecture, specifically for data scientists and AI developers on how they can deploy AI models, particularly LLMs seamlessly.

He also touched upon Intel® Developer Cloud, access to Ponte Vecchio instances, and practical demos using LLM’s with Langchain and OpenAI Stable Diffusion.

One of the panel discussions explored the topic of how AI has barged in on every field into everyone’s life and work place. The conversation was between industry experts Akanksha Singh, Jayachandran Ramachandran, Vinodh Ramachandran and Chirag Jain.

The discussion was largely around how the adoption of AI has affected employees and how to navigate this introduction positively.

In addition to this, it also touched on the legal, academic, enterprise perspectives and the panellists also elaborated on the best work practices.

“The management should provide resources and clearly explain the vision of the organisation. It is the responsibility of anyone in the leadership position to instil faith and rally the organisation towards a common goal,” Vinodh Ramachandran clearly summed up.

Lastly, Jonty Rhodes, the South African cricketer and legendary fielder, spoke on the role of gathering data and analytics that gives players an edge in cricket. The retired player is a coach to IPL teams who touched upon different aspects of analysing players. This inevitably improves strategy, he said but not without the perils of too much information, which leads to a decision paralysis.

Humble Rhodes was in high spirits and spoke about his love for India. He also gave away the Minsky Awards for some of the exemplary leaders and companies in AI and analytics for their contribution and impact.

The post Cypher 2023: Key Highlights (Day 1) appeared first on Analytics India Magazine.

Creating content can be exhausting and expensive, but you need new material for your website and social channels to keep people interested and ensure they’re up to date with your business. If you’re looking for a better way to create content, check out Scribbyo AI, which you can get a lifetime subscription to for just $79.97 through October 15 only.

Scribbyo uses algorithms to create imaginative, high-quality and unique content covering virtually any prompt. You can write material in 37 supported languages to reach audiences across the globe and create content for your blog, website or social media in a matter of minutes. It covers more than 50 templates to support your content creation.

But that’s just the tip of the iceberg. Scribbyo also offers AI image generation to complement your content, as well as an AI voice-over generator so you can generate realistic human voices in 140 accents and languages. This adds quality voiceover to your video content. The transcription feature can also work the other way with up to 99% accuracy.

Workspace calls Scribbyo AI the “Best AI Content Generator.” User Mark L. agrees, writing, “Scribbyo has been a lifesaver for me as a blogger. With its AI content generator, I can create high-quality and engaging content in just a few minutes, without having to worry about the quality. I also love the AI image creation feature and the ready-made prompt templates that make my job so much easier. I would definitely recommend Scribbyo to anyone in the content creation industry!”

Find out why people love Scribbyo when you pick up a lifetime subscription for an extra $20 off our already discounted price. Now through October 15, Scribbyo AI is just $79.97.

Prices and availability are subject to change.

Subscribe to the Daily Tech Insider Newsletter

Stay up to date on the latest in technology with Daily Tech Insider. We bring you news on industry-leading companies, products, and people, as well as highlighted articles, downloads, and top resources. You’ll receive primers on hot tech topics that will help you stay ahead of the game.

Delivered Weekdays Sign up today