Рубрика: AI

Mastering the space ecosystem is no small feat. From development to deployment, each mission demands years of intricate work. However, Hyderabad-based Dhruva Space has cracked the code. It has successfully completed three space missions in less than 12 months – a record for any private space company to date.

Alongside Bengaluru-based Digantara which works in situational awareness, Dhruva Space became one of the only two private organisations authorized by IN-SPACe to launch and carry out missions.

“This was a significant achievement for us as one of the first companies to receive authorisation and successfully launch two satellites,” said Abhay Egoor, co-founder and chief technology officer at Dhruva Space, in an exclusive interview with AIM.

These three missions include DSOD-1U Space Qualification on June 30, 2022, where the Satellite Orbital Deployer obtained authorisation during ISRO’s PSLV C53 mission. This was followed by the Thybolt Mission, launched on November 26, 2022, involving two amateur radio communication nanosatellites, Thybolt-1 and Thybolt-2.

The third mission in April featured successful experiments, including 3U and 6U Satellite Orbital Deployers and the Dhruva Space Orbital Link (DSOL). All experiments, including DSOL’s proven capability for satellite-based data relay applications, were declared successful on May 5, 2023, with the satellites completing about 5000 orbits to date.

Founded in 2012, Dhruva Space has 25 institutional investors including Indian Angel Network, Indian Angel Network Fund and Blue Ashva Capital. The founding team consists of Sanjay Nekkanti, chief executive officer, Krishna Teja Penamakuru, chief operating officer, Chaitanya Dora Surapureddy, chief financial officer and Egoor.

How Did They Make This Happen?

Egoor humbly attributes the success of this to strategic planning and the adept use of the local ecosystem.

“Instead of managing everything internally, we’ve partnered with a network of over 430 registered vendors covering tasks like mechanical machining, coatings, and electronics,” he added. This approach, streamlining processes to operate concurrently with multiple vendors, expedites manufacturing.

However, in-house quality control, adhering to stringent space-grade standards, is rigorously maintained. Many of these vendors boast prior experience working with ISRO, demonstrating familiarity with and adherence to stringent standards. The company’s continuous optimisation for mass production addresses the escalating demand for larger-scale satellites, facilitating efficient service delivery. The success achieved is credited to the unwavering dedication of the team and their collaborative efforts with the local ecosystem.

Working with ISRO

In August this year, Dhruva Space received the transfer of satellite technology, including the IMS-1 Satellite Bus Technology from NewSpace (NSIL)/ISRO, and it encompasses detailed design files and documentation for all satellite subsystems. This includes information on solar panels, electrical connections, and structural elements, providing a comprehensive blueprint for the satellite’s construction.

“However, the transfer is not a mere replication of the design; rather, it involves fine-tuning for mass manufacturing, often in collaboration with design firms to make efficient modifications. This is important for meeting the demand for multiple satellites within a short timeframe,” added Egoor.

The support provided by ISRO as part of the tech transfer agreement further accelerates Dhruva’s credibility to offer services in this satellite class. “Overall, technology transfer accelerates our ability to offer services in this satellite class and reduces the time and effort required for research and development from scratch,” he added.

Tracking the Tech

Dhruva’s comprehensive tech stack spans three key segments: space, launch, and ground.

In the space segment, the company specializes in designing and manufacturing satellite platforms, encompassing crucial subsystems like solar panels, structures, electrical power systems, onboard controllers, RF systems, and attitude determination and control systems.

“While we don’t build propulsion systems or payloads, we manage most other subsystems internally,” said Egoor.

For the launch segment, the company doesn’t construct rockets but focuses on separation systems, particularly for CubeSats. In the ground segment, they offer complete ground stations, including antennas, control systems, radio stack, and software for data download, deploying these solutions as both software as a service and turnkey products across the country.

Throughout the whole process, AI and ML play a crucial role, particularly in the satellite segment, as Egoor said. In satellite operations, AI and ML are used for onboard processing to minimize the need for downlinking large datasets to the ground, improving operational efficiency by conducting initial analyses of extensive datasets in orbit, resulting in a more streamlined downlink of critical information.

Dhruva is also integrating edge computing with AI into its satellite platforms, expanding applications beyond imagery to include diverse payloads like communications or scientific instruments.

Additionally, the team has developed a comprehensive software suite for the ground segment, incorporating AI and ML elements to optimise data analysis and streamline ground station operations, ensuring reliable and efficient satellite downlinks for end customers.

Turning attention to generative AI, Egoor said that even though it is in an early stage of development for space, he sees potential applications for it in the space industry. “The evolving landscape of the space segment, coupled with advancements in processing power and emerging edge computing technologies, presents a promising opportunity for the integration of generative AI,” he added.

Unlike in the past, where extensive server farms were necessary for such operations, today’s technology allows us to deploy these models on processors as compact as a mobile phone. Moreover, real-time updates based on additional data enable us to manage modeling and data analysis without relying on large server farms.

Soaring into Space Success

Dhruva Space has a lot of interesting projects coming up. Just last week, Dhruva unveiled its ambitious plans for a 280,000-square-foot design, engineering, assembly, integration, and testing facility for large-scale spacecraft manufacturing, marking a significant stride in India’s space privatization efforts in Hyderabad.

Discussing the same, Egoor said, “This facility, set to start construction by year-end, will streamline the entire process from solar panel fabrication to satellite assembly and testing.”

The facility, strategically located in Shamshabad, will address the growing demand for satellite-based services on Earth. The project, set on a 6.5-acre plot allocated by the Telangana Government, will be completed in phases over five years, with the initial phase featuring a 150,000-square-foot space, including a Solar Array Assembly Line.

In their upcoming mission, scheduled for early next year, the space tech startup announced the introduction of hosted payload services through the Leap initiative. They plan to construct an entire satellite for an Australian company aiming to validate its imaging camera payload.

“Operating on our new flagship platform, the P30, a 30-kilogram satellite, we will design and integrate various subsystems, including solar panels, structure, power systems, onboard controllers, RF link, and attitude control systems,” Egoor told us.

Following the satellite’s launch, the company will manage all operations and data delivery for the customer. Moreover, beyond hosted payloads, the company intends to offer the P30 platform globally, allowing customers to purchase and launch complete satellites tailored to their specific needs.

Read more: This Indian Space Tech Startup is Building the World’s First Multisensor Satellite

The post Dhruva’s Sprint to Space: How to Knock Triple Milestones in No Time appeared first on Analytics India Magazine.

The complexity and change experienced by organisations as they grow is one reason we are seeing similar cyber security risks to a decade ago, says Rapid7’s CISO Jaya Baloo. However, quantum computing is one emerging risk where we could stay ahead of the game.

Speaking on ethics in information security at the 2023 Australian Cyber Conference, Baloo said the Australian market has truly woken up to cyber risks in the last year due to a number of high-profile data breaches that have affected millions of Australians.

Baloo told TechRepublic proactive mapping of assets and vulnerabilities, consistency through times of organisational growth and planning ahead for risks like quantum computing could help Australian security pros step off what can feel like a “hamster wheel.”

Jump to

• Organisations lack understanding of assets and vulnerabilities
• Multicloud expansion is exacerbating data security risks
• The risks of quantum computing a test of industry proactivity

Organisations lack full understanding of assets and vulnerabilities

Despite talking to organisations about similar risks for a decade, Baloo said that many were “still surprised” when a lack of understanding of the assets they had and the vulnerabilities that were on those assets led to them being the victim of a cyber security incident.

“We still don’t have a full understanding of our footprint, a critical thing for an enterprise, and we wind up surprised if we have an exposed API, issues with credentials being made open or a dataset aggregated for an AI learning model that was open to everyone,” Baloo said. “It is not enough to have effective remediation.

“We should know ourselves, but we still don’t. For example we don’t understand our networks and systems, and we don’t deploy the same standards for internal products as we do to test environments — which we should, but we don’t.”

SEE: A definitive guide to evaluating cybersecurity solutions.

Old vulnerabilities were also creeping up into new products in new tech stacks, Baloo said, because, as an industry, “we haven’t done the security-by-design thing very well.”

Business growth making cyber risk control difficult

Part of the problem is a lack of discipline in the way companies have grown. Baloo said this leads to companies or departments adding new services, for example, or taking them away, without necessarily documenting these changes or following a thorough process.

This often happens when companies grow through acquisition or become a part of a bigger entity themselves, creating a lack of documentation on total external and internal assets.

“We don’t do that well, we don’t execute through these changes in a consistent fashion,” said Baloo.

SEE: Take advantage of TechRepublic Premium’s change control policy.

Baloo said attack surface management automations in the form of third-party risk scores were also not always correct in estimating what belonged to a company.

“We have an imperfect third-party external view and internal view, which is the most important stuff,” said Baloo.

Multicloud expansion is exacerbating data security risks

Cloud computing growth has exacerbated the risk of organisations losing track of their assets and vulnerabilities. Baloo said the ease of spinning up cloud assets, often not taken down, and slightly different services for logging, identity and monitoring added to overall complexity.

“Identity, for example, is set up differently (in different cloud environments), and that is the prerequisite for all the other stuff we do,” Baloo said. “If you are not doing that right from the get go and harmonising that across cloud stacks, it can be easy to screw everything up.”

Harmonise clouds to reduce complexity

Organisations should ask themselves what they are putting in the cloud and why, Baloo said. Pure “lift-and-shift” operations — which would see old applications just “flopped down somewhere else,” even when using some cloud native features — would be best avoided.

“In a multicloud environment, you need to ask how you harmonise the different cloud environments you are using,” Baloo said. “You should have a baseline for what you want on different platforms, how they are set up, then pull that back to centralised or native monitoring. We need to find a way to do this without it being incredibly complex.”

SEE: Here’s everything you need to know about multicloud.

If data is being shared cloud to cloud, Baloo said IT needed to know what that flow looks like.

“Even there can create points of failure,” said Baloo. “What are those from a topological point of view?”

The risks of quantum computing a test of industry proactivity

Quantum computing is one area where proactivity could put IT ahead of the game. With the first quantum computer potentially five to 10 years away, there is time to invest in replacing existing encryption algorithms before they are made redundant for defence by quantum computers.

SEE: Australia is looking at an “assume-breach” approach to combating cyber attacks.

Baloo said the question that should drive action is what data we want to protect and for how long. If Australian organisations want to be able to protect healthcare data for the lifetime of a patient, or even intergenerationally, Baloo said quantum computing now means “we don’t know how to do that.”

“Quantum computing is an area that I am worried will be just like AI,” said Baloo. “It won’t be prioritised as super important until it actually hits us. It is coming, so I would like to see us plan ahead. Let’s not be chickens with their heads cut off when it does hit us.”

Getting ahead of the quantum game

The solution will probably be a combination of both quantum communication networks, like those being developed in China, and post-quantum algorithms, Baloo suggested. However, the important thing is having enough time to undertake the transition before it is too late.

“We suck at change; we are terrible at it,” said Baloo. “Getting everyone in the same place and to the same level of understanding to invest in that transition is going to be a difficult thing to do. But if we wait until there is a quantum computer, then we are screwed.”

Subscribe to the Daily Tech Insider AU Newsletter

Stay up to date on the latest in technology with Daily Tech Insider Australian Edition. We bring you news on industry-leading companies, products, and people, as well as highlighted articles, downloads, and top resources. You’ll receive primers on hot tech topics that are most relevant to AU markets that will help you stay ahead of the game.

Delivered Thursdays Sign up today

Alluxio Claims 4X GPU Speed Boost for AI Training October 18, 2023 by Alex Woodie

(Aleksey Nikiforov/Shutterstock)

Customers that use the high-speed cache in the new Alluxio Enterprise AI platform can squeeze up to four times as much work out of their GPU setups than without it, Alluxio announced today. Alluxio also says the overall model training pipeline, meanwhile, can be sped up to 20x thanks to the data virtualization platform and its new DORA architecture.

Alluxio is better known in the advanced analytics market than in the AI and machine learning (ML) market, which is a result of where it debuted in the data stack. Originally developed as a sister project to Apache Spark by Haoyuan “HY” Li—who was also a co-creator of Spark at UC Berkeley’s AMPlab—Alluxio gained traction by serving as a distrbuted file system in modern hybrid cloud environments. The product's real forte was providing an abstraction layer to accelerate I/O for data between storage repositories like HDFS, S3, MinIO, and ADLS and processing engines like Spark, Trino, Presto, and Hive.

By providing a single global namespace across hybrid cloud environments built around the precepts of separation of compute and storage, Alluxio reduces complexity, bolsters efficiency, and simplifies data management for enterprise clients with sprawling data estates measuring in the hundreds of petabytes.

Alluxio now works with analytics and AI workloads

As deep learning matured, the folks at Alluxio realized that an opportunity existed to apply its core IP to help optimize the flow of data and compute for AI workloads too—predominantly computer vision use cases but also some natural language processing (NLP) too.

The company saw that some of its existing analytic customers had adjacent AI teams that were struggling to expand their deep learning environments beyond a single GPU. To cope with the challenge of coordinating data in multi-GPU environments, customers either bought high performance storage appliances to accelerate the flow of data into GPUs from their primary data lakes, or paid data engineers to write scripts for moving data in a more low-level and manual fashion.

Alluxio saw an opportunity to use its technology to do essentially the same thing—accelerate the flow of training data into the GPU—but with more orhcestration and automaton around it. This gave rise to the creation of Alluxio Enterprise AI, which Alluxio unveiled today.

The new product shares some technology with its existing product, which previously was called Alluxio Enterprise and now has been rebranded as Alluxio Enterprise Data, but there are important differences too, says Adit Madan, the director of product management at Alluxio.

“With Alluxio Enterprise AI, even though some of the functionality sounds and is very familiar to what was there in the Alluxio, this is a brand new systems architecture,” Madan tells Datanami. “This is a completely decentralized architecture that we are naming DORA.”

With DORA, which is short for Decentralized Object Repository Architecture, Alluxio is, for the first time, tapping into the underlying hardware, including GPUs and NVMe drives, instead of living purely at the software level.

“We are saying use Alluxio on your accelerated compute itself,” i.e. the GPU nodes, Madan says. “We will make use of NVME, the pool of NvME that you have available, and provide you with the I/O demand by simply pointing to your data lake without the need for another high performance storage solution underneath.”

Source: Alluxio

Additionally, Alluxio Enterprise AI leverages techniques that Alluxio has used on the analytics side for some time to “make the I/O more intelligent,” the company says. That includes optimizing the cache for data access patterns commonly found in AI environments, which are characterized by large file sequential access, large file random access, and massive small file access. “Think of it like we are extracting all of the I/O capabilities on the GPU cluster itself, instead of you having to buy more hardware for I/O,” Madan says.

Conceptually, Alluxio Enterprise AI works similarly to the company’s existing data analytics product—it accelerates the I/O and allows users to get more work done without worrying so much about how the data gets from point A (the data lake) to point B (the compute cluster). But at a technology level, there was quite a bit of innovation required, Madan says.

“To co-locate on the GPUs, we had to optimize on how much resources that we use,” he says. “We have to be really resource efficient. Instead of let's say consuming 10 CPUs, we have to bring it down to only consuming two CPUs on the GPU node to serve the I/O. So from a technical standpoint, there is a very significant difference there.”

Alluxio Enterprise AI can scale to more than 10 billion objects, which will make it useful for the many small files used in computer vision use cases. It's designed to integrate with PyTorch and Tensorflow frameworks. It's primarily intended to be used for training AI models, but it can be used for model deployment too.

Source: Alluxio

The results that Alluxio claims from the optimization are impressive. On the GPU side, Alluxio Enterprise AI delivers 2x to 4x more capacity, the company says. Customers can use that freed up capacity to either get more computer vision training work done or to cut their GPU costs, Madan says.

Some of Alluxio’s early testers used the new product in production settings that included 200 GPU servers. “It’s not a small investment,” Madan says. “We have some active engagement with smaller [customers running] a few dozen. And we have some engagements which are much larger with a few hundred servers, each of which cost a few hundred grand.”

One early tester is Zhihu, which runs a popular question and answer site from its headquarters in Beijing. "Using Alluxio as the data access layer, we’ve significantly enhanced model training performance by 3x and deployment by 10x with GPU utilization doubled," Mengyu Hu, a software engineer in Zhihu's data platform team, said in a press release. "We are excited about Alluxio’s Enterprise AI and its new DORA architecture supporting access to massive small files. This offering gives us confidence in supporting AI applications facing the upcoming artificial intelligence wave.”

This solution is not for everyone, Madam says. Customers that are using off-the-shelf AI models, perhaps with a vector database for LLM use cases, don’t need this. “If you're adapting your model [i.e. fine-tuning it] that's where the need is,” he says.

Related

About the author: Alex Woodie

Alex Woodie has written about IT as a technology journalist for more than a decade. He brings extensive experience from the IBM midrange marketplace, including topics such as servers, ERP applications, programming, databases, security, high availability, storage, business intelligence, cloud, and mobile enablement. He resides in the San Diego area.

Cloud computing has opened new doors for app development and hosting. Before cloud services became mainstream, developers had to maintain their own expensive servers. Now, cloud platforms like AWS and Azure provide easy database hosting without the high hardware costs. Cloud databases offer the flexibility and convenience of the cloud while providing standard database functionality. They can be relational, NoSQL, or any other database model, accessed via API or web interface.

In this review article, we will explore the top 7 cloud databases used by professionals to build robust applications. These leading cloud database platforms enable developers to efficiently store and manage data in the cloud. We will examine the key features, pros, and cons of each platform, so you can determine which one is the right fit for your app development needs.

Azure SQL Database is a fully managed relational cloud database that is part of Microsoft's Azure SQL family. It provides a database-as-a-service solution built specifically for the cloud, combining the flexibility of a multi-model database with automated management, scaling, and security. Azure SQL database is always up-to-date, with Microsoft handling all updates, backups, and provisioning. This enables developers to focus on building their applications without database administration overhead.

🔑 Azure SQL Database Key Points

• Serverless computing and hyperscale storage solutions are both flexible and responsive
• A fully managed database engine that automates updates, provisioning, and backups
• It has a built-in AI and high availability to ensure consistent peak performance and durability

✅ Pros

• User-friendly interface for creating data models
• Straightforward billing system
• Fully managed and secure SQL database
• Seamless migration from on-premise to cloud storage

❌ Cons

• Job and task managers work in different ways
• Limited database size
• Need for more efficient notification and logging system for database errors
• Costly scaling up and down without proper automation implementation

Amazon Redshift is a fully-managed, petabyte-scale cloud-based data warehousing solution designed to help organizations store, manage, and analyze large amounts of data efficiently. Built on top of the PostgreSQL open-source database system, Redshift uses columnar storage technology and massively parallel processing to deliver fast query performance on high volumes of data. Its distributed architecture allows it to elastically scale storage and processing power to accommodate growing data volumes. Its tight integration with other AWS services also enables seamless data loading from S3, EMR, DynamoDB, etc. The end result is a performant, cost-effective, and flexible cloud data warehouse solution suitable for large-scale data analytics.

🔑 Amazon Redshift Key Points

• It uses column-oriented databases
• Its architecture is based on massively parallel processing
• It includes machine learning to improve performance
• It is fault tolerant

✅ Pros

• Easy setup, deployment, and management
• Detailed documentation that makes it easy to learn
• Seamless integration with data stored in S3
• Simplified ETL setup

❌ Cons

• JSON support in SQL is limited
• Array type columns are missing and are automatically converted to strings
• The logging function is almost non-existent

Amazon DynamoDB is a fast, flexible, and reliable NoSQL database service that helps developers build scalable, serverless applications. It supports key-value and document data models, and can handle massive amounts of requests daily. DynamoDB automatically scales horizontally, ensuring availability, durability, and fault tolerance without any extra effort from the user. Designed for internet-scale applications, DynamoDB offers limitless scalability and consistent performance with up to 99.999% availability.

🔑 Amazon DynamoDB Key Points

• The ability to handle over 10 trillion requests per day
• Support for ACID transactions
• A multi-Region and multi-Master database
• NoSQL database

✅ Pros

• Fast and simple to operate
• Handle data that is dynamic and constantly changing
• Indexed data can be retrieved quickly
• Performs exceptionally well even when working with large-scale applications

❌ Cons

• If the resource is not monitored correctly, the expenses can be significant
• Does not support backup in different regions
• It can be expensive for projects that require multiple environments to be created

Google BigQuery is a powerful, fully-managed cloud-based data warehouse that helps businesses analyze and manage massive datasets. With its serverless architecture, BigQuery enables lightning-fast SQL queries and data analysis, processing millions of rows in seconds. You can store your data in Google Cloud Storage or in BigQuery's own storage, and it seamlessly integrates with other GCP products like Data Flow and Data Studio, making it a top choice for data analytics tasks.

🔑 Google BigQuery Key Points

• It can scale up to a petabyte, making it highly scalable
• It offers fast processing speeds, allowing you to analyze data in real-time
• It is available in both on-demand and flat-rate subscription models

✅ Pros

• Automatically optimizes queries to retrieve data quickly
• Great customer support
• Its data exploration and visualization capabilities are very useful
• It has a large number of native integrations

❌ Cons

• Uploading databases using Excel can be time-consuming and prone to errors
• Connecting to other cloud infrastructures like AWS can be difficult
• The interface can be difficult to use if you are not familiar with it

MongoDB Atlas is a cloud-based, fully managed MongoDB service that allows developers to quickly setup, operate, and scale MongoDB deployments in the cloud with just a few clicks. Developed by the same engineers that build the MongoDB database, Atlas provides all the features and capabilities of the popular document-based NoSQL database, without the operational heavy lifting required for on-premise deployments. Atlas simplifies MongoDB cloud operations by automating time-consuming administration tasks like infrastructure provisioning, database setup, security hardening, backups, and more.

🔑 MongoDB Atlas Key Points

• It's a document-oriented database
• Sharding feature allows for easy horizontal scalability
• The database triggers in MongoDB Atlas are powerful and can execute code when certain events occur
• Useful for time series data

✅ Pros

• It is easy to adjust the scale of the service based on your needs
• There are free and trial plans available for evaluation or testing purposes, which are quite generous
• Any database information that is uploaded to MongoDB Atlas is backed up
• JSON documents can be accessed from anywhere

❌ Cons

• It is not possible to directly download all information stored in MongoDB Atlas clusters
• Lacks more granular billing
• No cross table joins

Snowflake is a powerful, self-managed data platform designed for the cloud. Unlike traditional offerings, Snowflake combines a new SQL query engine with an innovative cloud-native architecture, providing a faster, easier-to-use, and highly flexible solution for data storage, processing, and analytics. As a true self-managed service, Snowflake takes care of hardware and software management, upgrades, and maintenance, allowing users to focus on deriving insights from their data.

🔑 Snowflake Key Points

• Provide query and table optimization
• It offers secure data sharing and zero-copy cloning
• Snowflake supports semi-structured data

✅ Pros

• Snowflake can ingest data from various cloud platforms, such as AWS, Azure, and GCP
• You can store data in multiple formats, including structured and unstructured
• Computers are dynamic, meaning you can choose a computer based on cost and performance
• It's great for managing different warehouses

❌ Cons

• Data visualization could use some improvement
• The documentation can be hard to understand
• Snowflake lacks CI/CD integration capabilities

Databricks SQL (DB SQL) is a powerful, serverless data warehouse that allows you to run all your SQL and BI applications at a massive scale, with up to 12x better price/performance than traditional solutions. It offers a unified governance model, open formats and APIs, and supports the tools of your choice, ensuring no lock-in. The rich ecosystem of tools supported by DB SQL, such as Fivetran, dbt, Power BI, and Tableau, allows you to ingest, transform, and query all your data in-place. This empowers every analyst to access the latest data faster for real-time analytics, and enables seamless transitions from BI to ML, unleashing the full potential of your data.

🔑 Databricks SQL Key Points

• Centralized governance
• Open and reliable data lake as the foundation
• Seamless integrations with the ecosystem
• Modern analytics
• Easily ingest, transform and orchestrate data

✅ Pros

• Enhanced collaboration between Data Science & Data Engineering teams
• Spark Jobs Execution Engine is highly optimized
• Analytics feature recently added for building visualization dashboards
• Native integration with managed MLflow service
• Data Science code can be written in SQL, R, Python, Pyspark, or Scala

❌ Cons

• Running MLflow jobs remotely is complicated and needs simplification
• All runnable code must be kept in Notebooks, which are not ideal for production
• Session resets automatically at times
• Git connections can be unreliable

Cloud databases have revolutionized how businesses store, manage, and utilize their data. As we have explored, leading platforms like Azure SQL Database, Amazon Redshift, DynamoDB, Google BigQuery, MongoDB Atlas, Snowflake, and Databricks SQL each offer unique benefits for app development and data analytics.

When choosing the right cloud database, key factors to consider are scalability needs, ease of management, integrations, performance, security, and costs. The optimal platform will align with your infrastructure and workload requirements.

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in Technology Management and a bachelor's degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.

More On This Topic

• 11 Best Practices of Cloud and Data Migration to AWS Cloud
• 11 Best Data Science Education Platforms
• 7 Best Platforms to Practice SQL
• Database Key Terms, Explained
• How To Build A Database Using Python
• Practising SQL without your own database

ChaosSearch Tackles Live Search, SQL, and Gen AI Analytics with LakeDB October 18, 2023 by Ali Azhar

ChaosSearch, a leading log analytics platform, last week announced the release of Chaos LakeDB – the first data lake database designed to power generative artificial intelligence (AI), SQL, and Live Search. Chaos LakeDB is available as a SaaS data platform for enterprises and as an embedded database from cloud platform providers.

Earlier this year, ChaosSearch unveiled the Chaos AI Assistant – powered by open AI and integrated into the ChaosSearch platform. Industry leaders such as Cisco and Equifax are already using Chaos LakeDB.

The ChaosSearch technology allows organizations to stream data directly into cloud object storage such as Amazon S3, turning it into a searchable data lake with unlimited hot data retention. With the launch of Chaos LakeDB, any cloud provider or enterprise software company can embed the ChaosSearch database into its existing offering. Embedding ChaosSearch would allow organizations flexibility in how they search data.

Chaos LakeDB is designed for various applications including general system observability, understanding product usage trends to derive business insights, and search log data for troubleshooting. It helps simplify architecture by removing the need for complex data movement or data pipelines. In addition, it consolidates diverse data streams and formats into one cohesive data lake database. The database also has the capability to automate the data pipeline, orchestrate workload across the lake backbone, and schema management.

“Our vision with Chaos LakeDB was to tackle the challenges of live analytics at scale. We recognized the challenges businesses faced with legacy systems and aimed to provide a solution that was not only efficient but also future-proof. Chaos LakeDB is a testament to our commitment to innovation, offering a unified platform where data is not just stored but is also activated, analyzed, and leveraged for actionable insights. In this AI-driven era, we believe that data should be at the forefront of decision-making, and with Chaos LakeDB, we’re making that a reality for businesses worldwide,” said Thomas Hazel, Chief Technology Officer at ChaosSearch.

The integration with Amazon Simple Storage Service (S3), the leading object store for AWS customers, Chaos LakeDP allows users to merge vast storage capabilities with the accessibility of cloud databases and helps minimize the need for complex ETL and ELT processes. The integration also ensures enhanced cost efficiency and performance at scale – crucial requirements for today’s data-intensive analytics and AI applications.

“Every modern business today is a data business, and where you store your data and how quickly you’re able to leverage it for analytics is incredibly important,” said James Kirschner, general manager, Amazon S3 at AWS. “Solutions like ChaosSearch’s Chaos LakeDB help improve how customers interact with their data, making it easier to manage, faster to leverage, and more cost-effective, while retaining Amazon S3’s industry-leading durability, availability, security, and scalability.”

There has been a rising demand for observability technology and this is evident in the recent acquisition of Splunk by Cisco representing $28 billion in equity value, and the New Relix private equity takeover earlier this year. It is clear that this is a space to keep an eye on.

Related

About the author: Alex Woodie

Alex Woodie has written about IT as a technology journalist for more than a decade. He brings extensive experience from the IBM midrange marketplace, including topics such as servers, ERP applications, programming, databases, security, high availability, storage, business intelligence, cloud, and mobile enablement. He resides in the San Diego area.