Рубрика: AI

OpenStack improves support for AI workloads Frederic Lardinois @fredericl / 8 hours

OpenStack allows enterprises to manage their own AWS-like private clouds on-premises. Even after 29 releases, it’s still among the most active open-source projects in the world and this week, the OpenInfra Foundation that shepherds the project announced the launch of version 29 of OpenStack. Dubbed ‘Caracal,’ this new release emphasizes new features for hosting AI and high-performance computing (HPC) workloads.

The typical OpenStack user is a large enterprise company. That may be a retailer like Walmart or a large telco like NTT. What virtually all enterprises have in common right now is that they’re thinking about how to put their AI models into production, all while keeping their data safe. For many, that means keeping total control of the entire stack.

OpenInfra Foundation COO Mark Collier

As Nvidia CEO Jensen Huang recently noted, we’re at the cusp of a multi-trillion dollar investment wave that will go into data center infrastructure. A large chunk of that is investments by the large hyperscalers, but a lot of it will also go into private deployments — and those data centers need a software layer to manage them.

That puts OpenStack into an interesting position right now as one of the only comprehensive alternatives to VMware’s offerings, which is facing its own issues as many VMware users aren’t all that happy about its sale to Broadcom. More than ever, VMware users are looking for alternatives. “With the Broadcom acquisition of VMware and some of the licensing changes they’ve made, we’ve had a lot of companies coming to us and taking another look at OpenStack,” OpenInfra Foundation executive director Jonathan Bryce explained.

Image Credits: Frederic Lardinois/TechCrunch

A lot of OpenStack’s growth in recent years was driven by its adoption in the Asia-Pacific region. Indeed, as the OpenInfra Foundation announced this week, its newest Platinum Member is Okestro, a South Korean cloud provider with a heavy focus on AI. But Europe, with its strong data sovereignty laws, has also been a growth market and the UK’s Dawn AI supercomputer runs OpenStack, for example.

“All the things are lining up for a big upswing and open-source adoption for infrastructure,” OpenInfra Foundation COO Mark Collier told TechCrunch. “That means OpenStack primarily, but also Kata Containers and some of our other projects. So it’s pretty exciting to see another wave of infrastructure upgrades give our community some important work to complete for many years to come.”

In practical terms, some of the new features added to this release include the ability to support vGPU live migrations in Nova, OpenStack’s core compute service. This means users now have the ability to move GPU workloads from one physical server to another with minimal impact on the workloads, something enterprises have been asking for since they want to be able to manage their costly GPU hardware as efficiently as possible. Live migration for CPUs has long been a standard feature of Nova, but this is the first time it’s available for GPUs as well.

The latest release also brings a number of security enhancements, including rule-based access control for more core OpenStack services like the Ironic bare-metal-as-a-service project. That’s in addition to networking updates to better support HPC workloads and a slew of other updates. You can find the full release notes here.

BURBANK, CA – JULY 10: A general view of atmosphere at the 7-Eleven 88th birthday celebration at 7-Eleven on July 10, 2015 in Burbank, California. (Photo by Chris Weeks/Getty Images for 7-Eleven)

This update is also the first since OpenStack moved to its ‘Skip Level Upgrade Release Process’ (SLURP) a year ago. The OpenStack project cuts a new release every six months, but that’s too fast for most enterprises — and in the early days of the project, most users would describe the upgrade process as ‘painful’ (or worse).

Today, upgrades are much easier and the project is also far more stable. The SLURP cadence introduces something akin to a long-term release version, where, on an annual basis, every second release is a SLURP release that’s easy to upgrade to, even as the teams still produce major updates on the original six-month cycle for those who want a faster cadence.

Throughout the years, OpenStack has gone through its up-and-down cycles in terms of perception. But it’s now a mature system and backed by a sustainable ecosystem — something that wasn’t necessarily the case at the height of its first hype cycle ten years ago. In recent years, it found a lot of success in the telco world, which allowed it to go through this maturation phase and today, it may just find itself in the right place and time to capitalize on the AI boom, too.

AI deepfakes were not on the risk radar of organisations just a short time ago, but in 2024, they are rising up the ranks. With AI deepfakes’ potential to cause anything from a share price tumble to a loss of brand trust through misinformation, they are likely to feature as a risk for some time.

Robert Huber, chief security officer and head of research at cyber security firm Tenable, argued in an interview with TechRepublic that AI deepfakes could be used by a range of malicious actors. While detection tools are still maturing, APAC enterprises can prepare by adding deepfakes to their risk assessments and better protecting their own content.

Ultimately, more protection for organisations is likely when international norms converge around AI. Huber called on larger tech platform players to step up with stronger and clearer identification of AI-generated content, rather than leaving this to non-expert individual users.

AI deepfakes are a rising risk for society and businesses

The risk of AI-generated misinformation and disinformation is emerging as a global risk. In 2024, following the launch of a wave of generative AI tools in 2023, the risk category as a whole was the second largest risk on the World Economic Forum’s Global Risks Report 2024 (Figure A).

Figure A

Over half (53%) of respondents, who were from business, academia, government and civil society, named AI-generated misinformation and disinformation, which includes deepfakes, as a risk. Misinformation was also named the biggest risk factor over the next two years (Figure B).

Figure B

Enterprises have not been so quick to consider AI deepfake risk. Aon’s Global Risk Management Survey, for example, does not mention it, though organisations are concerned about business interruption or damage to their brand and reputation, which could be caused by AI.

Huber said the risk of AI deepfakes is still emergent, and it is morphing as change in AI happens at a fast rate. However, he said that it is a risk that APAC organisations should be factoring in. “This is not necessarily a cyber risk. It’s an enterprise risk,” he said.

AI deepfakes provide a new tool for almost any threat actor

AI deepfakes are expected to be another option for any adversary or threat actor to use to achieve their aims. Huber said this could include nation states with geopolitical aims and activist groups with idealistic agendas, with motivations including financial gain and influence.

“You will be running the full gamut here, from nation state groups to a group that’s environmentally aware to hackers who just want to monetise depfakes. I think it is another tool in the toolbox for any malicious actor,” Huber explained.

SEE: How generative AI could increase the global threat from ransomware

The low cost of deepfakes means low barriers to entry for malicious actors

The ease of use of AI tools and the low cost of producing AI material mean there is little standing in the way of malicious actors wishing to make use of new tools. Huber said one difference from the past is the level of quality now at the fingertips of threat actors.

“A few years ago, the [cost] barrier to entry was low, but the quality was also poor,” Huber said. “Now the bar is still low, but [with generative AI] the quality is greatly improved. So for most people to identify a deepfake on their own with no additional cues, it is getting difficult to do.”

What are the risks to organisations from AI deepfakes?

The risks of AI deepfakes are “so emergent,” Huber said, that they are not on APAC organisational risk assessment agendas. However, referencing the recent state-sponsored cyber attack on Microsoft, which Microsoft itself reported, he invited people to ask: What if it were a deepfake?

“Whether it would be misinformation or influence, Microsoft is bidding for large contracts for their enterprise with different governments and reasons around the world. That would speak to the trustworthiness of an enterprise like Microsoft, or apply that to any large tech organisation.”

Loss of enterprise contracts

For-profit enterprises of any type could be impacted by AI deepfake material. For example, the production of misinformation could cause questions or loss of contracts around the world or provoke social concerns or reactions to an organisation that could damage their prospects.

Physical security risks

AI deepfakes could add a new dimension to the key risk of business disruption. For instance, AI-sourced misinformation could cause a riot or even the perception of a riot, causing either danger to physical persons or operations, or just the perception of danger.

Brand and reputation impacts

Forrester released a list of potential deepfake scams. These include risks to an organisation’s reputation and brand or employee experience and HR. One risk was amplification, where AI deepfakes are used to spread other AI deepfakes, reaching a broader audience.

Financial impacts

Financial risks include the ability to use AI deepfakes to manipulate stock prices and the risk of financial fraud. Recently, a finance employee at a multinational firm in Hong Kong was tricked into paying criminals US $25 million (AUD $40 million) after they used a sophisticated AI deepfake scam to pose as the firm’s chief financial officer in a video conference call.

Individual judgment is no deepfake solution for organisations

The big problem for APAC organisations is AI deepfake detection is difficult for everyone. While regulators and technology platforms adjust to the growth of AI, much of the responsibility is falling to individual users themselves to identify deepfakes, rather than intermediaries.

This could see the beliefs of individuals and crowds impact organisations. Individuals are being asked to decide in real-time whether a damaging story about a brand or employee may be true, or deepfaked, in an environment that could include media and social media misinformation.

Individual users are not equipped to sort fact from fiction

Huber said expecting individuals to discern what is an AI-generated deepfake and what is not is “problematic.” At present, AI deepfakes can be difficult to discern even for tech professionals, he argued, and individuals with little experience identifying AI deepfakes will struggle.

“It’s like saying, ‘We’re going to train everybody to understand cyber security.’ Now, the ACSC (Australian Cyber Security Centre) puts out a lot of great guidance for cyber security, but who really reads that beyond the people who are actually in the cybersecurity space?” he asked.

Bias is also a factor. “If you’re viewing material important to you, you bring bias with you; you’re less likely to focus on the nuances of movements or gestures, or whether the image is 3D. You are not using those spidey senses and looking for anomalies if it’s content you’re interested in.”

Tools for detecting AI deepfakes are playing catch-up

Tech companies are moving to provide tools to meet the rise in AI deepfakes. For example, Intel’s real-time FakeCatcher tool is designed to identify deepfakes by assessing human beings in videos for blood flow using video pixels, identifying fakes using “what makes us human.”

Huber said the capabilities of tools to detect and identify AI deepfakes are still emerging. After canvassing some tools available on the market, he said that there was nothing he would recommend in particular at the moment because “the space is moving too fast.”

What will help organisations fight AI deepfake risks?

The rise of AI deepfakes is likely to lead to a “cat and mouse” game between malicious actors generating deepfakes and those trying to detect and thwart them, Huber said. For this reason, the tools and capabilities that aid the detection of AI deepfakes are likely to change fast, as the “arms race” creates a war for reality.

There are some defences organisations may have at their disposal.

The formation of international AI regulatory norms

Australia is one jurisdiction looking at regulating AI content through measures like watermarking. As other jurisdictions around the world move towards consensus on governing AI, there is likely to be convergence about best practice approaches to support better identification of AI content.

Huber said that while this is very important, there are classes of actors that will not follow international norms. “There has to be an implicit understanding there will still be people who are going to do this regardless of what regulations we put in place or how we try to minimise it.”

SEE: A summary of the EU’s new rules governing artificial intelligence

Large tech platforms identifying AI deepfakes

A key step would be for large social media and tech platforms like Meta and Google to better fight AI deepfake content and more clearly identify it for users on their platforms. Taking on more of this responsibility would mean that non-expert end users like organisations, employees and the public have less work to do in trying to identify if something is a deepfake themselves.

Huber said this would also assist IT teams. Having large technology platforms identifying AI deepfakes on the front foot and arming users with more information or tools would take the task away from organisations; there would need to be less IT investment required in paying for and managing deepfake detection tools and the allocation of security resources to manage it.

Adding AI deepfakes to risk assessments

APAC organisations may soon need to consider making the risks associated with AI deepfakes a part of regular risk assessment procedures. For example, Huber said organisatinos may need to be much more proactive about controlling and protecting the content organisations produce both internally and externally, as well as documenting these measures for third parties.

“Most mature security companies do third party risk assessments of vendors. I’ve never seen any class of questions related to how they are protecting their digital content,” he said. Huber expects that third-party risk assessments conducted by technology companies may soon need to include questions relating to the minimisation of risks arising out of deepfakes.

Time series forecasting plays a vital role in crucial decision-making processes across various industries such as retail, finance, manufacturing, and healthcare. However, compared to domains like natural language processing and image recognition, the integration of advanced artificial intelligence (AI) techniques into time series forecasting has been relatively slow. Although foundational AI has made significant progress in areas like natural language processing and image recognition, its impact on time series forecasting has been limited until recently. Nevertheless, there is now an increasing momentum in the development of foundational models specifically tailored for time series forecasting. This article aims to delve into the evolving landscape of foundational AI for time series forecasting, exploring the recent advancements in this domain. However, before delving into these advancements, let's briefly introduce time series forecasting and its applications in various industries.

Time Series Forecasting and Applications

Time series data refers to a sequence of data points collected or recorded at regular time intervals. This type of data is prevalent across various domains, such as economics, weather, health, and more. Each data point in a time series is time-stamped, and the sequence is often used to analyze trends, patterns, and seasonal variations over time.

Time series forecasting involves using historical data to predict future values in the series. It is a critical method in statistics and machine learning that helps in making informed decisions based on past patterns. Forecasting can be as simple as projecting the same growth rate into the future or as complex as using AI models to predict future trends based on intricate patterns and external factors.

Some applications of time series forecasting are as follows:

• Financial Markets: In finance, time series forecasting is used to predict stock prices, exchange rates, and market trends. Investors and analysts use historical data to forecast future movements and make trading decisions.
• Weather Forecasting: Meteorological departments use time series data to predict weather conditions. By analyzing past weather data, they forecast future weather patterns, helping in planning and decision-making for agriculture, travel, and disaster management.
• Sales and Marketing: Businesses utilize time series forecasting to predict future sales, demand, and consumer behavior. This helps in inventory management, setting sales targets, and developing marketing strategies.
• Energy Sector: Energy companies forecast demand and supply to optimize production and distribution. Time series forecasting helps in predicting energy consumption patterns, enabling efficient energy management and planning.
• Healthcare: In the healthcare sector, time series forecasting is used to predict disease outbreaks, patient admissions, and medical inventory requirements. This assists in healthcare planning, resource allocation, and policy making.

Foundation Time Series Models

Foundational AI models are extensive, pre-trained models that form the basis for various artificial intelligence applications. They are trained on large and diverse datasets, enabling them to discern patterns, connections, and structures within the data. The term “foundational” refers to their capacity for being fine-tuned or modified for tasks or domains with minimal additional training. In the context of time-series forecasting, these models are constructed similarly to large language models (LLMs), utilizing transformer architectures. Like LLMs, they are trained to predict the subsequent or missing element in a data sequence. However, unlike LLMs, which process text as subwords through transformer layers, foundational time-series models treat sequences of continuous time points as tokens, allowing them to sequentially process time-series data.

Recently, various foundational models have been developed for time series data. With better understanding and choosing the appropriate foundational model, we can more effectively and efficiently leverage their capabilities. In the subsequent sections, we will explore the different foundational models available for time series data analysis.

• TimesFM: Developed by Google Research, TimesFM is a decoder-only foundational model with 200 million parameters. The model is trained on a dataset of 100 billion real-world time points, encompassing both synthetic and actual data from varied sources such as Google Trends and Wikipedia Pageviews. TimesFM is capable of zero-shot forecasting in multiple sectors, including retail, finance, manufacturing, healthcare, and the natural sciences, across different time granularities. Google intends to release TimesFM on its Google Cloud Vertex AI platform, providing its sophisticated forecasting features to external clients.
• Lag-Llama: Created by researchers from the Université de Montréal, Mila-Québec AI Institute, and McGill University, Lag-Llama is a foundational model designed for univariate probabilistic time series forecasting. Build on the foundation of Llama, the model employs a decoder-only transformer architecture which uses variable sizes time lags and time resolutions for forecasting. The model is trained on diverse time series datasets from several sources across six different groups including energy, transportation, economics, nature, air quality and cloud operations. The model is conveniently accessible through the Huggingface library.
• Moirai: Developed by Salesforce AI Research, Moirai is a foundational time series model designed for universal forecasting. Moirai is trained on the Large-scale Open Time Series Archive (LOTSA) dataset, which contains 27 billion observations from nine distinct domains, making it the largest collection of open time series datasets. This diverse dataset allows Moirai to learn from a wide range of time series data, enabling it to handle different forecasting tasks. Moirai uses multiple patch size projection layers to capture temporal patterns across various frequencies. An important aspect of Moirai is to use any-variate attention mechanism, allowing forecasts across any number of variables. The code, model weights, and data associated with Moirai are available in the GitHub repository called “uni2ts“

• Chronos: Developed by Amazon, Chronos is a collection of pre-trained probabilistic models for time series forecasting. Built on the T5 transformer architecture, the models use a vocabulary of 4096 tokens and have varying parameters, ranging from 8 million to 710 million. Chronos is pretrained on a vast array of public and synthetic data generated from Gaussian processes. Chronos differs from TimesFM in that it is an encoder-decoder model, which enables the extraction of encoder embeddings from time series data. Chronos can be easily integrated into a Python environment and accessed via its API.

• Moment: Developed collaboratively by Carnegie Mellon University and the University of Pennsylvania, Moment is a family of open-source foundational time series models. It utilizes variations of T5 architectures, including small, base, and large versions, with the base model incorporating approximately 125 million parameters. The model undergoes pre-training on the extensive “Time-series Pile,” a diverse collection of public time-series data spanning various domains. Unlike many other foundational models, MOMENT is pre-trained on a wide spectrum of tasks, enhancing its effectiveness in applications such as forecasting, classification, anomaly detection, and imputation. The complete Python repository and Jupyter notebook code are publicly accessible for utilizing the model.

The Bottom Line

Time series forecasting is a crucial tool across various domains, from finance to healthcare, enabling informed decision-making based on historical patterns. Advanced foundational models like TimesFM, Chronos, Moment, Lag-Llama, and Moirai offer sophisticated capabilities, leveraging transformer architectures and diverse training datasets for accurate forecasting and analysis. These models provide a glimpse into the future of time series analysis, empowering businesses and researchers with powerful tools to navigate complex data landscapes effectively.

Image generated with DALLE-3

Learning computer science from scratch can be quite difficult, and you might feel discouraged before you even begin. However, the open-source community on GitHub has created a wealth of resources that can guide you through this journey. In this blog post, we will explore 10 essential GitHub repositories that can help you learn the necessary concepts and tools to master computer science and secure a job at a top tech company.

1. Developer Roadmap

Link: kamranahmedse/developer-roadmap

The repository contains community-driven road maps that have become increasingly popular among developers, providing interactive and comprehensive guides for learning various technologies and programming languages.

These roadmaps cover a wide range of topics, including frontend, backend, DevOps, full stack, computer science, data structures, AI, data science, MLOps, QA, Python, software architecture, game development, software design, JavaScript, TypeScript, C++, and more.

2. Computer Science Degree Program

Link: ossu/computer-science

The Open Source Society University (OSSU) offers a free, comprehensive, and community-driven computer science curriculum that provides a complete education in fundamental computer science concepts. The curriculum is designed according to the degree requirements of undergraduate computer science majors at leading universities and taught by professors from prestigious institutions such as MIT, Harvard, and Princeton.

The coursework covers essential topics such as programming languages, algorithms, data structures, operating systems, computer architecture, and software engineering, with advanced electives available for specialization.

3. freeCodeCamp

Link: freeCodeCamp/freeCodeCamp

freeCodeCamp is the platform that offers free courses and tutorials on all kinds of computer technologies. It also provides a friendly and supportive community for individuals looking to learn coding skills. freeCodeCamp, run by a charity organization, aims to assist millions of adults in transitioning into tech careers.

It provides a comprehensive and self-paced curriculum in full-stack web development and machine learning, featuring thousands of interactive coding challenges to enhance coding abilities.

4. System Design Primer

Link: donnemartin/system-design-primer

The repository serves as a comprehensive resource for learning system design principles and preparing for system design interviews. It is maintained by Donne Martin, a software engineer who has worked at companies like Amazon and Google.

It covers a wide range of topics related to designing large-scale systems, including topics such as: system design basics, key characteristics of distributed systems, system design problems, Anki Flashcards, and system design interview questions with solutions.

5. Free Programming Books

Link: EbookFoundation/free-programming-books

It is a community-driven project aimed to provide a curated list of high-quality programming books that can be accessed and downloaded for free, covering a wide range of programming languages, frameworks, and related topics.

It provides free books in English and also other languages like Arabic, Korean, Italian, Tamil, and more. The repository also provides additional resources like multi language courses on various programming languages, Interactive programming resources, podcast, and programming playgrounds.

6. Awesome Computer Science Opportunities

Link: anu0012/awesome-computer-science-opportunities

It is a list of awesome computer science opportunities and resources for students and professionals.

It covers a wide range of topics, including competitive programming platforms, web development tutorials, mobile development courses, DevOps bootcamps, data science competitions, artificial intelligence resources, computer science fundamentals, open-source projects, cybersecurity blogs, online courses (MOOCs), fellowships and scholarships, programming events and hackathons, general opportunities like student packs, and community-driven demo projects.

The list aims to provide a comprehensive collection of valuable resources to help individuals learn, practice, excel in various areas of computer science.

7. Coding Interview University

Link: jwasham/coding-interview-university

The repository contains a study plan for becoming a software engineer at a large company, compiled by jwasham. It is meant for individuals who do not have a formal computer science education or degree just like the author.

The study plan is comprehensive and aims to cover most of the important topics typically taught in a university computer science program, with a focus on data structures, algorithms, and coding interview preparation.

It provides a structured approach, including a table of contents, daily plan, coding practice, and a list of topics to study.

8. Build Your Own X

Link: codecrafters-io/build-your-own-x

It is a repository that contains a collection of well-written, step-by-step guides for recreating various technologies from scratch. It aims to provide a great way to learn and understand different technologies by building them from the ground up.

The repository is structured into different categories, each focusing on a specific type of technology like 3D Renderer, Augmented Reality, Command-Line Tool, Database, Docker, Game, Git, and more. Each category contains a list of links to tutorials, articles, and resources that guide users through the process of building various technologies from scratch.

9. Public APIs

Link: public-apis/public-apis

Public APIs is a curated list of free APIs that developers can explore and integrate into their projects with ease. These APIs cover a wide range of categories, from animals and entertainment to finance and health, providing access to data and services that would otherwise be time-consuming and costly to develop in-house.

One of the key advantages of using public APIs is the ability to enhance your product's features rapidly and build MVP fast. For instance, if you're building a car-related app, you can seamlessly integrate a Public API that provides information about different types of cars. This not only saves development time but also ensures that your users have access to accurate and up-to-date data.

10. Awesome List

Link: sindresorhus/awesome

It is a curated list of awesome lists on GitHub that cover a wide range of interesting topics. These lists contain resources organized by various categories such as platforms, programming languages, development tools, and more.

The Awesome list is a massive community project where community members create a pull request and update the existing list. It is the only resource that you will ever need while learning about various tools and concepts of computer science. The awesome list is also useful for professionals who want to build durable and secure applications.

Final Thoughts

Whether you are a novice or an expert in the field, these resources are designed to meet a diverse range of skill levels and interests. They not only establish a strong foundation in fundamental computer science concepts but also provide roadmaps for the specialization and exploration of state-of-the-art technologies.

The true power of these GitHub repositories lies in their collaborative nature. They are maintained and contributed to by passionate developers and educators from around the world, ensuring that the content remains up-to-date, relevant, and constantly evolving.

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in technology management and a bachelor's degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.

More On This Topic

• 10 GitHub Repositories to Master Machine Learning
• 10 GitHub Repositories to Master MLOps
• Learn Data Science From These GitHub Repositories
• 25 Github Repositories Every Python Developer Should Know
• Learn Data Engineering From These GitHub Repositories
• Learn Machine Learning From These GitHub Repositories

Image by Author

For many of us, exploring the possibilities of LLMs has felt out of reach. Whether it's downloading complicated software, figuring out coding, or needing powerful machines — getting started with LLMs can seem daunting. But just imagine, if we could interact with these powerful language models as easily as starting any other program on our computers. No installation, no coding, just click and talk. This accessibility is key for both developers and end-users. llamaFile emerges as a novel solution, merging the llama.cpp with Cosmopolitan Libc into a single framework. This framework reduces the complexity of LLMs by offering a one-file executable called “llama file”, which runs on local machines without the need for installation.

So, how does it work? llamaFile offers two convenient methods for running LLMs:

• The first method involves downloading the latest release of llamafile along with the corresponding model weights from Hugging Face. Once you have those files, you're good to go!
• The second method is even simpler — you can access pre-existing example llamafiles that have weights built-in.

In this tutorial, you will work with the llamafile of the LLaVa model using the second method. It's a 7 Billion Parameter model that is quantized to 4 bits that you can interact with via chat, upload images, and ask questions. The example llamafiles of other models are also available, but we will be working with the LLaVa model as its llamafile size is 3.97 GB, while Windows has a maximum executable file size of 4 GB. The process is simple enough, and you can run LLMs by following the steps mentioned below.

Step 1: Download the llamafile

First, you need to download the llava-v1.5-7b-q4.llamafile (3.97 GB) executable from the source provided here.

Step 2: Grant Execution Permission (For macOS, Linux, or BSD Users)

Open your computer’s terminal and navigate to the directory where the file is located. Then run the following command to grant permission for your computer to execute this file.

chmod +x llava-v1.5-7b-q4.llamafile

Step 3: Rename the File (For Windows Users)

If you are on Windows, add “.exe” to the llamafile’s name on the end. You can run the following command on the terminal for this purpose.

rename llava-v1.5-7b-q4.llamafile llava-v1.5-7b-q4.llamafile.exe

Step 4: Run the llamafile

Execute the llama file by the following command.

./llava-v1.5-7b-q4.llamafile -ngl 9999

⚠️ Since MacOS uses zsh as its default shell and if you run across zsh: exec format error: ./llava-v1.5-7b-q4.llamafile error then you need to execute this:

bash -c ./llava-v1.5-7b-q4.llamafile -ngl 9999

For Windows, your command may look like this:

llava-v1.5-7b-q4.llamafile.exe -ngl 9999

Step 5: Interact with the User Interface

After running the llamafile, it should automatically open your default browser and display the user interface as shown below. If it doesn’t, open the browser and navigate to http://localhost:8080 manually.

Image by Author

Let's start by interacting with the interface with a simple question to provide some information related to the LLaVa model. Below is the response generated by the model:

Image by Author

The response highlights the approach to developing the LLaVa model and its applications. The response generated was reasonably fast. Let’s try to implement another task. We will upload the following sample image of a bank card with details on it and extract the required information from it.

Image by Ruby Thompson

Here’s the response:

Image by Author

Again, the response is pretty reasonable. The authors of LLaVa claim that it attains top-tier performance across various tasks. Feel free to explore diverse tasks, observe their successes and limitations, and experience the outstanding performance of LLaVa yourself.

Once your interaction with the LLM is complete, you can shut down the llama file by returning to the terminal and pressing "Control — C".

Wrapping Up

Distributing and running LLMs has never been more straightforward. In this tutorial, we explained how easily you can run and experiment with different models with just a single executable llamafile. This not only saves time and resources but also expands the accessibility and real-world utility of LLMs. We hope you found this tutorial helpful and would love to hear your thoughts on it. Additionally, if you have any questions or feedback, please don't hesitate to reach out to us. We're always happy to help and value your input.

Thank you for reading!

Kanwal Mehreen Kanwal is a machine learning engineer and a technical writer with a profound passion for data science and the intersection of AI with medicine. She co-authored the ebook "Maximizing Productivity with ChatGPT". As a Google Generation Scholar 2022 for APAC, she champions diversity and academic excellence. She's also recognized as a Teradata Diversity in Tech Scholar, Mitacs Globalink Research Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having founded FEMCodes to empower women in STEM fields.

More On This Topic

• How to Package and Distribute Machine Learning Models with MLFlow
• Learn How to Run Alpaca-LoRA on Your Device in Just a Few Steps
• Schedule & Run ETLs with Jupysql and GitHub Actions
• When to Retrain an Machine Learning Model? Run these 5 checks to…
• Run an LLM Locally with LM Studio
• How to Make Python Code Run Incredibly Fast