# Nexa AI: Create Local AI Chatbot for FREE! (EASY Guide)

Source: https://www.youtube.com/watch?v=0k_B6XCwzy8
Recap page: https://rapidrecap.app/video/0k_B6XCwzy8
Generated: 2025-11-07T18:06:26.848+00:00

---
## Quick Overview

The video demonstrates how to create a free, locally-run AI chatbot using the Nexa SDK, which is an open-source developer toolkit that allows running any AI model on any backend across various hardware, including NPUs, GPUs, and CPUs, by following installation steps for the CLI, running a local inference server, and then building a basic web UI using Chainlit and ChromaDB for Retrieval Augmented Generation (RAG) against private data like a PDF.

**Key Points:**
- Nexa SDK is an open-source toolkit enabling local inference for any AI model across diverse hardware (NPU, GPU, CPU) and platforms, featuring NPU-first support and compatibility with GGUF, MLX, and Nexa formats.
- The demonstration involved installing the Nexa CLI, running the local Nexa AI Server on http://127.0.0.1:18181/docs/ui, and successfully running inference on a Gemma model locally (02:57).
- A Chainlit application was built using an OpenAI-compatible client pointing to the local Nexa server (base_url="http://127.0.0.1:18181/v1"), enabling streaming responses (04:33).
- For RAG implementation, necessary packages (PyPDF2, chromadb, sentence-transformers) were installed, and ChromaDB was used as the vector store to index chunks generated from an uploaded PDF file (09:01).
- The RAG code incorporates functions to chunk text, generate embeddings using 'all-MiniLM-L6-v2', store chunks in ChromaDB, and query the database based on user input before passing context to the LLM for the final answer (08:28, 09:43).
- The final result shows the Chainlit UI successfully answering a question about the uploaded IBM PDF document using the locally served Nexa AI model (10:58).

![Screenshot at 00:08: The title card displaying the core promise of NexaSDK: "Run any AI model on any backend," which is the central theme of the tutorial.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-00-08.png)

**Context:** The video serves as a tutorial explaining how to leverage the Nexa SDK, an open-source AI inference engine, to build a custom Retrieval Augmented Generation (RAG) chatbot that runs entirely locally on a user's machine. This setup bypasses cloud services by using the local Nexa AI server and integrating it with Python libraries like Chainlit for the front-end UI and ChromaDB for vector storage of private documents.

## Detailed Analysis

The tutorial begins by introducing NexaSDK as an easy-to-use developer toolkit for running any AI model locally across NPUs, GPUs, and CPUs, powered by the NexaML engine built from scratch for peak performance. A comparison chart highlights NexaSDK's advantages over Ollama, llama.cpp, and LM Studio, particularly its NPU-first support and broader format compatibility (GGUF, MLX, Nexa) (00:12). The process starts with installing the Nexa CLI via one-click downloads for different operating systems (02:27). After installation, the user runs the local server using 'nexaserve' (03:26), which hosts an OpenAPI-compatible API on http://127.0.0.1:18181/docs/ui (03:30). Next, the tutorial shifts to building a RAG chatbot interface using Chainlit and integrating the local Nexa server via the OpenAI API structure (04:05). The Python code imports necessary libraries including AsyncOpenAI, ChromaDB, PyPDF2, and sentence-transformers (08:28). The core RAG logic involves functions to chunk text from uploaded files (like PDFs), generate embeddings using the 'all-MiniLM-L6-v2' model, and store these vectors in a ChromaDB collection named 'documents' (08:45). When a user asks a question, the query is embedded, the vector database is queried for the top 3 relevant chunks (09:43), and this context is appended to the message history before being sent to the Nexa AI model running via the local endpoint (09:53). The demonstration concludes with successfully uploading a PDF about 'agentir-ai-strategic-ascent-ibm.pdf', indexing its 12 chunks, and receiving a detailed, context-aware summary from the locally running chatbot (10:50).

### Nexa SDK Key Capabilities

- NPU-first support
- Supports GGUF, MLX, NEXA formats
- Unified inference engine built at kernel level
- Full multimodality support (Image, Audio, Text)
- Cross-platform support (Desktop, Mobile, Automotive, IoT)

### Local Server Setup

- Install Nexa CLI via one-click downloads (02:27)
- Start the local inference server using 'nexaserve' (03:26)
- Server runs locally accessible at http://127.0.0.1:18181/docs/ui (03:30)

### Chatbot Development Setup

- Install required packages: openai, chainlit, PyPDF2, chromadb, sentence-transformers (04:06, 10:06)
- Use AsyncOpenAI client pointing to the local Nexa base URL (http://127.0.0.1:18181/v1) (08:31)

### RAG Implementation - Data Ingestion

- Implement 'get_embedding' and 'chunk_text' functions (08:45)
- Use ChromaDB client to create/manage a 'documents' collection (09:15)
- Iterate through chunks, generate embeddings, and add them to ChromaDB (09:21)

### RAG Implementation - Querying

- In the 'main' function, retrieve collection and message history (09:35)
- Generate query embedding, query ChromaDB for top 3 results (09:43)
- Construct enhanced message using context and query (09:55)
- Send enhanced message to Nexa AI model via client.chat.completions.create with stream=True (10:02)

### Final Demonstration

- Upload the PDF 'agentir-ai-strategic-ascent-ibm.pdf' (10:36)
- System processes and indexes 12 chunks (10:41)
- User asks 'Tell me what is written in this IBM file' (10:47)
- The locally served chatbot streams an accurate summary based on the document context (10:58)

![Screenshot at 00:04: The initial interaction showing the user querying the AI about its capabilities, receiving a response confirming operational status.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-00-04.png)
![Screenshot at 00:06: The Nexa SDK GitHub repository overview, showing it is trusted by partners like Qualcomm, NVIDIA, AMD, and Intel.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-00-06.png)
![Screenshot at 00:15: The feature differentiation table where NexaSDK shows superior support across key areas compared to Ollama, llama.cpp, and LM Studio.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-00-15.png)
![Screenshot at 00:44: Python code snippet demonstrating the use of the OpenAI client structure to connect to the local Nexa AI server using base\_url and api\_key.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-00-44.png)
![Screenshot at 03:37: The Nexa AI Server documentation page \(Swagger UI\) displaying the available OpenAPI compatible endpoints like /v1/chat/completions.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-03-37.png)
![Screenshot at 05:55: The Chainlit application interface prompting the user to upload a text file to begin processing data for the RAG pipeline.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-05-55.png)
![Screenshot at 08:20: The RAG architecture diagram illustrating the two main steps: 1\) Indexing private data sources into embeddings stored in a DB, and 2\) User query retrieval and LLM generation.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-08-20.png)
![Screenshot at 09:18: Python code showing the vectorization process: chunking text from a PDF, generating embeddings, and adding them to the ChromaDB collection.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-09-18.png)
![Screenshot at 10:24: Terminal output showing the RAG application running, loading the sentence-transformer embedding model, and confirming the app is available on localhost:8000.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-10-24.png)
![Screenshot at 10:58: The final Chainlit UI displaying the RAG response, summarizing the contents of the uploaded IBM PDF document accurately.](https://ss.rapidrecap.app/screens/0k_B6XCwzy8/00-10-58.png)