# How to Run and Deploy LLMs on your iOS or Android Phone

Source: https://www.youtube.com/watch?v=sBfN7XvvZEo
Recap page: https://rapidrecap.app/video/sBfN7XvvZEo
Generated: 2025-12-19T18:12:58.368+00:00

---
## Quick Overview

Meta's Llama 3 8B model can be successfully run and deployed locally on both iOS and Android devices, offering 40 tokens per second performance while maintaining 70% of the accuracy of the full model by employing quantization and specialized execution engines like ExecuteTorch.

**Key Points:**
- The Llama 3 8B model, when quantized, achieves 40 tokens per second performance on mobile devices like the iPhone 15 Pro.
- Quantization (specifically 4-bit or 5-bit) allows the model to retain about 70% of its full accuracy while drastically reducing memory footprint.
- Meta utilizes its ExecuteTorch engine to manage on-device execution, ensuring the model runs efficiently on hardware accelerators like Apple's Neural Engine or Qualcomm's NPU.
- Running models locally offers significant advantages in privacy, as user inputs, queries, and custom data never leave the device, eliminating network latency.
- The deployment process involves installing a minimal SDK/NDK environment and transferring the quantized model file (e.g., .pte file) to the device.
- A key barrier to entry for on-device LLMs, especially on iOS, is the rigid requirement to use the Apple developer program enrollment for running even test builds.

![Screenshot at 00:05: The presenter highlights the breakthrough achievement of running powerful models like Llama 3 locally on mobile devices, demonstrating the core value proposition of the technique discussed.](https://ss.rapidrecap.app/screens/sBfN7XvvZEo/00-00-05.png)

**Context:** This video provides a technical deep dive into deploying large language models (LLMs), specifically Meta's Llama 3 8B, directly onto mobile hardware (iOS and Android). The discussion centers on overcoming hardware and software constraints through techniques like quantization to achieve real-time performance suitable for applications like private, on-device AI.

## Detailed Analysis

The video details the process and benefits of running powerful LLMs, specifically Meta's Llama 3 8B, locally on smartphones, contrasting it with the traditional cloud-based server model. The speaker emphasizes that this local deployment maintains privacy, eliminates network latency, and provides instant responses. The key technical enabler is quantization, which compresses the model (like using 4-bit or 5-bit precision instead of 16-bit) to fit within limited mobile memory while retaining approximately 70% of the original accuracy. Meta's ExecuteTorch engine is identified as the core tool managing execution on mobile hardware accelerators like Apple's Neural Engine or Qualcomm's NPUs. A successful deployment on an iPhone 15 Pro yielded 40 tokens per second. The speaker notes that while the process is documented, the complexity lies in deployment, especially the strict requirements for Apple's developer ecosystem (like requiring a paid developer program enrollment for testing), which introduces an entry barrier that Android avoids via command-line jobs. Ultimately, the goal is to make sophisticated AI accessible and private on the device itself.

### Llama 3 Local Deployment Breakthrough

- Meta ExecuTorch delivers powerful AI on mobile
- Quantization maintains 70% accuracy while reducing size
- 40 tokens/second achieved on iPhone 15 Pro

### Advantages of On-Device AI

- Total privacy as data never leaves the phone
- Instant response times eliminate network latency
- Enables use in offline environments like airplanes or underground

### Technical Requirements and Limitations

- Requires specific SDK/NDK setup with correct version numbers (e.g., Java 17, NDK)
- iOS deployment faces high friction due to Apple's paid developer program for testing
- Android deployment is slightly smoother via command line jobs

### Model Comparison and Performance

- Llama 3 8B (472MB quantized) runs well on phones, unlike larger models that require more memory
- The process democratizes access to powerful, private AI, moving beyond centralized server farms

![Screenshot at 00:00: The introductory screen featuring the podcast graphic and a call to action to 'Become A Member Today!' against a background of a green waveform.](https://ss.rapidrecap.app/screens/sBfN7XvvZEo/00-00-00.png)
![Screenshot at 00:10: Visual representation of the AI processing speed, referencing the 40 tokens per second achieved on a smartphone.](https://ss.rapidrecap.app/screens/sBfN7XvvZEo/00-00-10.png)
![Screenshot at 01:14: Text overlay or visual emphasis on the primary benefit: 'fundamentally privacy first' due to local processing.](https://ss.rapidrecap.app/screens/sBfN7XvvZEo/00-01-14.png)
![Screenshot at 03:36: Visual representation of the performance trade-off, contrasting the speed of quantized models versus the accuracy loss.](https://ss.rapidrecap.app/screens/sBfN7XvvZEo/00-03-36.png)
![Screenshot at 09:30: Visual of the discussion turning to the challenge of deployment rigidity, contrasting the Apple/iOS process with the Android command-line approach.](https://ss.rapidrecap.app/screens/sBfN7XvvZEo/00-09-30.png)