Hybrid on-device inference on Android: llama.cpp + LiteRT + NPU/GPU routing

Hi everyone, I’m the maintainer of Box — a fork of Google’s AI Edge Gallery that I’ve been extending into a fully offline AI assistant for Android. Full disclosure: I built this project. It runs entirely on-device (no cloud, no accounts, no external inference), and combines multiple local inference backends in a single app. What I’ve been experimenting with The goal was to see how far a fully offline mobile AI stack could be pushed using: llama.cpp (GGUF LLM inference) whisper.cpp (on-device STT

More from LocalLlama

Bruh

LocalLlama·2

Qwen3.6-27B-NVFP4 - images

LocalLlama·5

MiniMax M2.7 AWQ-4bit on 2x Spark vs 2x RTX 6000 96GB - performance and energy efficiency

LocalLlama·4

Qwen3.6-27B at 72 tok/s on RTX 3090 on Windows using native vLLM (no WSL, no Docker), portable launcher and installer

LocalLlama·6

A Dark-Money Campaign Is Paying Influencers to Frame Chinese AI as a Threat

LocalLlama·6

"LLM is created so engineer don't have to write a report", anyway found out ONLYOFFICE can connect to OpenAI compatible, using Qwen 3.6 to do elaboration.

LocalLlama·5