Run local models on your own GPU without the command line. AZYL.inference installs and runs Ollama for you, loads models when they’re asked for and lets them go when they’re not, and puts every setting that matters on one page, with a friendly face on whatever is running.
Written in Rust · one 6 MB binary · Ollama underneath · nothing leaves your machine
Read on
One tool wants to own your desktop. The other wants environment variables. The context window hides in a Modelfile, concurrency in a variable you set before the server starts, and what’s using your VRAM in a command you have to remember. Your agents reach the model through an OpenAI endpoint that can’t even say how much context it needs.
AZYL.inference runs the engine for you and puts every one of those settings on a page, where you can see them, change them, and watch what they do.
Ollama does the inference. AZYL.inference does everything around it: installing it, keeping it running, deciding what’s in memory, and showing you, at a glance, what it’s doing.
One click installs Ollama from its official release, with the ROCm libraries for AMD cards, into your home folder. AZYL starts it, stops it, updates it and brings it back after a crash. Its log is one click away.
A model loads the moment an app asks for it and unloads when it’s been idle long enough. Keep one warm for good, preload the ones you need at startup, or load and unload by hand.
Parallel requests per model, models in memory at once, context windows from 4K to 256K, flash attention and a leaner context cache. Save, and it restarts the engine for you.
Context, sampling and the system prompt are saved into the model itself, so apps on the OpenAI-compatible endpoint get them too. Make a named copy per agent, rename it, and nothing is duplicated.
It finds the GGUFs LM Studio downloaded, with their vision projectors, and imports them, keeping the sampling each file recommends. Or pull from the Ollama library and Hugging Face.
One switch publishes it privately with Tailscale. Only your Tailscale account gets in, with no password and nothing open to the internet, and there’s an Android app for it.
The AZYL dot sleeps when nothing is loaded, raises its shield while a model loads, waits with big eyes when one is ready, and gets busy while it generates. The numbers are right underneath.
A front-end that supervises a GPU server has to be the steady part. AZYL.inference is one small native binary that starts in milliseconds, holds no runtime, and cleans up after itself.
An async Rust server with the web app built in. Installed as a systemd user service, opened as its own window.
Ollama runs in its own process group, restarts after a crash, and stops with the app, even if the app is killed outright.
Ollama comes from its GitHub release, is checked against the published SHA-256, and is swapped in only once it’s whole.
The app and the engine listen on this computer only, and Ollama’s cloud features are off. Your prompts stay on your GPU.
Remote access goes through tailscale serve, and every request must carry an allowed Tailscale identity.
A DNS-rebinding guard, a custom header on every change and a strict content policy, so a web page can’t drive your models.
Measured on the machine in the screenshot: a 27B model (Q4_K_M) loads in about 12 seconds and generates around 37 tokens per second on a Radeon RX 7900 XTX; importing its 17 GB file from LM Studio takes under a minute.
Your models, your hardware, no terminal required.
AZYL.inference is free for personal and other noncommercial use. Download it for Linux, with the Android app alongside; for commercial use, get in touch.
Linux on x86_64, glibc 2.39 or newer (Fedora 40+, Ubuntu 24.04+, Debian 13+) · Android 11+ · PolyForm Noncommercial 1.0.0
The screenshot shows AZYL.inference running on a Radeon RX 7900 XTX with demo model names; the sketches above are illustrative.