2026/09/05¶
Creating a Local LLM server with Ubuntu Server¶
Last year when I was running NixOS as my daily driver, I set up a local LLM with SillyTavern as the front end. Since then I have held onto this idea of running the same thing over my LAN via a headless server.
So today I havce decided to decommission PVE2 and turn it into a dedicated AI server.
-
Wiped PVE2, this will create some work for me later with PVE1 since I had set up a proxmox cluster last year, but I that is my decision to repurpose this machine.
-
Installed Ubuntu Server 26.04 LTS minimised, with LUKS. I have a monitor and keyboard installed for now for LUKS but am interested to set up remote unlocking, and I prefer my drives to be encrypted.
-
Assigned a static IP reservation in the routers DHCP and then set an SSH alias in my fedora workstation.
-
Obligatory
sudo apt update && apt full upgrade -y
Setting up remote unlocking for LUKS¶
I have encrypted the fresh install of Ubuntu Server with LUKS, but I intend to keep this as a headless server, so I will be setting up remote unlocking with dropbear-initramfs.
- Installed
dropbear-initramfs - Created dedicated SSH key for remote unlocking
- Configured initramfs networking so the server is reachable before the root filesystem is mounted, rebuilt initramfs.
- Test reboot successful, unlocked ubuntu-server remotely from my fedora workstation over SSH.
Setting Up Server Side Local LLM¶
- Installed NVIDIA drivers
- Installed OLLAMA, pulled and installed a model
- Asked it to write a short poem about Shrek, it did successfully
- Confirmed running on GPU
Local Network Access¶
- Added a local DNS record using Pi-hole so SillyTavern can be accessed by hostname rather than IP address.
- Added a Caddy reverse proxy in front of SillyTavern.
- Confirmed access from my Fedora workstation and other devices on the LAN.