Getting a handle on my homelab with Proxmox and cloud-init

Introduction

In my first homelab post I talked about how I got Proxmox up and running and the hardware I picked up along the way. What I didn’t mention is how messy things got once I started actually running things on it.

For a long time every VM was built by hand. Download an ISO, click through the installer, create a user, copy over an SSH key, install the same handful of packages and then forget exactly what I did the next time around. Containers were even worse, I built them in the GUI and never really went back to them.

Cloud-init is what finally fixed that for me. It’s something I’d heard mentioned a lot but never looked into, and I wish I’d done it a lot sooner. This post covers how I’ve set it up, what it looks like in practice and a handful of things that bit me.

The old way

If you’ve never installed Debian by hand more than once or twice it probably sounds fine. I’ve done it more times than I’d like to admit, and it never got any less tedious.

Every single VM meant going through the same installer screens. Pick the language, pick the location, pick the keyboard layout, set the hostname, set the domain, set a root password, create a user, set a password for that user, choose the partitioning, choose the mirror, deselect the desktop environment, tick SSH server and then wait. I ticked the same options in the same order every time, and I still managed to miss one now and again and only noticed after the install had finished.

That was only the start of it though. Once Debian was installed I still had to get the machine into a usable state:

• Log in and work out what IP address it had been given.
• Copy my SSH key across.
• Install the same handful of packages as every other machine.
• Make up yet another password for the new user, and another one for root.

The passwords were the part I disliked most. Every new machine meant a new set of credentials that I had to store somewhere and remember to rotate. Managing them across more and more machines turned into a job in itself, and the more machines I added the worse it got. Then there was the enrolment step of getting each new machine into my management tooling so I could actually reach it and keep it updated. That was another manual job which I regularly forgot to do, so some machines ended up half set up and nobody was looking after them.

None of it was difficult, it was just repetitive, which is exactly the sort of thing a computer should be doing for me.

What is cloud-init?

Whats cloud-init? It’s the tool that most cloud providers use to configure a server the first time it boots. You hand it a bit of YAML describing the hostname, the user, SSH keys, packages and so on, and the machine sets itself up before you ever log in.

Proxmox has support for this built in. You attach a tiny cloud-init drive to a VM, fill in some fields in the GUI (or on the command line) and Proxmox turns those into the YAML for you.

Part 1 - Building the template

I used the Debian 13 (trixie) genericcloud image rather than the normal installer ISO. This was a deliberate choice. The cloud image boots straight into cloud-init with no interactive installer at all, so the only way the machine ever gets configured is through the user-data. You can grab it from the Debian cloud images page.

I did all of this over SSH on the Proxmox node. The commands below are the rough shape of what I ran rather than an exact transcript:

qm create 114 --name debian13-cloud-template --memory 1024 --cores 1 \
  --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single \
  --scsi0 local-lvm:0,import-from=/root/debian-13-genericcloud-amd64.qcow2,discard=on,ssd=1 \
  --ide2 local-lvm:cloudinit --citype nocloud --agent enabled=1 \
  --serial0 socket --vga serial0 --ostype l26 --ciuser debian
qm template 114

A few of those options are worth explaining:

• --ide2 local-lvm:cloudinit is the cloud-init drive. It’s just a small ISO attached to the VM as a CD.
• --citype nocloud uses the NoCloud datasource. Proxmox generates the ISO and cloud-init reads it on boot.
• --serial0 socket --vga serial0 sends the console over serial so qm terminal works without needing VNC. This is really handy when something goes wrong on first boot.
• qm template turns the VM into a template, which is what lets you clone it later.

I also set the CPU type to x86-64-v2-AES rather than leaving the default. The default kvm64 hides some instruction sets and a later version of MySQL refused to run on it.

One last thing I did was add the template to my nightly backup job. Templates can be backed up and it’s the base of every VM I’ll ever build, so it seemed silly not to.

Part 2 - What cloud-init actually sees

This is the bit that took me a while to get my head around. There are two layers of user-data.

The template has its own, which you can read with qm cloudinit dump 114 user:

#cloud-config
hostname: debian13-cloud-template
manage_etc_hosts: true
fqdn: debian13-cloud-template
user: debian
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

When you clone the template, Proxmox regenerates all of this from the clone’s own settings. Here’s what a freshly built VM gets:

#cloud-config
hostname: test-vm
manage_etc_hosts: true
fqdn: test-vm
user: alfred
ssh_authorized_keys:
  - from="<allowed-source-ip>",restrict ssh-ed25519 AAAA...redacted guests@lab
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

The network config is a separate file (qm cloudinit dump <vmid> network). Even though this is Debian 13, Proxmox still writes it in cloud-init’s older v1 schema:

version: 1
config:
    - type: physical
      name: eth0
      mac_address: 'bc:24:11:xx:xx:xx'
      subnets:
      - type: static
        address: '<vm-ip>'
        netmask: '255.255.255.0'
        gateway: '<gateway-ip>'
    - type: nameserver
      address:
      - '<gateway-ip>'
      search:
      - 'local'

Editing the template’s user-data doesn’t change any VMs you’ve already cloned. Each clone builds its own from its own settings.

The qm cloudinit dump command is the only readable view of any of this. Everything else gets URL-encoded in the VM config, so debugging by eye is a nightmare. Read the generated files rather than trusting what you think you set.

Part 3 - Cloning

Building a new VM is now a handful of commands, which I’ve wrapped up in a small Python script so I’m not typing them out each time:

qm clone 114 <vmid> --name test-vm --full
qm set <vmid> --net0 virtio,bridge=vmbr0,tag=30 \
               --ipconfig0 ip=<vm-ip>/24,gw=<gateway-ip> \
               --nameserver <gateway-ip> \
               --ciuser alfred --sshkeys /tmp/guest-key.pub \
               --cores 1 --memory 1024
qm resize <vmid> scsi0 8G
qm start <vmid>

A few things that aren’t obvious from the docs:

• The clone name isn’t cosmetic. --name feeds the hostname and fqdn in the generated user-data, and manage_etc_hosts writes /etc/hosts from it.
• --ipconfig0 is the whole static IP story. The address and gateway go in one string. The VLAN isn’t set here, it’s the tag=30 on --net0.
• --sshkeys takes a path on the node, not on your own machine. I scp the key across first.
• Containers and VMs share an ID space, so a VM and a container can’t both be 116. My script checks qm list and pct list for a free ID and has sanity limits on cores, RAM and disk size. It also does a dry run by default and only builds when I pass --confirm.

Restricting the SSH key from first boot

This is my favourite part. --sshkeys copies the file contents straight into the YAML, so key options like from="<allowed-source-ip>",restrict are written into authorized_keys exactly as I wrote them.

Before this, cloud-init injected a bare key and a script replaced it a few seconds after boot. That left a small window on every build where a copy of that key would have worked from anywhere. Now the restricted version is the only one that ever exists on the machine.

The catch is that nothing will tell you if you pass a bare key by mistake. I now check the key by fingerprint before building, using the private key itself rather than the .pub file:

ssh-keygen -y -f ~/.ssh/id_ed25519_guests | ssh-keygen -lf -
ssh-keygen -lf ~/.ssh/id_ed25519_guests.pub

Those two fingerprints must match. A stale .pub file looks completely fine while authorising nobody, and that mistake cost me a couple of rounds of confusion.

Containers don’t get cloud-init

Containers have no cloud-init at all, so I needed a different route to the same result. The IP goes on the --net0 line, since there’s no --ipconfig0, and the key goes in via --ssh-public-keys:

pct create 116 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst --hostname test-lxc \
  --unprivileged 1 --onboot 1 --cores 1 --memory 512 --swap 512 --rootfs local-lvm:8 \
  --net0 'name=eth0,bridge=vmbr0,tag=30,ip=<container-ip>/24,gw=<gateway-ip>' \
  --nameserver <gateway-ip> --ssh-public-keys /tmp/guest-key.pub
pct start 116

That writes the key into root’s authorized_keys, so I run a small enrolment script with pct exec afterwards to create my own user and then strip the root key back out.

Before and after

Before: There was no template at all. VMs came off a downloaded ISO and were set up by hand. Containers were built in the GUI and never enrolled into anything, and the account I used to manage everything was root.

After:

• Every VM comes off the same Debian template with the restricted key in place from the first boot.
• Containers get the same account, the same key and the same sudo setup, and root no longer holds a key at all.
• Each build adds itself to the matching backup job and checks that it worked.
• Each build is also recorded in my inventory, so it ends up in my Ansible inventory rather than just being a machine that happens to be running.
• Eleven hosts now respond to Ansible as a named user rather than root.

A container builds end to end in about 2 minutes 20 seconds without me touching the console. A VM takes closer to 12 minutes, though most of that is the first boot upgrade (more on that below).

Gotchas

These are the ones that cost me the most time:

1. DNS can silently kill a build. cloud-init only sets one nameserver. My Pi-hole only answers its own subnet and quietly drops everything else, so a VM on another VLAN sat on cloud-init status: running forever. Nothing mentioned DNS. Point VLAN guests at their own gateway, and check cloud-init status and resolvectl status rather than waiting for an error.
2. package_upgrade: true means a full apt upgrade on first boot. There’s no visible output unless you have the serial console attached, so budget a few minutes.
3. A fresh clone can’t answer qm agent ping, because the guest agent only arrives with my post-boot bootstrap. I was waiting around five minutes for something that couldn’t possibly respond yet.
4. A new guest can be unreachable from another subnet for a few minutes even though everything inside it is healthy. It cleared up as soon as the guest sent something out, since an ARP request is what teaches the switch where it is.
5. cloud-init already gives your user passwordless sudo. It writes /etc/sudoers.d/90-cloud-init-users from user: alone, so don’t add a second rule for the same account, or the two will drift apart.
6. OpenSSH 9.8 and later will drop a source that fails too often. That looks exactly like a broken key. If your SSH wrapper has no ConnectTimeout, one dead connection can also hang a build for 15 minutes.
7. Containers need extra care. The Debian LXC template may not ship with sudo or python3, so I install them through pct exec before trying SSH. Otherwise the host looks enrolled while Ansible’s become fails on it.

Wrapping up

Cloud-init has taken my homelab from a collection of machines I’d built by hand and half remembered to something I can rebuild on demand. If you’re already running Proxmox, building a template is an afternoon’s work and it’s well worth the time.

I’ve got a few more ideas for where to take this next, including more of the Ansible side, so I’ll cover those in a future post.