I’m thinking about starting a self hosting setup, and my first thought was to install k8s (k3s probably) and containerise everything.
But I see most people on here seem to recommend virtualizing everything with proxmox.
What are the benefits of using VMs/proxmox over containers/k8s?
Or really I’m more interested in the reverse, are there reasons not to just run everything with k8s as the base layer? Since it’s more relevant to my actual job, I’d lean towards ramping up on k8s unless there’s a compelling reason not to.
deleted by creator
I am with you. So easy to use. I have Dietpi running as lightweight OS on my VMs for when i don’t want LXCs.
What I did is install proxmox on the bare metal, setup a vm in which I put the containers.
Proxmox itself stays (almost) completely stock. The only changes I’ve made to it were to add the NUT client package so it could gracefully shut down if my NUT server indicates that the UPS is running out of power during an outage.
In your VMs you can do whatever. Setup OMV, or a stock Ubuntu or Debian vm and install your services on the VM or use Docker/Podman. Setup Fedora CoreOS or IoT vms and host all your services in Podman containers.
The great thing about Proxmox is you can do snapshot backups which take mere moments to complete. Then pass those off to a NAS where they can survive a irreparable loss of your Proxmox server.
You can also spin up new vms as needed to just try to fuck around with new techs or just a new way of setting up your home lab. It gives you a ton of flexibility and makes backing stuff up way easier.
Another great thing you can do is if 3 years down the line you are looking to replace your server hardware with some newer or more powerful stuff you can just add the new device as a node to the cluster. Then you can migrate all your existing VMs over to your new hardware and decommission your old one with very little to no downtime on anything.
This is my exact setup as well. Proxmox with one beefy vm dedicated just to docker and then a few other vms for non docker workloads (eg, home assistant, pihole, jelltfin). I can probably run those in docket as well, but the to worked better as vms when I set them up
Appreciate your take on this and specifically mentioning that you have a VM for Home Assistant. That was a lightbulb moment for me as I like how easy it is to manage updates as an OS install rather than in a Docker container. If I ever get around to rebuilding my server architecture I’m definitely going to do this!
The great thing about Proxmox is you can do snapshot backups which take mere moments to complete. Then pass those off to a NAS where they can survive a irreparable loss of your Proxmox server.
Hopefully you put a giant asterix by this point. You need the snapshot AND the original backup. Snapshots are only diffs and can’t survive without their base backup.
I have a similar setup, but 2 VMs on each of my 2 servers, then on server 1, I have VM A running one test K3s node and VM B running one live (Production) K3s node with the same on server 2, so I can take one server full down for maintenance, but keep my test and live sites running. It’s way overkill, but allows me to learn about how to set up and maintain resilient systems. One day, I’ll do the same for my network :-(
also curious abt this
I, personally, haven’t done a whole lot of VM work but I do run a metric ass-ton of containers. I can spool up servers in docker compose on absolutely dogshit hardware and have it run serviceably. Also, the immutability of the container OS is really nice for moving things around and/or getting them set up quickly.
Where did you learn so much about Docker? Having a server at home, I’m more inclined to spin up a VM. I would like to learn more about Docker.
Just get started somewhere. I ran traditional VMs for most things before and I would never go back unless it was necessary for something.
Easiest way is just to start using Docker for some service you’re hosting that has a public image available and go from there. If you want a more visual approach there’s stuff like Portainer you can use too.
Also get started early on with docker compose, it makes it much easier to organize your container configs.
If I’m honest, I’ve stumbled nose-first through pretty much everything I know. I am never afraid to break things as long as I learn from it.
I think it depends on your scale. If homelab stuff docker is awesome IMO.
K8s are more complex than containers using proxmox. If you are up for the challenge sure go crazy.
VMs are often imperative and can be quite easy and familiar to setup for most people, but can be harder or more time-consuming to reproduce, depending on the type of update or error to be fixed. They have their own kernel and can have window managers and graphical interfaces, and can therefore also be a bit resource heavy.
Containers are declarative and are quite easy to reproduce, but can be harder to setup, as you’ll have to work by trial-and-error from the CLI. They also run on your computers kernel and can be extremely slimmed down.
They are both powerful, depends how you want to maintain and interface with them, how resource efficient you want them to be, and how much you’re willing to learn if necessary.
That sums it up really well.
I generally tend to try to use containers for everything and only branch out to VMs if it doesn’t work or I need more separation.
This is my general recommendation as containers are easier to set up and in my opinion individual software packages are easier to maintain with things like compose. I have limited time for my self hosted instance and that took away a lot of work, especially when updating.
That sums it up really well.
I generally tend to try to use containers for everything and only branch out to VMs if it doesn’t work or I need more separation.
This is my general recommendation as containers are easier to set up and in my opinion individual software packages are easier to maintain with things like compose. I have limited time for my self hosted instance and that took away a lot of work, especially when updating.
It depends on your use case and what you are trying to achieve.
You do not need k8s (or k3s…) to use containers though. Plain old containers could also suffice, or Docker Swarm if you need some container orchestration functionality.
Trying to learn k8s would be a good reason to use k8s though :)
Why not do both? I run proxmox on my physical hardware, then have guest VMs within proxmox that run k8s.
Advantages of proxmox:
- Proxmox makes it easy to spin up VMs for non self host purposes (say I want to play with NixOS)
- Proxmox snapshots make migrations and configuration changes a bit safer (I recently messed up a postgres 15 migration and was able to roll back in a button press)
You can then just run docker images through Proxmox, but I like k8s (specifically k3s) because:
Advantages of k8s:
- Certmanager means your HTTP services automatically get assigned TLS certs essentially for free (once you’ve set up cert manager for the first time, anyway)
- I find k8s’ YML-based configuration easier to track and manage. I can spin my containers up fresh just from my config, without worrying about stray environment settings I might not have backed up.
- k8s makes it easy for me to reason about which services are exposed internally to each other, and which are exposed on the host outside of my k8s cluster.
- k8s services get persistent DNS and IPs within the cluster, so configuring nodes to talk to each other is very easy.
And yeah, this way I get to learn two technologies rather than one 😁
Not a proxmox pro by any means, but it can do both VMs and containers. I have a few VMs for various Linux distros to play around with. I also have one dedicated VM for all my security related tools.
Stuff like PI hole, jellyfin, logstash, etc. dont really have any need for a full OS, so a container works perfectly. Plus having a full OS with several things running on it makes it more difficult if you just need to restart one service
I started doing everything in VMs but over time realized some things were better to maintain as containers
I have a pretty low power server at home (Pentium G4560), and the previous one was even slower J3160, so I don’t want to unnecessarily hog the CPU with a VM, and the few services I need at home run perfectly fine in containers.
I run pihole, unbound, wireguard, plex, unifi controller in containers, and I run some additional services directly on the host (samba, transmission).
I have a Windows VM on my Windows PC for work, so it’s isolated from my main rig (various VPN clients and work files etc), and if I needed some Linux stuff on my Windows PC I’d also run a VM, but more VMs also mean more updating and patching, which is much easier with containers.
Just to add my two cents: When I started out I thought I’d need a datacenter, with 10 Gig connectivity and a lot of storage. Turns out, a Raspberry Pi 4 8GB would’ve been sufficient for the things I actually use.
My recommendation would be therefore to start minimalistic and build up according to your needs from there. Start with a Raspberry PI and Docker or use a used Micro SFF and go up from there, this advice would’ve saved me a lot of money and electricity.
I personally really, really like (Docker) containers and I host most of my stuff with it, on a Raspberry Pi and on (free tier) Oracle Cloud VPS’s. I also plan to (re)install Proxmox on a spare old laptop and run some stuff in VMs on that (namely Home Assistant) and might try a NixOS server too.
So really, use both. Use the right tool for the job. And you can also run containers in VMs and even use Ansible to configure everything with playbooks, allowing you to re-run said playbooks when things go wrong.
I use proxmox for the sole benefit of just spinning up a VM of choice without having to deal with usb-sticks, etc.
From there I just run everything with Docker containers, via Portainer.
This is exactly what I do for my personal servers (except with ESXi instead of proxmox).
You will probably want both VMs and containers, there are some things that are not well supported in containers (e.g. gitlab).
I run a couple k8s clusters for work and the complexity is beyond what most people starting out would want, I would imagine.
Unless you need something that has a helm chart but not docker support (e.g. gitlab) or you are really keen on learning, it can be quite a jump…
(For gitlab I still would recommend a VM with the omnibus installer over k8s unless you are big enough to have a separate team managing your k8s clusters. It would suck to have a PV issue and lose all your data.)
Why not do both ? As I understand it, to do kubernetes clusters, you must have at least 3 hosts. They don’t need to be 3 different physical hosts: they could be VM (hosted on Proxmox).
Proxmox also having a very strong implementation of ZFS, then it could be used as the storage « host », and it gives you also the option to do snapshots of the VM (and the storage pool), as well as replication/etc.
A k8s cluster can run on a single host if that’s what you want. I’m not sure if it would be worth the virtualisation cost to run it on VMs in the middle as well. If you were only ever going to run on a single host I probably wouldn’t use k8s though, I would just run containers. 🤷♂️
They don’t need to be 3 different physical hosts: they could be VM (hosted on Proxmox).
That is fine for training purposes, but not for real hosting. You typically don’t guard against software crashes, but against hardware failures/outages. And this it not given with all three nodes on the same physical system.
In that case you can simply skip the HA setup and go with a single node. Or not use k8s at all and just manage containers using ansible and systemd or whatever.
You only need 3 host if you want to load-balance
etcd, which I think totally unnecessary for selfhosting purpose. Some downtime when updating kubernetes is acceptable in selfhosted environment for personal purpose.
