My home lab has a mild amount of complexity and I’d like practice some good habits about documenting it. Stuff like, what each system does, the OS, any notable software installed and, most importantly, any documentation around configuration or troubleshooting.
i.e. I have an internal SMTP relay that uses a letsencrypt SSL cert that I need to use the DNS challenge to renew. I’ve got the steps around that sitting in a Google Doc. I’ve got a couple more google docs like that.
I don’t want to get super complicated but I’d like something a bit more structured than a folder full of google docs. I’d also like to pull it in-house.
Thanks
Edit: I appreciate all the feedback I’ve gotten on this post so far. There have been a lot of tools suggested and some great discussion about methods. This will probably be my weekend now.
What is it that you all are documenting? I’m just getting started and have so far just set up docker and several containers (Plex, the *arrs, qbittorrent, gluetun) and not sure if I should be writing something down?
I document my proxmox configuration, and again for each service running on it; ports, fqdn, let’s encrypt etc. Any problems or deviations from the standard installation and how I resolved it. How I mounted my media drive and and made it accessible in Plex and Jellyfin. Any configuration that I had to search for the solution, including the actual steps and not just links that might be gone in the future. The more services you accumulate the harder it is to start the documentation, so it’s a good idea to start sooner than later.
I’d like to document what I have, and what I need to do it spin it back up with minimal effort should I need to. Esp, anything that’s unique to my setup or that I did to work around a problem. I’ve shot myself in the foot many a time where I go back to something 6 months later and I’ve long since forgotten what I did. So, if it’s not commented in a config I’m figuring it out all over again.
I’m using netbox, and I’m in the process of info dumping my brain to a media wiki. I may add ansible into the mix in the near future.
My lab is a bit large and complex and I’m currently in the process of trying to train help to run it, from the pool of people that make use of it. They know how the front end works, a few of them need to learn the back end.
The only thing I save in Google Drive are my notes just in case of disaster.
Are you writing to Google drive directly from the cli? If so how? I regularly need to search, edit, copy, and paste to and from my notes; backup config files; save a neat little script I wrote; etc. all from the CLI. It would be awesome to have this searchable and online from a web browser too for when I’m not working in the terminal. For example, piping an error message to a file and grabbing/sanitizing that error to search later. I have ways, but their all a lot clunkier than simply have a Dropbox. I’m basically looking for something that works just like Dropbox, is not self hosted, and not as cumbersome to setup as NextCloud and the like.
It’s not automated. I just have the most important commands to fix/rebuild my sever in case of disasater.
Frankly the only thing I’d save in Google Docs are encrypted archives. Otherwise they’ll profile the documents to send ads to you. But it is a good back up in case lightning strikes your home or something.
I don’t save all my documents. Just my self-hosting, servers infraestructure notes. I don’t want to have the recovery intructions in the same machine I’m recovering
I use trillium that gets backed up every hour to my pc.
I also do a lot of python development so my project ideas get written down there too.
I’m not a fan of code is documentation because what happens when you step away for a month and you need to figure something out? In trillium I have a search bar. What do you have in the code?
I will second trilium. I use their sync server in a VM (which is backed up with the rest of my VM’s so its easy to drop back down should something happen). The app appeals to me, even after using Obisdian for the past 6 months (i’m a fan of markdown as well).
I use Ansible, Docker, and Emacs OrgMode files committed to Git. Diagrams are a mix of Miro and Graphviz. There’s also a few markdowns in there too. Joplin is used for rough notes only.
I use bookstack. Simple selfhosted wiki.
+1 for bookstack. I also selfhost a kanban with the services basic info and it’s related status (pilot/test, production and to be decommissioned). At the beginning I used Planka, but now switched to Nextcloud Deck.
I deploy as much as I possibly can via Ansible. Then the Ansible code serves as the documentation. I also keep the underlying OS the same on all machines to avoid different OS conventions. All my machines run Debian. The few things I cannot express in Ansible, such as network topology, I draw a diagram for in draw.io, but that’s it.
Also, why not automate the certificate renewal with certbot? I have two reverse proxies and they renew their certificates themselves.
My reverse proxy can do automated renewal just fine. The SMTP relay requires a DNS challenge that is manual.
Why not have the reverse proxy also do renewal for the SMTP relay certificate and just rsync it to the relay? For a while I had one of my proxies do all the renewals and the other would rsync it.
It certainly wouldn’t be because I’ve been doing it this way for so long that it never occurred to me. Nope. Certainly not that.
In fairness, I very recently switched from a cobbled together apache web server/rev proxy config I’ve been carrying along in some form for well over a decade (I remember converting the config to 2.4), to an NPM container. I had some initial trouble switching my certs over to NPM and haven’t revisited that yet.
I’m in the middle of a major overhaul of my tech stack. Fixing certs is on my short list.
Thanks for pointing out where I was stuck in my ways.
This is the way
psssst. ansible is red hat.
red hat bad.
What alternative to you suggest?
i m all good with selfhosted wiki
I meant to replace Ansible automation. Pointing out it’s RH is all well and good, but what’s the alternative?
Git based static site generator, like gohugo or Jekyll.
This is interesting. I already just keep a collection of markdown files… might as well make it an internal documentation site so it’s easier to browse 🤔
I use a combination of netbox for the physical/logical network and server connectivity, and outline for text documentation of the different components.
Woah thanks for the NetBox shout! Gonna check that out
I think I looked at netbox a while back. I may circle back to it for the actual physical layer. If I remember the ipam didn’t include network scanning tho.
I’ve been using Obsidian for a lot of other purposes for a couple years now, so I was comfortable adding my documentation into my existing vault there. I made a couple templates that I fill out for any hardware/software/networking equipment.
Since the app’s selling point is storing all your notes in plain text I wouldn’t put anything security-related in there without some encrypted container. I use KeePass for that part, and keep the file it generates in the same folder as Obsidian so I can link to it within notes. Click the link in the note, KeePass opens the vault and asks for its password.
I use obsidian too. It supports mermaid too so you can make your network diagram with it.
This is the 2nd ref I’ve seen to mermaid. I need to check that out.
I would not consider Mermaid complete enough for network diagramming. The very basics are possible, but try to describe anything more complicated throws off the placement and makes the pathing whacky.
Straight flow charts are the closest you can get to a network diagram, so if you try to draw a link that travels back up the chart, it breaks mermaid’s brain trying to figure out the order of decision points (network devices).
The allure of text based diagrams is so tantalizing - but if you need them to be functional, it’s not going to happen
There’s an issue tracking the need a new diagram type to handle it.
This is the first I’ve heard of Kroki. A quick glance at their site and wow! So many options for markup. I’ll be trying this out for sure
Mind sharing your template?
Sure.
I left everything in, so no doubt there’s stuff in there specific to my vault you won’t need like metadata - adjust these to your needs or use them as a starting point for something new. There’s no network device template, I usually use the hardware one and just delete the irrelevant bits.
Thanks!
I made myself a wiki in my helpdesk system - I use osticket to send me various email alerts to so I can track issues I need to fix, and they have a little wiki option.
Then one day that host was down and I needed some info and I was very irritated. Now all of those notes are in my Apple notes backed up in iCloud and searchable on whatever I’ve got handy so if I need info I can get the info
I played with GLPI just long enough to realize that was way more than I wanted or needed. I’d like to track changes but I don’t want to run a full ticketing/chg mgmt system to do it.
I use WikiJS for documentation. Simple, powerful and has a lot of features
+1 for WikiJS. As a bonus you can have WikiJS back itself up to plain text MarkDown files, so if things explode you can always just read those from wherever.
Another great feature I use is to have WikiJS back itself up into git. If I am going to a place with no internet access I can do a quick git pull and have a complete copy of my wiki including files on my laptop.
That sounds pretty handy.
Git, MinIO, Amazon S3, Filesystem and many more options to for backup🐱
Can’t wait for v3 finally
I run a local MediaWiki appliance from turnkeylinux, super easy to spin up in proxmox.
Joplin synced to webdav/nexcloud.
Same
