Used to be messing with kernel arguments and installing/tweaking boot parameters. That was until Grub broke, I learned systemd-boot and chrooting into the system via live USB

Now if I break anything it’s just a matter of “sigh, let me get the USB and type a few commands”

I once broke my Ubuntu install by trying to convert it KDE Neon, that reinstalled half my packages and left it in an basically unusable state. I then un-broke the install while upgrading multiple Ubuntu releases, that reinstalled the other half as well. It actually worked, and I’m still using that install.

Biggest thing I noticed after switching is forum posts. In Linux ones you usually get a fix where the Windows ones 9/10 they just advise you reformat

I feel seen here, I was building a Ubuntu server and messed up the firewall settings not being able to get an internet connection, hours of trying to get back to where I was I gave up and plan to just start from scratch next time.

Is there a way of taking system snapshots with Linux?

Some programs still relying on python2 when the operating system has long since upgraded to python3.

Not really an issue per se, I just had to switch those apps over to using the flatpak version which would have it installed as needed. (I’m looking at you GIMP)

I’ve found that the silliest desktop problems are usually the hardest to solve, and the “serious” linux system errors are the easiest.

System doesn’t boot? Look at error message, boot from a rescue disk, mount root filesystem and fix what you did wrong.

Wrong mouse cursor theme in some Plasma applications, ignoring your settings? Some weird font rendering issue? Bang your head against a wall exploring various dotfiles and rc files in your home directory for two weeks, and eventually give up and nuke your profile and reconfigure your whole desktop from scratch.

Some of the crap I had to do back in the late 00s to get wifi, sleep and power management even barely working on some machines felt like the hardest thing at the time. I wonder how I’d fare with those issues today, 17 years later, knowing quite a bit more about the underlying OS and working with the OS daily… I don’t know that I’d qualify that as difficult more than it was extremely tedious and a bunch of trial and error of configuration options I didn’t know anything about.

If we’re talking about modern day… not so much honestly. btrfs snapshots saved my ass a couple of times, the rare issue I encounter I just rollback and wait for an upstream fix, and the rest I typically ignore or use something else. Everything tends to run quite smooth for me as a general rule, though.

I managed a CentOS system where someone accidentally deleted everything from /usr, so no lib64, and no bin. I didn’t have a way to get proper files at the time, so I hooked the drive up to my Arch system, made sure glibc matched, and copied yum and other tools from Arch.

Booted the system, reinstalled a whole lot of yum packages, and… the thing still worked.

That’s almost equivalent to a reinstall, though. As a broke college student, I had a laptop with a loose drive, that would fall out very easily. I set it up to load a few crucial things into a ramdisk at boot, so that I could browse the web and take notes even if the drive was disconnected, and it would still load images and things. I could pull the cover off and push the drive back in place to save files, but doing that every time I had class got really tiring, so I wanted it to run a little like a live system.

A Gentoo upgrade package list with over 100 packages and conflicts all over the place. Then do it again when the list grows to the same size in a few months.

This is why I don’t use Gentoo anymore.

Are you including back in the day when we had to use windows device drivers via ndiswrappers?

I’ve managed to remove a critical library once but did manage to extract it from an RPM on another machine and manually install it. That was good enough to get me to the point where I could yum reinstall.

Pre-linux we had an HP workstation where the disc drive died and of course we had no backups. I managed to frankenstein the disc by connecting the platters on the broken disc to the circuit board of a working disc. This worked and I was able to back up the disk and reload on to a new drive.

And then we bought an 8mm tape drive for backups and I had to port some drivers to HP-UX to get it to work. But we had awesome backups after that!

My first Linux machine crashing. This was way before Redhat, Ubuntu, Arch, or OpenSUSE. This was installed from 60+ floppy disks on a 386-40 with 8MB of RAM.

This machine ran happily, but it crashed under heavy load. I checked out causing the load by using different applications, but could not nail it to a certain software. So the next thing I checked was the RAM. Memtest86 ran for a day without any problems. But the crashes still came. So I got the infrared camera from the lab to see if some hardware overheats. Nope, this went nowhere, either.

Then I tested the harddisk. Read test of the whole HD went without problems. I copied the data on a backup medium and did a write and read test by dd’ing /dev/zero over the whole disk, and then dd’ing the disk to /dev/null. Nothing did show up.

I reinstalled the Linux, and it crashed again. But this time, I noticed that something was odd with the harddisk. I added a second swap partition, disabled the first, and the machine ran without problems. Strange…

So I wrote a small program that tested the part of the disk occupied by the old swap space: Write data, read data, and log everything with timestamps. And there was the culprit: There was an area on the HD where I could write any data, but when I read blocks from that area, a) It took a very long time for the read, b) the blocks I read were containing all zero, regardless of what I had written, and worst of all c) there was no error indication whatsoever from the controller or drive. Down at the kernel level, the zeroed blocks were happily served by the HD with an “OK”. And the faulty area was right in the middle of the original swap partition.

I’ve generally had good luck with hardware and things just worked under linux. But one day I upgraded a few machines on my network to 2.5G ethernet. Several already had the ports, but my little NUC NAS box didn’t, so I installed a 2.5G usb ethernet dongle. No matter what I did, I couldn’t get it to work. It would show up and NM would act like it was up and there were no errors or anything, but it just wouldn’t actually function.

Eventually, I found out that it has a built in USB data partition that contains the drivers for windows. The card was coming up as a usb disk first when the hardware was assigned and not a network card which it should have been.

I had to write a blacklist the usb modules first, which I had done before, but I had to also write a udev rule to automatically add the network card and driver on boot. It wasn’t that difficult to actually do, but I had just never had to do anything with udev rules before. Took me a good three days of troubleshooting to finally get everything to work correctly on boot.

ACTION=="add", ATTRS{idVendor}=="20f4", ATTRS{idProduct}=="e02c", RUN+="/sbin/modprobe r8152" RUN+="/bin/sh -c 'echo 20f4 e02c > /sys/bus/usb/drivers/r8152/new_id'"

Around 2017 I spent three days on and off trying to diagnose why my laptop running elementary OS had no wifi support. I reinstalled the wifi drivers and everything countless times. It worked for many days initially then just didn’t one day when I got on the laptop. Turns out I had accidentally flipped the wifi toggle switch while it was in my bag. I forgot the laptop had one. Womp womp.

I once exited vim without having to look up the commands.

Hmm I have come up with a bunch of neat solutions over the years. Where to start?

One time I broke the sudoers file on a distro without a root account, thoroughly locking myself out. I used docker -v /:/chroot to get myself root access to my root filesystem where I fixed the sudoers file. Protip always use visudo