It just happened that due to some poweroutage or something my backup server’s HDD got messed up.
Upon booting from a rescue CD I was able to fsck it but there were a lot of inodes with problems, files/folders unattached etc.
After I accepted all fixes, I ended up with system that had boot errors complaining of various errors.
Mounting it in rescue cd I noticed the bin folder was missing.
So I installed the same CentOD version in a virtual machine and copied over the bin folder.
This brought up the login prompt but I was not able to login.
There was a user request but after that no password request, just got back to login prompt.
So I ended up rsync-ing most of the system folders one by one.
like (from the virtual machine)
rsync -c -r -l -v /sys/ root@a.b.c.d:/mnt/hdd/sys/
for bin, sbin, var, usr, etc. (/mnt/hdd/ is the mount point in rescue cd for the server’s hdd)
Once that happened, I got the login and password prompts but the session was closed right after.
So I logged in single mode and disabled selinux (after noticing the access problems in log messages)
This allowed logging in. Most services started up fine, except named, which required re-setting the owner on the config and rndc.key
It’s a very messy and results in a garbage system, but at least it’s up and running. Allowing me to migrate things much easier than messing around with files and configs manually.
Hope this helps someone, cause it took me almost an entire day until I managed to put the pieces together.
