How to Install and Configure an Ubuntu Server 22.04 LTS

[LHammonds]

Restore Partitions Using SystemRescueCD and FSArchiver

Partitions cannot be mounted when restoring to them. If services can be stopped that use files on a specific partition, it can be unmounted and restored. However, the root partition can never be restored while the server is online so these instructions will cover the common denominator which requires taking the server offline.

The server needs to be booted up with a CD but not just any CD will do because it needs to have FSArchiver on it. For this document, the ISO image from www.sysresccd.org will be used.

Once downloaded, the ISO can be burned to a CD-ROM disc or uploaded to your ISO repository such as a LUN. VMware and VirtualBox can attach an ISO image and mount it in the CD-ROM device to allow the virtual machine to boot the ISO image.

For this example, files on the root and home partitions will be restored using this summarized procedure:

1. Boot the system using the RescueCD.
2. Mount the bak partition which contains the backup archives.
3. Verify that the backup archives are in good condition.
4. Restore the partitions.
5. Mount the root and home partitions.
6. Verify the restore worked.
7. Unmount the partitions.
8. Reboot the system.

Be sure the partitions have been backed up and the files are sitting in the /bak volume.

Code: Select all

ls -l /bak/partitions/

total 1199788 -rw------- 1 root root 119002341 May 31 14:22 srv-ubuntu-boot.fsa -rw------- 1 root root 70 May 31 14:22 srv-ubuntu-boot.md5 -rw------- 1 root root 748 May 31 14:22 srv-ubuntu-boot.txt -rw------- 1 root root 10639 May 31 14:23 srv-ubuntu-home.fsa -rw------- 1 root root 70 May 31 14:23 srv-ubuntu-home.md5 -rw------- 1 root root 747 May 31 14:23 srv-ubuntu-home.txt -rw------- 1 root root 5072 May 31 14:30 srv-ubuntu-opt.fsa -rw------- 1 root root 69 May 31 14:30 srv-ubuntu-opt.md5 -rw------- 1 root root 745 May 31 14:30 srv-ubuntu-opt.txt -rw------- 1 root root 1968035 May 31 14:23 srv-ubuntu-root.fsa -rw------- 1 root root 70 May 31 14:23 srv-ubuntu-root.md5 -rw------- 1 root root 750 May 31 14:23 srv-ubuntu-root.txt -rw------- 1 root root 5072 May 31 14:23 srv-ubuntu-srv.fsa -rw------- 1 root root 69 May 31 14:23 srv-ubuntu-srv.md5 -rw------- 1 root root 745 May 31 14:23 srv-ubuntu-srv.txt -rw------- 1 root root 8253 May 31 14:29 srv-ubuntu-tmp.fsa -rw------- 1 root root 69 May 31 14:29 srv-ubuntu-tmp.md5 -rw------- 1 root root 744 May 31 14:29 srv-ubuntu-tmp.txt -rw------- 1 root root 760951206 May 31 14:28 srv-ubuntu-usr.fsa -rw------- 1 root root 69 May 31 14:28 srv-ubuntu-usr.md5 -rw------- 1 root root 748 May 31 14:28 srv-ubuntu-usr.txt -rw------- 1 root root 346538394 May 31 14:29 srv-ubuntu-var.fsa -rw------- 1 root root 69 May 31 14:29 srv-ubuntu-var.md5 -rw------- 1 root root 749 May 31 14:29 srv-ubuntu-var.txt

As a test of the restore, let's delete the two text files created in the previous section:

Code: Select all

rm /important.txt
rm /home/administrator/important.txt

These files should have been included in the backup image. When the restore is complete, these files should return.

Insert the SystemRescueCD CDROM (or mount the ISO image) and boot the server with it.

Here is the 1st screen:


Option #1 was chosen (SystemRescueCd: default boot options)

The next screen takes you to the command prompt:



Show what partitions the system has by using fsarchiver.

root@sysresccd /root % fsarchiver probe simple

[======DISK======] [=============NAME==============] [====SIZE====] [MAJ] [MIN] [sda ] [Virtual disk ] [ 30.00 GB] [ 8] [ 0] [sr0 ] [VMware SATA CD00 ] [ 720.00 MB] [ 11] [ 0] [=====DEVICE=====] [==FILESYS==] [======LABEL======] [====SIZE====] [MAJ] [MIN] [loop0 ] [squashfs ] [<unknown> ] [ 647.75 MB] [ 7] [ 0] [sda1 ] [<unknown> ] [<unknown> ] [ 1.00 MB] [ 8] [ 1] [sda2 ] [ext4 ] [boot ] [ 1.00 GB] [ 8] [ 2] [sda3 ] [LVM2_member] [<unknown> ] [ 28.00 GB] [ 8] [ 3] [dm-0 ] [ext4 ] [root ] [ 6.00 GB] [254] [ 0] [dm-1 ] [ext4 ] [home ] [ 1.00 GB] [254] [ 1] [dm-2 ] [ext4 ] [srv ] [ 1.00 GB] [254] [ 2] [dm-3 ] [ext4 ] [usr ] [ 5.00 GB] [254] [ 3] [dm-4 ] [ext4 ] [var ] [ 3.00 GB] [254] [ 4] [dm-5 ] [ext4 ] [tmp ] [ 3.00 GB] [254] [ 5] [dm-6 ] [ext4 ] [opt ] [ 1.00 GB] [254] [ 6] [dm-7 ] [ext4 ] [bak ] [ 4.00 GB] [254] [ 7]

Mount the bak partition which contains the backup archives.

root@sysresccd /root % mkdir /bak
root@sysresccd /root % mount --read-only /dev/dm-7 /bak
root@sysresccd /root % ls -l /bak/partitions

total 1199788 -rw------- 1 root root 119002341 May 31 14:22 srv-ubuntu-boot.fsa -rw------- 1 root root 70 May 31 14:22 srv-ubuntu-boot.md5 -rw------- 1 root root 748 May 31 14:22 srv-ubuntu-boot.txt -rw------- 1 root root 10639 May 31 14:23 srv-ubuntu-home.fsa -rw------- 1 root root 70 May 31 14:23 srv-ubuntu-home.md5 -rw------- 1 root root 747 May 31 14:23 srv-ubuntu-home.txt -rw------- 1 root root 5072 May 31 14:30 srv-ubuntu-opt.fsa -rw------- 1 root root 69 May 31 14:30 srv-ubuntu-opt.md5 -rw------- 1 root root 745 May 31 14:30 srv-ubuntu-opt.txt -rw------- 1 root root 1968035 May 31 14:23 srv-ubuntu-root.fsa -rw------- 1 root root 70 May 31 14:23 srv-ubuntu-root.md5 -rw------- 1 root root 750 May 31 14:23 srv-ubuntu-root.txt -rw------- 1 root root 5072 May 31 14:23 srv-ubuntu-srv.fsa -rw------- 1 root root 69 May 31 14:23 srv-ubuntu-srv.md5 -rw------- 1 root root 745 May 31 14:23 srv-ubuntu-srv.txt -rw------- 1 root root 8253 May 31 14:29 srv-ubuntu-tmp.fsa -rw------- 1 root root 69 May 31 14:29 srv-ubuntu-tmp.md5 -rw------- 1 root root 744 May 31 14:29 srv-ubuntu-tmp.txt -rw------- 1 root root 760951206 May 31 14:28 srv-ubuntu-usr.fsa -rw------- 1 root root 69 May 31 14:28 srv-ubuntu-usr.md5 -rw------- 1 root root 748 May 31 14:28 srv-ubuntu-usr.txt -rw------- 1 root root 346538394 May 31 14:29 srv-ubuntu-var.fsa -rw------- 1 root root 69 May 31 14:29 srv-ubuntu-var.md5 -rw------- 1 root root 749 May 31 14:29 srv-ubuntu-var.txt

Verify that the backup archives are in good condition:

root@sysresccd /root % md5sum --check /bak/partitions/srv-ubuntu-root.md5

/bak/partitions/srv-ubuntu-root.fsa: OK

root@sysresccd /root % md5sum --check /bak/partitions/srv-ubuntu-home.md5

/bak/partitions/srv-ubuntu-home.fsa: OK

Restore the root partition:

root@sysresccd /root % fsarchiver restfs --cryptpass="abc123" /bak/partitions/srv-ubuntu-root.fsa id=0,dest=/dev/dm-0

Statistics for filesystem 0 * files successfully processed:....regfiles=941, directories=297, symlinks=620, hardlinks=0, specials=8 * files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0

Restore the home partition:

root@sysresccd /root % fsarchiver restfs --cryptpass="abc123" /bak/partitions/srv-ubuntu-home.fsa id=0,dest=/dev/dm-1

Statistics for filesystem 0 * files successfully processed:....regfiles=11, directories=5, symlinks=0, hardlinks=0, specials=0 * files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0

Mount the root and home partitions:

root@sysresccd /root % mkdir /mnt/root
root@sysresccd /root % mkdir /mnt/home
root@sysresccd /root % mount --read-only /dev/dm-0 /mnt/root
root@sysresccd /root % mount --read-only /dev/dm-1 /mnt/home

Verify the restore worked:

root@sysresccd /root % ls -l /mnt/root/import*

-rw-r--r-- 1 root root 0 May 3 17:00 important.txt

root@sysresccd /root % ls -l /mnt/home/administrator/import*

-rw-r--r-- 1 root root 0 May 3 17:00 important.txt

Unmount the partitions:

root@sysresccd /root % umount /mnt/root
root@sysresccd /root % umount /mnt/home
root@sysresccd /root % umount /bak

Eject the CDROM/ISO and reboot the server.

If it worked right, the server will boot up (we restored the root file system) and the test files should have been restored.

NOTE: If you noticed any ext file systems that had <unknown> labels, you can update them using the tune2fs command which makes identification easier when restoring.

Example 1:

Code: Select all

sudo tune2fs /dev/sda2 -L boot

Example 2:

Code: Select all

sudo tune2fs /dev/dm-0 root

[LHammonds]

Migrating users/groups from one server to another

WARNING: These migration commands are not thoroughly tested by me...just jotting down these research notes for future testing.

Commands to run on old server

Copy accounts to passwd.mig using awk to filter out system accounts (i.e. only copy user accounts)

Code: Select all

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /tmp/passwd.mig

Copy groups to group.mig

Code: Select all

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/group > /tmp/group.mig

Copy shadow to shadow.mig

Code: Select all

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534) {print $1}' /etc/passwd | tee - |egrep -f - /etc/shadow > /tmp/shadow.mig

Copy gshadow to gshadow.mig (rarely used):

Code: Select all

sudo cp /etc/gshadow /tmp/gshadow.mig

Backup the home directories.

Code: Select all

sudo tar -zcvpf /tmp/home.tar.gz /home

If you have Samba users...

Code: Select all

sudo cp /etc/samba/smb.conf /tmp/smb.mig
sudo cp /etc/samba/smbusers /tmp/smbusers.mig
sudo cp /etc/samba/smbpasswd /tmp/smbpasswd.mig

Transfer the migration files to new server.

Code: Select all

sudo scp /tmp/*.mig administrator@newserver:/tmp/.
sudo scp /tmp/home.tar.gz administrator@newserver:/tmp/.

Commands to run on new server

Backup current files (and their backups).

Code: Select all

sudo mkdir /root/save
sudo cp /etc/passwd /etc/passwd- /etc/shadow /etc/shadow- /etc/group /etc/group- /etc/gshadow /etc/gshadow- /root/save

If you have Samba users, backup your current files.

Code: Select all

sudo cp /etc/samba/smb.conf /etc/samba/smbusers /etc/samba/smbpasswd /root/save

Merge/Overwrite files.

Code: Select all

sudo cat /tmp/passwd.mig >> /etc/passwd
sudo cat /tmp/group.mig >> /etc/group
sudo cat /tmp/shadow.mig >> /etc/shadow
sudo cp /tmp/gshadow.mig /etc/gshadow

Ensure file ownership and permissions are correct.

Code: Select all

sudo chown root:shadow /etc/shadow /etc/gshadow
sudo chmod 640 /etc/shadow /etc/gshadow
sudo chown root:root /etc/passwd /etc/group
sudo chmod 644 /etc/passwd /etc/group

If you have Samba users...

Code: Select all

sudo cp /tmp/smb.mig /etc/samba/smb.conf
sudo cp /tmp/smbusers.mig /etc/samba/smbusers
sudo cp /tmp/smbpasswd.mig /etc/samba/smbpasswd

Extract home directories.

Code: Select all

cd /
sudo tar -zxvf /tmp/home.tar.gz

Reboot system for new user accounts to take effect.

Code: Select all

sudo reboot

[LHammonds]

Replacing a server with a new one

If you want to keep SSH keys and the fingerprint from changing when swapping out an old server for a new one (e.g. old FTPS server to new FTPS which interfaces with many 3rd-party vendors), these are the steps needed to keep the system looking the same.

The old server name and the new server name must match. Example: srv-ftps

Make sure the sshd configuration files on both servers are pointing to the same files (filenames and paths)

Code: Select all

sudo vi /etc/ssh/sshd_config

Look for section like this:

Code: Select all

# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

If there are differences, figure out if they matter and if so, make sure the transferred files on the new server are being referenced correctly in the config file on the new server.

Commands to run on old server

Transfer files to new server

Code: Select all

sudo scp /etc/ssh/ssh_host* administrator@newserver:/tmp/.

Commands to run on new server

Backup existing files

Code: Select all

sudo mkdir /root/ssh-backup
sudo cp /etc/ssh/ssh_host* /root/ssh-backup/.

Set correct file ownership and permissions

Code: Select all

sudo chown root:root /tmp/ssh_host*
sudo chmod 600 /tmp/ssh_host*
sudo chmod 644 /tmp/ssh_host*.pub

Move and overwrite files to correct location

Code: Select all

sudo mv /tmp/ssh_host* /etc/ssh/.

Restart SSH daemon

Code: Select all

sudo systemctl restart ssh

You will also want to copy over individual user's public keys in their ~/.ssh/authorized_keys but if you transferred the contents of the /home folder, this was already done.

[LHammonds]

Boot Issues

Code: Select all

systemd-analyze plot > /srv/samba/share/boot.xml

Open boot.xml in a web browser to examine the boot process to see if anything is taking longer than it should.

Code: Select all

systemd-analyze blame

This shows a list of all running processes ordered by the time it took to initialize. Be careful not to judge a process too harshly at 1st sight...it may have been waiting on another process to initialize 1st.

Code: Select all

dmesg > /tmp/boot.txt

Open boot.txt and observe the order of events from the moment it starts its boot all the way to the end. The events are ordered by time and it shows how much time has passed since the machine was turned on for that particular event to occur.

Disk Space Issues

To quickly find out status on partition sizes, use this command:

Code: Select all

df -h

To help find out what is eating up your space in a particular folder, you can use this command:

Code: Select all

du -sh /path/to/folder

To make faster use of du, use the ncdu utility I recommended to install in an earlier section. If you start at the root like the example below, you can use your arrow keys to select the problematic folders and keep drilling down until you find the culprit you are looking for. It automatically sorts the results so you are typically already on the biggest folder and just need to press ENTER to drill down to the next folder.

Code: Select all

ncdu /

[LHammonds]

Security Topics to Research and Incorporate
PSAD intrusion detection - TheFanClub walkthru (Note, current version is 2.4.4 released on Feb 2017, Ubuntu repository version is 2.2.3 released in March, 2014)
Tiger intrusion detection - homepage
Tripwire intrusion detection - DigitalOcean VPN, Unixmen
LogWatch - DigitalOcean walkthru, Unixmen, linode
Passing Passwords
Let's Encrypt (FREE SSL) and Certbot
etckeeper for config revision control