How to Install and Configure an Ubuntu Server 18.04 LTS

User avatar
LHammonds
Site Admin
Site Admin
Posts: 670
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Partition-Level Restores

Post: # 633Post LHammonds
Thu Apr 19, 2018 8:32 pm

Restore Partitions Using SystemRescueCD and FSArchiver

Partitions cannot be mounted when restoring to them. If services can be stopped that use files on a specific partition, it can be unmounted and restored. However, the root partition can never be restore while the server is online so these instructions will cover the common denominator which requires taking the server offline.

The server needs to be booted up with a CD but not just any CD will do because it needs to have FSArchiver on it. For this document, the ISO image from www.sysresccd.org will be used.

Once downloaded, the ISO can be burned to a CD-ROM disc or uploaded to your ISO repository such as a LUN. VMware and VirtualBox can attach an ISO image and mount it in the CD-ROM device to allow the virtual machine to boot the ISO image.

For this example, the root and srv partitions will be restored.

Be sure the partitions have been backed up and the files are sitting in the /bak volume.

Code: Select all

ls -l /bak
-rw------- 1 root root 65888238 Apr 20 17:11 srv-ubuntu-boot.fsa -rw------- 1 root root 59 Apr 20 17:11 srv-ubuntu-boot.md5 -rw------- 1 root root 745 Apr 20 17:11 srv-ubuntu-boot.txt -rw------- 1 root root 9027 Apr 20 17:13 srv-ubuntu-home.fsa -rw------- 1 root root 59 Apr 20 17:13 srv-ubuntu-home.md5 -rw------- 1 root root 748 Apr 20 17:13 srv-ubuntu-home.txt -rw------- 1 root root 3564202 Apr 20 17:15 srv-ubuntu-opt.fsa -rw------- 1 root root 58 Apr 20 17:15 srv-ubuntu-opt.md5 -rw------- 1 root root 749 Apr 20 17:15 srv-ubuntu-opt.txt -rw------- 1 root root 266145534 Apr 20 17:12 srv-ubuntu-root.fsa -rw------- 1 root root 59 Apr 20 17:12 srv-ubuntu-root.md5 -rw------- 1 root root 751 Apr 20 17:12 srv-ubuntu-root.txt -rw------- 1 root root 5336 Apr 20 17:15 srv-ubuntu-srv.fsa -rw------- 1 root root 58 Apr 20 17:15 srv-ubuntu-srv.md5 -rw------- 1 root root 747 Apr 20 17:15 srv-ubuntu-srv.txt -rw------- 1 root root 8853 Apr 20 17:13 srv-ubuntu-tmp.fsa -rw------- 1 root root 58 Apr 20 17:13 srv-ubuntu-tmp.md5 -rw------- 1 root root 747 Apr 20 17:13 srv-ubuntu-tmp.txt -rw------- 1 root root 270704126 Apr 20 17:15 srv-ubuntu-usr.fsa -rw------- 1 root root 58 Apr 20 17:15 srv-ubuntu-usr.md5 -rw------- 1 root root 750 Apr 20 17:15 srv-ubuntu-usr.txt -rw------- 1 root root 77218643 Apr 20 17:13 srv-ubuntu-var.fsa -rw------- 1 root root 58 Apr 20 17:13 srv-ubuntu-var.md5 -rw------- 1 root root 749 Apr 20 17:13 srv-ubuntu-var.txt
As a little test of the restore, let's delete the two text files created in the previous section:

Code: Select all

rm /important.txt
rm /srv/samba/share/important.txt
These files should have been included in the backup image. When the restore is complete, these files should return.

Insert the CDROM (or mount the ISO image) and boot the server with it.

Here is the 1st screen:
Image

Option #1 was chosen (SystemRescueCd: default boot options)

The next screen takes you to the command prompt:

Image

root@sysresccd /root % mkdir /mnt/test
root@sysresccd /root % fsarchiver probe simple
[======DISK======] [=============NAME==============] [====SIZE====] [MAJ] [MIN] [sda ] [Virtual disk ] [ 10.00 GB] [ 8] [ 0] [sdb ] [Virtual disk ] [ 12.00 GB] [ 8] [ 16] [sdc ] [Virtual disk ] [ 12.00 GB] [ 8] [ 32] [sr0 ] [VMware IDE CDR10 ] [ 545.51 MB] [ 11] [ 0] [=====DEVICE=====] [==FILESYS==] [======LABEL======] [====SIZE====] [MAJ] [MIN] [loop0 ] [squashfs ] [<unknown> ] [ 469.86 MB] [ 7] [ 0] [sda1 ] [ext2 ] [boot ] [ 476.00 MB] [ 8] [ 1] [sda5 ] [LVM2_member] [<unknown> ] [ 9.53 GB] [ 8] [ 5] [sdb1 ] [LVM2_member] [<unknown> ] [ 12.00 GB] [ 8] [ 17] [sdc1 ] [LVM2_member] [<unknown> ] [ 12.00 GB] [ 8] [ 33] [dm-0 ] [swap ] [<unknown> ] [ 1.86 GB] [253] [ 0] [dm-1 ] [ext4 ] [root ] [ 3.00 GB] [253] [ 1] [dm-2 ] [ext4 ] [usr ] [ 4.00 GB] [253] [ 2] [dm-3 ] [ext4 ] [var ] [ 3.00 GB] [253] [ 3] [dm-4 ] [ext4 ] [tmp ] [ 2.00 GB] [253] [ 4] [dm-5 ] [ext4 ] [bak ] [ 4.00 GB] [253] [ 5] [dm-6 ] [ext4 ] [srv ] [ 2.00 GB] [253] [ 6] [dm-7 ] [ext4 ] [opt ] [ 2.00 GB] [253] [ 7] [dm-8 ] [ext4 ] [home ] [ 1.00 GB] [253] [ 8]
root@sysresccd /root % mkdir /bak
root@sysresccd /root % mount --read-only /dev/dm-5 /bak
root@sysresccd /root % ls -l /bak
-rw------- 1 root root 65888238 Apr 20 22:11 srv-ubuntu-boot.fsa -rw------- 1 root root 59 Apr 20 22:11 srv-ubuntu-boot.md5 -rw------- 1 root root 745 Apr 20 22:11 srv-ubuntu-boot.txt -rw------- 1 root root 9027 Apr 20 22:13 srv-ubuntu-home.fsa -rw------- 1 root root 59 Apr 20 22:13 srv-ubuntu-home.md5 -rw------- 1 root root 748 Apr 20 22:13 srv-ubuntu-home.txt -rw------- 1 root root 3564202 Apr 20 22:15 srv-ubuntu-opt.fsa -rw------- 1 root root 58 Apr 20 22:15 srv-ubuntu-opt.md5 -rw------- 1 root root 729 Apr 20 22:15 srv-ubuntu-opt.txt -rw------- 1 root root 266145534 Apr 20 22:12 srv-ubuntu-root.fsa -rw------- 1 root root 59 Apr 20 22:12 srv-ubuntu-root.md5 -rw------- 1 root root 732 Apr 20 22:12 srv-ubuntu-root.txt -rw------- 1 root root 5336 Apr 20 22:15 srv-ubuntu-srv.fsa -rw------- 1 root root 58 Apr 20 22:15 srv-ubuntu-srv.md5 -rw------- 1 root root 728 Apr 20 22:15 srv-ubuntu-srv.txt -rw------- 1 root root 270704126 Apr 20 22:15 srv-ubuntu-usr.fsa -rw------- 1 root root 58 Apr 20 22:15 srv-ubuntu-usr.md5 -rw------- 1 root root 730 Apr 20 22:15 srv-ubuntu-usr.txt -rw------- 1 root root 77218643 Apr 20 22:13 srv-ubuntu-var.fsa -rw------- 1 root root 58 Apr 20 22:13 srv-ubuntu-var.md5 -rw------- 1 root root 730 Apr 20 22:13 srv-ubuntu-var.txt
root@sysresccd /root % md5sum --check /bak/srv-ubuntu-srv.md5
/bak/srv-ubuntu-srv.fsa: OK
root@sysresccd /root % fsarchiver restfs --cryptpass="abc123" /bak/srv-ubuntu-srv.fsa id=0,dest=/dev/dm-6
Statistics for filesystem 0 * files successfully processed:....regfiles=1, directories=4, symlinks=0, hardlinks=0, specials=0 * files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0
root@sysresccd /root % mount --read-only /dev/dm-6 /mnt/test
root@sysresccd /root % ls -l /mnt/test/samba/share
-rw-r--r-- 1 root root 0 Apr 20 17:00 important.txt
root@sysresccd /root % umount /mnt/test
root@sysresccd /root % md5sum --check /bak/srv-ubuntu-root.md5
/bak/srv-ubuntu-root.fsa: OK
root@sysresccd /root % fsarchiver restfs --cryptpass="abc123" /bak/srv-ubuntu-root.fsa id=0,dest=/dev/dm-1
Statistics for filesystem 1 * files successfully processed:....regfiles=8938, directories=1693, symlinks=855, hardlinks=11, specials=80 * files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0
root@sysresccd /root % mount --read-only /dev/dm-1 /mnt/test
root@sysresccd /root % ls -l /mnt/test
drwxr-xr-x 3 root root 4096 Apr 20 12:25 bak drwxr-xr-x 2 root root 4096 Apr 20 12:37 bin drwxr-xr-x 4 root root 4096 Apr 20 12:36 boot drwxr-xr-x 14 root root 4096 Apr 20 09:17 dev drwxr-xr-x 90 root root 4096 Apr 20 09:17 etc drwxr-xr-x 4 root root 1024 Apr 20 10:20 home -rw-r--r-- 1 root root 0 Apr 20 17:00 important.txt lrwxrwxrwx 1 root root 33 Apr 20 10:33 initrd.img -> /boot/initrd.img-4.15.0-15-generic drwxr-xr-x 17 root root 4096 Apr 20 12:34 lib drwxr-xr-x 2 root root 4096 Apr 20 10:16 lib64 drwx------ 2 root root 16384 Apr 20 10:15 lost+found drwxr-xr-x 4 root root 4096 Apr 20 10:16 media drwxr-xr-x 3 root root 4096 Apr 20 12:35 mnt drwxr-xr-x 3 root root 4096 Apr 20 10:15 opt dr-xr-xr-x 111 root root 4096 Apr 9 09:17 proc drwx------ 3 root root 4096 Apr 20 19:57 root drwxr-xr-x 15 root root 4096 Apr 20 09:18 run drwxr-xr-x 2 root root 12288 Apr 20 12:37 sbin drwxr-xr-x 2 root root 4096 Apr 20 10:16 snap drwxr-xr-x 4 root root 4096 Apr 20 08:34 srv drwxr-xr-x 13 root root 4096 Apr 9 09:17 sys drwxrwxrwt 4 root root 4096 Apr 20 09:18 tmp drwxr-xr-x 11 root root 4096 Apr 20 10:16 usr drwxr-xr-x 14 root root 4096 Apr 20 13:08 var lrwxrwxrwx 1 root root 30 Apr 20 10:33 vmlinuz -> boot/vmlinuz-4.15.0-15-generic
root@sysresccd /root % umount /mnt/test

Eject the CDROM/ISO and reboot the server.

If it worked right, the server will boot up (we restored the root file system) and the test files should have been restored.

NOTE: If you noticed any ext file systems that had <unknown> labels, you can update them using the tune2fs command.
Example 1:

Code: Select all

tune2fs /dev/sda1 -L boot
Example 2:

Code: Select all

tune2fs /dev/dm-0 root

User avatar
LHammonds
Site Admin
Site Admin
Posts: 670
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Migrating Users from one server to another

Post: # 634Post LHammonds
Thu Apr 19, 2018 8:33 pm

Migrating users/groups from one server to another

WARNING: These migration commands are completely untested by me...just jotting down these research notes for future testing.

Commands to run on old server

Copy accounts to passwd.mig using awk to filter out system accounts (i.e. only copy user accounts)

Code: Select all

awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /tmp/passwd.mig
Copy groups to group.mig

Code: Select all

awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/group > /tmp/group.mig
Copy shadow to shadow.mig

Code: Select all

awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534) {print $1}' /etc/passwd | tee - |egrep -f - /etc/shadow > /tmp/shadow.mig
Copy gshadow to gshadow.mig (rarely used):

Code: Select all

cp /etc/gshadow /tmp/gshadow.mig
Backup the home directories.

Code: Select all

tar -zcvpf /tmp/home.tar.gz /home
If you have Samba users...

Code: Select all

cp /etc/samba/smb.conf /tmp/smb.mig
cp /etc/samba/smbusers /tmp/smbusers.mig
cp /etc/samba/smbpasswd /tmp/smbpasswd.mig
Transfer the migration files to new server.

Code: Select all

scp /tmp/*.mig administrator@newserver:/tmp/.
scp /tmp/home.tar.gz administrator@newserver:/tmp/.
Commands to run on new server

Backup current files (and their backups).

Code: Select all

mkdir /root/save
cp /etc/passwd /etc/passwd- /etc/shadow /etc/shadow- /etc/group /etc/group- /etc/gshadow /etc/gshadow- /root/save
If you have Samba users, backup your current files.

Code: Select all

cp /etc/samba/smb.conf /etc/samba/smbusers /etc/samba/smbpasswd /root/save
Merge/Overwrite files.

Code: Select all

cat /tmp/passwd.mig >> /etc/passwd
cat /tmp/group.mig >> /etc/group
cat /tmp/shadow.mig >> /etc/shadow
cp /tmp/gshadow.mig /etc/gshadow
Ensure file ownership and permissions are correct.

Code: Select all

chown root:shadow /etc/shadow /etc/gshadow
chmod 640 /etc/shadow /etc/gshadow
chown root:root /etc/passwd /etc/group
chmod 644 /etc/passwd /etc/group
If you have Samba users...

Code: Select all

cp /tmp/smb.mig /etc/samba/smb.conf
cp /tmp/smbusers.mig /etc/samba/smbusers
cp /tmp/smbpasswd.mig /etc/samba/smbpasswd
Extract home directories.

Code: Select all

cd /
tar -zxvf /tmp/home.tar.gz
Reboot system for new user accounts to take effect.

Code: Select all

reboot

User avatar
LHammonds
Site Admin
Site Admin
Posts: 670
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Server Replacement

Post: # 635Post LHammonds
Thu Apr 19, 2018 8:35 pm

Replacing a server with a new one

If you want to keep SSH keys and the fingerprint from changing when swapping out an old server for a new one (e.g. old FTPS server to new FTPS which interfaces with many 3rd-party vendors), these are the steps needed to keep the system looking the same.

The old server name and the new server name must match. Example: srv-ftps

Make sure the sshd configuration files on both servers are pointing to the same files (filenames and paths)

Code: Select all

vi /etc/ssh/sshd_config
Look for section like this:

Code: Select all

# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
If there are differences, figure out if they matter and if so, make sure the transferred files on the new server are being referenced correctly in the config file on the new server.

Commands to run on old server

Transfer files to new server

Code: Select all

scp /etc/ssh/ssh_host* administrator@newserver:/tmp/.
Commands to run on new server

Backup existing files

Code: Select all

mkdir /root/ssh-backup
cp /etc/ssh/ssh_host* /root/ssh-backup/.
Set correct file ownership and permissions

Code: Select all

chown root:root /tmp/ssh_host*
chmod 600 /tmp/ssh_host*
chmod 644 /tmp/ssh_host*.pub
Move and overwrite files to correct location

Code: Select all

mv /tmp/ssh_host* /etc/ssh/.
Restart SSH daemon

Code: Select all

service ssh restart
You will also want to copy over individual user's public keys in their ~/.ssh/authorized_keys but if you transferred the contents of the /home folder, this was already done.

User avatar
LHammonds
Site Admin
Site Admin
Posts: 670
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Trouble-Shooting

Post: # 636Post LHammonds
Thu Apr 19, 2018 8:37 pm

Boot Issues

Code: Select all

systemd-analyze plot > /srv/samba/share/boot.xml
Open boot.xml in a web browser to examine the boot process to see if anything is taking longer than it should.

Code: Select all

systemd-analyze blame
This shows a list of all running processes ordered by the time it took to initialize. Be careful not to judge a process too harshly at 1st sight...it may have been waiting on another process to initialize 1st.

Code: Select all

dmesg > /tmp/boot.txt
Open boot.txt and observe the order of events from the moment it starts its boot all the way to the end. The events are ordered by time and it shows how much time has passed since the machine was turned on for that particular event to occur.

User avatar
LHammonds
Site Admin
Site Admin
Posts: 670
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Future Research Items

Post: # 637Post LHammonds
Thu Apr 19, 2018 8:37 pm

Security Topics to Research and Incorporate
Gamin File Change Monitoring - Gamin web site
PSAD intrusion detection - TheFanClub walkthru (Note, current version is 2.4.4 released on Feb 2017, Ubuntu repository version is 2.2.3 released in March, 2014)
Tiger intrusion detection - homepage
Tripwire intrusion detection - DigitalOcean VPN, Unixmen
LogWatch - DigitalOcean walkthru, Unixmen, linode
Passing Passwords
Let's Encrypt (FREE SSL) and Certbot
etckeeper for config revision control

Post Reply