How to Install and Configure an Ubuntu Server 22.04 LTS

User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

How to Install and Configure an Ubuntu Server 22.04 LTS

Post: # 1003Post LHammonds »

Greetings and salutations,

I hope this thread will be helpful to those who follow in my foot steps as well as getting any advice based on what I have done / documented.

To discuss this thread, please participate here: Ubuntu Forums

High-level overview

This document will cover installation of a dedicated Ubuntu server. This will be the "base" installation of the server as a prerequisite for other documents that will build upon it (e.g. MariaDB, Apache, MediaWiki, etc.). The server will be installed inside a virtual machine using vSphere running on ESXi servers. Notes will also be supplied for doing the same thing for Oracle's VirtualBox on a Windows 10 PC. Although there are some VMware-specific and VirtualBox-specific steps, they are very few and the majority of this documentation will work for other Virtual Machines or even directly installed onto a physical machine (e.g. bare-metal install).

This document will also cover some custom scripts to help automate tasks such as backing up, automatically growing the file system when free space is low, etc.

Tools utilized in this process
Helpful links

The list below are sources of information that helped me configure this system as well as some places that might be helpful to me later on as this process continues.
Assumptions

This documentation will need to make use of some very-specific information that will most-likely be different for each person / location. This variable data will be noted in this section and highlighted in red throughout the document as a reminder that you should plug-in your own value rather than actually using these "place-holder" values.

Under no circumstance should you use the actual values listed below. They are place-holders for the real thing. This is just a checklist template you need to have answered before you start the install process.

Wherever you see RED in this document, you need to substitute it for what your company uses.
  • Ubuntu Server name: srv-ubuntu
  • Internet domain: mydomain.com
  • Ubuntu Server IP address: 192.168.107.2
  • Ubuntu Server IP subnet mask: 255.255.255.0
  • Ubuntu Server IP gateway: 192.168.107.1
  • Internal DNS Server 1: 192.168.107.212
  • Internal DNS Server 2: 192.168.107.213
  • External DNS Server 1: 1.1.1.1 (Cloudflare)
  • External DNS Server 2: 8.8.8.8 (Google)
  • Ubuntu Admin ID: administrator
  • Ubuntu Admin Password: myadminpass
  • Email Server (remote): 192.168.107.25
  • Windows Share ID: myshare
  • Windows Share Password: mysharepass
It is also assumed that the reader knows how to use the VI editor. If not, you will need to beef up your skill set or use a different editor in place of it. The vim-nox package that is installed later includes "vimtutor" which is also a good place to learn how to use the vi editor.
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Analysis and Design

Post: # 1004Post LHammonds »

Analysis and Design

The Ubuntu Server Long-Term Support (LTS) is great choice for companies because it is a solid operating system that happens to be free. If professional support is needed, there is an option to buy support for the Long-Term Support (LTS) versions of the operating system.

The large decision over the configuration of Ubuntu is how the hard drive space is sliced up (partitioned). This documentation will focus on partitioning the drives in such a way that it allows for growth depending on what is needed for the specific application.

This following design allows for dynamic growth and fine-tuning if need be. Being caught offguard with a scenario where space is filled up with no immediate option other than deleting files is never a good thing. Long-term life and growth of the system as well as budgeting concerns have to be taken into consideration.

Isolating the root volume to mainly just static data that will not grow much over time is the central concern. Pushing the other folders into their own volumes will be done so their dynamic growth will not affect the root partition. Filling up the root volume on a *nix system is a very bad thing and should be avoided at all costs. The file systems will also not take up 100% of the logical volume. This will allow the file systems (through automated scripts) to grow as needed and give the administrators some time to add more drives if necessary or shrink other volumes to get more space.

The volumes will initially be sliced up as follows:
  • boot - This will remain static in size. It is also the only space residing outside the Logical Volume Manager (LVM)
  • root volume - Operating system and everything else which should remain fairly static.
  • home volume - This is where personal files will be stored but likely not be used in most server configurations.
  • srv volume - This will contain the files stored in the Samba share.
  • usr volume - This will contain mostly static data and should not grow unexpectedly.
  • var volume - This is the app/database/log storage and will continue to grow over time.
  • tmp volume - This location will be used for temporary storage. Size should be adjusted to match however it is being used.
  • opt volume - This will contain specific software you add but may not be utilized at all depending on configuration.
  • bak volume - This will contain a local backup of the server/applications/data. So space needs to be about double the size of your application data (typically double the /var and /opt size).
  • Offsite Storage - This will be handled elsewhere. A backup server will access this server to pull backup data. It is wise to make sure no servers can access the backup server for security reasons.
The partitions will be increased later as needed but will start off with a minimum size due to how the installer expands the file system to fill up all available volume space by default.

To get a good idea of the initial hard drive layout and to understand the process better, here is a graphical representation of the initial design for the server:

Image

When looking at the graphical image, you can see how we start off at the bottom with a physical hard disk which is represented as a physical volume. We then create two partitions out of that physical volume. A large logical volume group (LVG) is created from the LVM partition. Inside the LVG, we have 8 logical volumes (LV) and each contains a single file system (FS).

These numbers will be used for the initial build of the system:

boot = 1 GB
root = 5 GB
home = 0.5 GB
srv = 0.5 GB
usr = 4 GB
var = 2 GB
tmp = 2 GB
opt = 0.5 GB
bak = 2 GB

Important information
  • When creating servers from scratch, many admins will utilize fewer logical volumes which is fine when you know the system that will be created. I create many logical volumes so that I can use this server as a template to deploy other servers very quickly...which will have different storage requirements and this configuration allows maximum flexibility with a single template image.
  • When the logical volumes and file systems are initially created, they consume the maximum amount of space allocated which means the file system size will initially equal the logical volume size. These partition sizes above are artificially small for that reason. These will be later modified so that the logical volume will be larger than the file system so that the file system has room to expand when needed in a safe and automated manner.
  • If you want, you can initially allocate a larger disk such as a 50 GB drive rather than adding the 2nd and 3rd disk as noted in these steps. These are just technical examples on how to manage and expand your storage when needed. On physical installations, it is not best practice to add a physical disk to an existing volume since 1 disk failure will affect the entire volume.
  • The /tmp folder is strictly temporary. By default, each time the server reboots, this folder is deleted and re-created.
  • The /bak folder will retain the most recent backup and is considered the "local" copy of the backup and where the remote backup server will pull backups from.
VMware Virtual Machine Settings
WARNING: If you are getting "multipathd" errors in syslog, you need to edit the .vmx configuration on ESXi to include the following entry because Ubuntu is now referencing hard disks by UUID:

Code: Select all

disk.EnableUUID = "TRUE"
  • Configuration: Custom
  • Name: srv-ubuntu
  • Datastore: V3700-SASR5
  • Virtual Machine Version: 8
  • Guest Operating System: Linux, Version: Ubuntu Linux (64-bit)
  • Number of virtual processors: 1
  • Memory Size: 1 GB
  • Number of NICs: 1
  • NIC 1: VM Network
  • Adapter: VMXNET3, Connect at Power On: Checked
  • SCSI controller: LSI Logic Parallel
  • Select a Disk: Create a new virtual disk
  • Create a Disk: 30 GB, No thin provisioning, No cluster features, Store with the virtual machine
  • Advanced Options: Virtual Device Node = SCSI (0:0)
  • Remove Floppy Drive
  • Mount CD/DVD Drive to Ubuntu ISO (ubuntu-22.04-live-server-amd64.iso). Make sure CD/DVD is set to Connect at power on
  • Set boot options to Force BIOS Setup so you can set CDROM to boot before the Hard Disk
VirtualBox Virtual Machine Settings
  • Name: srv-ubuntu
  • Operating System: Linux
  • Version: Ubuntu (64 bit)
  • Memory: 1 GB
  • Check - Start-up Disk
    - Create new hard disk
    - VMDK
    - Dynamically allocated
    - Size: 30 GB
  • Select srv-ubuntu and click Settings (CTRL+S)
    - System, Processor, Enable PAE/NX
    - Network, Attached to: Bridged Adapter, Advanced, Adapter Type: Intel PRO/1000 MT Server
    - Storage, IDE Controller, Choose a virtual CD/DVD disk file, ubuntu-22.04-live-server-amd64.iso
Install PuTTY

When running inside a virtual machine, the response time for screen refreshes can be painfully slow to view man (manual) pages and navigating in VI (text editor). However, when using PuTTY via SSH (or Terminal), it is a far better solution for your Ubuntu console because it handles the screen draws much faster when scrolling and allows copying and pasting text between windows.

For example, selecting and copying a command in this document and then right-clicking in the PuTTY window will paste the command and have it ready to execute. Any text/lines highlighted with the mouse will be automatically copied into clipboard memory on the PuTTY window.

Download the portable edition and run the install...except it does not really "install" like a normal program, it simply extracts to a specified folder and will run from that folder even if you put it on a USB stick and carry over to a new computer (requires no install to run and thus leaves no footprint on your Windows system)
  1. Start PuTTY
  2. Under Window - Translation - Remote character set, select UTF-8
  3. Type the following and click the Save button:
    Host Name: SRV-Ubuntu (or the IP such as 192.168.107.2)
    Port: 22
    Connection type: SSH
    Saved Sessions: SRV-Ubuntu
  4. Now all you have to do is double-click on the session and it will connect to your server (when online).
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Install Ubuntu Server

Post: # 1005Post LHammonds »

Install Ubuntu Server
  1. Power on the Virtual Machine (VM)
  2. Try or Install Ubuntu Server - Press {ENTER}
  3. Language - Press {ENTER} to accept English
  4. Keyboard - Press {ENTER} to accept English (US)
  5. Choose type of install - Press {ENTER} to accept the following default:
    (X) Ubuntu Server
    ( ) Ubuntu Server (minimized)
  6. Network connections - Select the network card (ens160 for example), Edit IPv4, set the following and select Done:
    - IPv4 Method: Manual
    - Subnet: 192.168.107.0/24
    - Address: 192.168.107.2
    - Gateway: 192.168.107.1
    - Name servers: 192.168.107.212,192.168.107.213,1.1.1.1,8.8.8.8
    - Search domains:
  7. Configure proxy - Press {ENTER} for empty proxy address
  8. Mirror address - Press {ENTER} for default of http://us.archive.ubuntu.com/ubuntu
  9. Installer update available - Press {ENTER} for default of Continue without updating
  10. Guided storage configuration - Select Custom storage layout and then select Done
  11. Storage configuration - Set the following and select Done:
    Select free space -> Add GPT Partition, set the following and choose Create:
    • Size: 1G
    • Format: ext4
    • Mount: /boot
    Select free space -> Add GPT Partition, set the following and choose Create:
    • Size: 28G
    • Format: Leave unformatted
    Select Create volume group (LVM), set the following and choose Create:
    • Name: LVG
    • [X] partition 3 28.000G
    • [ ] Create encrypted volume
    Select free space under the LVM volume group -> Create Logical Volume, set the following and choose Create:
    • Name: root
    • Size: 5G
    • Format: ext4
    • Mount: /
    Select free space under the LVM volume group -> Create Logical Volume, set the following and choose Create:
    • Name: home
    • Size: 0.5G
    • Format: ext4
    • Mount: /home
    Select free space under the LVM volume group -> Create Logical Volume, set the following and choose Create:
    • Name: srv
    • Size: 0.5G
    • Format: ext4
    • Mount: /srv
    Select free space under the LVM volume group -> Create Logical Volume, set the following and choose Create:
    • Name: usr
    • Size: 4G
    • Format: ext4
    • Mount: /usr
    Select free space under the LVM volume group -> Create Logical Volume, set the following and choose Create:
    • Name: var
    • Size: 2G
    • Format: ext4
    • Mount: /var
    Select free space under the LVM volume group -> Create Logical Volume, set the following and choose Create:
    • Name: tmp
    • Size: 2G
    • Format: ext4
    • Mount: Other -> /tmp
    Select free space under the LVM volume group -> Create Logical Volume, set the following and choose Create:
    • Name: opt
    • Size: 0.5G
    • Format: ext4
    • Mount: Other -> /opt
    Select free space under the LVM volume group -> Create Logical Volume, set the following and choose Create:
    • Name: bak
    • Size: 2G
    • Format: ext4
    • Mount: Other -> /bak
    Here is what it should look like:

    Code: Select all

    USED DEVICES
      DEVICE                                      TYPE                  SIZE
     [LVG (new)                                   LVG volume group     27.996G
      root          new, to be formatted as ext4, mounted at /          5.000G
      home          new, to be formatted as ext4, mounted at /home    512.000M
      srv           new, to be formatted as ext4, mounted at /srv     512.000M
      usr           new, to be formatted as ext4, mounted at /usr       4.000G
      var           new, to be formatted as ext4, mounted at /var       2.000G
      tmp           new, to be formatted as ext4, mounted at /tmp       2.000G
      opt           new, to be formatted as ext4, mounted at /opt     512.000M
      bak           new, to be formatted as ext4, mounted at /bak       2.000G
    
    [ 36000c2945e543be985a4567efde913ce           local disk           30.000G
      partition 1   new, BIOS grub spacer                               1.000M
      partition 2   new, to be formatted as ext4, mounted at /boot      1.000G
      partition 3   new, PV of LVM volume group LVG                    28.000G
    
  12. Confirm destruction action - Select Continue
  13. Profile setup - Set the following and choose Done:
    • Your name: Administrator
    • Your server's name: srv-ubuntu
    • Pick a usesrname: administrator
    • Choose a password: myadminpass
    • Confirm your password: myadminpass
  14. SSH Setup - Set the following and select Done:
    • [X] Install OpenSSH server
    • Import SSH identity: No
  15. Featured Server Snaps - Choose none of them - Select Done
    [ ] micro8s
    [ ] nextcloud
    [ ] wekan
    [ ] kata-containers
    [ ] docker
    [ ] canonical-livepatch
    [ ] rocketchat-server
    [ ] mosquitto
    [ ] etcd
    [ ] powershell
    [ ] stress-ng
    [ ] sabnzbd
    [ ] wormhole
    [ ] aws-cli
    [ ] google-cloud-sdk
    [ ] slcli
    [ ] doctl
    [ ] conjure-up
    [ ] postgresql10
    [ ] heroku
    [ ] keepalived
    [ ] prometheus
    [ ] juju
  16. Installation Complete - from the VM menu, select VM --> Edit Settings and select CD/DVD Drive 1 and change to "Client Device" which will effectively remove the ISO. Now press {ENTER} to reboot.
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Initial Configurations

Post: # 1006Post LHammonds »

Initial Configurations
  1. At the login prompt, login with your administrator account (administrator / myadminpass)
  2. Check the server timezone settings:

    Code: Select all

    timedatectl
                   Local time: Tue 2022-05-31 19:52:20 UTC
               Universal time: Tue 2022-05-31 19:52:20 UTC
                     RTC time: Tue 2022-05-31 19:52:20
                    Time zone: Etc/UTC (UTC, +0000)
    System clock synchronized: yes
                  NTP service: active
              RTC in local TZ: no
    If this is not correct, you will need to set your region/city. Before setting region/city, list the available options to make sure you find the right combination:

    Code: Select all

    timedatectl list-timezones
    Mine should be America/Chicago so I will run the following command:

    Code: Select all

    sudo timedatectl set-timezone America/Chicago
  3. Let's list the block IDs:
    blkid /dev/mapper/LVG-home: UUID="de94f98a-ef83-4779-9104-f920829757cf" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-bak: UUID="86b2b348-a89f-4bfb-8624-4be4565d7a19" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-var: UUID="29b831f2-69f9-4bc5-8b65-22c6138c8485" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-srv: UUID="c53e72cd-6f9e-4d7a-99a3-3247c49666a7" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-root: UUID="d1392cba-b4af-4721-9088-810eef9cb21f" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-opt: UUID="ec5cf098-1b88-4fa2-a478-b6b77fec6dd9" BLOCK_SIZE="4096" TYPE="ext4" /dev/sda2: UUID="5bce6059-5191-4284-a662-18e8c5705928" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="de99337b-7605-4559-a3cb-491190d5e5b2" /dev/sda3: UUID="OFQMa0-B26I-7Ti0-xd4r-ysjA-QUMP-CSdmyD" TYPE="LVM2_member" PARTUUID="8a923539-562a-4911-9c5a-697a63dd8eb5" /dev/mapper/LVG-tmp: UUID="418165ca-9abd-471b-bed0-587255e66f98" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-usr: UUID="066326e0-66c3-4113-83e8-c02327b0b47a" BLOCK_SIZE="4096" TYPE="ext4"
  4. The installer did not let us add labels during install. Labels help identify partitions when it comes time to restore from backup (and helps with fstab). Let's add labels to all ext4 partitions:

    Code: Select all

    sudo e2label /dev/sda2 boot
    sudo e2label /dev/mapper/LVG-bak bak
    sudo e2label /dev/mapper/LVG-opt opt
    sudo e2label /dev/mapper/LVG-srv srv
    sudo e2label /dev/mapper/LVG-tmp tmp
    sudo e2label /dev/mapper/LVG-usr usr
    sudo e2label /dev/mapper/LVG-var var
    sudo e2label /dev/mapper/LVG-root root
    sudo e2label /dev/mapper/LVG-home home
  5. Let's take another look at the block ID list and verify our labels were added:
    blkid /dev/mapper/LVG-home: LABEL="home" UUID="de94f98a-ef83-4779-9104-f920829757cf" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-bak: LABEL="bak" UUID="86b2b348-a89f-4bfb-8624-4be4565d7a19" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-var: LABEL="var" UUID="29b831f2-69f9-4bc5-8b65-22c6138c8485" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-srv: LABEL="srv" UUID="c53e72cd-6f9e-4d7a-99a3-3247c49666a7" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-root: LABEL="root" UUID="d1392cba-b4af-4721-9088-810eef9cb21f" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-opt: LABEL="opt" UUID="ec5cf098-1b88-4fa2-a478-b6b77fec6dd9" BLOCK_SIZE="4096" TYPE="ext4" /dev/sda2: LABEL="boot" UUID="5bce6059-5191-4284-a662-18e8c5705928" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="de99337b-7605-4559-a3cb-491190d5e5b2" /dev/sda3: UUID="OFQMa0-B26I-7Ti0-xd4r-ysjA-QUMP-CSdmyD" TYPE="LVM2_member" PARTUUID="8a923539-562a-4911-9c5a-697a63dd8eb5" /dev/mapper/LVG-tmp: LABEL="tmp" UUID="418165ca-9abd-471b-bed0-587255e66f98" BLOCK_SIZE="4096" TYPE="ext4" /dev/mapper/LVG-usr: LABEL="usr" UUID="066326e0-66c3-4113-83e8-c02327b0b47a" BLOCK_SIZE="4096" TYPE="ext4"
  6. Sometimes the df command will show partitions such as /usr using the long "by-id" line. I like to modify fstab to make sure that does not happen and make fstab look a bit cleaner.
    Here is the original fstab:

    Code: Select all

    # /etc/fstab: static file system information.
    #
    # Use 'blkid' to print the universally unique identifier for a
    # device; this may be used with UUID= as a more robust way to name devices
    # that works even if disks are added and removed. See fstab(5).
    #
    # <file system> <mount point>   <type>  <options>       <dump>  <pass>
    # / was on /dev/LVG/root during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-P94Z1eNFjc37i3UKq5Ct6fx65kddzWOr4iZHmPE4OWsYxl0UueE5NckR23GOfpSq / ext4 defaults 0 1
    # /home was on /dev/LVG/home during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-P94Z1eNFjc37i3UKq5Ct6fx65kddzWOrcXyjxZA1k3D1Rsda36LYmldo4yHgi3pc /home ext4 defaults 0 1
    # /srv was on /dev/LVG/srv during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-P94Z1eNFjc37i3UKq5Ct6fx65kddzWOrn1wOmMnvOJhp8ehVTFtjImAJ1ugDBojJ /srv ext4 defaults 0 1
    # /usr was on /dev/LVG/usr during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-P94Z1eNFjc37i3UKq5Ct6fx65kddzWOr2hM8MKfLi1d4hHqhx7QdrcP0pSd9MW3i /usr ext4 defaults 0 1
    # /var was on /dev/LVG/var during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-P94Z1eNFjc37i3UKq5Ct6fx65kddzWOrh46TmpwoxWFUKnlakRSbX7NV4qBobNbk /var ext4 defaults 0 1
    # /tmp was on /dev/LVG/tmp during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-P94Z1eNFjc37i3UKq5Ct6fx65kddzWOrLNiNrqfWZIHlP9x3z84rDfy78wh7MrO8 /tmp ext4 defaults 0 1
    # /opt was on /dev/LVG/opt during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-P94Z1eNFjc37i3UKq5Ct6fx65kddzWOrUn98Iw8HHZlhvsYriBrSzX362IADaYqt /opt ext4 defaults 0 1
    # /bak was on /dev/LVG/bak during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-P94Z1eNFjc37i3UKq5Ct6fx65kddzWOrjgLzqvjuZ6e4sKwm1cFFrcM95KH053zl /bak ext4 defaults 0 1
    # /boot was on /dev/sda2 during curtin installation
    /dev/disk/by-uuid/27c19a11-83f8-4a2a-8f81-53fa853fe4e3 /boot ext4 defaults 0 1
    /swap.img       none    swap    sw      0       0
    
    Here is my modified fstab of the above:

    Code: Select all

    # <file system> <mount point>   <type>  <options>       <dump>  <pass>
    LABEL=root / ext4 defaults 0 1
    LABEL=home /home ext4 defaults 0 2
    LABEL=srv /srv ext4 defaults 0 2
    LABEL=usr /usr ext4 defaults 0 2
    LABEL=var /var ext4 defaults 0 2
    LABEL=tmp /tmp ext4 defaults 0 2
    LABEL=opt /opt ext4 defaults 0 2
    LABEL=bak /bak ext4 defaults 0 2
    LABEL=boot /boot ext4 defaults 0 1
    /swap.img none swap sw 0 0
    
  7. Before rebooting to let the fstab changes take effect, I like to run this command to make sure I didn't make a typo anywhere:
    NOTE: Warnings about non-bind mount source for swap files are normal and can be ignored.

    Code: Select all

    sudo findmnt --verify --verbose
  8. Edit the network configuration file and inspect it for desired settings:

    Code: Select all

    sudo vi /etc/netplan/00-installer-config.yaml
  9. Here is an example of the contents after I make changes: (be sure to modify the red items match your specific server)
    network: ethernets: ens160: optional: true dhcp4: false dhcp6: false addresses: - 192.168.107.2/24 routes: - to: default via: 192.168.107.1 nameservers: addresses: - 192.168.107.212 - 192.168.107.213 - 1.1.1.1 - 8.8.8.8 search: [] version: 2
    NOTE #1: The above YAML format is extremely sensitive to spaces. Each indentation needs to be exactly 2 spaces. Visit NetPlan.io for more information.

    NOTE #2: The network card name (ens160) might be something different on your machine, be sure to use the name of your card. Example: ls /sys/class/net

    NOTE #3: The "optional: true" is there to allow the boot process to not hang if a service wants network access before the network has a chance to start.

    NOTE #4: You may need to manually remove the DHCP record (lease) associated to this Ubuntu server from your DHCP server so the correct IP can be found by other machines on the network. This can be avoided by temporarily configuring the VM Network Adapter connection to be "Host Only Network" instead of "VM Network" so the server is isolated during setup...at least until you reach the testing of the static IP below.

    NOTE #5: You might also need to manually add a HOST(A) record to your Windows DNS server (for srv-ubuntu.mydomain.com and srv-ubuntu.work.mydomain.com)
  10. Verify that the configuration file syntax is good:

    Code: Select all

    sudo netplan --debug generate
  11. If there were no errors, restart the network by typing the following:

    Code: Select all

    sudo netplan --debug apply
  12. Verify settings:

    Code: Select all

    ip address | grep inet
  13. Make sure the interface is up:

    Code: Select all

    ip link | grep -i -w UP
  14. Sanity check! Type ping www.google.com or similar and see if ping and DNS works. To test without using DNS to resolve names to IPs, type ping 1.1.1.1
  15. Make sure any file created by the root account is set to only be accessible to root by default:

    Code: Select all

    sudo su
    echo 'umask 0077' >> ~/.bashrc
    exit
    
  16. Disable command history for your account and the root user on production systems to prevent hackers from seeing the commands you have typed in the past which might expose passwords:

    Code: Select all

    echo 'set +o history' >> ~/.bashrc
    sudo su
    echo 'set +o history' >> ~/.bashrc
    exit
    
  17. Make sure menus will correctly draw lines instead of displaying ascii codes for your account and the root account:

    Code: Select all

    echo 'export NCURSES_NO_UTF8_ACS=1' >> ~/.bashrc
    sudo su
    echo 'export NCURSES_NO_UTF8_ACS=1' >> ~/.bashrc
    exit
    
  18. Remove cloud-init (optional) - My systems are not on Amazon or similar cloud services and have no need to utilize cloud-init.

    Code: Select all

    sudo apt --purge autoremove cloud-init
    sudo rm -rf /etc/cloud/
    sudo rm -rf /var/lib/cloud/
  19. Remove snap (optional) - My systems do not use LivePatch nor LXD. I also do not intend to install applications with snap. Snap causes my systems to boot slower so here is how I remove it:
    1. List installed snap packages:

      Code: Select all

      snap list
      Name    Version        Rev    Tracking       Publisher   Notes
      core20  20220512       1494   latest/stable  canonical✓  base
      lxd     5.0.0-b0287c1  22923  5.0/stable/…   canonical✓  -
      snapd   2.55.5         15904  latest/stable  canonical✓  snapd
      
    2. Remove the installed snaps with "snapd" being last and "core20" being next to last and once all snaps are removed, uninstall snap.

      Code: Select all

      sudo snap remove --purge lxd
      sudo snap remove --purge core20
      sudo snap remove --purge snapd
      sudo apt --purge autoremove snapd
  20. Shutdown and power off the server:

    Code: Select all

    sudo shutdown -P now
  21. At this point forward, you can use PuTTY to access the console rather than the console itself for better performance, ability to scroll, etc.
  22. In VM menu, select VM --> Snapshot --> Take Snapshot. Give it a name like STEP 1 and description of Ubuntu Server 22.04 LTS, Clean install, Static IP: 192.168.107.2 and click OK
  23. NOTE: Please remember to delete all snapshots when you are 100% done with the setup and happy with the results. Not deleting snapshots can secretly consume all your storage space over time. These snapshots are only there so you can revert back to a prior step quickly and easily without the need to completely re-install.
Add more SUDO users and lockdown SSH

The root user is locked by default which is good. When you installed the server, you created the administrator account which can run sudo commands with root authority. Let's add one more user that can use SUDO and ensure only these user accounts can login via SSH to the server.

Create a new user called "newadmin"

Code: Select all

sudo adduser newadmin
Now add them to the SUDO group which will allow them to use the SUDO command.

Code: Select all

sudo usermod -aG sudo newadmin
Now modify SSH service to only allow these 2 users to login via SSH.

Code: Select all

sudo vi /etc/ssh/sshd_config
Add the following line anywhere in the file:

Code: Select all

AllowUsers administrator newadmin
Reload the SSH config for the change to take affect:

Code: Select all

sudo systemctl reload sshd
Now only administrator and newadmin can login to the server via SSH. If you create another user, that user will not be able to login even with the correct password. It will just say "Access denied."

The firewall and fail2ban sections later on will further increase SSH security.

You can also use SSH key-based authentication and disable user/password authentication.

Operating System Patches
  1. Start the Ubuntu server and connect using PuTTY.
  2. Install the patches by typing the following commands:

    Code: Select all

    sudo apt update
    sudo apt upgrade
    sudo apt dist-upgrade
  3. Shutdown and power off the server

    Code: Select all

    sudo shutdown -P now
  4. In VM menu, select VM --> Snapshot --> Take Snapshot. Give it a name like STEP 2 and description of Ubuntu Server 22.04 LTS, Patches applied, Static IP: 192.168.107.2. The Snapshot Manager should now have a nice hierarchy of snapshots (STEP 1 --> STEP 2 --> You are here)
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Volume / Disk Management

Post: # 1007Post LHammonds »

Volume / Disk Management

Earlier, it was mentioned that the partition design needed to have some breathing room in each volume so that the file system inside can grow as needed. When the volumes were created during setup, the file systems were automatically expanded to fill the entire volume. We will now correct this by adding more "drives" to the system and then extend each logical volume to gain some breathing space.

Most logical volumes will be increased in size and then the file systems contained in them will be increased but not to the maximum amount.

This design will allow growth when needed and ensure that there will be time to add additional hard drives BEFORE they are needed which will keep the administrators from being stuck between a rock and a hard place! Nobody wants to lose a job because somebody did not estimate growth correctly or the budget did not allow for large capacity when the system first rolled out.

We started off with a 30 GB drive to hold these volumes and the changes below will use 24 GB.

Here are the planned adjustments for each logical volume:

root = 5 GB to 6 GB
home = 0.5 GB to 1 GB
srv = 0.5 GB to 1GB
usr = 4 GB to 5 GB
var = 2 GB to 3 GB
tmp = 2 GB to 3 GB
opt = 0.5 GB to 1 GB
bak = 2 GB to 4 GB

Here are the planned adjustments for each file system:

root = 5 GB (no change)
home = 0.5 GB (no change)
srv = 0.5 GB (no change)
usr = 4 GB (no change)
var = 2 GB (no change)
tmp = 2 GB (no change)
opt = 0.5 GB (no change)
bak = 2 GB to 3 GB

Here is a graphical representation of what will be accomplished:

Image

If we were to type df -h right now, we should see something like this:

Code: Select all

df -h
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                  98M  1.2M   96M   2% /run
/dev/mapper/LVG-root  4.9G  1.3G  3.4G  27% /
/dev/mapper/LVG-usr   3.9G  2.2G  1.5G  59% /usr
tmpfs                 486M     0  486M   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sda2             974M  125M  782M  14% /boot
/dev/mapper/LVG-var   2.0G  433M  1.4G  24% /var
/dev/mapper/LVG-home  488M   56K  452M   1% /home
/dev/mapper/LVG-srv   488M   24K  452M   1% /srv
/dev/mapper/LVG-tmp   2.0G   72K  1.8G   1% /tmp
/dev/mapper/LVG-opt   488M   24K  452M   1% /opt
/dev/mapper/LVG-bak   2.0G   24K  1.8G   1% /bak
tmpfs                  98M  4.0K   98M   1% /run/user/1000
To get a list of volume paths to use in the next commands, use lvscan to show your current volumes and their sizes.

Code: Select all

sudo lvscan
  ACTIVE            '/dev/LVG/root' [5.00 GiB] inherit
  ACTIVE            '/dev/LVG/home' [512.00 MiB] inherit
  ACTIVE            '/dev/LVG/srv' [512.00 MiB] inherit
  ACTIVE            '/dev/LVG/usr' [4.00 GiB] inherit
  ACTIVE            '/dev/LVG/var' [2.00 GiB] inherit
  ACTIVE            '/dev/LVG/tmp' [2.00 GiB] inherit
  ACTIVE            '/dev/LVG/bak' [2.00 GiB] inherit
  ACTIVE            '/dev/LVG/opt' [512.00 MiB] inherit
Type the following to set the exact size of the volume by specifying the end-result size you want:

Code: Select all

sudo lvextend -L6G /dev/LVG/root
sudo lvextend -L1G /dev/LVG/home
sudo lvextend -L1G /dev/LVG/srv
sudo lvextend -L5G /dev/LVG/usr
sudo lvextend -L3G /dev/LVG/var
sudo lvextend -L3G /dev/LVG/tmp
sudo lvextend -L1G /dev/LVG/opt
sudo lvextend -L4G /dev/LVG/bak
or you can grow each volume by the specified amount (the number after the plus sign):

Code: Select all

sudo lvextend -L+1G /dev/LVG/root
sudo lvextend -L+0.5G /dev/LVG/home
sudo lvextend -L+0.5G /dev/LVG/srv
sudo lvextend -L+1G /dev/LVG/usr
sudo lvextend -L+1G /dev/LVG/var
sudo lvextend -L+1G /dev/LVG/tmp
sudo lvextend -L+0.5G /dev/LVG/opt
sudo lvextend -L+2G /dev/LVG/bak
To see the new sizes, use lvscan

Code: Select all

sudo lvscan
  ACTIVE            '/dev/LVG/root' [6.00 GiB] inherit
  ACTIVE            '/dev/LVG/home' [1.00 GiB] inherit
  ACTIVE            '/dev/LVG/srv' [1.00 GiB] inherit
  ACTIVE            '/dev/LVG/usr' [5.00 GiB] inherit
  ACTIVE            '/dev/LVG/var' [3.00 GiB] inherit
  ACTIVE            '/dev/LVG/tmp' [3.00 GiB] inherit
  ACTIVE            '/dev/LVG/opt' [1.00 GiB] inherit
  ACTIVE            '/dev/LVG/bak' [4.00 GiB] inherit
The last thing to do now is the actual growth of the file systems. We want to grow the existing file systems but only to a certain amount so we do not take up all the space in the volume. We want room for growth in the future so we have time to order and install new drives when needed.

Code: Select all

sudo resize2fs /dev/LVG/bak 3G
If we need to increase space in /usr at a later point, we can issue the following command without any downtime (we will automate this in a nifty script later):

Code: Select all

sudo resize2fs /dev/LVG/usr 4.2G
We could continue to increase this particular file system all the way until we reach the limit of the volume which is 5 GB at the moment.

If we were to type df -h right now, we should see something like this:

Code: Select all

df -h
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                  98M  1.2M   96M   2% /run
/dev/mapper/LVG-root  4.9G  1.3G  3.4G  27% /
/dev/mapper/LVG-usr   3.9G  2.2G  1.5G  59% /usr
tmpfs                 486M     0  486M   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sda2             974M  125M  782M  14% /boot
/dev/mapper/LVG-var   2.0G  434M  1.4G  24% /var
/dev/mapper/LVG-home  488M   56K  452M   1% /home
/dev/mapper/LVG-srv   488M   24K  452M   1% /srv
/dev/mapper/LVG-tmp   2.0G   72K  1.8G   1% /tmp
/dev/mapper/LVG-opt   488M   24K  452M   1% /opt
/dev/mapper/LVG-bak   2.9G  6.0M  2.8G   1% /bak
tmpfs                  98M  4.0K   98M   1% /run/user/1000
Remember, df -h will tell you the size of the file system and lvscan will tell you the size of the volumes where the file systems live in.

TIP: If you want to see everything in a specific block size, such as everything showing up in megabytes, you can use df --block-size m

Swap File Management

If you do not specify a swap partition during the initial setup, a swap file system will be created automatically and point to a file called "swap.img" in the root filesystem.

Type swapon --summary to see the status of the swap system:

Code: Select all

swapon --summary

Filename                                Type            Size            Used   Priority
/swap.img                               file            1266684         0      -2
Specific needs vary but the current rule of thumb for Linux servers is to have a swap file 1/2 the size of the amount of RAM in your system.

Let's assume we want a 1GB swap file and we want it on the /bak partition. Keep in mind that it is normally recommended to keep the swapfile with the root partition for performance reasons. Here are the steps to setup this scenario:
  1. Make sure you have space on /bak by typing df -h /bak
  2. Run the following commands for the new swap file:

    Code: Select all

    sudo fallocate --length 1G /bak/swapfile1g
    sudo chown root:root /bak/swapfile1g
    sudo chmod 600 /bak/swapfile1g
    sudo mkswap /bak/swapfile1g --label swap
    sudo swapon /bak/swapfile1g
  3. Look at the current swap settings:

    Code: Select all

    swapon --summary
    Filename                                Type            Size    Used    Priority
    /swap.img                               file            1266684         0      -2
    /bak/swapfile1g                         file            1048572 0       -3
  4. Now disable the old swap file using these commands:

    Code: Select all

    sudo swapoff /swap.img
    sudo rm /swap.img
  5. Remove the old swapfile from /etc/fstab and add the new one.

    Code: Select all

    sudo vi /etc/fstab
    Remove:

    Code: Select all

    /swap.img none swap sw 0 1
    Add:

    Code: Select all

    /bak/swapfile1g none swap sw 0 1
  6. Since changes were made to fstab, check to make sure there are no typo errors:
    NOTE: Warning about non-bind mount source for the swapfile is normal and can be ignored.

    Code: Select all

    sudo findmnt --verify --verbose
  7. Look at the current swap settings again:

    Code: Select all

    # swapon --summary
    Filename                                Type            Size    Used    Priority
    /bak/swapfile1g                         file            1048500 0       -2
  8. Reboot the server and run the summary command again to verify that your /etc/fstab changes worked.
Adding More Hard Drives

For this exercise, we will add two additional hard drives. The addition of these drives are NOT necessary and this section can be skipped. The extra drives are only to demonstrate how to add additional hard drives to the system.

Adding more space in VMware or VirtualBox is easy. In this exercise, each drive will be added as a separate disk just as if we were to add a physical drive to a physical server. However, if you were to add a physical disk, it is bad practice to add a disk to an existing volume...when one drive fails, the entire volume goes offline.

vSphere Steps
  1. Shutdown and power off the server by typing sudo shutdown -P now
  2. In the vSphere client, right-click the Virtual Machine and choose Edit Settings.
  3. On the hardware tab, click the Add button and select Hard Disk. Click Next, choose "Create a new virtual disk", click Next, set the size to 10 GB, click Next, Next, Finish.
  4. Add another 15 GB disk using the same steps above and click OK to close the settings and allow VMware to process the changes.
VirtualBox Steps
  1. Shutdown and power off the server by typing sudo shutdown -P now
  2. In the VirtualBox Manager, select the Virtual Machine and click Settings.
  3. On the Storage tab, select Controller: SATA and click the Add new storage attachment button and select Add Hard Disk. Click Create new disk, VDI, Next, Fixed size, Next, give it a Name/Location/Size of 10 GB, click Create.
  4. Add another 15 GB disk using the same steps above and click OK to close the settings and allow VirtualBox to process the changes.
Collect information about the newly added drives.
  1. Start the server and connect using PuTTY and your administrator credentials.
  2. Make note of how much "Free PE / Size" you have in your logical volume group when using the vgdisplay command. When done adding the new drives, the free space listed here will increase by the amount added.
  3. Use pvdisplay which should show something similar to this:
    sudo pvdisplay --- Physical volume --- PV Name /dev/sda3 VG Name LVG PV Size 28.00 GiB / not usable 0 Allocatable yes PE Size 4.00 MiB Total PE 7167 Free PE 1023 Allocated PE 6144 PV UUID f7NB0i-fOy2-MQjF-bz69-KIpE-zVpr-UTH29e
    The important bits of info here are the PV Name and VG Name for our existing configuration.
  4. Use fdisk -l which should show something similar to this (however I abbreviated it to show just the important parts):
    sudo fdisk -l Disk /dev/sda: 30 GiB, 32212254720 bytes, 62914560 sectors Device Boot Start End Sectors Size Id Type /dev/sda1 2048 4095 2048 1M BIOS boot /dev/sda2 4096 2101247 2097152 1G Linux filesystem /dev/sda3 2101248 60821503 58720256 28G Linux filesystem Disk /dev/sdb: 10 GiB, 10884901888 bytes, 21165824 sectors Disk /dev/sdc: 15 GiB, 15884901888 bytes, 28165824 sectors
    The important bits of info here are the device paths for the new drives which I highlighted in red.
Prepare the first drive (/dev/sdb) to be used by the LVM

Type the following:

Code: Select all

sudo fdisk /dev/sdb
n (Create New Partition)
p (Primary Partition)
1 (Partition Number)
{ENTER} (use default for first cylinder)
{ENTER} (use default for last cylinder)
t (Change partition type)
8e (Set to Linux LVM)
p (Preview how the drive will look)
w (Write changes)
Prepare the second drive (/dev/sdc) to be used by the LVM

Do the exact same steps as above but start with sudo fdisk /dev/sdc

Create physical volumes using the new drives

If we type sudo fdisk -l, we now see /dev/sdb1 and /dev/sdc1 which are Linux LVM partitions.

Type the following to create physical volumes:

Code: Select all

sudo pvcreate /dev/sdb1
sudo pvcreate /dev/sdc1
Now add the physical volumes to the volume group (LVG) by typing the following:

Code: Select all

sudo vgextend LVG /dev/sdb1
sudo vgextend LVG /dev/sdc1
You can run the sudo vgdisplay command to see that the "Free PE / Size" has increased.

Now that the space of both drives have been added to the logical volume group called LVG, we can now allocate that space to grow the logical volumes and then the file systems as needed.

Shutdown and power off the server by typing sudo shutdown -P now

In VM menu, select VM --> Snapshot --> Take Snapshot. Give it a name like STEP 3 and description of Ubuntu Server 22.04 LTS, Storage space adjusted, Static IP: 192.168.107.2. The Snapshot Manager should now have a nice hierarchy of snapshots (STEP 1 --> STEP 2 --> STEP 3 --> You are here)
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Software Configurations

Post: # 1008Post LHammonds »

Software Configurations
  1. Turn on the server and connect using PuTTY. with your administrator credentials.
  2. At the $ prompt, type the following to install various utilities which are described below:

    Code: Select all

    sudo apt -y install vim-nox p7zip-full fsarchiver sendemail dialog ncdu
    • vim-nox for use instead of the built-in VI editor. more info
    • p7zip-full is a 7-zip archive utility.
    • fsarchiver is a backup utility.
    • sendemail is a command-line email utility.
    • dialog is used to build menu selections.
    • ncdu shows disk usage in menu format.
  3. Type sudo vi /etc/hosts and add your email server:
    192.168.107.25 srv-mail
  4. Test the ability to send email by typing:

    Code: Select all

    sendemail -f root@myserver -t MyTargetAddress@MyDomain.com -u "This is the Subject" -m "This is the body of the email" -s srv-mail:25
  5. To make changes to the behavior of the vim-nox editor, edit its configuration file:

    Code: Select all

    vi ~/.vimrc
    Automatic indents can ruin code that you copy/paste and this will disable auto-indents:

    Code: Select all

    filetype indent off
    This sets indents to be 2 spaces when you press TAB and not a tab character:
    Add the following:

    Code: Select all

    set tabstop=2
    set shiftwidth=2
    set expandtab
    This changes the commented text from the hard-to-see dark blue to light blue:

    Code: Select all

    hi Comment ctermfg=LightBlue
VMware Tools

Starting with Ubuntu 16.04, open-vm-tools are installed automatically. We should not need to perform any actions in this section.

VirtualBox Guest Additions - Installation

The Guest Additions need to be installed if the VM is using a VirtualBox host. This will insure maximum performance in a virtual environment. The specific features this enables are: mouse pointer integration, shared clipboard, drag and drop, shared folders, better video support, seamless windows, generic host/guest communication channels, time synchronization and automated logins.
  1. Connect to the server using PuTTY or Terminal.
  2. At the login prompt, login with your administrator account (administrator / myadminpass)
  3. You need to perform the following commands to fulfill the prerequisites:

    Code: Select all

    sudo apt install dkms
    sudo reboot
  4. Connect to the server using PuTTY.
  5. At the login prompt, login with your administrator account (administrator / myadminpass)
  6. From the VirtualBox menu, click Devices, Install Guest Additions
  7. At the command line, type the following:

    Code: Select all

    sudo mount /dev/cdrom /media/cdrom
    sudo /media/cdrom/VBoxLinuxAdditions.run
    sudo umount /media/cdrom
  8. To see the status, stop or start the service, you can use these commands:

    Code: Select all

    systemctl status vboxadd-service
    sudo systemctl stop vboxadd-service
    sudo systemctl start vboxadd-service

VirtualBox Guest Additions - Upgrading

If VirtualBox is updated on the host machine, each VM also needs the upgraded Guest Additions.

Then mount the CDROM and run the installer just like the above. Reboot after it is upgraded.
  1. Connect to the server using PuTTY or Terminal.
  2. At the login prompt, login with your administrator account (administrator / myadminpass)
  3. From the VirtualBox menu, click Devices, Install Guest Additions
  4. At the console, type the following:

    Code: Select all

    sudo mount /dev/cdrom /mnt/cdrom
    sudo /mnt/cdrom/VBoxLinuxAdditions.run
    sudo reboot

VirtualBox Guest Additions - Uninstallation

If a VM will be migrated from VirtualBox to something like a VMware, the Guest Additions on the VM will need to be removed.
  1. Connect to the server using PuTTY or Terminal.
  2. At the login prompt, login with your administrator account (administrator / myadminpass)
  3. Type the following:

    Code: Select all

    cd /opt/VBox*
    sudo ./uninstall.sh
AppArmor Profiles

It might be necessary to disable specific AppArmor profiles for certain processes such as mysqld.
  1. Show the current status of AppArmor:

    Code: Select all

    sudo apparmor_status
    apparmor module is loaded.
    31 profiles are loaded.
    31 profiles are in enforce mode.
       /snap/snapd/15534/usr/lib/snapd/snap-confine
       /snap/snapd/15534/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
       /snap/snapd/15904/usr/lib/snapd/snap-confine
       /snap/snapd/15904/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
       /usr/bin/man
       /usr/lib/NetworkManager/nm-dhcp-client.action
       /usr/lib/NetworkManager/nm-dhcp-helper
       /usr/lib/connman/scripts/dhclient-script
       /usr/lib/snapd/snap-confine
       /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
       /{,usr/}sbin/dhclient
       lsb_release
       man_filter
       man_groff
       nvidia_modprobe
       nvidia_modprobe//kmod
       snap-update-ns.lxd
       snap.lxd.activate
       snap.lxd.benchmark
       snap.lxd.buginfo
       snap.lxd.check-kernel
       snap.lxd.daemon
       snap.lxd.hook.configure
       snap.lxd.hook.install
       snap.lxd.hook.remove
       snap.lxd.lxc
       snap.lxd.lxc-to-lxd
       snap.lxd.lxd
       snap.lxd.migrate
       snap.lxd.user-daemon
       tcpdump
    0 profiles are in complain mode.
    0 profiles are in kill mode.
    0 profiles are in unconfined mode.
    0 processes have profiles defined.
    0 processes are in enforce mode.
    0 processes are in complain mode.
    0 processes are unconfined but have a profile defined.
    0 processes are in mixed mode.
    0 processes are in kill mode.
    
  2. List the profiles:

    Code: Select all

    ls -l /etc/apparmor.d
    total 76
    drwxr-xr-x 2 root root  4096 Apr 21 01:01 abi
    drwxr-xr-x 4 root root  4096 Apr 21 01:01 abstractions
    drwxr-xr-x 2 root root  4096 Apr 21 01:02 disable
    drwxr-xr-x 2 root root  4096 Mar 10 01:35 force-complain
    drwxr-xr-x 2 root root  4096 Apr 21 01:02 local
    -rw-r--r-- 1 root root  1339 Mar 10 01:35 lsb_release
    -rw-r--r-- 1 root root  1189 Mar 10 01:35 nvidia_modprobe
    -rw-r--r-- 1 root root  3222 Aug  9  2021 sbin.dhclient
    drwxr-xr-x 5 root root  4096 Apr 21 01:01 tunables
    -rw-r--r-- 1 root root  3448 Mar 17 19:03 usr.bin.man
    -rw-r--r-- 1 root root  1421 Jun 20  2021 usr.bin.tcpdump
    -rw-r--r-- 1 root root 28376 Apr  8 14:48 usr.lib.snapd.snap-confine.real
    -rw-r--r-- 1 root root  1592 Nov 16  2021 usr.sbin.rsyslogd
    
  3. Let's disable the mysql profile:

    Code: Select all

    sudo aa-disable /etc/apparmor.d/usr.sbin.mysqld
    
  4. Reboot the server and run the apparmor_status command again to verify that the profile is still disabled after a reboot.
  5. To re-enable the profile we just disabled, type the following:

    Code: Select all

    sudo aa-enable /etc/apparmor.d/usr.sbin.mysqld
    
  6. To disable apparmor (and all profiles), type the following:

    Code: Select all

    sudo systemctl disable apparmor
  7. To re-enable apparmor, type the following:

    Code: Select all

    sudo systemctl enable apparmor
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Scripting

Post: # 1009Post LHammonds »

Scripting

Many of the sections below utilize BASH scripts as part of the solution/automation.

To speed up installation for myself and others, the commands below will download and extract the base set of scripts I use as a foundation for all my servers. You can download all of them at once or skip this step and copy/paste the scripts one at a time throughout this tutorial.

Code: Select all

cd /tmp
wget https://files.hammondslegacy.com/linux/init-scripts-2204.tar.gz
tar -xvf /tmp/init-scripts-2204.tar.gz -C /
rm /tmp/init-scripts-2204.tar.gz
NOTE: If you are leary about extracting the contents of the archive before seeing it, use the following command to peek inside before extracting:

Code: Select all

tar -ztvf /tmp/init-scripts-2204.tar.gz
The directory structure will be as follows after extraction:

Code: Select all

/var/scripts/common
/var/scripts/data
/var/scripts/prod
/var/scripts/test
The "common" directory contains code that is commonly used in other scripts.
The "data" directory contains stored information (currently, just backups of crontab schedule)
The "prod" directory contains all production-ready scripts
The "test" directory contains scripts that are under development or never need to be run in production-mode.

Most of my scripts will import a file called "standard.conf" from the common script folder.

/var/scripts/common/standard.conf (GitHub Download)
############################################################# ## Name : standard.conf ## Version : 1.3 ## Date : 2020-04-09 ## Author : LHammonds ## Purpose : Common variables and functions for various scripts. ## Compatibility : Verified on Ubuntu Server 22.04 LTS ######################## CHANGE LOG ######################### ## DATE VER WHO WHAT WAS CHANGED ## ---------- --- --- --------------------------------------- ## 2012-05-11 1.0 LTH Created script. ## 2014-04-21 1.1 LTH Added company variable. ## 2016-06-01 1.2 LTH Changed variables to CamelCase. ## 2020-04-09 1.3 LTH Added text color codes. ############################################################# ## Global Variables ## Company="abc" TempDir="/tmp" LogDir="/var/log" ShareDir="/srv/samba/share" MyDomain="mydomain.com" AdminEmail="admin@${MyDomain}" ReportEmail="LHammonds <lhammonds@${MyDomain}>" BackupDir="/bak" OffsiteDir="/mnt/backup" OffsiteTestFile="${OffsiteDir}/offline.txt" ArchiveMethod="tar.7z" ## Choices are tar.7z or tgz Hostname="$(hostname -s)" ScriptName="$0" ScriptDir="/var/scripts" MailFile="${TempDir}/mailfile.$$" ## Text color codes for use with "echo -e" COLORRESET='\033[0m' RED='\033[00;31m' GREEN='\033[00;32m' YELLOW='\033[00;33m' BLUE='\033[00;34m' PURPLE='\033[00;35m' CYAN='\033[00;36m' LIGHTGRAY='\033[00;37m' LRED='\033[01;31m' LGREEN='\033[01;32m' LYELLOW='\033[01;33m' LBLUE='\033[01;34m' LPURPLE='\033[01;35m' LCYAN='\033[01;36m' WHITE='\033[01;37m' ## Global Functions ## function f_sendmail() { ## Purpose: Send administrative email message. ## Parameter #1 = Subject ## Parameter #2 = Body sendemail -f "${AdminEmail}" -t "${ReportEmail}" -u "${1}" -m "${2}\n\nServer: ${Hostname}\nProgram: ${ScriptName}\nLog: ${LogFile}" -s srv-mail:25 1>/dev/null 2>&1 } function f_sendusermail() { ## Purpose: Send end-user email message. ## Parameter #1 = To ## Parameter #2 = Subject ## Parameter #3 = Body sendemail -f "${AdminEmail}" -t "${1}" -u "${2}" -m "${3}" -s srv-mail:25 1>/dev/null 2>&1 } function f_mount() { ## Mount the pre-configured remote share folder. ## NOTE: The Linux mount point should have a file called "offline.txt" mount -t cifs //srv-backup/MyShare ${OffsiteDir} --options nouser,rw,nofail,noexec,credentials=/etc/cifspw } function f_umount() { ## Dismount the remote share folder. ## NOTE: The unmounted folder should have a file called "offline.txt" umount ${OffsiteDir} }
Here is a script to help automate the creation of this structure on the various servers if you do not use the "init-scripts" archive:

setup-folders.sh

Code: Select all

#!/bin/bash
if [ ! -d /var/scripts/prod ]; then
  mkdir -p /var/scripts/prod
fi
if [ ! -d /var/scripts/test ]; then
  mkdir -p /var/scripts/test
fi
if [ ! -d /var/scripts/common ]; then
  mkdir -p /var/scripts/common
fi
if [ ! -d /var/scripts/data ]; then
  mkdir -p /var/scripts/data
fi
chown root:root -R /var/scripts
chmod 0755 -R /var/scripts
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Installing Ubuntu Server - Scheduling

Post: # 1010Post LHammonds »

Crontab Schedule

The crontab schedule can be edited directly by typing "crontab -e" but that can be a bit dangerous. It would be safer to edit a file and then load that file into the schedule. This will allow backups of the schedule to be made. If there is ever a problem with the schedule, it can be re-loaded with a known-good schedule or at least back to the way it was before the last change. This requires the person doing the editing to always work with a copy of the schedule 1st.

Here is an example crontab scheduling file for the root user:

/var/scripts/data/crontab.root (GitHub Download)

Code: Select all

#############################################################
## Name    : Crontab.root
## Author  : LHammonds
## Version : 1.2
## Date    : 2022-05-31
## Purpose : Crontab Schedule for root user
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2012-05-20 1.0 LTH Created schedule.
## 2020-04-09 1.1 LTH Minor updates.
## 2022-04-31 1.2 LTH Partition updates to match 22.04 instructions.
#############################################################

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Crontab SYNTAX:
#       __________ Minute (0-59)
#      / _________ Hour (0-23)
#     / /  _______ Day Of Month (1-31)
#    / /  /   ____ MONth (1-12)
#   / /  /   /   _ Day Of Week (0-7) (Sun = 0 or 7)
#  / /  /   /   /  -------------------------------------------------------------
# m h dom mon dow  command <arguments> > /dev/null 2>&1
#
# Backup database server
#
#0 23 * * * /var/scripts/prod/db-backup.sh > /dev/null 2>&1
#
# Backup specific database on demand
#
#0-59 * * * * /var/scripts/prod/db-ondemand-backup.sh > /dev/null 2>&1
#
# Daily checks for available space
#
0 1 * * * /var/scripts/prod/check-storage.sh root 500 100 > /dev/null 2>&1
5 1 * * * /var/scripts/prod/check-storage.sh home 100 50 > /dev/null 2>&1
10 1 * * * /var/scripts/prod/check-storage.sh srv 100 50 > /dev/null 2>&1
15 1 * * * /var/scripts/prod/check-storage.sh usr 100 50 > /dev/null 2>&1
20 1 * * * /var/scripts/prod/check-storage.sh var 100 50 > /dev/null 2>&1
25 1 * * * /var/scripts/prod/check-storage.sh tmp 100 50 > /dev/null 2>&1
30 1 * * * /var/scripts/prod/check-storage.sh opt 100 50 > /dev/null 2>&1
35 1 * * * /var/scripts/prod/check-storage.sh bak 500 500 > /dev/null 2>&1
#
# Daily software upgrade check
#
0 3 * * * /var/scripts/prod/apt-upgrade.sh > /dev/null 2>&1
30 3 * * * /var/scripts/prod/reboot-check.sh > /dev/null 2>&1
Once the file is created, make sure appropriate permissions are set by typing the following:

Code: Select all

chown root:root /var/scripts/data/crontab.root
chmod 0600 /var/scripts/data/crontab.root
To enable the root schedule using this file, type the following:

Code: Select all

crontab -u root /var/scripts/data/crontab.root
To disable the root schedule, type the following:

Code: Select all

touch /tmp/deleteme
crontab -u root /tmp/deleteme
rm /tmp/deleteme
If you need to modify the schedule, make a backup copy 1st. For example:

Code: Select all

cp /var/scripts/data/crontab.root /var/scripts/data/2022-05-31-crontab.root
vi /var/scripts/data/crontab.root
(make your changes)
crontab -u root /var/scripts/data/crontab.root
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Operator Scripts

Post: # 1011Post LHammonds »

Operator Scripts

I like scripts to be as generic as possible when copying among multiple servers.

I push the service start/stop specifics into their own scripts which can be unique to each server depending on what services need to be stopped/started.

On a MySQL/MariaDB server, the scripts that start and stop services would just contain the "mysql" line and the other service controls that do not apply are commented out.

NOTE: This script is custom-tailored to each server it is placed on to safely stop the services that are unique to it.
/var/scripts/prod/servicestop.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : servicestop.sh
## Version       : 1.2
## Date          : 2022-05-31
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : None
## Purpose       : Stop primary services.
## Run Frequency : As needed
## Exit Codes    : None
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ----------------------------
## 2013-01-08 1.0 LTH Created script.
## 2019-09-24 1.1 LTH Switched "service" to "systemctl" format.
## 2022-05-31 1.2 LTH Replaced echo statements with printf.
#############################################################
## NOTE: Configure whatever services you need stopped here.
printf "Stopping services...\n"
#systemctl stop vsftpd
#systemctl stop nagios
#systemctl stop apache2
#systemctl stop mysql
sleep 1
NOTE: This script is custom-tailored to each server it is placed on to start the services that are unique to it. Although the services are likely to auto-start with the server, this is mainly used if only restarting the services and not the entire server.
/var/scripts/prod/servicestart.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : servicestart.sh
## Version       : 1.2
## Date          : 2022-05-31
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : None
## Purpose       : Start primary services.
## Run Frequency : As needed
## Exit Codes    : None
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2018-04-19 1.0 LTH Created script.
## 2019-09-24 1.1 LTH Switched "service" to "systemctl" format.
## 2022-05-31 1.2 LTH Replaced echo statements with printf.
#############################################################
## NOTE: Add whatever services you need started here.
printf "Starting services...\n"
#systemctl start mysql
#systemctl start apache2
#systemctl start nagios
#systemctl start vsftpd
sleep 1
The service restart, reboot and shutdown scripts can simply call the service stop and start scripts and should never need to be modified from default.

NOTE: This script is generic enough that it should not need to be modified when deployed to any server.
/var/scripts/prod/servicerestart.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : servicerestart.sh
## Version       : 1.1
## Date          : 2018-04-19
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : None
## Purpose       : Stop/Start primary services.
## Run Frequency : As needed
## Exit Codes    : None
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2013-01-08 1.0 LTH Created script.
## 2018-04-19 1.1 LTH Spit stop/start code into individual scripts.
#############################################################
## Import standard variables and functions. ##
source /var/scripts/common/standard.conf
clear
${ScriptDir}/prod/servicestop.sh
${ScriptDir}/prod/servicestart.sh
/var/scripts/prod/reboot.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : reboot.sh
## Version       : 1.4
## Date          : 2022-05-31
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : Run as root
## Purpose       : Stop services and reboot server.
## Run Frequency : As needed
## Exit Codes    : None
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2013-01-07 1.0 LTH Created script.
## 2017-12-18 1.1 LTH Added logging.
## 2018-04-19 1.2 LTH Various minor changes.
## 2020-09-02 1.3 LTH Added broadcast notice to all connected SSH users.
## 2022-05-31 1.4 LTH Replaced echo statements with printf.
#############################################################

## Import standard variables and functions. ##
source /var/scripts/common/standard.conf

## Define local variables.
LogFile="${LogDir}/${Company}-reboot.log"

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  printf "\nERROR: Root user required to run this script.\n"
  printf "Type 'sudo su' to temporarily become root user.\n"
  exit 1
fi

clear
printf "`date +%Y-%m-%d_%H:%M:%S` - Reboot initiated.\n" | tee -a ${LogFile}
${ScriptDir}/prod/servicestop.sh
## Broadcasting message to any other users logged in via SSH.
printf "WARNING: Rebooting server. Should be back online in 20 seconds.\n" | wall
printf "Rebooting...\n"
printf "3\n"
sleep 1
printf "2\n"
sleep 1
printf "1\n"
sleep 1
shutdown -r now
/var/scripts/prod/shutdown.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : shutdown.sh
## Version       : 1.1
## Date          : 2022-05-31
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : Run as root
## Purpose       : Stop services and power off server.
## Run Frequency : As needed
## Exit Codes    : None
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2018-04-19 1.0 LTH Created script.
## 2022-05-31 1.1 LTH Replaced echo statements with printf.
#############################################################

## Import standard variables and functions. ##
source /var/scripts/common/standard.conf

## Define local variables.
LogFile="${LogDir}/${Company}-shutdown.log"

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  printf "\nERROR: Root user required to run this script.\n"
  printf "Type 'sudo su' to temporarily become root user.\n"
  exit 1
fi

clear
printf "`date +%Y-%m-%d_%H:%M:%S` - Shutdown initiated.\n" | tee -a ${LogFile}
${ScriptDir}/prod/servicestop.sh
printf "Shutting down...\n"
printf "3\n"
sleep 1
printf "2\n"
sleep 1
printf "1\n"
sleep 1
shutdown -P now
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Scripts - APT Upgrade

Post: # 1012Post LHammonds »

APT Upgrade

This script that can be scheduled to run daily to check for OS/software updates in the repositories and then install them if available.

NOTE: This script will assume the answer is yes to any question the package manager asks with exception to modified configuration files. If you modify a configuration file, the package manager will want to overwrite your changes with the packages template. You typically do NOT want to do this because you will lose your customized changes. There is at least one scenario you need to be careful about...major changes in the config file. If breaking changes were made in the latest version, you will have to rename your existing config file, find/copy the package's template config file and manually merge your custom changes into the new config file.

The following is an example of a crontab entry to schedule the script to run once per day @ 3am.

/var/scripts/data/crontab.root

Code: Select all

0 3 * * * /var/scripts/prod/apt-upgrade.sh > /dev/null 2>&1
/var/scripts/prod/apt-upgrade.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : apt-upgrade.sh
## Version       : 1.5
## Date          : 2022-05-31
## Author        : LHammonds
## Purpose       : Keep system updated (rather than use unattended-upgrades)
## Compatibility : Verified on Ubuntu Server 18.04 thru 22.04 LTS
## Requirements  : Sendemail, run as root
## Run Frequency : Recommend once per day.
## Parameters    : None
## Exit Codes    :
##    0 = Success
##    1 = ERROR: Lock file detected.
##    2 = ERROR: Not run as root user.
##    4 = ERROR: APT update Error.
##    8 = ERROR: APT upgrade Error.
##   16 = ERROR: APT autoremove Error.
##   32 = ERROR: APT autoclean Error.
##   64 = ERROR: APT clean Error.
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2012-06-01 1.0 LTH Created script.
## 2013-01-08 1.1 LTH Allow visible status output if run manually.
## 2013-03-11 1.2 LTH Added company prefix to log files.
## 2017-03-16 1.3 LTH Made compatible with 16.04 LTS.
## 2019-09-23 1.4 LTH Set to not overwrite existing config file.
## 2022-05-31 1.5 LTH Changed echo statements to printf.
#############################################################

## Import standard variables and functions. ##
source /var/scripts/common/standard.conf

## Define local variables.
LogFile="${LogDir}/${Company}-apt-upgrade.log"
LockFile="${TempDir}/${Company}-apt-upgrade.lock"
ErrorFlag=0
ErrorMsg=""
ReturnCode=0
AptCmd="$(which apt)"
AptGetCmd="$(which apt-get)"

#######################################
##            FUNCTIONS              ##
#######################################
function f_cleanup()
{
  if [ -f ${LockFile} ]; then
    ## Remove lock file so subsequent jobs can run.
    rm ${LockFile} 1>/dev/null 2>&1
  fi
  ## Temporarily pause script in case user is watching output.
  sleep 2
  if [ ${ErrorFlag} -gt 0 ]; then
    ## Display error message to user in case being run manually.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: ${ErrorMsg}\n" | tee -a ${LogFile}
    ## Email error notice.
    f_sendmail "ERROR ${ErrorFlag}: Script aborted" "${ErrorMsg}"
  fi
  exit ${ErrorFlag}
}

#######################################
##           MAIN PROGRAM            ##
#######################################
clear
if [ -f ${LockFile} ]; then
  # Lock file detected.  Abort script.
  printf "** Script aborted **\n"
  printf "This script tried to run but detected the lock file: ${LockFile}\n"
  printf "Please check to make sure the file does not remain when check space is not actually running.\n"
  ErrorMsg="This script tried to run but detected the lock file: ${LockFile}\n\nPlease check to make sure the file does not remain when check space is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LockFile}'"
  ErrorFlag=1
  f_cleanup
else
  printf "`date +%Y-%m-%d_%H:%M:%S` ${ScriptName}\n" > ${LockFile}
fi
## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  printf "ERROR: Root user required to run this script.\n"
  ErrorMsg="Root user required to run this script."
  ErrorFlag=2
  f_cleanup
fi

## Make sure the cleanup function is called from this point forward.
trap f_cleanup EXIT

printf "`date +%Y-%m-%d_%H:%M:%S` - Begin script.\n" | tee -a ${LogFile}
printf "`date +%Y-%m-%d_%H:%M:%S` --- Apt-Get Update\n" | tee -a ${LogFile}
${AptGetCmd} update > /dev/null 2>&1
ReturnCode=$?
if [[ "${ReturnCode}" -gt 0 ]]; then
  ErrorMsg="Apt-Get Update return code of ${ReturnCode}"
  ErrorFlag=4
  f_cleanup
fi
printf "`date +%Y-%m-%d_%H:%M:%S` --- Apt-Get Upgrade\n" | tee -a ${LogFile}
printf "==================================================\n" >> ${LogFile}
${AptGetCmd} --assume-yes --option Dpkg::Options::="--force-confdef" --option Dpkg::Options::="--force-confold" upgrade >> ${LogFile} 2>&1
ReturnCode=$?
if [[ "${ReturnCode}" -gt 0 ]]; then
  ErrorMsg="Apt-Get Upgrade return code of ${ReturnCode}"
  ErrorFlag=8
  f_cleanup
fi
printf "`date +%Y-%m-%d_%H:%M:%S` --- Apt-Get Dist-Upgrade\n" | tee -a ${LogFile}
printf "==================================================\n" >> ${LogFile}
${AptGetCmd} --assume-yes --option Dpkg::Options::="--force-confdef" --option Dpkg::Options::="--force-confold" dist-upgrade >> ${LogFile} 2>&1
ReturnCode=$?
if [[ "${ReturnCode}" -gt 0 ]]; then
  ErrorMsg="Apt-Get Dist-Upgrade return code of ${ReturnCode}"
  ErrorFlag=8
  f_cleanup
fi
printf "==================================================\n" >> ${LogFile}
printf "`date +%Y-%m-%d_%H:%M:%S` --- Apt-Get Autoremove\n" | tee -a ${LogFile}
printf "==================================================\n" >> ${LogFile}
${AptGetCmd} --assume-yes autoremove >> ${LogFile} 2>&1
ReturnCode=$?
if [[ "${ReturnCode}" -gt 0 ]]; then
  ErrorMsg="Apt-Get Autoremove return code of ${ReturnCode}"
  ErrorFlag=16
  f_cleanup
fi
printf "==================================================\n" >> ${LogFile}
printf "`date +%Y-%m-%d_%H:%M:%S` --- Apt-get Autoclean\n" | tee -a ${LogFile}
printf "==================================================\n" >> ${LogFile}
${AptGetCmd} autoclean >> ${LogFile} 2>&1
ReturnCode=$?
if [[ "${ReturnCode}" -gt 0 ]]; then
  ErrorMsg="Apt-Get Autoclean return code of ${ReturnCode}"
  ErrorFlag=32
  f_cleanup
fi
printf "==================================================\n" >> ${LogFile}
printf "`date +%Y-%m-%d_%H:%M:%S` --- Apt-get Clean\n" | tee -a ${LogFile}
printf "==================================================\n" >> ${LogFile}
${AptGetCmd} clean >> ${LogFile} 2>&1
ReturnCode=$?
if [[ "${ReturnCode}" -gt 0 ]]; then
  ErrorMsg="Apt-Get Clean return code of ${ReturnCode}"
  ErrorFlag=64
  f_cleanup
fi
printf "==================================================\n" >> ${LogFile}
printf "`date +%Y-%m-%d_%H:%M:%S` - End script.\n" | tee -a ${LogFile}

## Perform cleanup routine.
f_cleanup
Here is the typical output:

/var/log/abc-apt-upgrade.log

Code: Select all

2019-09-23_17:29:15 - Begin script.
2019-09-23_17:29:15 --- Apt-Get Update
2019-09-23_17:29:18 --- Apt-Get Upgrade
--------------------------------------------------
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages have been kept back:
  linux-generic linux-headers-generic linux-image-generic
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
2019-09-23_17:29:19 --- Apt-Get Dist-Upgrade
--------------------------------------------------
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following NEW packages will be installed:
  linux-headers-4.15.0-64 linux-headers-4.15.0-64-generic
  linux-image-4.15.0-64-generic linux-modules-4.15.0-64-generic
  linux-modules-extra-4.15.0-64-generic
The following packages will be upgraded:
  linux-generic linux-headers-generic linux-image-generic
3 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 66.2 MB of archives.
After this operation, 332 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-64-generic amd64 4.15.0-64.73 [13.1 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-64-generic amd64 4.15.0-64.73 [7,965 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-64-generic amd64 4.15.0-64.73 [32.9 MB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.64.66 [1,868 B]
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.64.66 [2,392 B]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-64 all 4.15.0-64.73 [11.1 MB]
Get:7 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-64-generic amd64 4.15.0-64.73 [1,231 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.64.66 [2,352 B]
Fetched 66.2 MB in 5s (13.1 MB/s)
Selecting previously unselected package linux-modules-4.15.0-64-generic.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 111865 files and directories currently installed.)
Preparing to unpack .../0-linux-modules-4.15.0-64-generic_4.15.0-64.73_amd64.deb ...
Unpacking linux-modules-4.15.0-64-generic (4.15.0-64.73) ...
Selecting previously unselected package linux-image-4.15.0-64-generic.
Preparing to unpack .../1-linux-image-4.15.0-64-generic_4.15.0-64.73_amd64.deb ...
Unpacking linux-image-4.15.0-64-generic (4.15.0-64.73) ...
Selecting previously unselected package linux-modules-extra-4.15.0-64-generic.
Preparing to unpack .../2-linux-modules-extra-4.15.0-64-generic_4.15.0-64.73_amd64.deb ...
Unpacking linux-modules-extra-4.15.0-64-generic (4.15.0-64.73) ...
Preparing to unpack .../3-linux-generic_4.15.0.64.66_amd64.deb ...
Unpacking linux-generic (4.15.0.64.66) over (4.15.0.60.62) ...
Preparing to unpack .../4-linux-image-generic_4.15.0.64.66_amd64.deb ...
Unpacking linux-image-generic (4.15.0.64.66) over (4.15.0.60.62) ...
Selecting previously unselected package linux-headers-4.15.0-64.
Preparing to unpack .../5-linux-headers-4.15.0-64_4.15.0-64.73_all.deb ...
Unpacking linux-headers-4.15.0-64 (4.15.0-64.73) ...
Selecting previously unselected package linux-headers-4.15.0-64-generic.
Preparing to unpack .../6-linux-headers-4.15.0-64-generic_4.15.0-64.73_amd64.deb ...
Unpacking linux-headers-4.15.0-64-generic (4.15.0-64.73) ...
Preparing to unpack .../7-linux-headers-generic_4.15.0.64.66_amd64.deb ...
Unpacking linux-headers-generic (4.15.0.64.66) over (4.15.0.60.62) ...
Setting up linux-modules-4.15.0-64-generic (4.15.0-64.73) ...
Setting up linux-headers-4.15.0-64 (4.15.0-64.73) ...
Setting up linux-image-4.15.0-64-generic (4.15.0-64.73) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-4.15.0-60-generic
I: /initrd.img.old is now a symlink to boot/initrd.img-4.15.0-60-generic
I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-64-generic
I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-64-generic
Setting up linux-headers-4.15.0-64-generic (4.15.0-64.73) ...
Setting up linux-headers-generic (4.15.0.64.66) ...
Setting up linux-modules-extra-4.15.0-64-generic (4.15.0-64.73) ...
Setting up linux-image-generic (4.15.0.64.66) ...
Setting up linux-generic (4.15.0.64.66) ...
Processing triggers for linux-image-4.15.0-64-generic (4.15.0-64.73) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-64-generic
/etc/kernel/postinst.d/x-grub-legacy-ec2:
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /vmlinuz-4.15.0-60-generic
Found kernel: /vmlinuz-4.15.0-58-generic
Found kernel: /vmlinuz-4.15.0-64-generic
Found kernel: /vmlinuz-4.15.0-60-generic
Found kernel: /vmlinuz-4.15.0-58-generic
Replacing config file /run/grub/menu.lst with new version
Updating /boot/grub/menu.lst ... done

/etc/kernel/postinst.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.0-64-generic
Found initrd image: /boot/initrd.img-4.15.0-64-generic
Found linux image: /boot/vmlinuz-4.15.0-60-generic
Found initrd image: /boot/initrd.img-4.15.0-60-generic
Found linux image: /boot/vmlinuz-4.15.0-58-generic
Found initrd image: /boot/initrd.img-4.15.0-58-generic
done
--------------------------------------------------
2019-09-23_17:30:31 --- Apt-Get Autoremove
--------------------------------------------------
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
  linux-headers-4.15.0-58 linux-headers-4.15.0-58-generic
  linux-image-4.15.0-58-generic linux-modules-4.15.0-58-generic
  linux-modules-extra-4.15.0-58-generic
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 333 MB disk space will be freed.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 147371 files and directories currently installed.)
Removing linux-headers-4.15.0-58-generic (4.15.0-58.64) ...
Removing linux-headers-4.15.0-58 (4.15.0-58.64) ...
Removing linux-modules-extra-4.15.0-58-generic (4.15.0-58.64) ...
Removing linux-image-4.15.0-58-generic (4.15.0-58.64) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-4.15.0-58-generic
/etc/kernel/postrm.d/x-grub-legacy-ec2:
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /vmlinuz-4.15.0-64-generic
Found kernel: /vmlinuz-4.15.0-60-generic
Found kernel: /vmlinuz-4.15.0-58-generic
Replacing config file /run/grub/menu.lst with new version
Found kernel: /vmlinuz-4.15.0-64-generic
Found kernel: /vmlinuz-4.15.0-60-generic
Replacing config file /run/grub/menu.lst with new version
Updating /boot/grub/menu.lst ... done

/etc/kernel/postrm.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.0-64-generic
Found initrd image: /boot/initrd.img-4.15.0-64-generic
Found linux image: /boot/vmlinuz-4.15.0-60-generic
Found initrd image: /boot/initrd.img-4.15.0-60-generic
done
Removing linux-modules-4.15.0-58-generic (4.15.0-58.64) ...
--------------------------------------------------
2019-09-23_17:30:48 --- Apt-get Autoclean
--------------------------------------------------
Reading package lists...
Building dependency tree...
Reading state information...
--------------------------------------------------
2019-09-23_17:30:48 --- Apt-get Clean
--------------------------------------------------
--------------------------------------------------
2019-09-23_17:30:48 - End script.
Reboot Check

You can schedule the server to run your custom reboot script right after an upgrade to see if it needs to be rebooted automatically. Use at your own discretion since many admins do not like having a server reboot without them being present just to make extra sure the service comes back up without any issues or at least be ready to handle them immediately.

You can run this reboot check every time the upgrade script but it will only initiate the reboot sequence if the upgrade placed a notification file saying it needs a reboot to complete the upgrade.

The following is an example of a crontab entry to schedule the script to run shortly after the upgrade script:

/var/scripts/data/crontab.root

Code: Select all

0 3 * * * /var/scripts/prod/apt-upgrade.sh > /dev/null 2>&1
30 3 * * * /var/scripts/prod/reboot-check.sh > /dev/null 2>&1
/var/scripts/prod/reboot-check.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : reboot-check.sh
## Version       : 1.3
## Date          : 2022-05-31
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : Run as root
## Purpose       : Stop services and reboot server but only if necessary.
## Run Frequency : Daily after update.
## Exit Codes    : None
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2017-12-13 1.0 LTH Created script.
## 2019-09-24 1.1 LTH Added email notification.
## 2019-10-01 1.2 LTH Added better event logging.
## 2022-05-31 1.3 LTH Replaced echo statement with printf.
#############################################################

## Import standard variables and functions. ##
source /var/scripts/common/standard.conf

## Define local variables.
LogFile="${LogDir}/${Company}-reboot-check.log"

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  printf "\nERROR: Root user required to run this script.\n"
  printf "Type 'sudo su' to temporarily become root user.\n"
  exit 1
fi

printf "`date +%Y-%m-%d_%H:%M:%S` Current Kernel: `/bin/uname -nr`\n" | tee -a ${LogFile}
if [ -f /var/run/reboot-required ]; then
  printf "`date +%Y-%m-%d_%H:%M:%S` - Reboot required.\n" | tee -a ${LogFile}
  cat /var/run/reboot-required.pkgs >> ${LogFile}
  f_sendmail "[INFO] ${Hostname} Reboot Notice" "${Hostname} rebooted at `date +%Y-%m-%d_%H:%M:%S`"
  ${ScriptDir}/prod/reboot.sh
else
  printf "`date +%Y-%m-%d_%H:%M:%S` - No reboot required.\n" | tee -a ${LogFile}
fi
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Scripts - Check Storage

Post: # 1013Post LHammonds »

Check Storage Space

In favor of managing by exception, this script that can be scheduled to run daily to check the file systems to see if they are getting close to filling up and will automatically expand them a little bit and give you an email notice. Everything is done at the megabyte level. If you do not want the script to perform the increase, simply add a pound sign in front of the resize2fs command on line 62 to comment it out. Might also want to modify the log and email messages so it does not look like it actually "performed" the resize but instead is telling YOU how to perform the resize.

Here are the lines added to the root crontab schedule which will check each file system.

The script check the specified file system and see if the amount of space free is less than the threshold (e.g. 100 MB). If the file system has free space that is less than the threshold, it will attempt to add the specified amount (e.g. 50 MB).

crontab

Code: Select all

0 1 * * * /var/scripts/prod/check-storage.sh root 500 100 > /dev/null 2>&1
5 1 * * * /var/scripts/prod/check-storage.sh home 100 50 > /dev/null 2>&1
10 1 * * * /var/scripts/prod/check-storage.sh srv 100 50 > /dev/null 2>&1
15 1 * * * /var/scripts/prod/check-storage.sh usr 100 50 > /dev/null 2>&1
20 1 * * * /var/scripts/prod/check-storage.sh var 100 50 > /dev/null 2>&1
25 1 * * * /var/scripts/prod/check-storage.sh tmp 100 50 > /dev/null 2>&1
30 1 * * * /var/scripts/prod/check-storage.sh opt 100 50 > /dev/null 2>&1
35 1 * * * /var/scripts/prod/check-storage.sh bak 500 500 > /dev/null 2>&1
/var/scripts/prod/check-storage.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : check-storage.sh
## Version       : 1.3
## Date          : 2022-05-31
## Author        : LHammonds
## Purpose       : Check available space for a file system and
##                 expand if necessary.
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : None
## Run Frequency : Recommend once per day for each FS to monitor.
## Parameters    :
##    1 = (Required) File System name (e.g. var)
##    2 = (Required) File System Threshold in MB (e.g. 50)
##    3 = (Required) Amount to increase File System in MB (e.g. 50)
## Exit Codes    :
##    0 = Success (either nothing was done or FS expanded without error)
##    1 = ERROR: Missing or incorrect parameter(s)
##    2 = ERROR: Invalid parameter value(s)
##    4 = ERROR: Lock file detected
##    8 = ERROR: Resize2fs error
##   16 = SEVERE: No room to expand
##   32 = ERROR: Script not run by root user
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2012-05-11 1.0 LTH Created script.
## 2013-03-11 1.1 LTH Added company prefix to logs.
## 2017-03-17 1.2 LTH Adjusted variable casing.
## 2022-05-31 1.3 LTH Replaced echo statement with printf.
#############################################################

## Import standard variables and functions. ##
source /var/scripts/common/standard.conf

## Define local variables.
LogFile="${LogDir}/${Company}-check-storage.log"
LockFile="${TempDir}/${Company}-check-storage.lock"
ErrorFlag=0
ReturnCode=0

#######################################
##            FUNCTIONS              ##
#######################################

function f_cleanup()
{
  if [ -f ${LockFile} ];then
    ## Remove lock file so other check space jobs can run.
    rm ${LockFile} 1>/dev/null 2>&1
  fi
  exit ${ErrorFlag}
}

function f_showhelp()
{
  echo -e "\nUsage : ${ScriptName} FileSystemName ThresholdSizeInMB AmountToIncreaseByInMB\n"
  echo -e "\nExample: ${ScriptName} var 50 50\n"
}

function f_auto-increment()
{
  let RoomInLV=${LVSize}-${FSSize}
  if [[ ${RoomInLV} -gt ${FSIncreaseBy} ]]; then
    ## There is room in the LV to increase space to the FS.
    resize2fs ${FSVol} ${NewFSSize}M
    ReturnCode=$?
    echo "`date +%Y-%m-%d_%H:%M:%S` --- resize2fs ${FSVol} ${NewFSSize}M, ReturnCode=${ReturnCode}" | tee -a ${LogFile}
    if [[ ${ReturnCode} -ne 0 ]]; then
      ## There was an error in resize2fs.
      return ${ReturnCode}
    fi
  else
    ## There is not enough room in the LV to increase space in the FS.
    return 50
  fi
  return 0
}

#######################################
##           MAIN PROGRAM            ##
#######################################

if [ -f ${LockFile} ]; then
  # Lock file detected.  Abort script.
  echo "Check space script aborted"
  echo "This script tried to run but detected the lock file: ${LockFile}"
  echo "Please check to make sure the file does not remain when check space is not actually running."
  f_sendmail "ERROR: check storage script aborted" "This script tried to run but detected the lock file: ${LockFile}\n\nPlease check to make sure the file does not remain when check space is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LockFile}'"
  ErrorFlag=4
  f_cleanup
else
  echo "`date +%Y-%m-%d_%H:%M:%S` ${ScriptName}" > ${LockFile}
fi

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  echo "ERROR: Root user required to run this script."
  echo ""
  ErrorFlag=32
  f_cleanup
fi

## Check existance of required command-line parameters.
case "$1" in
  "")
    f_showhelp
    ErrorFlag=1
    f_cleanup
    ;;
  --help|-h|-?)
    f_showhelp
    ErrorFlag=1
    f_cleanup
    ;;
  *)
    FSName=$1
    ;;
esac
case "$2" in
  "")
    f_showhelp
    ErrorFlag=1
    f_cleanup
    ;;
  --help|-h|-?)
    f_showhelp
    ErrorFlag=1
    f_cleanup
    ;;
  *)
    FSThreshold=$2
    ;;
esac
case "$3" in
  "")
    f_showhelp
    ErrorFlag=1
    f_cleanup
    ;;
  --help|-h|-?)
    f_showhelp
    ErrorFlag=1
    f_cleanup
    ;;
  *)
    FSIncreaseBy=$3
    ;;
esac

## Check validity of File System name.
case "${FSName}" in
  "root")
    FSVol="/dev/LVG/root"
    FSMap="/dev/mapper/LVG-root"
    ;;
  "home")
    FSVol="/dev/LVG/home"
    FSMap="/dev/mapper/LVG-home"
    ;;
  "tmp")
    FSVol="/dev/LVG/tmp"
    FSMap="/dev/mapper/LVG-tmp"
    ;;
  "opt")
    FSVol="/dev/LVG/opt"
    FSMap="/dev/mapper/LVG-opt"
    ;;
  "bak")
    FSVol="/dev/LVG/bak"
    FSMap="/dev/mapper/LVG-bak"
    ;;
  "usr")
    FSVol="/dev/LVG/usr"
    FSMap="/dev/mapper/LVG-usr"
    ;;
  "var")
    FSVol="/dev/LVG/var"
    FSMap="/dev/mapper/LVG-var"
    ;;
  "srv")
    FSVol="/dev/LVG/srv"
    FSMap="/dev/mapper/LVG-srv"
    ;;
  *)
    echo "ERROR: ${FSName} does not match a known file system defined in this script."
    f_showhelp
    ErrorFlag=2
    f_cleanup
    ;;
esac

## Check validity of threshold value.
test ${FSThreshold} -eq 0 1>/dev/null 2>&1
if [[ $? -eq 2 ]]; then
  ## Threshold parameter is not an integer.
  echo "ERROR: ${FSThreshold} is not an integer."
  f_showhelp
  ErrorFlag=2
  f_cleanup
fi

## Check validity of increment value.
test ${FSIncreaseBy} -eq 0 1>/dev/null 2>&1
if [[ $? -eq 2 ]]; then
  ## FSIncreaseBy parameter is not an integer.
  echo "ERROR: ${FSIncreaseBy} is not an integer."
  f_showhelp
  ErrorFlag=2
  f_cleanup
fi

## Get available space for the file system.
FSAvailable="`df --block-size=m ${FSMap} | awk '{ print $4 }' | tail -n 1 | sed 's/M//'`"

## Get the current size of the File System.
FSSize="`df --block-size=m ${FSMap} | awk '{ print $2 }' | tail -n 1 | sed 's/M//'`"

## Get the current size of the Logical Volume for the File System
LVSize="`lvs --noheadings --nosuffix --units=m ${FSMap} | awk '{ print $4}' | sed 's/[.].*//'`"

## Calculate the new size of the FS in case we need it.
let NewFSSize=${FSSize}+${FSIncreaseBy}

if [[ ${FSAvailable} -lt ${FSThreshold} ]]; then
  echo "`date +%Y-%m-%d_%H:%M:%S` - Starting expansion of ${FSVol}" | tee -a ${LogFile}
  echo "`date +%Y-%m-%d_%H:%M:%S` --- LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvail=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB" | tee -a ${LogFile}
  ## Run the auto-expansion function.
  f_auto-increment
  ReturnCode=$?
  case ${ReturnCode} in
  0)
    f_sendmail "NOTICE: File System Expanded" "${FSVol} was expanded because it was nearing max capacity.  Please review disk space usage and plan appropriately. LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB"
    ;;
  50)
    echo "`date +%Y-%m-%d_%H:%M:%S` - SEVERE: No room to expand ${FSVol}" | tee -a ${LogFile}
    ErrorFlag=16
    f_sendmail "SEVERE: No room to expand ${FSVol}" "There is not enough room in the Logical Volume to expand the ${FSVol} File System.  Immediate action is required.  Make sure there is free space in the Volume Group 'LVG' and then expand the Logical Volume...then expand the File System.\n\nLVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB.\n\nType 'vgs' to see if there is any free space in the Volume Group which can be given to the Logical Volume.\n\nType 'lvs' to see the current sizes of the LVs.\n\nType 'lvdisplay' to see a list of Logical Volumes so you can get the LV Name which is used in the lvextend and resize2fs commands.\n\nType 'lvextend -L+50M /dev/LVG/var' if you want to extend the var Logical Volume by 50 megabytes (assuming there is 50MB available in the Volume Group).\n\nType 'df --block-size=m' to see a list of file systems and their associated size and available space.\n\nType 'resize2fs /dev/LVG/var ${NewFSSize}M' to set the size of var to ${NewFSSize} megabytes. Make sure you set the size to the desired end-result which should be LARGER than the current FS size so you do not lose data."
    ;;
  *)
    echo "`date +%Y-%m-%d_%H:%M:%S` - ERROR: Expansion failure for ${FSVol}" | tee -a ${LogFile}
    ErrorFlag=8
    f_sendmail "ERROR: File System Expansion Failed" "${FSVol} Expansion failed with return code of ${ReturnCode}.  LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB"
    ;;
  esac
  echo "`date +%Y-%m-%d_%H:%M:%S` - Finished expansion of ${FSVol}" | tee -a ${LogFile}
else
  echo "`date +%Y-%m-%d_%H:%M:%S` - ${FSVol} ${FSAvailable}M>${FSThreshold}M No action required." | tee -a ${LogFile}
fi

## Perform cleanup routine.
f_cleanup
Here is the typical output when it does not have to increase the FS:

/var/log/abc-check-storage.log

Code: Select all

2022-05-31_01:00:00 - /dev/LVG/root 3653M>500M No action required.
2022-05-31_01:05:00 - /dev/LVG/home 452M>100M No action required.
2022-05-31_01:10:00 - /dev/LVG/srv 452M>100M No action required.
2022-05-31_01:15:00 - /dev/LVG/usr 1472M>100M No action required.
2022-05-31_01:20:00 - /dev/LVG/var 1284M>100M No action required.
2022-05-31_01:25:00 - /dev/LVG/tmp 1828M>100M No action required.
2022-05-31_01:30:00 - /dev/LVG/opt 452M>100M No action required.
2022-05-31_01:35:00 - /dev/LVG/bak 1623M>500M No action required.
Here is a sample of what the log will look like when it performs increases:

/var/log/abc-check-storage.log

Code: Select all

2022-05-03_09:40:10 - Starting expansion of /dev/LVG/var
2022-05-31_09:40:10 --- LVSize=3072MB, FSSize=1843MB, FSAvail=507MB, FSThreshold=600MB, FSIncreaseBy=100MB
2022-05-31_09:40:10 --- resize2fs /dev/LVG/var 1943M, ReturnCode=0
2022-05-31_09:40:11 - Finished expansion of /dev/LVG/var
2022-05-31_09:40:37 - Starting expansion of /dev/LVG/tmp
2022-05-31_09:40:37 --- LVSize=2048MB, FSSize=979MB, FSAvail=917MB, FSThreshold=2000MB, FSIncreaseBy=100MB
2022-05-31_09:40:37 --- resize2fs /dev/LVG/tmp 1079M, ReturnCode=0
2022-05-31_09:40:37 - Finished expansion of /dev/LVG/tmp
2022-05-31_09:40:56 - Starting expansion of /dev/LVG/tmp
2022-05-31_09:40:56 --- LVSize=2048MB, FSSize=1030MB, FSAvail=966MB, FSThreshold=2000MB, FSIncreaseBy=100MB
2022-05-31_09:40:56 --- resize2fs /dev/LVG/tmp 1130M, ReturnCode=0
2022-05-31_09:40:56 - Finished expansion of /dev/LVG/tmp
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Scripts - Operator Menu

Post: # 1014Post LHammonds »

Operator Menu

To make my servers easier to administer within my group (who are not all *NIX guys), I made an operator script that can be used to manipulate the server.

The main features I wanted to make easier for other members of the IT group are:

1. Update the operating system (manually)
2. Disk Status
3. Memory Status
4. Stop/Restart primary services
5. Cleanly shutdown services and reboot server
6. Cleanly shutdown services and power off server

I accomplished this by creating a script and making a shortcut to it so the only thing they need to type on each server is "opm" which is short for "Operator Menu"

Here is how to do something similar for your servers.

Create the script file and set the permissions so that only the root user can run the script and make a shortcut for it:

Code: Select all

touch /var/scripts/prod/opm.sh
chmod 0700 /var/scripts/prod/opm.sh
ln -s /var/scripts/prod/opm.sh /usr/sbin/opm
Now open the script in your favorite editor and add the following:
/var/scripts/prod/opm.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : opm.sh
## Version       : 1.3
## Date          : 2022-05-31
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : dialog (apt-get dialog) and root privileges
## Purpose       : Display menu to control the server
## Run Frequency : As needed
## Exit Codes    : None
## SymLink Cmd   : ln -s /var/scripts/prod/opm.sh /usr/sbin/opm
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2013-01-07 1.0 LTH Created script.
## 2017-03-17 1.1 LTH Updated variable standards.
## 2018-04-19 1.2 LTH Various minor changes.
## 2022-05-31 1.3 LTH Replaced echo statement with printf.
#############################################################

## Store menu options selected by the user.
TempDir="/tmp"
ScriptDir="/var/scripts/prod"
InputFile="${TempDir}/opm-input.$$"

## Storage file for displaying cal and date command output.
OutputFile="${TempDir}/opm-output.$$"

## Get text editor or fall back to vi_editor.
vi_editor=${EDITOR-vi}

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  printf "\nERROR: Root user required to run this script.\n"
  printf "Type 'sudo su' to temporarily become root user.\n"
  exit 1
fi

## Trap and delete temp files.
trap "rm $OutputFile; rm $InputFile; exit" SIGHUP SIGINT SIGTERM

function f_display_output(){
  ## Purpose - display output using msgbox
  ##  $1 -> set msgbox height
  ##  $2 -> set msgbox width
  ##  $3 -> set msgbox title
  local h=${1-10}     ## box height default 10
  local w=${2-41}     ## box width default 41
  local t=${3-Output} ## box title
  dialog --backtitle "Operator Menu for $(hostname -f)" --title "${t}" --clear --msgbox "$(<$OutputFile)" ${h} ${w}
} ## f_display_output

function f_showdate(){
  ## Purpose - display current system date & time
  printf "Today is $(date) @ $(hostname -f).\n" >$OutputFile
  f_display_output 6 60 "Date and Time"
} ## f_showdate

function f_checkdisk(){
  ## Purpose: Display disk status.
  clear
  printf "df --block-size=M\n"
  df --block-size=M
  printf "\n"
  read -p "Press [Enter] key to continue..."
} ## f_checkdisk

## Loop the menu display.
while true
do
  ## Display main menu.
  dialog --clear  --no-cancel --backtitle "Operator Menu for $(hostname -f)" \
  --title "[ M A I N - M E N U ]" \
  --menu "You can use the UP/DOWN arrow keys, the first \n\
  letter of the choice as a hot key, or the \n\
  number keys 1-9 to choose an option.\n\
  Choose the TASK" 19 50 7 \
  Exit "Exit menu" \
  OSUpdate "Update Operating System" \
  CheckDisk "Check Disk Status" \
  MEMCheck "Look at running processes" \
  ServiceRestart "Stop/Start Main Services" \
  RebootServer "Cleanly reboot server" \
  PoweroffServer "Cleanly Power-off server" \
  Date/time "Displays date and time" 2>"${InputFile}"

  menuitem=$(<"${InputFile}")

  ## Make decision.
  case $menuitem in
    OSUpdate) ${ScriptDir}/apt-upgrade.sh;;
    CheckDisk) f_checkdisk;;
    MEMCheck) htop;;
    ServiceRestart) ${ScriptDir}/servicerestart.sh;;
    RebootServer) ${ScriptDir}/reboot.sh;;
    PoweroffServer) ${ScriptDir}/shutdown.sh;;
    Date/time) f_showdate;;
    Exit) clear; printf "Clean menu exit.\n"; break;;
  esac
done

## Delete temp files.
[ -f $OutputFile ] && rm $OutputFile
[ -f $InputFile ] && rm $InputFile

Customizing Operator Menu

Let's say you have "mytop" installed on your MySQL/MariaDB server and you want that added to your menu. This is how you would do that.

Edit the opm.sh file and find this section:

Code: Select all

  OSUpdate "Update Operating System" \
  CheckDisk "Check Disk Status" \
  MEMCheck "Look at running processes" \
  ServiceRestart "Stop/Start Main Services" \
  Reboot "Cleanly reboot server" \
  Date/time "Displays date and time" 2>"${Input}"
Now add the following line so it shows up in your menu:

Code: Select all

  MySQLCheck "Look at MySQL processes" \
End result looking like this:

Code: Select all

  OSUpdate "Update Operating System" \
  CheckDisk "Check Disk Status" \
  MEMCheck "Look at running processes" \
  MySQLCheck "Look at MySQL processes" \
  ServiceRestart "Stop/Start Main Services" \
  Reboot "Cleanly reboot server" \
  Date/time "Displays date and time" 2>"${Input}"
Now that the menu entry is there, we need the menu item to actually do something so find this section below:

Code: Select all

    OSUpdate) ${ScriptDir}/abc-apt-upgrade.sh;;
    CheckDisk) f_checkdisk;;
    MEMCheck) htop;;
    Reboot) ${ScriptDir}/reboot.sh;;
    ServiceRestart) ${ScriptDir}/servicerestart.sh;;
    Date/time) f_showdate;;
    Exit) clear; echo "Clean menu exit."; break;;
Add the following line:

Code: Select all

    MySQLCheck) mytop;;
The end result looking like this:

Code: Select all

    OSUpdate) ${ScriptDir}/abc-apt-upgrade.sh;;
    CheckDisk) f_checkdisk;;
    MEMCheck) htop;;
    MySQLCheck) mytop;;
    Reboot) ${ScriptDir}/reboot.sh;;
    ServiceRestart) ${ScriptDir}/servicerestart.sh;;
    Date/time) f_showdate;;
    Exit) clear; echo "Clean menu exit."; break;;
Save and close the script.

Now when you run opm, you will see the MySQL/MariaDB entry which will run mytop when you select it.

With this menu example, I have given you 3 example types of how you can do something. You can call the program directly, like mytop or htop, or you can call a function to process several lines of script like f_checkdisk or f_showdate or you can call an external script like reboot.sh or servicerestart.sh which can further isolate individual server needs in other scripts.
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Security - Firewall

Post: # 1015Post LHammonds »

Firewall

Linux uses IPTables as its firewall but learning the ins and outs of the syntax can be daunting. Luckily for us, Ubuntu has a front-end for it called Uncomplicated Firewall (UFW for short).

If you are not using IPv6 (most networks are still just using IPv4), you can comment out the IPv6 setting so you don't get doubles of each rule (one for IPv4 and one for IPv6).

To disable IPv6 rules:

Code: Select all

sudo vi /etc/default/ufw
Find:

Code: Select all

IPV6=yes
Change to:

Code: Select all

#IPV6=yes
Here is a script I use to configure the firewall rules and turn it on...which will remain on even during a reboot.

You can copy this script and use it for your own purposes and tweak it for your environment. Enough commands are used/documented in it that you should be able to modify it to fit your particular server. For example, if running a Minecraft server, you could add a command to allow TCP port 25565. If you are running a web server, you could remove the comment from the TCP ports 80, 8080 and 443.

I NEVER simply copy this script and run it. Each and every server requires a custom variation of this script tailored for it. The application section has examples for commonly used services such as web, database, etc. Feel free to uncomment lines you can use and modify to suit your needs.

/var/scripts/prod/en-firewall.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : en-firewall.sh
## Version       : 1.2
## Date          : 2022-05-31
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Requirements  : Run as root
## Purpose       : Restore and enable firewall.
## Run Frequency : As needed
## Exit Codes    : None
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2015-08-28 1.0 LTH Created script.
## 2017-04-13 1.1 LTH Added comments in rules.
## 2022-05-31 1.2 LTH Replaced echo statements with printf.
#############################################################

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  printf "\nERROR: Root user required to run this script.\n"
  printf "Type 'sudo su' to temporarily become root user.\n"
  exit
fi

clear
printf "\nResetting Firewall to factory default\n"
printf "y\n" | ufw reset 1>/dev/null 2>&1
ufw default deny incoming 1>/dev/null 2>&1
ufw default allow outgoing 1>/dev/null 2>&1
printf "Allowing SSH from only LAN connections\n"
ufw allow from 192.168.107.0/24 to any port 22 comment 'SSH via LAN' 1>/dev/null 2>&1
ufw allow from 192.168.108.0/24 to any port 22 comment 'SSH via LAN' 1>/dev/null 2>&1
printf "Allowing Samba file sharing connections\n"
ufw allow proto tcp to any port 135,139,445 comment 'Samba Share' 1>/dev/null 2>&1
ufw allow proto udp to any port 137,138 comment 'Samba Share' 1>/dev/null 2>&1
printf "Allowing NFS Sharing\n"
ufw allow proto tcp to any port 2049 comment 'NFS Share' 1>/dev/null 2>&1
ufw allow proto udp to any port 2049 comment 'NFS Share' 1>/dev/null 2>&1
printf "Allowing Nagios connections\n"
#ufw allow from 192.168.107.21 to any port 12489 comment 'Nagios' 1>/dev/null 2>&1
#ufw allow from 192.168.107.21 proto tcp to any port 5666 comment 'Nagios' 1>/dev/null 2>&1
printf "Adding Application-specific rules\n"
#printf "Adding MariaDB/MySQL rules\n"
#ufw allow from 192.168.107.0/24 proto tcp to any port 3306 comment 'MariaDB via LAN' 1>/dev/null 2>&1
#ufw allow from 192.168.108.0/24 proto tcp to any port 3306 comment 'MariaDB via LAN' 1>/dev/null 2>&1
#printf "Adding FTP/FTPS rules\n"
#ufw allow proto tcp to any port 990 comment 'FTPS' 1>/dev/null 2>&1
#ufw allow proto tcp to any port 21 comment 'FTP' 1>/dev/null 2>&1
#ufw allow proto tcp to any port 2000:2020 comment 'FTP Passive' 1>/dev/null 2>&1
#printf "Adding Web Server rules\n"
#ufw allow proto tcp to any port 80 comment 'HTTP Service' 1>/dev/null 2>&1
#ufw allow proto tcp to any port 8080 comment 'HTTP ALT Service' 1>/dev/null 2>&1
#ufw allow proto tcp to any port 443 comment 'HTTP Service' 1>/dev/null 2>&1
printf "Enabling firewall\n"
printf "y\n" | ufw enable 1>/dev/null 2>&1
printf "Firewall enabled and all rules have been configured.\n"
exit 0
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Security - Fail2Ban

Post: # 1016Post LHammonds »

Fail2Ban

Fail2Ban is an intrusion prevention system that can be used to protect servers from different kinds of attacks.

Fail2ban scans log files for various services, such as SSH, FTP, SMTP, Apache and block the IP address that makes too many password failures.

Regular expressions (RegEx) are used to filter the log entries. RegEx101.com is a handy site where you can copy/paste log entries and then craft regular expressions that match the lines you want to find and avoid the ones you don't.

Install Fail2Ban

Code: Select all

sudo apt install fail2ban
Configure Fail2Ban
The default configuration file should not be altered since upgrades tend to overwrite this file with new settings.

Instead, use a custom configuration file that will simply override or append various settings to the default.

Fail2Ban reads .conf files first and then .local files last which can override any settings initially set in .conf files.

First, look at the default file:

Code: Select all

vi /etc/fail2ban/jail.conf
Notice the following partial list of default values. These are based on the OS and version at the time of this writing:

Code: Select all

[DEFAULT]
#ignoreself = true
#ignoreip = 127.0.0.1/8 ::1
ignorecommand =
bantime  = 10m
findtime  = 10m
maxretry = 5
filter = %(__name__)s[mode=%(mode)s]
destemail = root@localhost
sender = root@<fq-hostname>
mta = sendmail
action = %(action_)s
[sshd]
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s
Now create a local jail file that will be used to override any settings in the default that we want changed or added.

For this example, we will ignore all failed login attempts from our local server subnet (192.168.107.xxx) and a specific admin workstation (192.168.1.69)

Code: Select all

sudo touch /etc/fail2ban/jail.local
sudo chown root:root /etc/fail2ban/jail.local
sudo chmod 644 /etc/fail2ban/jail.local
sudo vi /etc/fail2ban/jail.local
Let's make some changes that fit our server as it stands right now:

Code: Select all

[DEFAULT]
## 127.0.0.1/8 = ignore login failures on the local machine
## 192.168.107.0/24 = ignore login failures on an entire subnet
## 192.168.1.69 = ignore login failures for this specific IP
ignoreip = 127.0.0.1/8 192.168.107.0/24 192.168.1.69
bantime = 30m
## "findtime" is the length of time between login attempts before a ban is set.
findtime = 10m
## "maxretry" is how many attempts can be made to access the server from a single IP before a ban is imposed.
maxretry = 7

## "destemail" is the email address where you would like to receive the emails.
destemail = webmaster@mydomain.com
## "sender" is the FROM: address when it arrives in your mailbox.
sender = myserver@mydomain.com
## Use the lightweight sendemail instead of sendmail
mta = sendemail
## Email notify with whois report and relevant log lines when a ban occurs
action = %(action_mwl)s
We are using sendemail for our scripts so we will also use it for fail2ban. We will need to make a configuration file that tells fail2ban how to make use of it though.

Code: Select all

sudo touch /etc/fail2ban/action.d/sendemail-whois-lines.conf
sudo chown root:root /etc/fail2ban/action.d/sendemail-whois-lines.conf
sudo chmod 644 /etc/fail2ban/action.d/sendemail-whois-lines.conf
sudo vi /etc/fail2ban/action.d/sendemail-whois-lines.conf

Code: Select all

[Definition]
actionstart =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: started" -m "The jail <name> has been started successfully.\n\nFail2Ban"
actionstop =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: stopped" -m "The jail <name> has been stopped.\n\nFail2Ban"
actioncheck =
actionban =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: banned <ip>" -m "The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>.\n\nHere is more information about <ip>:\n `/usr/bin/whois <ip>`\n\n Lines containing IP:<ip> in <logpath>\n`/bin/grep '\<<ip>\>' <logpath>`\n\n\n\nFail2Ban"
actionunban =

[Init]
## Amended to be the same as the SMTP user
sender = fail2ban@mydomain.com
## SMTP password for user
#password = XXXXXXX
## SMTP server - use port 587 for Google rather than 25 (times out too often) or 465 (crashes sendemail)
#smtp = smtp.googlemail.com:587
smtp = srv-mail

## Name for this server - handy when there are lots of servers sending emails to the destemail
servername = srv-ubuntu
Useful commands

Here are a few of the most-used commands. Reference: Fail2Ban commands.
Stop, start or restart the fail2ban service:

Code: Select all

sudo systemctl stop fail2ban
sudo systemctl start fail2ban
sudo systemctl restart fail2ban
This command will show which services are being watched (sshd in this example):

Code: Select all

sudo fail2ban-client status
Example output:

Code: Select all

Status
|- Number of jail:      1
`- Jail list:   sshd
This command gets the status on the sshd service being watched:

Code: Select all

sudo fail2ban-client status sshd
Example output:

Code: Select all

Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:
Firewall Rules

Installing Fail2Ban will create the following firewall rules which can be seen with the following command:

Code: Select all

sudo iptables -S

Code: Select all

-N f2b-sshd
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A f2b-sshd -j RETURN
If you have already enabled Uncomplicated Firewall (UFW), you will see several ufw-prefixed rules as well.

Test Fail2Ban

To manually ban an IP:

Code: Select all

sudo fail2ban-client set sshd banip 192.168.1.69
To manually unban an IP:

Code: Select all

sudo fail2ban-client set sshd unbanip 192.168.1.69
On the server, run the following command to watch the Fail2Ban log file:

Code: Select all

tail -f /var/log/fail2ban.log
On a different machine (that is not on your ignoreip list), try to ssh into your server with the incorrect ID or password multiple times until you have reached the "maxretry" attempts and you get locked out...thus causing an entry in the log.

References

Fail2Ban Manual
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Security - SSH Public and Private Keys

Post: # 1017Post LHammonds »

SSH Public and Private Keys

If you have more than one Ubuntu server, you probably will want to setup a trusted SSH authentication between the two servers so you can securely and automatically transfer files between them with commands such as scp or sftp.

SSH Keys for Low-Rights Users

If you already have an SSH key generated, generating another will likely overwrite the prior one and break authentication whatever machines it was connected to. So rather than generating another, just use what you already have.

Make sure you do not already have an SSH key generated.

Code: Select all

ls -l ~/.ssh/id_*.pub
ls: cannot access '/root/.ssh/id_*.pub': No such file or directory
Generate the private/public key on each server:
  1. Connect to a server using PuTTY.
  2. At the login prompt, login with your administrator account (administrator / myadminpass)
  3. Generate a key by typing the following and accept the default values on everything (even the blank passphrase):

    Code: Select all

    mkdir ~/.ssh
    chmod 700 ~/.ssh
    ssh-keygen -t rsa -b 4096 -C "administrator"
  4. Do the same thing on all your other servers.
Transfer the public key to your trusted servers:
  1. Type the following command for each server you have (use your own server name or IP instead of my example names):
    ssh-copy-id administrator@srv-database ssh-copy-id administrator@srv-wiki ssh-copy-id administrator@srv-ftp ssh-copy-id administrator@srv-nagios
  2. Do the same thing on all your other servers.
SSH Keys for Root User

The root account is typically locked by default so you cannot login with root directly. In order to initially copy the public key using ssh-copy-id, you will need to enable the root account temporarily on all servers.

If you already have an SSH key generated, generating another will likely overwrite the prior one and break authentication whatever machines it was connected to. So rather than generating another, just use what you already have.

Make sure you do not already have an SSH key generated.

Code: Select all

ls -l ~/.ssh/id_*.pub
ls: cannot access '/root/.ssh/id_*.pub': No such file or directory
Temporarily enable login with the root account on each server:
  1. Connect to a server using PuTTY.
  2. At the login prompt, login with your administrator account (administrator / myadminpass)
  3. At the $ prompt, temporarily grant yourself super user privileges by typing sudo su {ENTER} and then provide the administrator password (myadminpass).
  4. Type the following and set the password to the same as your administrator account (myadminpass):

    Code: Select all

    passwd root
  5. Edit /etc/ssh/sshd_config, and comment out the following from:

    Code: Select all

    PermitRootLogin prohibit-password
    to:

    Code: Select all

    #PermitRootLogin prohibit-password
  6. Just below that line, add the following:

    Code: Select all

    PermitRootLogin yes
  7. You might also need to add root to the AllowUsers line:

    Code: Select all

    AllowUsers administrator root
  8. Restart SSH:

    Code: Select all

    systemctl restart ssh
  9. Do the same for each of your servers
Generate the private/public key on each server:
  1. Generate a key by typing the following and accept the default values on everything (even the blank passphrase):

    Code: Select all

    mkdir ~/.ssh
    chmod 700 ~/.ssh
    ssh-keygen -t rsa -b 4096 -C "root"
  2. Do the same thing on all your other servers.
Transfer the public key to your trusted servers:
  1. Type the following command for each server you have (use your own server name or IP instead of my example names):
    ssh-copy-id root@srv-database ssh-copy-id root@srv-wiki ssh-copy-id root@srv-ftp ssh-copy-id root@srv-nagios
  2. Do the same thing on all your other servers.
Lock the root account on each server:
  1. Type the following:

    Code: Select all

    passwd -l root
  2. Do the same for each of your servers
Set root login to SSH only on each server:
  1. Edit /etc/ssh/sshd_config, and uncomment the following from:

    Code: Select all

    #PermitRootLogin prohibit-password
    to:

    Code: Select all

    PermitRootLogin prohibit-password
  2. Just below that line, delete the following:

    Code: Select all

    PermitRootLogin yes
  3. Restart SSH:

    Code: Select all

    systemctl restart ssh
  4. Do the same for each of your servers
Now you should be able to securely interact between servers without having to type a password.

Examples:
scp /etc/hosts root@srv-database:/tmp/hosts.txt ssh administrator@srv-database
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Security - Messages

Post: # 1018Post LHammonds »

SUDO Warning Message
  1. Enable the default warning message whenever "sudo" is used. Edit sudoers file by typing:

    Code: Select all

    sudo EDITOR=vim visudo
    Add the following line near the other defaults (instead of "always" you can use "once" too):

    Code: Select all

    Defaults lecture=always
  2. To make sure the syntax of the edited file is not incorrect, check it with the following command and look for "parsed: OK"
    NOTE: If you mess up the syntax in this file, it can cause you problems using the "sudo" command.

    Code: Select all

    sudo visudo -c
  3. To customize the message, you can create a custom message file. Edit the lecture file by typing:

    Code: Select all

    sudo touch /etc/lecture
    sudo chown root:root /etc/lecture
    sudo chmod 644 /etc/lecture
    sudo vi /etc/lecture
    Add the following text (which is default) or whatever you want:

    Code: Select all

    We trust you have received the usual lecture from the local System
    Administrator. It usually boils down to these three things:
    
        #1) Respect the privacy of others.
        #2) Think before you type.
        #3) With great power comes great responsibility.
  4. Now point to the newly-created file in the sudoers config file.

    Code: Select all

    sudo EDITOR=vim visudo

    Code: Select all

    Defaults lecture_file=/etc/lecture
SSH Login Message
  1. To enable login banners that are displayed before entering a password, edit the following:

    Code: Select all

    sudo vi /etc/issue.net
  2. Add your message however you like to word it and save/close the file:

    Code: Select all

    *********************************************************************
      ALERT! You are entering into a secured area!
      Your IP, Login Time, Username have been recorded.
      This service is restricted to authorized users only.
      All activities on this system are logged.
      Unauthorized access will be reported to the law enforcement agencies.
    *********************************************************************
    
  3. Enable banners to be displayed by editing the SSH config file:

    Code: Select all

    sudo vi /etc/ssh/sshd_config

    Code: Select all

    Banner /etc/issue.net
  4. Restart the SSH service:

    Code: Select all

    sudo systemctl restart sshd
Custom Login Logo

If you want an Ubuntu Logo and additional info added to your login, visit this thread.

Image
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

File Sharing via Samba

Post: # 1019Post LHammonds »

Configure Ubuntu for File Sharing

This file sharing section is optional but can be handy if you need to swap files between the Linux server and a Windows machine.

This documentation will utilize this share for passing pre-configured files (configs, scripts, etc.) to make it faster/easier during installation.
  1. Start the Ubuntu server and connect using PuTTY.
  2. At the login prompt, login with your administrator account (administrator / myadminpass)
  3. Install Samba by typing the following:

    Code: Select all

    sudo apt -y install samba cifs-utils
    NOTE: To share a folder with Windows, you just need the samba package, to connect to a Windows share, you need both samba and cifs-utils
  4. Type the following commands:

    Code: Select all

    sudo cp /etc/samba/smb.conf /etc/samba/smb.conf.bak
    sudo mkdir -p /srv/samba/share
    sudo chown nobody:nogroup /srv/samba/share/
    sudo chmod 0777 /srv/samba
    sudo chmod 0777 /srv/samba/share
  5. Edit the configuration file:

    Code: Select all

    sudo vi /etc/samba/smb.conf
  6. Change:

    Code: Select all

    workgroup = WORKGROUP
    to:
    workgroup = work
    (you are using the domain alias)
  7. Add the following section to the end of the file:

    Code: Select all

    [share]
    comment = Ubuntu File Server Share
    path = /srv/samba/share
    browsable = yes
    guest ok = yes
    read only = no
    create mask = 0660
    directory mask = 0755
  8. Save and exit the file.
  9. Restart the samba services to utilize the new configuration by typing:

    Code: Select all

    sudo systemctl restart smbd
    sudo systemctl restart nmbd
  10. Update your firewall rules to allow Samba file sharing:

    Code: Select all

    sudo ufw allow proto tcp to any port 135,139,445 comment 'Samba Share'
    sudo ufw allow proto udp to any port 137,138 comment 'Samba Share'
    
  11. You should now be able to click Start --> Run and type \\srv-ubuntu or \\192.168.107.2 {ENTER} and see an explorer window with a Share folder. Drag-n-drop a file into the Share folder. If it worked, it will not display an error message and you should be able to view it from the server by typing ls -l /srv/samba/share/
  12. Shutdown and power off the server by typing shutdown -P now {ENTER}
  13. In VM menu, select VM --> Snapshot --> Take Snapshot. Give it a name like STEP 4 and description of Ubuntu Server 22.04 LTS, File share configured, Static IP: 192.168.107.2. The Snapshot Manager should now have a nice hierarchy of snapshots (STEP 1 --> STEP 2 --> STEP 3 --> STEP 4 --> You are here)
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

File Sharing via NFS (Linux to Linux)

Post: # 1020Post LHammonds »

NFS Mount (Sharing among Linux)

If you want to share files between Linux servers, you can use NFS. Here is how you can do that.

On the machine that will host the files to be shared:
  1. Install the required software:

    Code: Select all

    sudo apt install nfs-kernel-server
  2. Configure a folder to be shared:

    Code: Select all

    sudo touch /etc/exports
    sudo chown root:root /etc/exports
    sudo chmod 644 /etc/exports
    sudo vi /etc/exports
  3. Add the following line, adjust as necessary and then save and close the file:

    Code: Select all

    /srv/samba/share        *(rw,sync,no_root_squash,no_subtree_check)
  4. Make the folder and configure ownership and permissions:

    Code: Select all

    sudo mkdir -p /srv/samba/share
    sudo chown nobody:nogroup /srv/samba/share
    sudo chmod 777 /srv/samba/share
  5. Restart services for changes to take effect:

    Code: Select all

    sudo systemctl restart nfs-kernel-server
  6. If your UFW firewall is enabled, you will need to open port 2049 TCP and UDP. Example:

    Code: Select all

    sudo ufw allow proto tcp to any port 2049 comment 'NFS Share'
    sudo ufw allow proto udp to any port 2049 comment 'NFS Share'
On the machine that will connect to the other remotely:
  1. Install the required software:

    Code: Select all

    sudo apt install nfs-common
  2. Create the mount point folder:

    Code: Select all

    sudo mkdir -p /mnt/nfs
    sudo touch /mnt/nfs/offline.txt
  3. Now connect to the server. We will assume the server's IP doing the sharing is 192.168.107.100 in this example.

    Code: Select all

    sudo mount 192.168.107.100:/srv/samba/share /mnt/nfs
  4. If the mount was successful, you will not see the "offline.txt" file we created at the mount point.
  5. To disconnect from the server, type the following:

    Code: Select all

    sudo umount /mnt/nfs
  6. If the unmount was successful, you will see the "offline.txt" file.
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

File Sharing via Windows Shares

Post: # 1021Post LHammonds »

Configure Windows Server as a Remote Mount Point

If you have a Windows 2008/2012 server that serves as your offsite backup repository, this section describes how to configure and mount the remote server for storage use.

Part of the backup process involves copying the backup files to an offsite storage server.

First, let's document the variables for this solution below and highlight them in red throughout the document for easy identification.

The values below are merely samples which you need to change in order to match your environment:

Windows AD Domain Name: work
Windows AD Share ID: ubuntushare
Windows AD Share Password: ubuntupassword
Windows Server Name: SRV-Backup
Windows Server IP: 192.168.107.218
Windows Share Name: ubuntu
Windows Physical Share Location: D:\Ubuntu\

Create a share on a Windows 2008/2012 server
  1. In Windows Explorer, right-click on the D:\Ubuntu folder and select Properties
  2. Click the Sharing tab
  3. Click the Advanced Sharing button
  4. Place a checkmark beside Share this folder
  5. Change the Share name to ubuntu
  6. Set the Comment to Ubuntu Backup
  7. Click the Permissions button
  8. Select Everyone and click the Remove button
  9. Click the Add button
  10. Type in your Ubuntu share account: work\ubuntushare and click the Check Names button, click OK
  11. Place a checkmark for Allow Full Control and click OK, click OK, click OK
Create an NFS mount to the Windows 2008/2012 server

Connecting to a Windows share requires the samba and cifs-utils packages to be installed. If you did not install them (from a prior section), type apt -y install samba cifs-utils
  1. At the login prompt, login with your administrator account (administrator / myadminpass)
  2. Type the following commands:

    Code: Select all

    sudo mkdir -p /mnt/backup
    sudo chown root:root /mnt/backup
    sudo chmod 0755 /mnt/backup
    sudo echo "This file is used to tell if the mount is active or not" > /mnt/backup/offline.txt
    sudo chown root:root /mnt/backup/offline.txt
    sudo chmod 0444 /mnt/backup/offline.txt
    sudo touch /etc/cifspw
    sudo chown root:root /etc/cifspw
    sudo chmod 0600 /etc/cifspw
    sudo vi /etc/cifspw
    
    Add the following text, save and exit the file:
    username=ubuntushare domain=work password=ubuntupassword
  3. Type sudo vi /etc/hosts and add the following line anywhere in the file:
    192.168.107.218 srv-backup
  4. At this point, you might want to type ping srv-backup to make sure you typed the right IP address as well as seeing a good response.
  5. To mount this system for backups, type the following command:

    Code: Select all

    sudo mount -t cifs //srv-backup/ubuntu /mnt/backup --options vers=2.0,nouser,rw,nofail,noexec,credentials=/etc/cifspw
  6. To test it, type cp /etc/hosts /mnt/backup/hosts.txt and look on the Windows server and see if the file shows up. Then type rm /mnt/backup/hosts.txt and verify that the file was deleted on the windows server.
  7. This would also be a good time to verify that you cannot see the "offline.txt" file that will be used by the backup script. Type ls -l /mnt/backup/*.txt
  8. To dismount the windows share, type the following command:

    Code: Select all

    sudo umount /mnt/backup
The scripts will call a common mount and unmount function to connect to this share only when needed.

However, if you would rather have it mounted all the time (even after a reboot), do the following (but remember to not use the mount/umount functions in the scripts later):
  1. Type sudo vi /etc/fstab and add the following line at the bottom of the file:
    //srv-backup/ubuntu /mnt/backup cifs vers=2.0,nouser,rw,nofail,noexec,credentials=/etc/cifspw 0 0
  2. Type sudo mount -a and if it does not spew out any error messages, it will quietly mount the share.
  3. To test it, type cp /etc/hosts /mnt/backup/hosts.txt and look on the Windows server and see if the file shows up. Then type rm /mnt/backup/hosts.txt and verify that the file was deleted on the windows server.
  4. If you need to unmount it, type sudo umount /mnt/backup and it will remain unmounted until you reboot. To make it permanent, you need to remove the line you added in the /etc/fstab file.
Sometimes it is helpful during tests to manually toggle the mount on or off so here is a script you might find helpful. You can find the contents of "standard.conf" file in the scripting section.

/var/scripts/prod/togglemount.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : togglemount.sh
## Version       : 1.4
## Date          : 2022-05-31
## Author        : LHammonds
## Compatibility : Verified on Ubuntu Server 22.04 LTS
## Purpose       : Toggle the mount status of a pre-configured backup mount.
## Run Frequency : Manual as needed.
## Exit Codes    :
##   0 = success
##   1 = failure
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2011-11-05 1.0 LTH Created script.
## 2017-03-17 1.1 LTH Updated variable standards.
## 2019-07-18 1.2 LTH Updated reference file online.txt to offline.txt
## 2020-05-01 1.3 LTH Test file now uses the common variable name.
## 2022-05-31 1.4 LTH Replaced echo statements with printf.
#############################################################

## Import common variables and functions. ##
source /var/scripts/common/standard.conf
ErrorFlag=0

if [ -f ${OffsiteTestFile} ]; then
  printf "Remote share is not mounted.  Mounting share now...\n"
  f_mount
  sleep 2
  if [ -f ${OffsiteTestFile} ]; then
    printf "Mount failed.  Listing contents:\n"
    ErrorFlag=1
  else
    printf "Mount successful.  Listing contents:\n"
  fi
else
  printf "Remote share is mounted.  Dismounting share now...\n"
  f_umount
  sleep 2
  if [ -f ${OffsiteTestFile} ]; then
    printf "Dismount successful.  Listing contents:\n"
  else
    printf "Dismount failed.  Listing contents:\n"
    ErrorFlag=1
  fi
fi
ls -l ${OffsiteDir}
exit ${ErrorFlag}
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1056
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Partition-Level Backups

Post: # 1022Post LHammonds »

Backup Partitions Using LVM Snapshots and FSArchiver

This method will allow online backup of the server at the partition level. It is designed to run via crontab schedule but can also be run manually.

This should be considered a full backup which means you will probably need to rely on other methods for granular backups and restores such as using rsync at the file level.

This method is great for backing up a system just prior to and just after a major upgrade of the OS or application. It is not very helpful for retrieving individual files although it could be done but would require a bit of work by temporarily restoring to an unused area, retrieving the file(s) and then destroying the temporary partition.

The /bak partition is skipped because that is where the archives are being stored.

The /tmp partition is skipped because there should not be anything in there that needs to be restored...but feel free to include it if you like.

The script below was built around a few very basic commands that do the bulk of the work but most of the code is for error handling.

Here are examples of the commands:

Code: Select all

## Create the snapshot volume of the partition to be backed up.
lvcreate --size=5G --snapshot --name="tempsnap" /dev/LVG/root

## Create the compressed and encrypted archive of the snapshot.
fsarchiver savefs --compress=7 --jobs=1 --cryptpass="abc123" --label="insert comment here" /bak/root.fsa /dev/LVG/tempsnap

## Create an informational text file about the archive.
fsarchiver archinfo --cryptpass="abc123" /bak/root.fsa > /bak/root.txt 2>&1

## Remove the snapshot.
lvremove --force /dev/LVG/tempsnap

## Create a checksum file about the archive.
md5sum /bak/root.fsa > /bak/root.md5

## Verify that the checksum file can validate against the archive.
md5sum --check /bak/root.md5
Here is an example of a crontab entry to run the script once a day.

/var/scripts/data/crontab.root

Code: Select all

0 4 * * * /var/scripts/prod/back-parts.sh > /dev/null 2>&1
Here is the script.

/var/scripts/prod/back-parts.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name : back-parts.sh (Backup Partitions)
## Version : 1.6
## Date : 2022-05-31
## Author : LHammonds
## Purpose : Backup partitions
## Compatibility : Verified on Ubuntu Server 22.04 LTS
##                 Verified with fsarchiver 0.8.6)
## Requirements : Fsarchiver, Sendemail, run as root
## Run Frequency : Once per day or as often as desired.
## Parameters : None
## Exit Codes :
## 0  = Success
## 1  = ERROR: Lock file detected
## 2  = ERROR: Must be root user
## 4  = ERROR: Missing software
## 8  = ERROR: LVM problems
## 16 = ERROR: File creation problems
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2013-01-09 1.0 LTH Created script.
## 2017-03-16 1.1 LTH Updated variable standards.
## 2017-08-31 1.2 LTH Added create folder if not exist.
## 2017-10-04 1.3 LTH Set file permissions.
## 2020-01-01 1.4 LTH Remove any prior temp snapshots before starting.
## 2020-05-27 1.5 LTH Removed offsite copy section. The offsite
##                    location will pull files for better security.
## 2022-05-31 1.6 LTH Replaced echo statements with printf.
#############################################################

## Import standard variables and functions. ##
source /var/scripts/common/standard.conf

## Define local variables.
LogFile="${LogDir}/${Company}-back-parts.log"
LockFile="${TempDir}/${Company}-back-parts.lock"
TargetDir="${BackupDir}/partitions"
LVG="/dev/LVG"
TempLV="${LVG}/tempsnap"
MaxTempVolSize=1G
ErrorFlag=0
ReturnCode=0
CryptPass="abc123"

#######################################
##            FUNCTIONS              ##
#######################################

function f_cleanup()
{
  if [ -f ${LockFile} ];then
    ## Remove lock file so other check space jobs can run.
    rm ${LockFile} 1>/dev/null 2>&1
  fi
  if [ ${ErrorFlag} != 0 ]; then
    f_sendmail "ERROR: Script Failure" "Please review the log file on ${Hostname}${LogFile}"
    printf "`date +%Y-%m-%d_%H:%M:%S` - Backup aborted.\n" >> ${LogFile}
  fi
  exit ${ErrorFlag}
}

function f_archive_fs()
{
  FSName=$1
  FSPath=$2

  ## Purge old backup files.
  if [ -f ${TargetDir}/${Hostname}-${FSName}.fsa ]; then
    rm ${TargetDir}/${Hostname}-${FSName}.fsa
  fi
  if [ -f ${TargetDir}/${Hostname}-${FSName}.txt ]; then
    rm ${TargetDir}/${Hostname}-${FSName}.txt
  fi
  if [ -f ${TargetDir}/${Hostname}-${FSName}.md5 ]; then
    rm ${TargetDir}/${Hostname}-${FSName}.md5
  fi

  ## Unmount FileSystem.
  umount /${FSName}

  LVLabel="${Hostname}:${FSPath}->/${FSName}"
  ## Create the compressed and encrypted archive of the snapshot.
  fsarchiver savefs --compress=7 --jobs=1 --cryptpass="${CryptPass}" --label="${LVLabel}" ${TargetDir}/${Hostname}-${FSName}.fsa ${FSPath} > /dev/null 2>&1
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Creation of the archive failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of ${TargetDir}/${FSName}.fsa failed, Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=16
    f_cleanup
  fi

  ## Create an informational text file about the archive.
  fsarchiver archinfo --cryptpass="${CryptPass}" ${TargetDir}/${Hostname}-${FSName}.fsa > ${TargetDir}/${Hostname}-${FSName}.txt 2>&1
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Creation of info text failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of info file failed for ${TargetDir}/${FSName}.txt, Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=16
    f_cleanup
  fi

  ## Create a checksum file about the archive.
  md5sum ${TargetDir}/${Hostname}-${FSName}.fsa > ${TargetDir}/${Hostname}-${FSName}.md5
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Creation of md5 checksum failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of checksum failed for ${TargetDir}/${FSName}.md5, Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=16
    f_cleanup
  fi

  ## Verify that the checksum file can validate against the archive.
  md5sum --check --status ${TargetDir}/${Hostname}-${FSName}.md5
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Verification failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: md5 validation check failed for ${TargetDir}/${FSName}.md5. Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=16
    f_cleanup
  fi

  ## Set file permissions.
  chmod 0600 ${TargetDir}/${Hostname}-${FSName}.*

  BackupSize=`ls -lak --block-size=m "${TargetDir}/${Hostname}-${FSName}.fsa" | awk '{ print $5 }'`

  printf "`date +%Y-%m-%d_%H:%M:%S` --- Created: ${TargetDir}/${Hostname}-${FSName}.fsa, ${BackupSize}\n" >> ${LogFile}

  ## Remount FileSystem.
  mount /${FSName}
}

function f_archive_vol()
{
  LVName=$1
  LVPath=${LVG}/${LVName}

  ## Purge old backup files.
  if [ -f ${TargetDir}/${Hostname}-${LVName}.fsa ]; then
    rm ${TargetDir}/${Hostname}-${LVName}.fsa
  fi
  if [ -f ${TargetDir}/${Hostname}-${LVName}.txt ]; then
    rm ${TargetDir}/${Hostname}-${LVName}.txt
  fi
  if [ -f ${TargetDir}/${Hostname}-${LVName}.md5 ]; then
    rm ${TargetDir}/${Hostname}-${LVName}.md5
  fi

  ## Create the snapshot volume of the partition to be backed up.
  lvcreate --size=${MaxTempVolSize} --snapshot --name="tempsnap" ${LVPath} > /dev/null 2>&1
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Creation of temporary volume failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of temp volume failed for ${LVPath}, size=${MaxTempVolSize}, Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=8
    f_cleanup
  fi

  ## Give the OS a moment to let the LV create command do its thing.
  sleep 2

  LVLabel="${Hostname}:${LVPath}"
  ## Create the compressed and encrypted archive of the snapshot.
  fsarchiver savefs --compress=7 --jobs=1 --cryptpass="${CryptPass}" --label="${LVLabel}" ${TargetDir}/${Hostname}-${LVName}.fsa ${TempLV} > /dev/null 2>&1
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Creation of the archive failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of ${TargetDir}/${Hostname}-${LVName}.fsa failed, Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=16
    f_cleanup
  fi

  ## Create an informational text file about the archive.
  fsarchiver archinfo --cryptpass="${CryptPass}" ${TargetDir}/${Hostname}-${LVName}.fsa > ${TargetDir}/${Hostname}-${LVName}.txt 2>&1
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Creation of info text failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of info file failed for ${TargetDir}/${Hostname}-${LVName}.txt, Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=16
    f_cleanup
  fi

  ## Create a checksum file about the archive.
  md5sum ${TargetDir}/${Hostname}-${LVName}.fsa > ${TargetDir}/${Hostname}-${LVName}.md5
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Creation of md5 checksum failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of checksum failed for ${TargetDir}/${Hostname}-${LVName}.md5, Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=16
    f_cleanup
  fi

  ## Set file permissions.
  chmod 0600 ${TargetDir}/${Hostname}-${LVName}.*

  ## Remove the snapshot.
  lvremove --force ${TempLV} > /dev/null 2>&1
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Removal of temporary volume failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Removal of temp volume failed. ${TempLV}. Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=8
    f_cleanup
  fi

  ## Give the OS a moment to let the LV remove command do its thing.
  sleep 2

  ## Verify that the checksum file can validate against the archive.
  md5sum --check --status ${TargetDir}/${Hostname}-${LVName}.md5
  ReturnCode=$?
  if [ ${ReturnCode} != 0 ]; then
    ## Verification failed.
    printf "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: md5 validation check failed for ${TargetDir}/${Hostname}-${LVName}.md5. Return Code = ${ReturnCode}\n" >> ${LogFile}
    ErrorFlag=16
    f_cleanup
  fi

  BackupSize=`ls -lak --block-size=m "${TargetDir}/${Hostname}-${LVName}.fsa" | awk '{ print $5 }'`

  printf "`date +%Y-%m-%d_%H:%M:%S` --- Created: ${TargetDir}/${Hostname}-${LVName}.fsa, ${BackupSize}\n" >> ${LogFile}

}

#######################################
##           MAIN PROGRAM            ##
#######################################

if [ -f ${LockFile} ]; then
  # Lock file detected.  Abort script.
  printf "Backup partitions script aborted\n"
  printf "This script tried to run but detected the lock file: ${LockFile}\n"
  printf "Please check to make sure the file does not remain when backup partitions is not actually running.\n"
  f_sendmail "ERROR: Backup partitions script aborted" "This script tried to run but detected the lock file: ${LockFile}\n\nPlease check to make sure the file does not remain when backup partitions is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LockFile}'"
  ErrorFlag=1
  exit ${ErrorFlag}
else
  printf "`date +%Y-%m-%d_%H:%M:%S` ${ScriptName}\n" > ${LockFile}
fi

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  printf "`date +%Y-%m-%d_%H:%M:%S` ERROR: Root user required to run this script.\n" >> ${LogFile}
  ErrorFlag=2
  f_cleanup
fi

## Requirement Check: Software
command -v fsarchiver > /dev/null 2>&1 && ReturnCode=0 || ReturnCode=1
if [ ${ReturnCode} = 1 ]; then
  ## Required program not installed.
  printf "`date +%Y-%m-%d_%H:%M:%S` ERROR: fsarchiver not installed.\n" >> ${LogFile}
  ErrorFlag=4
  f_cleanup
fi

## Make sure target folder exists.
if [ ! -d ${TargetDir} ]; then
  mkdir -p ${TargetDir}
fi
## Remove old snapshot from a prior run if it exists.
lvremove --force ${TempLV} > /dev/null 2>&1

StartTime="$(date +%s)"
printf "`date +%Y-%m-%d_%H:%M:%S` - Partition backup started.\n" >> ${LogFile}

f_archive_fs boot /dev/sda2
f_archive_vol root
f_archive_vol home
f_archive_vol srv
f_archive_vol usr
f_archive_vol var
f_archive_vol tmp
f_archive_vol opt
#f_archive_vol swap

## Calculate total time for backup.
FinishTime="$(date +%s)"
ElapsedTime="$(expr ${FinishTime} - ${StartTime})"
Hours=$((${ElapsedTime} / 3600))
ElapsedTime=$((${ElapsedTime} - ${Hours} * 3600))
Minutes=$((${ElapsedTime} / 60))
Seconds=$((${ElapsedTime} - ${Minutes} * 60))

printf "`date +%Y-%m-%d_%H:%M:%S` --- Total backup time: ${Hours} hour(s) ${Minutes} minute(s) ${Seconds} second(s)\n" >> ${LogFile}

printf "`date +%Y-%m-%d_%H:%M:%S` - Partition backup finished.\n" >> ${LogFile}
f_cleanup
Here is an example of the log file:

/var/log/abc-back-parts.log

Code: Select all

2022-05-31_19:22:05 - Partition backup started.
2022-05-31_19:22:37 --- Created: /bak/partitions/srv-ubuntu-boot.fsa, 114M
2022-05-31_19:23:03 --- Created: /bak/partitions/srv-ubuntu-root.fsa, 2M
2022-05-31_19:23:08 --- Created: /bak/partitions/srv-ubuntu-home.fsa, 1M
2022-05-31_19:23:13 --- Created: /bak/partitions/srv-ubuntu-srv.fsa, 1M
2022-05-31_19:28:14 --- Created: /bak/partitions/srv-ubuntu-usr.fsa, 726M
2022-05-31_19:29:55 --- Created: /bak/partitions/srv-ubuntu-var.fsa, 331M
2022-05-31_19:30:00 --- Created: /bak/partitions/srv-ubuntu-tmp.fsa, 1M
2022-05-31_19:30:05 --- Created: /bak/partitions/srv-ubuntu-opt.fsa, 1M
2022-05-31_19:30:05 --- Total backup time: 0 hour(s) 8 minute(s) 0 second(s)
2022-05-31_19:30:05 - Partition backup finished.
An example email notification when a fatal error occurred:

Code: Select all

From: admin@mydomain.com
Sent: Tuesday, May 31, 2022 5:20:45 PM
To: lhammonds@mydomain.com
Subject: ERROR: Backup partitions script aborted

This script tried to run but detected the lock file: /tmp/abc-back-parts.lock

Please check to make sure the file does not remain when backup partitions is not actually running.

If you find that the script is not running/hung, you can remove it by typing 'rm /tmp/abc-back-parts.lock'

Server: srv-ubuntu
Program: /var/scripts/prod/back-parts.sh
Log: /var/log/abc-back-parts.log
An example email notification when no errors occur and email notifications turned on:

Code: Select all

From: admin@mydomain.com
To: lhammonds@mydomain.com
Sent: Tuesday, May 31, 2022 5:20:45 PM
Subject: Backup Completed

INFO: The partition backup job has completed without any errors.

Server: srv-ubuntu
Program: /var/scripts/prod/back-parts.sh
Log: /var/log/abc-back-parts.log
Troubleshooting Snapshot Failures

If the snapshot volume could not be automatically removed, here is how you do it:

Code: Select all

sudo dmsetup ls
LVG-tempsnap (253:10) LVG-bak (253:7) LVG-home (253:1) LVG-opt (253:6) LVG-root (253:0) LVG-root-real (253:8) LVG-tempsnap-cow (253:9) LVG-srv (253:2) LVG-tmp (253:5) LVG-usr (253:3) LVG-var (253:4)

Code: Select all

sudo dmsetup remove LVG-tempsnap
sudo dmsetup remove LVG-tempsnap-cow
sudo lvremove LVG-tempsnapshot
The script can fail for multiple reasons...mostly due to lack of space.

Use "vgdisplay" to make sure you have enough free/unallocated space in the volume group. Your max tempsnap size cannot exceed the amount of unallocated/unused space in the volume group.

Make sure you have enough room in /bak to hold the partitions as well as your remote location. A new server without any software/data installed will take up at least 1.2 GB of space when archived.

Backup Test

Before the partitions are backed on your server, create a couple of empty test files to verify that the restore in the next section will work.

Type the following commands:

Code: Select all

touch /important.txt
touch /home/administrator/important.txt
Make sure the above files are included in your backup before testing the restore in the next section.
Post Reply