How to install and setup an HA DICOM router on Ubuntu Server 20.04 LTS

[LHammonds]

------------- WORK-IN-PROGRESS -------------

Greetings and salutations,

I hope this thread will be helpful to those who follow in my foot steps.

High-level overview

Orthanc is an open-source, lightweight DICOM server. This software will be used to setup a DICOM router for use between a modality workstation (like X-Ray, CR, CT, MRI, Mammo, etc.) and an imaging server (like UltraLinq, ProtonPACS, Sectra, Centricity, Merge PACS, Synapse, etc.)

We will be creating a 4-server system. Two servers will be proxy load-balancers and 2 will be the DICOM routers. More routers can be used to help offload processing / traffic.



Tools utilized in this process

• Ubuntu Server 20.04.1 LTS
• Orthanc 1.5.8 (or 1.8.0 if you compile from source code)
• HAProxy
• Keepalived
• Portable PuTTY 0.74
• Hypervisor Host: Proxmox 6.2 or VMware ESXi 6.0

Helpful links

The list below are sources of information that was helpful in the creation of this document.

• Orthanc Documentation
• Orthanc Users Google Group

Assumptions

This documentation will need to make use of some very-specific information that will most-likely be different for each person / location. And as such, this information will be noted in this section. They will be highlighted in red throughout the document as a reminder that you should plug-in your own value rather than actually using these "place-holder" values.

Under no circumstance should you use the actual values listed below. They are place-holders for the real thing. This is just a checklist template you need to have answered before you start the install process.

Wherever you see RED in this document, you need to substitute it for you will use in your environment.

• Load Balancer Virtual IP: 192.168.1.100
• Load Balancer #1 Name: SRV-LB1: 192.168.1.101 (Primary / Active)
• Load Balancer #2 Name: SRV-LB2: 192.168.1.102 (Failover / Inactive)
• Router #1 Name: SRV-DICOM1: 192.168.1.103
• Router #2 Name: SRV-DICOM2: 192.168.1.104
• PACS AE Title PACSPROD, IP 192.168.1.200, Port 104
• Web Interface Credentials: ID myadminid, Password myadminpass

It is also assumed the reader knows how to use the VI editor. If not, you will need to beef up your skill set or use a different editor in place of it.

It is also assumed that all servers were setup following my Ubuntu Server instructions.

[LHammonds]

Orthanc

On the Orthanc servers, install the software:

Code: Select all

sudo apt install orthanc

Open the configuration file:

Code: Select all

sudo vi /etc/orthanc/orthanc.json

NOTE: If you need to generate the original file again, you can run "Orthanc --config=/etc/orthanc/orthanc.json"

Make the following changes (substituting the place-holder values for your real values):
NOTE: You might want to set "DicomAet" to be the same for all the routers in the cluster since any one of them can end up doing the processing of a dicom job (which needs to be the specific AE Title that the sending modality knows it by)

Code: Select all

  "Name" : "srv-dicom1"
  "DicomAet" : "SRVDICOM",
  "RemoteAccessAllowed" : true,
  "AuthenticationEnabled" : true,
  "RegisteredUsers" : {
    "myadminid" : "myadminpass"
  },
  "DicomModalities" : {
    "PACSProd" : [ "PACSPROD", "192.168.1.200", 104]
  ],
"LuaScripts" : [
  "/var/scripts/prod/dicom-autoroute.lua"
  ],

Create the auto-route script and open for editing:

Code: Select all

sudo touch /var/scripts/prod/dicom-autoroute.lua
sudo chown root:root /var/scripts/prod/dicom-autoroute.lua
sudo chmod 644 /var/scripts/prod/dicom-autoroute.lua
sudo vi /var/scripts/prod/dicom-autoroute.lua

Edit the auto-route script and add the following:

Code: Select all

-- Purpose: Route incoming DICOM image and then delete from Orthanc.
function OnStoredInstance(instanceId, tags, metadata)
  Delete(SendToModality(instanceId, 'PACSProd'))
end

Restart the service:

Code: Select all

sudo service orthanc restart

If you have problems with the service starting, take a look at the log file:

Code: Select all

vi /var/log/orthanc/Orthanc.log

[LHammonds]

Orthanc Firewall Rules

On each Orthanc server, edit the firewall script that was created during the initial setup of the server (if you followed my instructions):

Code: Select all

sudo vi /var/scripts/prod/en-firewall.sh

Add the following rules under the Application-specific section which allows connectivity from from anything on your local subnet:

Code: Select all

echo "Adding Orthanc rules"
ufw allow from 192.168.1.0/24 proto tcp to any port 4242 comment 'DICOM' 1>/dev/null 2>&1
ufw allow from 192.168.1.0/24 proto tcp to any port 8042 comment 'Orthanc Web' 1>/dev/null 2>&1

Or you can reduce access even further by limiting to just specific IP addresses such as your modality workstations and admin workstations like this:

Code: Select all

echo "Adding Orthanc rules"
ufw allow from 192.168.1.30 proto tcp to any port 4242 comment 'DICOM' 1>/dev/null 2>&1
ufw allow from 192.168.1.31 proto tcp to any port 4242 comment 'DICOM' 1>/dev/null 2>&1
ufw allow from 192.168.1.32 proto tcp to any port 4242 comment 'DICOM' 1>/dev/null 2>&1
ufw allow from 192.168.1.33 proto tcp to any port 8042 comment 'Orthanc Web' 1>/dev/null 2>&1
ufw allow from 192.168.1.34 proto tcp to any port 8042 comment 'Orthanc Web' 1>/dev/null 2>&1

Run the updated rules:

Code: Select all

sudo /var/scripts/prod/en-firewall.sh

Test Connectivity

Open a browser on your workstation and go to http://192.168.1.103:8084/ and login with the credentials set in the configuration file and see if the main page shows up.

[LHammonds]

Install Keepalived on Primary

On the primary load balancer server, run this command:

Code: Select all

sudo apt install keepalived

Create the default user that runs the script checks:

Code: Select all

sudo useradd -g users -M keepalived_script

Primary Load Balancer Firewall Rules

Edit the firewall script that was created during the initial setup of the server (if you followed my instructions):

Code: Select all

sudo vi /var/scripts/prod/en-firewall.sh

Add (or enable) the following:

NOTE: If you have more than 1 failover load balancer (VRRP Router), be sure to add their IP addresses as well.

Code: Select all

echo "Adding Dicom Server rules"
ufw allow proto tcp to any port 4242 comment 'DICOM' 1>/dev/null 2>&1
echo "Adding VRRP rules"
ufw allow to 224.0.0.18 comment 'VRRP Broadcast' 1>/dev/null 2>&1
ufw allow from 192.168.1.101 comment 'VRRP Router' 1>/dev/null 2>&1

Run the updated rules:

Code: Select all

sudo /var/scripts/prod/en-firewall.sh

Primary Keepalive Config

On the primary server (srv-lb1), create the keepalive configuration file:

Code: Select all

sudo touch /etc/keepalived/keepalived.conf
sudo chown root:root /etc/keepalived/keepalived.conf
sudo chmod 600 /etc/keepalived/keepalived.conf

Edit the configuration file:

Code: Select all

sudo vi /etc/keepalived/keepalived.conf

Add the following to the file (substituting for your own values for email, smtp_server, virtual IP and Ethernet interface name):

Code: Select all

global_defs {
  notification_email {
    my_email@mydomain.com
  }
  notification_email_from keepalived@mydomain.com
  smtp_server 192.168.1.25
  smtp_connect_timeout 30
  script_user keepalived_script users
  enable_script_security
}
vrrp_script chk_haproxy {
  script "/usr/bin/killall -0 haproxy"
  interval 1            # check every second
  weight 2              # add 2 points of priority if OK
}
vrrp_instance primary {
  interface ens32
  state MASTER
  smtp_alert
  virtual_router_id 51  # Should be same on all LBs
  priority 101          # 101 on master, 100 on slaves
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  virtual_ipaddress {
    192.168.1.100
  }
  track_script {
    chk_haproxy
  }
}

Restart the service:

Code: Select all

sudo systemctl restart keepalived

[LHammonds]

Install Keepalived on Failover

On each failover server, this command:

Code: Select all

sudo apt install keepalived

Create the default user that runs the script checks:

Code: Select all

sudo useradd -g users -M keepalived_script

Failover Load Balancer Firewall Rules

Code: Select all

sudo vi /var/scripts/prod/en-firewall.sh

Add the following:

Code: Select all

echo "Adding DICOM Server rules"
ufw allow proto tcp to any port 4242 comment 'DICOM' 1>/dev/null 2>&1
echo "Adding VRRP rules"
ufw allow to 224.0.0.18 comment 'VRRP Broadcast' 1>/dev/null 2>&1
ufw allow from 192.168.1.101 comment 'VRRP Router' 1>/dev/null 2>&1

Run the updated rules:

Code: Select all

sudo /var/scripts/prod/en-firewall.sh

Failover Keepalive Config

On each failover server (srv-lb2), create the keepalive configuration file:

Code: Select all

sudo touch /etc/keepalived/keepalived.conf
sudo chown root:root /etc/keepalived/keepalived.conf
sudo chmod 600 /etc/keepalived/keepalived.conf

Edit the configuration file:

Code: Select all

sudo vi /etc/keepalived/keepalived.conf

Add the following to the file (substituting for your own values for email, smtp_server, virtual IP and Ethernet interface name):

Code: Select all

global_defs {
  notification_email {
    my_email@mydomain.com
  }
  notification_email_from keepalived@mydomain.com
  smtp_server 192.168.1.25
  smtp_connect_timeout 30
  script_user keepalived_script users
  enable_script_security
}
vrrp_script chk_haproxy {
  script "/usr/bin/killall -0 haproxy"
  interval 1            # check every second
  weight 2              # add 2 points of priority if OK
}
vrrp_instance failover {
  interface ens32
  state BACKUP
  smtp_alert
  virtual_router_id 51  # Should be same on all LBs
  priority 100          # 101 on master, 100 on slaves
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  virtual_ipaddress {
    192.168.1.100
  }
  track_script {
    chk_haproxy
  }
}

Restart the service:

Code: Select all

sudo systemctl restart keepalived

[LHammonds]

Test Keepalived

You should be able to ping the virtual IP address at this point. If you reboot srv-lb1 while continuously pinging the virtual IP, you might see 1 or 2 drops in the ping until a failover takes over for the primary. When the primary comes back, you might see another 1 or 2 drops in the ping as the virtual IP moves back from the failover to the primary.

This is what the master NIC should look like (while active):

Code: Select all

# ip addr show ens32

Code: Select all

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:50:56:bf:27:cc brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.101/24 brd 192.168.1.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet 192.168.1.100/32 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:febf:27cc/64 scope link
       valid_lft forever preferred_lft forever

This is what the slave NIC should look like (while inactive):

Code: Select all

# ip addr show ens32

Code: Select all

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:50:56:bf:6a:52 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.102/24 brd 192.168.1.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:febf:6a52/64 scope link
       valid_lft forever preferred_lft forever

If you configured email notification, you will get emails whenever a load balancer changes state.

Example email when srv-lb1 becomes the primary:

Code: Select all

Subject: [srv-lb1] VRRP Instance failover1 - Entering MASTER state
Body: => VRRP Instance is now owning VRRP VIPs <=

Example email when srv-lb2 becomes the backup:

Code: Select all

Subject: [srv-lb2] VRRP Instance failover1 - Entering BACKUP state
Body: => VRRP Instance is no longer owning VRRP VIPs <=

[LHammonds]

Install HAProxy

Run this command on both load balance servers:

Code: Select all

sudo apt install haproxy

Name Resolution

You can modify your host file for name resolution and just use the names of servers in your configuration files rather than the IP addresses. If a server IP changes in the future, you only need to modify the host which is much easier than tracking down various application configuration files. You could do this with an internal DNS server but I prefer using the local host file for fastest resolution.

Edit the local host file (on all load balance servers):

Code: Select all

sudo vi /etc/hosts

Add the Orthanc servers (substituting for your own values):

Code: Select all

192.168.107.103 srv-dicom1
192.168.107.104 srv-dicom2

Allow Virtual IP Binding

If you try to bind the inactive proxy to an IP it is not currently using (because the other proxy is using it) then it will fail to load the service. We need to modify the system to allow binding to a non-local IP that is not currently active.

On both load balance servers, edit the sysctl.conf file:

Code: Select all

sudo vi /etc/sysctl.conf

Add the following to the end of the file:

Code: Select all

net.ipv4.ip_nonlocal_bind=1

Activate the change:

Code: Select all

sysctl -p

HAProxy Configuration

This is the default ownership and file permission settings of the configuration file:

Code: Select all

sudo chown root:root /etc/haproxy/haproxy.cfg
sudo chmod 644 /etc/haproxy/haproxy.cfg

Backup the original configuration file (on both servers):

Code: Select all

sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.bak

Edit the configuration file (on all load balance servers):

Code: Select all

sudo vi /etc/haproxy/haproxy.cfg

Add the following to the bottom (substituting your own values)

Code: Select all

frontend fe-lb
  ## Bind to port 4242 on the virtual IP
  bind 192.168.1.100:4242
  mode tcp
  ## Must use tcplog when using tcp mode.
  option tcplog
  default_backend be-dicom

## Balance between the various backend servers (unencrypted).
backend be-dicom
  mode tcp
  ## Must use tcplog when using tcp mode.
  option tcplog
  ## Various policies for determining how to route traffic to the servers.
#  balance first
#  balance leastconn
  balance roundrobin
#  balance source
#  balance static-rr
  server dicom1 srv-dicom1 weight 1 check port 4242 rise 2 fall 3 inter 2000 fastinter 1000 downinter 5000
  server dicom2 srv-dicom2 weight 1 check port 4242 rise 2 fall 3 inter 2000 fastinter 1000 downinter 5000

## HAProxy stats web gui - This entire section is optional.
listen stats
  ## Setup listener on port 9000 on any interface.
  bind *:9000
  ## http mode so a web browser can be used to access it.
  mode http
  ## Enable metrics to be recorded.
  stats enable
  ## Configure the URI.  Example: http://192.168.107.100:9000/stats
  stats uri /stats
  ## How long the browser waits before refreshing the page.
  stats refresh 30s
  ## Title for popup window.
  stats realm HAProxy\ Statistics
  ## Hide the HAProxy version. Ex: version 1.8.8-1ubuntu0.4, released 2019/01/24
  stats hide-version
  ## Define user credentials.
  stats auth haproxy:haproxy
  stats auth viewer:viewer
  ## Allows taking down and bringing up backend servers.
  stats admin if TRUE

Validate the changes to the configuration files:

Code: Select all

sudo haproxy -f /etc/haproxy/haproxy.cfg -c

On both servers, restart the proxy service:

Code: Select all

sudo systemctl restart haproxy

Verify that the service started and is running (active):

Code: Select all

systemctl status haproxy

You can also verify it is listening on the expected ports (if you have net-tools installed):

Code: Select all

sudo netstat -ntlp | grep haproxy

Output:

Code: Select all

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      20102/haproxy
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      20102/haproxy
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      20102/haproxy

Stats Firewall Rules

If you use the optional "stats" section which listens on port 9000, you will need to add a firewall rule. If not using the proxy stats, ignore this section.

On each load balance server, edit the firewall script that was created during the initial setup of the server (if you followed my instructions):

Code: Select all

sudo vi /var/scripts/prod/en-firewall.sh

Add the following rules under the Application-specific section which allows connectivity from from anything on your local subnet:

Code: Select all

echo "Adding HAProxy Stats rules"
ufw allow from 192.168.1.0/24 proto tcp to any port 9000 comment 'ProxyStats' 1>/dev/null 2>&1

Or you can reduce access even further by limiting to just specific IP addresses such as your admin workstations like this:

Code: Select all

echo "Adding HAProxy Stats rules"
ufw allow from 192.168.1.30 proto tcp to any port 9000 comment 'ProxyStats' 1>/dev/null 2>&1
ufw allow from 192.168.1.31 proto tcp to any port 9000 comment 'ProxyStats' 1>/dev/null 2>&1
ufw allow from 192.168.1.32 proto tcp to any port 9000 comment 'ProxyStats' 1>/dev/null 2>&1

Run the updated rules:

Code: Select all

sudo /var/scripts/prod/en-firewall.sh

Open a web browser and have a look at the statistics page at http://192.168.1.100:9000/stats/