Server hardening and optimization for the would be node operator

In this article we will look at some measures you can do to harden your Ubuntu server from attacks. We will also look at a few tweaks to system performance to help with node operations. The article is not ment as an exchaustive list as server hardening is a huge topic, but more ment to show some common security measurements you can take as a would be node operator for Proof of Stake nodes.

It is assumed you have logged in and have access to your servers for the commands we will use in the examples. The example is from the latest Ubuntu 22.04 but should work on all Ubuntu releases. It is assumes you have secured your SSH login if you have a remote server. Ideally if you have a local server you do not need even a SSH login, or you could alternatively also use a VPN to protect the SSH. Cardano Cafe wrote a small piece on such a VPN setup.

First things first it is important to protect yourself from denial of service attacks. Usually this is done by having outward facing nodes that are connected to some internal node that actually handles the block production of the blockchain. For example in Cardano you usually have atleast one of what is called Relay nodes and these get information from the outside world while the node block producer itself gets information from the relay. You can handle this with systemd based services but if having a larger infrastructure you can look into deployment solutions such as docker or kubernetes. In those cases you also need to make sure the docker/kubernetes infrastructure on the server is as secure as the rest of the services on the server. A VPN between the relay and the node could be good but keep in mind this creates some overhead in latency. Howeever with wireguard that is built into the kernel this overhead is small and it if the server has more internet traffic it could be positive to seperate the node traffic so it is easier to log this as well.

This outward facing vs internal also brings us to the first thing to do:
Setting up a firewall. A simple firewall in Ubuntu is the Uncomplicated Firewall or UFW. Let us say your node has to have open port X and you have a ssh login on port Y you could set the following:

Sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow _PORT_Y_NUMBER
sudo ufw allow _PORT_X_NUMBER
sudo ufw enable
sudo ufw status verbose

This should now show that the firewall is active. If you have ip6 and not using it it is recommended you delete the ip6 permissions. You can get a numbered list and then delete the specific rule:

sudo ufw status numbered
sudo ufw delete _your_number_to_delete

You can also be more specific if you know the ip addresses of the nodes that will connect to your network such as:

Sudo ufw allow from _ip.ip.ip.ip to any port _yourport

Now with firewall in place you should check your ports. First check the local ports being listened to:

netstat -tulpn
ss -tulpn

Then also check if you can access the port from other computers using a port checker such as https://www.yougetsignal.com/tools/open-ports/

Disable root account. This is good because it forces you to only step up privleges when needed. (Make sure you have a set up a local user with sudo privileges first) Set:

sudo passwd -l root

If you for some reason need to reeneable root set it with:

sudo passwd -u root

If you did not have a local user you can do this with adduser then after modify it to be part of the sudo group with:

usermod -aG sudo your_user_name

Make sure to keep the system up to date and run periodical updates. Obviously this is important to get the latest security patches.

sudo apt-get update -y && sudo apt-get upgrade -y
sudo apt-get autoremove
sudo apt-get autoclean
sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

Server document for reflection on security. In a simple form this could be that you write down a document with who can access the server. What is the survilance and log methods of the server. Ideally if having multiple people running the server or having automated logs you should also test how long it takes to detect a breach by simulating an attack.

Server audit with Lynis is one way to improve your security. Lynis is an auditing, server harderning and compliance test software. You can install it with:

sudo apt install lynis

Then you can run an audit with:

sudo lynis audit system

This will set you off to a journey on how to improve your security even more and is too comprehensive for this short article to go into all the details of but will give you a hardening index and tips on how to improve security even more.

Example of a Lynis audit where the server has a way to go before we can consider it a hardened server

Transactions and checking scripts used for transactions. A typical critical operation for a blockchain node operation is running scripts for transaction. This is the moment that founds can be moved and should have extra scrutiny. A tip here can be to audit the code of the script and then make sure that you are warned of any difference in the script. To detect a change in a script folder you can do the following:

find /directoryto/check -type f -exec md5sum {} \; > checklist.chk

If you download new scripts audit then and repeat the output of the checklist as above. Then when you are running the script make sure to check the md5sum:

md5sum -c checklist.chk

Here you create a file checklist.chk with the md5sum values of all the files in the directory and then you can run md5sum -c checklist.chk

Keys security. Any keys you have used for transactions or node operations should ideally be stored on an air gapped computer. (Not at all connected to the internet and files are transfered on USB that are also checked before transfering) and preferably also secured by a hardware device as well. Ideally you also have run the scripts by hand and created the transactions yourself before you run any such scripts off other users so you know what is going on and can audit the code. You also should considerd the 5$ wrench attack and for example seperate seed words for hardware devices in such a way that no single location have all the seed phrases and if possible also set up a multi hw account where more than one person has to sign before a transaction is submitted.

Now with a security audit, a plan for server security and some thinking on the security of your keys you have a good platform to evolve your security and the key to evolving it is to have a security fist mindset as well as being critical to any operations done and thinking what could go wrong and how could an attacker use this step. List down every operation you do and consider these angles. And again this is not ment as a full list of security measures but more as an introduction to security aspects of being a node operator. With that said lets move on to a few optimization tips:

Use Chrony for timesynching with Network Time Protcol. Install with:

sudo apt-get install chrony -y

Edidt the chrony conf file with your favorite text editor such as nano or nice editor with sudo privleges to /etc/chrony/chrony.conf. Here is an example from a guide by ILAP for Shelley testnet:

# 3 sources per time servers.
pool ntp.ubuntu.com        iburst maxsources 3
pool time.nist.gov         iburst maxsources 3
pool us.pool.ntp.org       iburst maxsources 3

keyfile /etc/chrony/chrony.keys

driftfile /var/lib/chrony/chrony.drift

logdir /var/log/chrony

maxupdateskew 10.0

rtcsync

# Make steps in 100ms.
makestep 0.1 3

Make sure to restart chrony to get the configuration in effect. Also the pool list you should change to ntp servers that have good latency to your node. Here is an example of a list of NTP servers to check for latency. Ideally you could also get a local timeserver, you can find these for example on Ebay or sites such as Time Machines or if adventures you can build your own.

You can check statistics and activity of your NTP sources with:

chronyc sourcestats -v
chronyc activity
chronyc tracking

Get Tuned to automate some of the tuning of your Ubuntu server. Tuned can adapt the operation system to better perform under certain workloads such as node operations that require network-latency performance. You can install with:

sudo apt install tuned tuned-utils tuned-utils-systemtap
sudo systemctl enable --now tuned

By default profile is balanced and you can check with tuned-adm profiles. in this example we set it tuned to network-latency at the cost of increased power consumption:

tuned-adm active
tuned-adm profile
sudo tuned-adm profile network-latency

Tweak the sysctl config file to use Google’s congestion control algorithm, add security measures and tune performance. More sysctl settings can be found for example here and here. With your favorite editor edit /etc/sysctl.conf and add:

# Use Google's congestion control algorithm
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Then reload sysctl config:
sudo sysctl -p /etc/sysctl.conf

While we are at it let us add some security helping with syn flood attacks, reducing attack surface with no ip source rooting, ip spoofing protection, no ICMP redirects, log all spoofed, routed and redirected packets, no proxy ARP, and much more:

#Prevent SYN attack, enable SYNcookies (they will kick-in when the #max_syn_backlog reached)
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_syn_backlog = 4096
# Disables IP source routing
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Add if ip6 used:
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
# Enable Log Spoofed Packets, Source Routed Packets, Redirect #Packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
# Don't relay bootp
net.ipv4.conf.all.bootp_relay = 0
# Don't proxy arp for anyone
net.ipv4.conf.all.proxy_arp = 0
# Turn on the tcp_timestamps, accurate timestamp make TCP congestion # control algorithms work better
net.ipv4.tcp_timestamps = 1
# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1 
# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Enable a fix for RFC1337 - time-wait assassination hazards in TCP
net.ipv4.tcp_rfc1337 = 1
# Accept packets with SRR option? No
net.ipv4.conf.all.accept_source_route = 0
# Ignore all ICMP ECHO and TIMESTAMP requests sent to it via 
# broadcast/multicast
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Ignore bad ICMP errors
net.ipv4.icmp_ignore_bogus_error_responses=1

Then reload sysctl config:
sudo sysctl -p /etc/sysctl.conf

Bonus: Using the Grafanalabs tutorial as a basis for adding monitoring of the server with Prometheus, Node Exporter, and Docker Compose from the sshlog dashboard in the previous SSH tutorial found here.

Edit the ssh docker-compose file you created in the SSH tutorial and add the following under services:

  #----------------------------------------------#
  # Prometheus : monitoring & time series db
  #----------------------------------------------#
prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    ports:
      - 9090:9090
  #----------------------------------------------#
  # Node-exporter : Monitoring host metrics
  #----------------------------------------------#
node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    ports:
      - 9100:9100

Under volumes add :

prometheus_data:

Finished it should look like this

Add a prometheus.yml with the following:

global:
  scrape_interval: 1m
scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 1m
    static_configs:
    - targets: ["localhost:9090"]
- job_name: "node"
    static_configs:
    - targets: ["#NAME_YOUR_NODE_EXPORTER_CONTAINER:9100"]

Now all you have to do is start up the docker containers with docker-compose up as in the SSH tutorial.

Make sure to rename to whatever container you use in my case it is sshlog_node-exporter_1. you can find this with docker ps

Next in Grafana import the datasource under add source prometheus:

Make sure to change the prometheus_container name to the correct one. Save & test it.

Now you can import a dashboard for hardware monitoring. Lets import a Full Node Dashboard by its grafana.com id in this case 1860 :

Select Prometheus as the datasource. And there you have it a full node dashboard with plenty of usefull metrics to tweak around with and maybe create alerts to your telegram or slack to:

If this guide was usefull and you hold ADA in the Cardano blockchain, I Hope you will consider delegating to the staking pool I run for NBX called ADA North Pool, with ticker ANP. See this guide for how to delegate.