Virtual Cardano part 2. Guide on adding peers to topology and further monitoring of relay.

This is a continuation of the previous article Virtual Cardano. Guide on setting up Cardano relay node on a Proxmox Virtual machine. If you followed those steps you would have an up and running Cardano relay node on your virtual machine inside the Proxmox host.

If you like this article consider delegating to ADA North Pool Cardano staking pool.

The next natural step to take is to fix more peers for this node. Lets start with the topology file:

Currently only contacting the IOHK official relays and in this case 2 relays from an array of relays (the valency option).

So what do you do to get more relays if you are not part of any alliance of staking pools sharing relays? Luckily the Guild Operators have a great script to solve this problem. The topology updater for the time being fetches peers until Cardano releases its Peer2Peer network (in testing) where this is handled automatically.

In your relays scripting directory follow the steps from the guild page to grab the topology updater: (make sure you have jq installed as well as it is a requirement).

curl -s -o topologyUpdater.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/topologyUpdater.sh 
curl -s -o env https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/env 
chmod 750 topologyUpdater.sh

Now you need to edit the env variable. Here is an example of edits on my anpserver4–1 relay1:

Add your own relays port number under your port number.

You will also need to edit the topologyUpdater.sh script:

Add your own IP or DNS. Please use services that will help you migate denial of service attacks as these IP/DNS can be found.

Tip to remember here is the more outgoing connections you have the more memory limted you will be as well as a slight latency penalty. However you also gain in propagation of blocks in the network. An ideal setup could have many relays with not too many connections on each individual relay but also connected to the otehr relays. (5 relays with 10 peers + internal relays for example).

Now we need to setup a cronjob so the script will run automatically. edit with crontab -e. In my case I run two scripts to update two relay nodes on the computer:

Every hour on the 25th minute mark you will be doing an update to the topology files with this cronjob.

Now we just need to verify that we have a new peers in the topology file and restart the nodes. To not corrupt the database it is recommended to stop the relay first then restart it as this gives the node proper chance to shut down in a safe way. Your topology file should reflect the new peers after around 3 hours so give the cronjob a chance to report and check back on your node after that timeperiod if it has updated the topology file.

Next lets get some proper monitoring data from our relays. There are many guides on this and you can choose between for example installing it to your ubuntu distro or running it as a container. In our case we will try to keep the host clean of docker as there are some bugs with firewall usage of docker if you are not careful — so for simplicity we will install it directly to the OS since this is a cardano relay node. This follows the same steps as the coincashew guide but there are plenty of other guides out there with the same steps.

Grab prometheus and node exporter (will export hardware data)

sudo apt-get install -y prometheus prometheus-node-exporter

Get the keys for grafana to be added to source list of software, update and install grafana then enable the services:

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" > grafana.list
sudo mv grafana.list /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana
sudo systemctl enable grafana-server.service
sudo systemctl enable prometheus.service
sudo systemctl enable prometheus-node-exporter.service

Edit the /etc/prometheus/prometheus.yml and add your relays: (in my case the prometheus port of our node is on 12788 and 12789 you can edit this in the configuration file for the node relay. The 9100 is the local host machine node exporting information about the computer such as hardware utilization of ram and cpu.

Now restart the services:

sudo systemctl restart grafana-server.service
sudo systemctl restart prometheus.service
sudo systemctl restart prometheus-node-exporter.service

Grafana is hosted on localhost port 3000 so you can forward this for example in Putty to get to the administration. Default user and pw is admin and you change this after the inital login.

forwarding both the prometheus data as well as the localhost grafana port.

After login in you need to connect grafana to the prometheus data so select add your first datasource:

Select Prometheus and make sure its getting data from http://localhost:9090

You can now import a dashboard. Coincashew has one made you can download in the guide they have. Other options is the IOHK cardano-ops dashboard: https://github.com/input-output-hk/cardano-ops/blob/master/modules/grafana/cardano/cardano-application-dashboard-v2.json and you will end up with something like this if you use the choincashew dashboard: (you can also just customize this yourself its fairly easy once you get the hang of Grafana to customize the different graphs).

Some data will not be shown if you are not a block producer and can be cancelled such as kes rotation dates.

Now lets do a final tweaking for the relay node by edditing its logging options. Cardano nodes has an amazing amount of logging options. See for example the 180+ pages pdf you can find on the iohk-monitoring github. https://github.com/input-output-hk/iohk-monitoring-framework

With that said not all monitoring is important as a relay node and you also want to optmize performance of the node by turning off logging when it is not needed. Ideally you could also have a relay that did the logging to a temporary ram drive that you copied data from on a regular basis. As cardano foundations guide on logging also suggests too high levels of debugging can harm efficiency. https://cardano-foundation.gitbook.io/stake-pool-course/stake-pool-guide/logging-monitoring/logging and the logging guidelines gives some good idea of the differente levels of logging. https://input-output-hk.github.io/cardano-wallet/contributing/Logging-Guidelines. A relay that is not a block producing node might not need to trace the mempool and transactions as well for example and this could be turned off with tracemempool:false, TraceTxInbound: false, TraceTxOutbound:false, TraceTxSubmissionProtcol: false. We also might want to keep the verbosity of messages to a minimal (MinimalVerbosity).

Just restart the node and it should have the new logging settings in effect.

With that you have a solid foundation for contributing to the network with relay nodes on the Cardano network. Some of these principles such as using Prometheus with Grafana monitoring is skills you can utilize for other node operations as well. If you like these types of articles or have suggestions for other types of such articles feel free to suggest it in the comment sections.