Rsyslog Configuration Tutorial

This week I was tasked with setting up a central logging server for our Elasticsearch cluster to monitor slow queries. The service that consumes our Elasticsearch cluster has a timeout of 1 second so we wanted a way to monitor timeouts in realtime and investigate the underlying nodes with increased error rates. In our cloud these were normally due to noisy neighbors on the same physical host and we could just terminate and rebuild the node in question to bring it back to full speed. To start I decided to turn to good old rsyslog to centralize the log data, logstash for parsing logs, elasticsearch for searching over the logs, and kibana for the visualizations.

This is the first step in a 3 part series to get our log data in to kibana to visualize. Getting rsyslog to cooperate though was a grueling task due to a high amount of bad and outdated documentation on google. Below I’ll go through the process I chose to use to get it up and running and hopefully it’ll be helpful to you. I’ll start with the rsyslog client (nodes sending logs) and then move to the server.

Installing Rsyslog

First unless you have a good reason to use a previous version you should use the most up to date and stable version of rsyslog. If you’re using an old version of Ubuntu it is most likely bundled with an older version of rsyslog. In particular version 5.x uses an old syntax which is outdated and won’t work on newer versions. Version 7.x I had issues with the configuration files not reloading properly but with the latest version I’ve had few to no issues.

If you see the below type syntax on a blog or in the documentation this is the old syntax they used to use and you shouldn’t follow it. This bit me multiple times.

Ubuntu 14.04 and beyond defaults to rsyslog 8.4.4 currently so you won’t need to add the apt repo like I did below. Unfortunately my cluster had a mix of 12.04 as well so I had to add the apt repo to be sure.

Configuring the Client Node

Now that we have 8.4.4 installed we need to configure rsyslog to monitor the log files. Below I have a sample configuration of what I used to monitor Elasticsearch’s slow search query log.

1. The first line just loads the imfile module which contains the code for monitoring a file. It basically stores metadata about what lines have already been sent to the remote server.
2. The second line defines an input with the details. The type specifies what kind of input it is.
3. The third line is the location of the file to monitor.
4. The fourth line is what will be used as a prefix on the central monitoring server. ie If my message is “This query is slow” and I send this output to X.X.X.X then on X.X.X.X you should see NODE_NAME-search-slow: This query is slow.
5. The state file records what position in the log it has already read. IMPORTANT: This location must be unique. Really weird things can happen if 2 monitored files write to the same state file. Also if the log file is editted in anyway you should clear the state file info. By default it should be written to /var/spool/STATE_FILE. The $WorkDirectory statement in /etc/rsyslog.conf sets where the state files are stored in case you want to change it.
6. Lastly this specifies which inputs to send and where. . means send all inputs, @ denotes that I want to use UDP or if you use @@ it uses tcp, X.X.X.X:514 is the normal IP and port settings.

This just shows an example of monitoring a single file but you can define multiple files to be monitored in the same configuration. After I write the configuration file I’ll place it in /etc/rsyslog.d/elasticsearch.conf. By default rsyslog will read all configuration files in /etc/rsyslog.d/ so you just need to place it in there and it should work.

You should always restart the service to pick up any new configuration changes. You can also verify the configuration is correct with rsyslogd -N1. This will just parse all the configuration files and return any errors.

Troubleshooting the Client

If your client isn’t sending the data it should be you can add the below to your /etc/rsyslog.conf and then restart it. This will write debug info into /tmp/rsyslog.debug and should show you what it’s monitoring and which messages it’s sending.

Configuring the Server Node

The server node is much easier to set up. Edit /etc/rsyslog.conf on the server which should be the same X.X.X.X ip as above. Uncomment either the UDP or TCP module depending on what you set above. I did UDP with a single @ so I uncommented that one. 514 is the port it is listening on.

The next part is just a personal preference but I partitioned the logs on the central server by hostname. Here we specify the file name will depend on the hostname variable of the syslog message. So if the host sending the log message is spiderman the output from that client would be written to /var/log/spiderman/syslog.log. The last line just says to use this template FILENAME for all logs.

Restart the service.

Now you should start seeing logs under /var/log/ by hostname. If you don’t you can use the same debug tip from above to try and debug the server as well.

Hope this helps others and I’ll post the next steps I took to get the data into elasticsearch and using the kibana dashboard to monitor queries.