Articles / SSL Encrypting Syslog with …

SSL Encrypting Syslog with Stunnel

In this article, I describe how to encrypt syslog messages on a network. Encryption is vital to keeping the confidential content of syslog messages secure. I describe the overall approach and provide a HOWTO to do it with the help of rsyslogd and stunnel.

Background

Syslog is a clear-text protocol. Anyone with a sniffer can have a peek at your data. In some environments, this is no problem at all. In others, it is a huge setback, probably even preventing deployment of syslog solutions. Thankfully, there is an easy way to encrypt syslog communication. I will describe one approach in this article.

The most straightforward solution would be to take advantage of the fact that syslogd itself encrypts messages. Unfortunately, while its encryption is standardized in RFC 3195, there is currently no syslogd that implements RFC 3195's encryption features, so this route leads to nothing. Another approach would be to use vendor- or project-specific syslog extensions. There are a few around, but the problem here is that they have compatibility issues.

However, there is one surprisingly easy and interoperable solution: though not standardized, many vendors and projects implement plain TCP syslog. In a nutshell, plain TCP syslog is a mode in which standard syslog messages are transmitted via TCP and records are separated by newline characters. This mode is supported by all major syslogds (both on Linux/Unix and Windows) as well as log sources (for example, EventReporter for Windows Event Log forwarding). Plain TCP syslog offers reliability, but it does not offer encryption in itself. However, since it operates on a TCP stream, it is now easy to add encryption. There are various ways to do that. In this article, I will describe how it is done with stunnel. (Another alternative would be IPSec, for example).

Stunnel is Open Source and is available both for Unix/Linux and Windows. It provides a way to use SSL communication for any non-SSL-aware client and server, in this case, our syslogd.

Stunnel works much like a wrapper. Both on the client and on the server machine, tunnel portals are created. The non-SSL-aware client and server software is configured not to talk directly to the remote partner, but to the local (s)tunnel portal instead. Stunnel, in turn, takes the data received from the client, encrypts it via SSL, and sends it to the remote tunnel portal, and that remote portal sends it to the recipient process on the remote machine. The transfer to the portals is done via unencrypted communication, so it is vital that the portal and the program that is talking to it are on the same machine; otherwise, data would travel partly unencrypted. Tunneling, as done by stunnel, requires connection-oriented communication. This is why you need to use TCP-based syslog. (As a side note, you can also encrypt a plain-text RFC 3195 session via stunnel, though this definitely is not what the protocol designers had in mind.)

In the rest of this article, I assume that you use my project rsyslog on both the client and the server. For the examples, I use Debian. Interestingly, there are some annoying differences between stunnel implementations. For example, on Debian, a comment line starts with a semicolon (";"). On Red Hat, it starts with a hash sign ("#"). You need to watch for subtle issues when setting up your system.

Overall System Setup

I assume two machines, one named "client" and the other named "server". In practice, you will probably have multiple clients but only one server. Syslog traffic will be transmitted via stunnel over the network. Port 60514 is used for that purpose. The machines are set up as follows:

Client

  • rsyslog forwards messages to the stunnel local portal at port 61514.
  • The local stunnel forwards data via the network to port 60514 to its remote peer.

Server

  • stunnel listens on port 60514 to connections from its client peers.
  • All connections are forwarded to the locally-running rsyslog listening at port 61514.

Setting up the system

For Debian, you need the "stunnel4" package. The "stunnel" package is the older 3.x release, which will not support the configuration I describe below. Other distributions might have other names. For example, on Red Hat it is just "stunnel". Make sure that you install the appropriate package on both the client and the server. It is also a good idea to check whether there are updates for either stunnel or OpenSSL (which stunnel uses). There are often security fixes available, and often the latest fixes are not included in the default package.

In my sample setup, I use only the bare minimum of options. For example, I do not make the server check client certificates. Also, I do not talk much about certificates at all. If you intend to really secure your system, you should probably learn about certificates and how to manage and deploy them. This is beyond the scope of this article. For additional information, http://www.stunnel.org/faq/certs.html is a good starting point.

You also need to install rsyslogd on both machines. Do this before starting with the configuration. You should familiarize yourself with its configuration file syntax, so that you know which actions you can trigger with it. Rsyslogd can work as a drop-in replacement for stock sysklogd, so if you know the standard syslog.conf syntax, you do not need to learn any more to follow this article.

Server Setup

On the server, you need to have a digital certificate. The certificate enables SSL operation, as it provides the necessary cryptographic keys to secure the connection. Many versions of stunnel come with a default certificate, often found in /etc/stunnel/stunnel.pem. If you have it, it is good for testing only. If you use it in production, it is very easy to break into your secure channel, as everybody is able to get your private key. I didn't find an stunnel.pem on my Debian machine. I guess the Debian folks removed it because of its insecurity.

You can create your own certificate with a simple OpenSSL tool. You need to do it if you have none, and I highly recommend that you create one in any case. To create it, cd to /etc/stunnel and type:

openssl req -new -x509 -days 3650 -nodes -out stunnel.pem -keyout stunnel.pem

That command will ask you a number of questions. If you are unsure what to answer, read http://www.stunnel.org/faq/certs.html. After the command has finished, you should have a usable stunnel.pem in your working directory.

Next, create a configuration file for stunnel. You can used the following basic file:

; Certificate/key is needed in server mode
      cert = /etc/stunnel/stunnel.pem
      
      ; Some debugging stuff useful for troubleshooting
	debug = 7
      foreground=yes
      
      [ssyslog]
      accept  = 60514
    connect = 61514

Save this file to, for example, /etc/stunnel/syslog-server.conf. Please note that the settings in italics are for debugging only. They run stunnel with a lot of debug information in the foreground. This is very valuable while you set up the system and very useless once everything works well. Be sure to remove these lines when going to production.

Finally, you need to start the stunnel daemon. Under Debian, this is done via "stunnel4 /etc/stunnel/syslog.server.conf". If you have enabled the debug settings, you will immediately see a lot of nice messages.

Now, configure rsyslog to do everything you want. If in doubt, you can simply copy /etc/syslog.conf to /etc/rsyslog.conf, and you probably have what you want. The really important thing in rsyslogd configuration is that you must make it listen to TCP port 61514 (remember, this is where stunnel sends the messages). Thankfully, this is easy to achieve: Just add "-t 61514" to the rsyslogd startup options in your system startup script. Then start (or restart) rsyslogd.

The server should now be fully operational.

Client Setup

The client setup is simpler. Most importantly, you do not need a certificate (of course, you can use one if you would like to authenticate the client, but this is beyond the scope of this article). The main thing you need to do is create the stunnel configuration file.

; Some debugging stuff useful for troubleshooting
	debug = 7
      foreground=yes
      
      client=yes
      
      [ssyslog]
      accept  = 127.0.0.1:61514
      connect = 192.0.2.1:60514
    

Again, the text in italics is for debugging purposes only. I suggest you leave it in during your initial testing and then remove it. The most important difference from the server configuration outlined above is the "client=yes" directive. It is what makes this stunnel behave like a client. The "accept" directive binds stunnel only to the local host, so it is protected from receiving messages from the network (somebody might fake being the local sender). The address "192.0.2.1" is the address of the server machine. You must change it to match your configuration. Save this file to /etc/stunnel/syslog-client.conf.

Start stunnel via "stunnel4 /etc/stunnel/syslog-client.conf". You should see some startup messages. If no errors appear, you have a running client stunnel instance.

Finally, you need to tell rsyslogd to send data to the remote host. In stock syslogd, you do this via the "@host" forwarding directive. The same works with rsyslog, but it supports extensions to use TCP. Add the following line to your /etc/rsyslog.conf:

*.*      @@127.0.0.1:61514

Please note the double "at" signs (@@). This is not a typo. It tells rsyslog to use TCP instead of UDP delivery. In this example, all messages are forwarded to the remote host. Obviously, you may want to limit this via the usual rsyslog.conf settings (if in doubt, man rsyslog.conf).

You do not need to add any special startup settings to rsyslog on the client. Start or restart rsyslog so the new configuration settings take place.

Done!

After following these steps, you should have a working secure syslog forwarding system. To verify it, you can type "logger test" or a similar command on the client. It should show up in the respective server log file. If you dig out your sniffer, you should see that the traffic on the wire is protected. In the configuration used above, the two stunnel endpoints should be quite chatty, so you can follow the action on your system.

If you have only basic security needs, you can probably just remove the debug settings and take the rest of the configuration to production. If you are security-sensitive, you should have a look at the various stunnel settings that help you further secure the system.

Preventing Systems from Talking Directly to the rsyslog Server

It is possible for remote systems (or attackers) talk to the rsyslog server by directly connecting to its port 61514. Currently, rsyslog does not offer the ability to bind to the local host only. This feature is planned, but as long as it is missing, rsyslog must be protected via a firewall. This can easily be done via, for example, iptables. Just be sure not to forget it.

Conclusion

With minimal effort, you can set up a secure logging infrastructure employing SSL-encrypted syslog message transmission. You also have the benefit of reliable TCP delivery, which is far less prone to message loss than UDP.

Recent comments

12 Apr 2006 02:35 Avatar sivaaskora9

Re: Reliability vs. Confidentiality

thanks for your explanation , good work indeed and thats working in the
specified mode

18 Feb 2006 03:22 Avatar johnstraws

Re: See also...


>

> % I looked around and it seem that all

> % encrypted connection programs are tcp

> % based.

>

> OpenSSL implements DTLS (Datagram TLS =~

> UDP over SSL) since 0.9.8 (july 2005).

i had followed the links you had given here and they had very much useful information thanks for sharing

26 Jan 2006 01:13 Avatar gg234

nice one
really nice one and very good one

26 Oct 2005 09:08 Avatar rgerhards

Re: Reliability vs. Confidentiality
This is a very good point. Obviously, it is operator-choice. For one, it should be made sure that only those messages be transmitted that actually need to be transmitted. Then, there is the question of what is more important - delivery of every message or the ability to run syslogd with limited ressources. In one approach (in rsyslogd), I have given ressource limitation the priority. It allows you to transmit messages only if the receiver would not block (and thus in short turn slow down or even freeze the system). This sounds like no improvement over UDP based syslog. But it still is much better. For one, you have error-correction in all these cases where you do not deliberatly discard a message. Also, you have the ability to know when things go wrong, whereas you would not notice UDP message loss. So reliable syslog can bring big improvements in in logging health. But I agree, an informed decision must be made. And the software (namely syslogd) must allow to make this decision.

07 Sep 2005 09:09 Avatar fpunit

Reliability vs. Confidentiality
There have been many efforts in providing reliable, confidential transport infrastructure for syslog messages, but they all stumble upon the big problem of dealing with resource contention whenever the syslog server needs to deal with the extra overhead of encrypting/acknowledging all traffic it generates.

Remember: you cannot control how many messages your syslog server will be generating, but you can control the amount of resources it will use. Whenever you increase the cost of sending all these messages, you are also increasing your risk of having a stalled syslog server (instead of one where packets are being lost) because of the associated overhead...

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.