Projects / Logger

Logger

Logger is a log queueing and optimization utility. It can completely replace sys/klogd, handle logs larger than 2GB, perform automatic nightly log rotation without affecting services, and transfer logs between machines via TCP connections. Any application that writes a text log can be maintained by Logger. It is light, small, and fast, emphasizes security, and reduces drive usage through memory queueing.

Tags
Licenses
Operating Systems
Implementation

RSS Recent releases

  •  26 Dec 2008 11:14

Release Notes: A permissions bug when creating the named pipes was fixed.

  •  27 Oct 2008 20:00

Release Notes: This critical upgrade addresses the issue with the random segfaults. The TCP connections are now UDP. This change has been made in such a way that the config file doesn't need to be changed. If you want to, though, simply replace TCP with UDP. The segfaults actually occurred due to the TCP connection stack running out of space. Switching to UDP is the best approach until something like T/TCP or similar is available in Linux.

  •  25 Sep 2008 21:06

Release Notes: A critical but extremely hard to reproduce bug has been fixed. The bug centered around the situation of one or more receivers not responding, causing the transmission to continue to consume memory until it segfaulted (exhausted all memory). The decision-making algorithm for this was rewritten to avoid the problem.

  •  18 Sep 2008 21:02

Release Notes: A critical bugfix to add a time wait feature for polling downed Logger servers. This prevents possible denial of service of the polling system. This also prevents all TCP sockets on the pollibf system from being consumed by the polling system.

  •  05 Jul 2008 20:31

Release Notes: This version has proven extremely stable and handles connection interrupts properly.

RSS Recent comments

04 Mar 2008 11:19 Avatar BpMatthias

Linux Gazette

Hello,

I have an article in this month's Linux Gazette on setting up Logger at:

Linux Gazete (linuxgazette.net/148/d...)

Its a simple HowTo guide. Please check it out and let me know what you think.

22 Feb 2008 14:27 Avatar BpMatthias

LogPipe utility
LogPipe is a simple utility included with Logger. It allows single shot pipes

to take advantage of memory queueing techniques for reduced I/O.

The COMPILE script automatically compiles LogPipe. It can be copied into

/usr/local/bin with the following:

cp LogPipe /usr/local/bin

Usage is as follows:

LogPipe <number of queued lines> <file name>

The number of queued lines is the maximum number for LogPipe to hold into

memory before writing to <file>

Example:

tcpdump -q -n | LogPipe 100000 tcpdump.log

LogPipe must be at the end of the command chain. It does not do any automatic

rotation or such.

30 Jan 2008 14:07 Avatar BpMatthias

Replacing sys/klogd
Here is an example config for replacing sys/klogd.

### Output Queues - MUST be listed first

Queue Klog 0 0 0600 1 /var/log/Klog.log

Queue Slog 0 0 0600 25 /var/log/Syslog.log

### Sys/KLog Entries

Kernel Sys1 Klog

SysLog Sys1 Slog

Sys1 is a prefix and can be a * for no prefix. I perfer to put the machine name for the prefix.

I also use this method on my laptop to keep all logging on my desktop gateway. This config lines are very similar to the above:

Queue Klog 0 0 0600 1 @10.100.0.1:4000

Queue Slog 0 0 0600 25 10.100.0.1:4001

### Sys/KLog Entries

Kernel Sys1 Klog

SysLog Sys1 Slog

Be sure you put the tcp command in your desktop Logger.conf that matches your system's setup.

Also, please backup your system startup files before you make any changes to the boot scripts. Always test first....

21 Dec 2007 14:32 Avatar BpMatthias

Re: Logger and a web server cluster
It should be noted that this example could easily be applied to Exim, SendMail, Squid, Tiny Proxy, WU Ftp, ProFTP or any log writing software used in a clusterd or "farm" environment.

21 Dec 2007 14:27 Avatar BpMatthias

Re: Logger and a web server cluster
A typo that I missed, the section regarding Logger.conf on the web servers should read:

For each of the four web servers, the Logger.conf would be:

### Output Queues - MUST be listed first

Queue ApacheLog 0 0 0644 1 @10.100.0.0:14721

### Apache Entries

Pipe /tmp/Mounts/tmpfs/ApacheLog * ApacheLog

Screenshot

Project Spotlight

GeneaPro

Genealogy software based on the GenTech Genealogical Data Model.

Screenshot

Project Spotlight

Webconverger

A Web kiosk Linux distribution.