pmacct is a small set of passive network monitoring tools to account, filter, classify, aggregate, and export IPv4 and IPv6 traffic. A pluggable and flexible architecture allows storing collected network data in memory tables, RDBMSs (MySQL, SQLite, PostgreSQL, BDB), and flat files, and also export via IPFIX, NetFlow, or sFlow protocols to remote collectors. pmacct features fully customizable historical data breakdown, sampling, BGP correlation, tagging, and triggers. Libpcap, ULOG, sFlow v2/v4/v5, NetFlow v1/v5/v7/v8/v9, and IPFIX are supported data capturing methods.
GoAccess is a real-time Web log analyzer and interactive viewer for almost every Web server. It runs in a terminal and provides fast and valuable HTTP statistics for system administrators that require log monitoring and visual reports on the fly. GoAccess can monitor unique visitors, browsers, spiders, OS, hosts and IP geolocation, keyphrases, referring sites, status codes, etc. It has support for IPv6 and it parses nearly any Web log format.
Fido is a multi-threaded file watcher which searches files in real time for user-defined patterns. When it locates a match, it runs a user-defined program. It is useful for monitoring log files for issues and responding to them. It was designed to recognize log file rotation and start monitoring from the beginning of the new file.
Squid Analyzer parses the native access log format of the Squid proxy and reports general statistics about hits, bytes, users, networks, top URLs, and top second level domains. Statistic reports are oriented toward user and bandwidth control; this is not a pure cache statistics generator.
grepcidr can be used to filter a list of IP addresses against one or more Classless Inter-Domain Routing (CIDR) specifications, or arbitrary networks specified by an address range. As with grep, there are options to invert matching and load patterns from a file. grepcidr is capable of comparing thousands or even millions of IPs to networks with little memory usage and in reasonable computation time. It has endless uses in network software, including mail filtering and processing, network security, log analysis, and many custom applications.
360-FAAR (Firewall Analysis Audit and Repair) is an offline, command line, Perl firewall policy manipulation tool to filter, compare to logs, merge, translate, and output firewall commands for new policies, in Checkpoint dbedit, Cisco ASA, or ScreenOS commands. It is all contained in one file. It can read policy and logs for: Checkpoint FW1 (in odumper.csv / logexport format), Netscreen ScreenOS (in get config / syslog format), and Cisco ASA (show run / syslog format). It uses both inclusive and exclusive CIDR and text filters, permitting you to split large policies into smaller ones for virutalization at the same time as removing unused connectivity. It supports policy to log association, object translation, rulebase reordering and simplification, rule moves, and duplicate matching automatically. It allows you to seamlessly move rules to where you need them. 'print' mode creates a spreadsheet for your audit needs with one command.
Free-SA is tool for statistical analysis of daemons' log files, similar to SARG. Its main advantages over SARG are much better speed (7x-20x), more support for reports, and W3C compliance of generated HTML/CSS reports. It can be used to help control traffic usage, to control Internet access security policies, to investigate security incidents, to evaluate server efficiency, and to detect troubles with configuration.
superseriousstats is a small and efficient program for creating a Web page with statistics from various types of IRC logs. It keeps track of its parse history and only processes new activity before storing any accumulated data in a SQLite or MySQL database. It is suitable for high volume IRC channels and large log archives, and is relatively easy to integrate with IRC services (e.g. bots) that interact with the database and provide last seen information and many other statistics directly in your channel.
nxlog is a modular, multi-threaded, high-performance log management solution with multi-platform support. In concept, it is similar to syslog-ng or rsyslog, but is not limited to Unix/syslog only. It can collect logs from files in various formats, receive logs from the network remotely over UDP, TCP, or TLS/SSL on all supported platforms. It supports platform-specific sources such as the Windows Eventlog, Linux kernel logs, Android device logs, local syslog, etc. Writing and reading logs to/from databases is also supported for many database servers. The collected logs can be stored into files, databases, or forwarded to a remote log server using various protocols. The old BSD Syslog and the newer IETF syslog standard (RFC 3164 and RFC 5424-5426) are fully supported by nxlog in addition to XML, JSON, CSV, GELF, and other custom formats. A key concept in nxlog is to be able to handle and preserve structured logs so there is no need to convert everything to syslog and then parse these logs again at the other side. It has powerful message filtering, log rewrite, and conversion capabilities. Using a lightweight, modular, and multi-threaded architecture which can scale, nxlog can process hundreds of thousands of events per second.