pdbtool stylists wanted

I have just pushed to my syslog-ng 3.1 git repo some patches to add more detailed debugging/troubleshooting capabilities to patterndb through pdbtool match. Now you can easily see how your patterns matched a given message, which part of the message was matched by literal part of the pattern and which by a parser. All this new feature with a wonderful colorized output if requested, though the colors are still bit ugly...

Up to now the easiest way of creating patterns was to do trial & error sequence and try to figure out which part of the pattern was broken. Now you can exactly see where the matching stopped. As one picture is more verbose than hundreds of words, here is some screenshoot of pdbtool match.

The patterndb.xml in use:













A successful match:













A failed match:






It is also possible to output the match in parse-able format to be used in scripts or as a backend of some pattern authoring tool. Here is the output for that:









I am still not sure about the colored output nor about the machine parse-able output format, so any feedback, comment, idea or suggestion is very much appreciated.

You can grab the latest source code from my git repo.

Happy matching!

SCB 2.0.2 and SSB 1.1.0.a releases

It is release time again!

For syslog-ng Store Box it is a sad news as the 1.1.0 release did contain some small, but annoying bugs so we addressed them in an update release which is available from our website. Please read the announcement before upgrading and please also note that downgrading to 1.0 is not possible. Also it is good idea to try to import your ssb 1.0 configuration into an ssb 1.1 vmware to check if there is no XML upgrade problem. See the release notes for details:
https://www.balabit.com/downchangelog.bbx?cl=/downloads/ssb/1.1.0.a/changelog-en.txt

The good news is on the other hand for Shell Control Box. I have just sent out the announcement of the 2.0.2 maintenance release where we tried to address two blocker upgrade related bugs with also more faster and verbose upgrade process. We also fixed some bugs around ssh and x11 protocol handling and fixed some non-critical security issues in the OS packages so it makes sense to upgrade. Release notes:
https://www.balabit.com/downchangelog.bbx?cl=/downloads/scb/2.0.2/changelog-en.txt

If you are using RDP traffic please note that WindowsXP SP3 clients and Windows Server 2008 R2 editions are not working through SCB. We are aware of this problem and working on a fix to address the problem.

Pattern Database first snapshot available

Last week BalaBit made available some 8000 patterns (covering more than 200 applications) for syslog-ng patterndb (or db_parser as you like to call it). The patterns are available under the Creative Commons
Attribution-Noncommercial-Share Alike 3.0 (CC by-NC-SA) license. The patterns in their current form are just snapshots of the ongoing effort of providing good quality patterns for various applications. You can download the snapshot of patterns from our website: http://www.balabit.com/downloads/files/patterndb-snapshot/patterndb-20091209.zip

The patterns are partially hand-crafted and also automatically generated from logfiles and from logcheck regexp based database. Some of the patterns also contains example messages which we are using to automatically test the pattern and syslog-ng's db_parser. You can merge the xml files using "pdbtool merge".

I would also like to setup a public git repository where anyone interested can follow the patterndb development and can submit patterns or fixes. A patterndb website containing all patterndb related information, links, forums, wikis and other useful documentations is under construction as well. Till than the syslog-ng mailing list a good place for questions, ideas and discussions.

As always feedbacks are very welcomed!

Happy parsing!

syslog-ng Store Box 1.1 released

I have just sent out the announcement of SSB 1.1. This is the first feature release that we have released so far. (Feature releases have shorter development and also support period, while they introduce new features earlier for interested parties. See more details on our new release structure.)

For detailed description on new features and changes see the "What is new in SSB 1.1" document. For the inpatients here is the highlight of the most important changes:

High Availability improvements

SSB is able to use the production interfaces (external/management) as redundant heartbeat links between the nodes of the SSB cluster. This will prevent split-brain scenarios in case the primary HA link fails.

SSB is able to monitor the next-hop routers from the nodes and trigger takeovers if the monitored routers become unaccessible from the master node, while available from slave node.


Enhanced reporting and statistics

It is possible to display the number of collected log messages as bar or pie charts in the reports and on the dashboards. Reports and statistics include charts on Top Talkers, Top Host names, Top Programs and others.

Users are also able to create and customize periodic reports to include charts and lists from syslog-ng statistics, as well as statistics on the collected log messages (including messages stored in SQL databases and the indexed logspaces).


Searching and indexing

Users have now the ability to create and save reusable filters on the Search pages. Permissions can be assigned to global filters to grant access to certain log messages for users who have no SSB access otherwise.

Users can now display the distribution of the log messages based on certain parameters (like sender address, hostname, program, facility etc.) as graphical charts. The charts can be included in custom periodic reports as well to provide more details on log messages.

Search queries on logspaces could be restricted to search only certain columns, but not in all part of the message.

It is possible to index and search encrypted logspaces. Decryption keys can be assigned to logspaces or users can upload keys to their personal (passphrase protected) key-stores.


Message classification extensions

SSB 1.1 supports version 3.1 of the pattern database format and functionality, including full tagging and value assignment support. The tags and name/value pairs assigned to log messages can be used in logpath filters, and also stored and searched later.


Besides introducing new features we have somewhat changed our internal development processes. We started using new development methodology: scrum and tried to use more automatic test systems in our Q&A process besides manual testing. We hope that these changes will lead to improved quality and to releases much more on time.

As always feedbacks and suggestions are always welcome.

Enjoy! :)

"Kispál és a Borz" again

We were at the Kispál és a Borz (a cult alternative Hungarian rock band) concert yesterday at Pecsa. I haven't been to Kispál for ages and now that I went again I must admit I really enjoyed it. They played some new songs which I did not know, but most songs I did remember. Little nostalgia...