ABOUT
Archive for the ‘Technology’ Category
Libre Software Meeting 2011
Like two years ago I will be giving a talk at the Libre Software Meeting at Strasbourg on the 12nd of July. This I will talk about the latest changes in syslog-ng and especially about the correlation feature added Bazsi recently. If you will be around come and let’s have a chat!
Shell Control Box 3.1 release
The latest Shell Control Box release (3.1) has been out for more than a month now, so it is definitely time to write about the new features it introduces. The development started approximately one year ago and there are for sure some very interesting and useful changes in the release.
The biggest and probably most important news is the support for the Citrix ICA ™ protocol that is used in Citrix’s XenApp ™ servers (formerly Presentation Server). In SCB 3.0 release we introduced support for the VMWare View ™ protocol and with this release we moved forward supporting desktop virtualization environments. With this latest addition Shell Control Box is able to control and audit 6 remote access/administration protocols, such as SSH, RDP, TELNET, TN3270, VNC, ICA covering all the widely used solutions and providing a unified solution for all control and audit purposes.
Support for Citrix ICA protocol is very similar to the current RDP protocol, it is possible to control which channels are permitted, which are denied independently of the server while all passing session capture could be stored in audit trail files (encrypted, signed, timestamped) for the usuall movie-like replay and search latter. Searching the graphical screen is possible with the same OCR based technology that is used in case of RDP or VNC and tailored content based activity reporting is available as well.
SCB 3.1 supports XenApp version 5 and 6, Presentation Server 4.5, Citrix Program Neighborhood ™ and the Citrix Online Client ™ as well. SCB could be deployed both in transparent and non-transparent (bastion) mode, while both simple, reliable and SSL encrypted ICA protocol transports are supported.
As a result of the Citrix ICA protocol support SCB 3.1 is also Citrix Ready verified.
This release also includes support for Terminal Services Gateway (TSGW) technology which allows inband destination selection for RDP protocol. The Microsoft Terminal Services Client (mstsc) can be configured to use SCB as a Terminal Services Gateway to access Terminal Services or Remote Desktops. Thus, the integration of SCB in environments with a large number of RDP servers became much easier in non-transparent or so-called bastion mode.
By using SCB as a TSGW server, inband gateway authentication of the users becomes also much easier for RDP connections. In TSGW mode connection between the client and SCB is established using HTTPS protocol, while connection towards the server is plain RDP. This way proxying client connection through a corporate network or allowing SSL-VPN like connection from externals become an easier integration task into existing environment.
The third major change introduced with this release was to move from a 32bit based system to a completely 64bit based one. As a result of this development, performance of SCB increased and now SCB could fully utilize all the capabilities of the underlying hardware. The upgrade from 32bit to 64bit is seamless, though the upgrade process could take longer than in normal cases.
As always we are eager to hear your feedback on this release or on Shell Control Box in general!
Happy auditing!
Shell Control Box 3.0.0 released
Approximatively one year ago we released SCB 2.0.0 which brought many new features like RDP 6, X11, TN3270 and VNC auditing capabilities besides many improvements all around the product. Now, after one year of development we are very happy to announce the general availability of the latest stable release: SCB 3.0.0
Besides adding some new and real-cool features we tried to stabilize the product even further by fixing bugs and spending way more time on Q&A; processes. This scb release also benefits many of the syslog-ng Store Box development efforts as we have ported small features and many general fixes from SSB.
VMWare View is the desktop virtualization (VDI) and management platform of VMware which utilizes RDP as one of the desktop sharing protocol. Starting with this version SCB officially supports VMWare View environments when used with RDP protocol. All important features of RDP are available like auditing, four-eyes, channel-policies etc.Adding support for VMWare View is a very important step in the development of SCB to extend it’s control & audit capabilities of desktop virtualization and “desktop as managed service” installations as a first step towards supporting cloud environments.
SFTP and SCP file-transfer “playback”
save the transferred files for later auditing purposes.With this new feature both managed file transfer (MFT) and ad-hoc network copies could be monitored in forensics situation or in case of any data leakage incident. An auditor could search for file transfers on the search interface of SCB and download audit-trails to save the actual
transferred content with the audit-player for further inspection.
(external/internal/management) for redundant connectivity besides the dedicated HA interface. This change dramatically improves the robustness of SCB clusters in case of network connectivity failures between the nodes.The HA functionality also gain a new feature to improve the responsiveness of SCB to external network failure situation by monitoring next-hop routers and triggering a takeover in case a
monitored device becomes inaccessible from the master node. With this change the availability of an SCB cluster is increased in cases of network connectivity failures such as switch, NIC, router blackouts.
RDP7 and compression support
New supported platforms
With this release the underlying hardware of SCB has been changed from the Oracle (SUN) Fire series to a new Intel based appliance series. (Support for the old SUN boxes will continue for at least three years from now and hardware upgrade is also available.) The new appliances are
more powerful both in CPU power (with the new Intel 5620 XEON CPUs) and in capacity (4-24GB RAM and 1-10TB disk space) resulting in better performance while providing redundant power supply and being more rack-space efficient by having 10TB effective disk space just in 2U.
Besides the physical platform change, starting from now SCB becomes officially supported on VMWare ESX systems as a virtual appliance with some platform specific limitations. This new form will make SCB more attractive for smaller customers and for virtual, cloud environments.
Search related enhancements
The Search interface of SCB went through a general rework to improve it’s usability. Long rows could be expanded and collapsed, while important columns could be freezed to help browsing audited connections by displaying always the relevant information on the screen.
The Search page has been extended to provide basic top/least statistics and unique lists over the audited channels, such as top source addresses, top usernames, unique list of remote exec commands etc. The information is available as bar- and pie-charts and as simple lists exportable from the web interface.
pdbtool test and pattern database reloaded
I did not have much time recently to work on or blog about patterndb or any other syslog-ng related stuff. Luckily on the syslog-ng mailing list there were some activity and questions regarding db_parser/patterndb and some others stepped up and tried to help. What is even better that Bazsi did spent some time on actually working on putting together a schema for patterndb and even added some messages with proper classification as well. So we have schema and some patterns for user accounting logs, especially for ssh messages. There is a git tree where you can track the progress or send patches:
session closed for user @ANYSTRING:usracct.username:@ session closed for user bazsi bazsi logout $PID $PROGRAM usracct
Testing message program='sshd' message='Failed password for bazsi from 127.0.1.1 port 44637 ssh2'Match name='.classifier.rule_id', value='aecda233-3d80-48cd-a72b-4896f58069c8', expected='aecda233-3d80-48cd-a72b-4896f58069c8'Match name='usracct.username', value='bazsi', expected='bazsi'Match name='usracct.authmethod', value='password', expected='password'Match name='usracct.device', value='127.0.1.1', expected='127.0.1.1'Match name='usracct.service', value='ssh2', expected='ssh2'Testing message program='sshd' message='Accepted password for bazsi from 127.0.0.1 port 48650 ssh2'Match name='.classifier.rule_id', value='4dd5a329-da83-4876-a431-ddcb59c2858c', expected='4dd5a329-da83-4876-a431-ddcb59c2858c'Match name='usracct.username', value='bazsi', expected='bazsi'Match name='usracct.authmethod', value='password', expected='password'Match name='usracct.device', value='127.0.0.1', expected='127.0.0.1'Match name='usracct.service', value='ssh2', expected='ssh2'0
Of course the pdbtool test could be useful for your own rulesets if test messages are available in the patterndb xml.
syslog-ng Store Box 2.0 released
It has been more than a half year since SSB 1.1 feature release and more than 1,5 years since the first SSB 1.0 release. We worked hard adding new features and fixing bugs in the product to have the second stable release ready as we have planned.
syslog-ng Store Box Roadmap for 2010
Last time I wrote about our Shell Control Box roadmap, so it was time now to publish the SSB roadmap as well. Meanwhile the 2010Q2 release is progressing very well, we have already in beta stage, so we are running some more tests and fixing some bugs before the GA release.
As with the SCB roadmap this list only highlights the bigger, more interesting features we planned for this year. Our plan is to have features releases quarterly delivering the new features more frequently. Here is the plan:
2010Q2 release
New hardware platforms
New hardware platforms will be introduced in SSB to meet the requirements of large volume installations. The new platforms will feature bigger CPUs, more RAM, redundant power supply and 1TB, 5TB or 10TB disk space. The new platforms will not be based on Sun/Oracle hardware, SSB will be moved to custom Intel based appliances.
Improved log indexing and search
SSB will be extended to have the ability to index and search any part of the messages based on user configuration. Statistics and reports could be created on any indexed field as well. The search interface is reworked in many aspects to provide smoother searching and better user experience.
2010Q3 release
VMWare virtual appliance
The SSB virtual appliance for VMWare servers will be introduced as a new platform. The virtual appliance will be a pre-installed and pre-configured system especially modified to fit into a virtual environment.
RPC API + SDK
SSB will be extended with a remote API, which can be used to run queries and retrieve log messages as the search interface does. An SDK will be provided with sample utilities, which can be used easily to script certain search functionalities. Modifying parts of the configuration of SSB will be possible via the API as well.
2010Q4 release
SQL collection “online”
SSB will have the ability to pull messages from SQL database tables in real-time, just as receiving messages over the network. Template configuration for specific applications will be provided to match the database schemes of the applications.
64bit Operating System & Performance improvements
SSB will be moved to a new 64bit based operating system based on the latest LTS Ubuntu release. With the new 64bit OS, SSB will be able to fully utilize all the resources of the hardware to speed up log handling, indexing and searching.
Shell Control Box Roadmap for 2010
We have updated our roadmap and release schedule for 2010, but I did not have the time to publish it on my blog. It is already May, but hopefully it is still interesting as the cool features are just coming now.
As with our other products we moved to a new feature/stable release version policy scheme so the features gonna be released in smaller feature releases. Besides the features releases we keep releasing the usual maintenances for the supported version with bug fixes only.
This roadmap contains only the bigger or more interesting features. We plan to release a feature release in approximately every quarter. So, here is the plan:
2010Q2 release
High Availability improvements
SCB will be able to use the production interfaces (external/management) as redundant heartbeat links between the nodes of the SCB cluster. This will prevent split-brain scenarios if the primary HA link fails.
SCB will be able to monitor the next-hop routers from the nodes and trigger takeovers if the monitored routers become unaccessible from the master node, while available from slave node.
SCP/SFTP “replay”
As of now SCP and SFTP traffic can be saved into an audit-trail file, but retrieving the transferred files is not supported by the Audit Player. The Audit Player will be able to analyze the recorded SCP and SFTP traffic and save the transferred files.
VMWare View support
VMWare View is VMWare’s Virtual Desktop Infrastructure desktop delivery solution. It uses multiple protocols as a desktop sharing core such as Microsoft’s RDP. SCB will be able to control and audit these RDP connections in a VMWare View environment.
RDP 7 support
With the recent release of Microsoft Windows 7 new extensions and protocol enhancements have been introduced in Remote Desktop Protocol. SCB will be able to interoperate with these new protocol enhancements.
New hardware platforms
New hardware platforms will be introduced in SCB to meet the requirements of large volume installation. The new platforms will feature bigger CPUs, more RAM, redundant power supply and 1TB, 5TB or 10TB disk space. The new platforms will not be based on Sun/Oracle hardware, SCB will move to custom Intel based appliances.
2010Q3 release
TS-Gateway
SCB will be able to act as a Terminal Server Gateway server to overcome the username-length limitation of the client in Bastion mode. This way client can connect to SCB using HTTPS+RDP and SCB will connect to the destination server using RDP.
Citrix ICA support
SCB will be able to control and audit the Citrix ICA protocol.
VMWare virtual appliance
SCB virtual appliance for VMWare servers will be introduced as a new platform. The virtual appliance will be a pre-installed and pre-configured system especially modified to fit into a virtual environment.
2010Q4 release
RPC API + SDK
SCB will be extended with a remote API, which can be used to run queries and retrieve results as the search interface does. An SDK will be provided with sample utilities, which can be used easily to script certain search functionalities. Modifying the configuration of SCB will be possible via the API as well.
Besides the API a reverse-API will be introduced to take policy decisions outside of the box. Using the reverse-API one will be able create a “policy-server” where site-specific policies can be enforced.
HTTPS audit
SCB will be able to audit HTTPS management traffic. With that release the Audit Player will not support replaying the traffic as seen in the browser of the client; only basic data retrieval will be provided.
patterndb on lwn.net
Robert wrote an excellent article for lwn.net on patterndb: Log message classification with syslog-ng. It is worthwhile reading it!
slngctl: syslog-ng debug, trace and statistics utility
PZolee wrote a quite lengthy blog post on troubleshooting and debuging syslog-ng back in December 2009. It popped into my mind that running syslog-ng with debug/trace/verbose option enabled is very handy, but sometimes you do not want to stop syslog-ng just to restart it in a more verbose mode. My idea was to enable syslog-ng to real-time modify these verbosity settings through the control socket which is currently only used for exporting statistical information.
I extended the control socket communication to have the ability of modify these logging settings. To make life easier I have created a small utility called slngctl which can be used to communicate with the running syslog-ng process. Using slngctl it also easy to query the statistics collected by syslog-ng. Also on longer term I think other useful small features could be added the slngctl.
slngctl commands:
marci@octane:$ slngctlPossible commands are: stats Dump syslog-ng statistics verbose Enable/query verbose messages debug Enable/query debug messages trace Enable/query trace messages
Querying statistics:
marci@octane:$ slngctl statsdestination;d_out;;a;processed;0global;payload_reallocs;;a;processed;0source;s_tcp;;a;processed;0global;msg_clones;;a;processed;0global;sdata_updates;;a;processed;0center;;received;a;processed;0center;;queued;a;processed;0
Getting current verbose settings:
marci@octane:$ slngctl verboseVERBOSE=1
Enabling trace run-time:
marci@octane:$ slngctl traceTRACE=0marci@octane:$ slngctl trace -s 1marci@octane:$ slngctl traceTRACE=1
As always you can find the source code in my 3.1 git repository and all feedbacks are very welcome.
pdbtool stylists wanted
I have just pushed to my syslog-ng 3.1 git repo some patches to add more detailed debugging/troubleshooting capabilities to patterndb through pdbtool match. Now you can easily see how your patterns matched a given message, which part of the message was matched by literal part of the pattern and which by a parser. All this new feature with a wonderful colorized output if requested, though the colors are still bit ugly…
Up to now the easiest way of creating patterns was to do trial & error sequence and try to figure out which part of the pattern was broken. Now you can exactly see where the matching stopped. As one picture is more verbose than hundreds of words, here is some screenshoot of pdbtool match.
The patterndb.xml in use:
A successful match:
A failed match:
It is also possible to output the match in parse-able format to be used in scripts or as a backend of some pattern authoring tool. Here is the output for that:
I am still not sure about the colored output nor about the machine parse-able output format, so any feedback, comment, idea or suggestion is very much appreciated.
You can grab the latest source code from my git repo.
Happy matching!