ABOUT
Archive for February, 2009
My syslog-ng git repo
As syslog-ng is a truly open-source project we felt important with Bazsi to send my patches for mainline inclusion not in-house rather through the public mailing list and in a public git repository. So the time has come to open my syslog-ng open-source git repository. Anyone interested in it can access it at git://git.balabit.hu/marci/syslog-ng-3.0.git
For the first patch I was doing today some performance tests where use_dns(yes) and persistent dns cache (dns_cache_hosts) were configured and I got awful results. When I disabled dns_cache_hosts the message rate doubled. After a quick check I realized that for every message a dns cache lookup was performed which stat()-ed my dns_cache_hosts file. However it does not make much sense to check the file for modification so often, I guess once a second is enough most of the cases.
In my test configuration I received msgs over syslog() source and sent to an empty destination. The server was a 2way quad-core 1.9GHz Opteron box running 32bit syslog-ng and messsages were generated with loggen. From an average 80,000-90,000 msg/s using the fix I could do 220,000-230,000 msg/s. Interesting that on my laptop (dual-core 2.4GHz Intel 64bit) the performance gain was smaller from 90,000 to 100,000 msg/s. I am still wondering about this big difference, specially because my laptop has higher frequency CPU…
Back to work again
I am back from skiing. In one piece which is generally a good thing as far as I am considered.
The snow was good so we had a great time and finally we did not have to queue for the lifts for hours as in Slovakia. We were at Gerlitzen (http://gerlitzen.com/) in Austria near Villach and I can recommend the place specially if you do not want to get lost in an over 100km ski runs place.
The tickets hold some kind of chips inside so I could check on the internet where I was skiing. (Maybe this is not the best from a personal data protection point of view, but much fun looking at the graphs.) On my best day I did ski 44kms and during the week I did around 170kms which is pretty good for me.
My best day (the long gap is lunch):
Graduating and Introduction
I just realized that I forgot to introduce myself, but I hope it is not too late to do so.
I am Marton Illes (friends call me Marci, which is the nickname for Marton in hungarian) and I am a product architect at BalaBit responsible mainly for the Shell Control Box and syslog-ng Store Box products. I am involved deeply in development, but time-to-time I help with pre-sales and do a little support as well.
The blog was started because our marketing guy (Attila) convinced me that it would be cool to write about my daily job and also it is a good place to rant about stupid problems I ran into. I must admit that I was wondering for some time about starting my blog, so here it is, hope you will enjoy it.
Attila put together the blog design (thanks for the nice pink box 
and give me a degree in advance. The truth is that I just had my graduation yesterday so now I am very happy and also have a little hang over… As a recreation I will leave for a week skiing on Sunday, so no updates next week!
Spanning tree vs. High Availability
The Shell Control Box and syslog-ng Store Box use a common framework (called XCB) to handle some low-level functions like HA. The idea was to create a master-slave fail-over cluster which requires very little configuration and does most of thing automatically.
Basically we use hearbeat to control the resources and DRBD to mirror the disk space over the network. Thanks for being an appliance we could easily dedicate a separate interface for HA purposes. While IP addresses of other interfaces are handled with hearbeat the IP addresses of the HA interface needs to be configured during the boot process. As minimal user interaction was a strong aim in our case, we did not want the user to configure IP addresses manually rather we created a simple negotiation “protocol” which handles it in boot time. We selected 1.2.4.1 and 1.2.4.2 addresses for HA. (I know they might not be the best choice, but using addresses from legacy private networks would definitely cause some collision at customers.)
The negotiation is very simple, as the HA nodes (maximum 2 at this time) are connected over a cross-over cable or a switched environment, they simply need to select the IP address the other node is not using. To do so, after the HA link is set up the nodes send out an arping (arp request) for 1.2.4.1. If a node gets a reply this means the other node already used the first address so it uses the second address: 1.2.4.2. However if there is no reply, which means 1.2.4.1 is still not used the node configures 1.2.4.1 for itself, but at this point it rechecks the situation with an other apring. If at this time it receives a reply than a more care full negotiation is required (which can be run on both nodes independently and provides the same results independent from the node running on). To handle this race, the node with a “smaller” mac-address takes 1.2.4.1 and the other takes 1.2.4.2. This hopefully yields to a configuration where the nodes have different HA IP addresses.
The process is simple and has been working well as long as the network connection works. Today we run into a strange situation where the nodes were not connected via a cross-over cable rather through switches as the HA nodes were located in different cities and the WAN connection involved switches. After booting up the nodes both were using 1.2.4.1 but testing the WAN connection did not show any error. Doing some more troubleshooting using tcpdump on one of the nodes showed that the initial arp request for 1.2.4.1 (from arping) did not arrive at all, though latter arp-requests for 1.2.4.2 did arrive. Strange. We come to the conclusion that after link is set up no packet made it to the other node for 30-40 seconds. Luckily the tcpdump showed some other traffic and we could spot STP (spanning tree protocol) traffic as well.
Checking again STP documentation we realized that this a normal behavior for STP enabled switches to disable traffic on ports after link up for 30-50 seconds during for listening and learning states before going into forwarding state. This explains why the negotiation arp-request did not make through and why both nodes ended with 1.2.4.1 addresses.
From this point the solution was easy, either reconfigure the switches to use rapid STP where the down time is under 1 second or enable “fast” for the specific ports. Normally switching to RSTP anytime is not an option in a production environment, but enabling fast mode for given ports was OK. After the change the switch forwarded packets so the nodes could successfully do the negotiation and the HA was again up and running.
It was a nice challenge, but after all everything works fine and we have way more experience on STP than we had before. Thanks for everybody helping us.