Archive for November, 2009
We had our usual annual wine-tasting party on Thursday, this time at a new location at the stylish Manna Lounge. (All the other locations went bankrupt and had to close the place. Maybe it is a bad sign when BalaBit chooses a restaurant or bar?) Anyhow the event went pretty well, we had some real good food, wines and some Hungarian “palinka“.
I gave a very short presentation on SCB using and narrating our demo video “Joe’s last day“. It was Attila’s idea and I must admit that it was really refreshing to do some unorthodox presentation this time.
Recently we run into some putty related performance problem when putty was used with Shell Control Box. Putty is a very nice tool for simple ssh terminal connections, but in our case we wanted to do X-forward together with Xming and that seemed to be a problem. After some more careful testing it seemed that not only X-forward is affected, but any higher throughput scenario (like sftp/scp, simple port-forward) as well.
There are two major reasons that can impact the performance of putty. These problems are not really visible in case of simple terminal connections where the transferred data is not significant, but clearly visible in mass-transfer scenarios. Also if there is a proxy (which buffers transfer-data) in between the client and the server the impact is magnified.
From putty ssh.c:
“When using a CBC-mode cipher, it’s necessary to ensure that an attacker can’t provide data to be encrypted using an IV that they know. We ensure this by prefixing each packet that might contain user data with an SSH_MSG_IGNORE.”
See for more details:
This approach adds an extra packet with apprx. 32bytes of data to each “normal” packet. SCB however forwards IGNORE and “normal” data packets in separate TCP packets and therefore doubles the packet count and the acknowledgment packet count as well.
Putty also recommends using CTR mode ciphers which can be enforced on SCB so this part seems “work-aroundable”.
Putty as of version 0.60 uses 16KB receiving window size, while openssh uses 2MB window size for ssh channels. This window size limits the number of bytes a certain channel can buffer before processing the data. (Similar to TCP window size.)
As the receiving window fills up the sending side must not send more data till the receiving window size is adjusted. This causes latency in the communication as at least a full round-trip time will elapse till the sender can send data again. This affects high-throughput scenarios
such as file-transfers (sftp/scp), port-forwards and X11 forward (which is technically a simple port-forward). 16KB window is very small 10-11 packet can fill it up.
WinSCP which utilizes putty’s pscp/plink as an ssh implementation has also performance problems, see this FAQ entry: http://winscp.net/eng/docs/faq_slow
“Also note that as SSH code of WinSCP is based on PuTTY, file transfers with SSH-based protocols can hardly be faster than PuTTY (PSCP/PSFTP) is. Hence there is no point asking for speed improvements, if you get the same rate with PuTTY.”
Between the download (where putty window matters) and upload (where the server window matters) scenarios in winscp/sftp cases 10x speed differences were messured by winscp users and by us as well. http://winscp.net/forum/viewtopic.php?t=4684
In case of SCB the situation is even worse because of the two independent ssh connections on the client and the server side. The receiving window size is advertised on the other side connection as well. In case of X11 the server sends 16KB data which is forwarded to putty, but the server needs to wait the window adjustment from SCB, which needs to wait for the adjustment from putty. Adding extra buffering to SCB does not help the situation considerably as SCB is still limited to send just 16KB of data to putty and after filling the extra buffer the server would also slow down plus SCB would be just busy receiving data from the server.
The situation gets worse when the connection speed between SCB and the client is slow or the RTT is high.
See also this putty semi-bug report:
A possible work-around would be to increase the window size of putty. In the current development SVN tree the window size is increased to 2MB in case only one channel is used in a connection. (Unfortunately this i only true in case of SFTP/SCP, but not in case of X11.)
However it is still possible to patch putty to use bigger window size or use an other ssh client.
For file-transfer cases scp also yields to a better performance than sftp, as despite it’s name sftp is much more like an nfs like (open/seek/read/write) protocol where request/response commands add an extra round-trip time while scp simply throws the data into the pipe.