Ethernet Issues v2: Retransmits

A few days before the time of this writing, certain problems arose at a high traffic (2.5 TB/day) and high load (50-75%) site.

What were the symptoms? The original settings ultimately led to TCP Retransmits, slowed down the specific users requests by aproximately 67 seconds on MacOS X, tho only by a few seconds in windows which maybe made the problem so hard to spot in the first place.

However,  LTE connections and other locations home networks seemed to have been unaffected by the issue, and an older linux router which seemed to eventually have compensated the mentioned problem until recently again made it hard to spot.

Normally, it is the other way around, but this time, a skilled customer took a deeper look into the issue (thnx, alex!) and gave us a hint having identified the probable involvement of the tcp_tw_recycle setting which can be checked by e.g.

# cat /proc/sys/net/ipv4/tcp_tw_recycle
1

What it does:

“Enable fast recycling of sockets in TIME-WAIT status. The default value is 0 (disabled). It should not be changed without advice/request of technical experts.”

Having mentioned tw_recycle, two other relevant settings are tcp_tw_reuse  and tcp_max_tw_buckets  which seems to be set to 262144 on newer and to as low as 16384, 65536 or 131072 on older systems. So yes, again there could be a connection to a high traffic site having roughly 77k connection states for some while, correlating w/ symptom appearance.

So, just to be safe, I would recommend setting it to a high value by

echo 262144 > /proc/sys/net/ipv4/tcp_max_tw_buckets

One could think that lots of TIME-WAIT states only arise if you have many clients w/ (too) idle connections coming from one NATed network, and that under normal circumstances the fast recycling may make sense. However, with the connection tracking table max. entries set to let’s say 512k, most systems perhaps never need to recycle the WAIT-STATES.

In the end, lets not forget about the fact that if we find problems, some trouble may have already taken place, but this is also the base for fixing it in most if not all cases.

OpenPGP Key Recreation and Revocation

Despites nearly having forgotten to blog about it, time has come to get myself a stronger OpenPGP keypair. But what about the folks I already established a secure connection with using the old key 0x800e21f5 and what about the rest of the internet? It’s not as complic as one might think.

Note: The following Howto is meant for more advanced users and involves the shell. Novice users should just continue to use roundcube for all their PGP related things.

 

1. Key Creation

Key creation is very simple if you use GnuPG on Linux:

0x220b:~$ gpg --gen-key

You can leave the default options (RSA/RSA,  4096bit, never expires) until it comes to name, e-mail and comment, where you should fill in your personal data associated w/ the use of the key. In most cases, one e-mail address is not enough, but you can just add one like this:

0x220b:~$ gpg --edit-key 6C71D217
gpg> showpref
[uneingeschränkt] (1). Peter Ohm (NetworkSEC/NWSEC) <p.ohm@networksec.de>
 Verschlü.: AES256, AES192, AES, CAST5, 3DES
 Digest: SHA256, SHA1, SHA384, SHA512, SHA224
 Komprimierung: ZLIB, BZIP2, ZIP, nicht komprimiert
 Eigenschaften: MDC, Keyserver no-modify
gpg> adduid

Now enter the other e-mail addy and a relevant comment if you wish. So, we now got a fresh key – but what about the old one(s)? At first, we should use them to sign the new one:

0x220b:~$ gpg --default-key 800e21f5 --sign-key 6C71D217
0x220b:~$ gpg --default-key 7BB7A759 --sign-key 6C71D217

and then finally give everybody access to our new public key by:

0x220b:~$ gpg --keyserver pgp.mit.edu --send-key 6C71D217

 

2. Key Revocation

Okay, now everybody must be able to know that the old keys are not used any longer. This can easily be achieved by first creating a revocation certificate for each of them, then importing that into the own keyring and finally exporting the revoked keys to the internet. Lets do it w/ a small shell skript and gpg2:

#!/bin/bash
for i in 7bb7a759 800e21f5
do
 gpg2 --output revoke.asc --gen-revoke $i
 gpg2 --import revoke.asc 
 gpg2 --keyserver pgp.mit.edu --send-keys $i
done

 

3. More Key Distribution

I also recommend to send everybody you already set up an encrypted communications channel with your new public key as they will be the only ones possibly using the old key material (most OpenPGP clients refuse to use revoked keys for encryption) and as it’s especially them who would need to be informed about any changes.

So, even if anybody interested in establishing a secure communications channel did not yet get your new public key, all that remains to be done is:

gpg --keyserver pgp.mit.edu --recv-keys 6C71D217
gpg --keyserver pgp.mit.edu --refresh-keys

…and don’t forget to attach your own public key if its a first-time contact.