Apache 2.4 + mod_fcgid

During a recent hardening/maintenance session,  the httpd was upgraded to the 2.4 version, which initially produced lots of php segfaults, most probably  b/c content and configs were not sufficiently adjusted by the admin(s) beforehand.

After some config + content adjustments during a 2nd maintenance session, the setup ran quite well, but during periods of high load, lots of different error messages were shown. Different types that kept our attention were:

[Sun Mar 05 23:39:40.008981 2017] [fcgid:emerg] [pid 6608:tid 139850686646016] (35)Resource deadlock avoided: [client x.x.x.x:48916] mod_fcgid: can't get pipe mutex
[Sun Mar 05 23:43:43.343519 2017] [fcgid:warn] [pid 7829:tid 139850577540864] (104)Connection reset by peer: [client x.x.x.x:49693] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function, referer: ....
[Mon Mar 06 00:03:53.432715 2017] [fcgid:emerg] [pid 8618:tid 139850669860608] (35)Resource deadlock avoided: [client x.x.x.x:47173] mod_fcgid: can't lock process table in pid 8618, referer: ....

Also, the server had at least 20 php zombie processes running which cannot get killed but exhaust ressources and pile up:

1745 ?        Z      0:06 [php-cgi] <defunct> 
1753 ?        Z      0:06 [php-cgi] <defunct> 
3340 ?        Z      0:09 [php-cgi] <defunct> 
3341 ?        Z      0:09 [php-cgi] <defunct> 
3509 ?        Z      0:02 [php-cgi] <defunct>

So,  it seemed to have something to do w/ the way mod_fcgi starts php and locks underlying processes.

After a little research here and there, a checkout of how the system is configured gave us:

user@host:/opt/apache2/conf# grep -i mutex */*
extra/httpd-ssl.conf:SSLMutex "file:/opt/httpd-2.2.29/logs/ssl_mutex"
extra/httpd-ssl-domain.conf:Mutex file:/opt/apache2/logs/ssl_mutex

Hmm. One config even points to the wrong directory, and both use the “file” fcntl locking mechanism which seems to initially cause the errors.

Possible solutions would be to use

Mutex flock:${APACHE_LOCK_DIR} default

or as recommended

Mutex sem
SSLMutex sem

instead. Also, if the “sem” config-switch produces new errors, it may be b/c the kernels semaphore arrays are limited and should be extended by

sysctl -w kernel.sem="250 32000 32 1024"

So, just to be on the safe side, let’s do both, wait for a good timeslot to maintain, and reload the new config by service apache reload or via init which uses apachectl anyways:

/etc/init.d/apache reload

Great, all php zombie processes also vanished:

www-data 16178 0.9 0.2 95092 21788 ? S 02:41 0:00 /opt/php5/bin/php-cgi -c /opt/apache2/conf
www-data 16505 1.8 0.2 95100 21628 ? S 02:41 0:01 /opt/php5/bin/php-cgi -c /opt/apache2/conf
www-data 16685 3.9 0.2 93856 20480 ? S 02:41 0:03 /opt/php5/bin/php-cgi -c /opt/apache2/conf
www-data 16819 3.7 0.2 95104 21448 ? S 02:41 0:03 /opt/php5/bin/php-cgi -c /opt/apache2/conf
www-data 16846 1.7 0.2 95364 21692 ? S 02:41 0:01 /opt/php5/bin/php-cgi -c /opt/apache2/conf

The rest now seems to work as well, no more error messages thrown so far! Wait – after a while, one error reoccurs, and this very last section should be pretty self-explanatory:

#soll wohl auf 500 stehen
# d1g 060317
#FcgidMaxRequestsPerProcess 500
FcgidMaxRequestsPerProcess 0

Ethernet Issues v2: Retransmits

A few days before the time of this writing, certain problems arose at a high traffic (2.5 TB/day) and high load (50-75%) site.

What were the symptoms? The original settings ultimately led to TCP Retransmits, slowed down the specific users requests by aproximately 67 seconds on MacOS X, tho only by a few seconds in windows which maybe made the problem so hard to spot in the first place.

However,  LTE connections and other locations home networks seemed to have been unaffected by the issue, and an older linux router which seemed to eventually have compensated the mentioned problem until recently again made it hard to spot.

Normally, it is the other way around, but this time, a skilled customer took a deeper look into the issue (thnx, alex!) and gave us a hint having identified the probable involvement of the tcp_tw_recycle setting which can be checked by e.g.

# cat /proc/sys/net/ipv4/tcp_tw_recycle
1

What it does:

“Enable fast recycling of sockets in TIME-WAIT status. The default value is 0 (disabled). It should not be changed without advice/request of technical experts.”

Having mentioned tw_recycle, two other relevant settings are tcp_tw_reuse  and tcp_max_tw_buckets  which seems to be set to 262144 on newer and to as low as 16384, 65536 or 131072 on older systems. So yes, again there could be a connection to a high traffic site having roughly 77k connection states for some while, correlating w/ symptom appearance.

So, just to be safe, I would recommend setting it to a high value by

echo 262144 > /proc/sys/net/ipv4/tcp_max_tw_buckets

One could think that lots of TIME-WAIT states only arise if you have many clients w/ (too) idle connections coming from one NATed network, and that under normal circumstances the fast recycling may make sense. However, with the connection tracking table max. entries set to let’s say 512k, most systems perhaps never need to recycle the WAIT-STATES.

In the end, lets not forget about the fact that if we find problems, some trouble may have already taken place, but this is also the base for fixing it in most if not all cases.

OpenPGP Key Recreation and Revocation

Despites nearly having forgotten to blog about it, time has come to get myself a stronger OpenPGP keypair. But what about the folks I already established a secure connection with using the old key 0x800e21f5 and what about the rest of the internet? It’s not as complic as one might think.

Note: The following Howto is meant for more advanced users and involves the shell. Novice users should just continue to use roundcube for all their PGP related things.

 

1. Key Creation

Key creation is very simple if you use GnuPG on Linux:

0x220b:~$ gpg --gen-key

You can leave the default options (RSA/RSA,  4096bit, never expires) until it comes to name, e-mail and comment, where you should fill in your personal data associated w/ the use of the key. In most cases, one e-mail address is not enough, but you can just add one like this:

0x220b:~$ gpg --edit-key 6C71D217
gpg> showpref
[uneingeschränkt] (1). Peter Ohm (NetworkSEC/NWSEC) <p.ohm@networksec.de>
 Verschlü.: AES256, AES192, AES, CAST5, 3DES
 Digest: SHA256, SHA1, SHA384, SHA512, SHA224
 Komprimierung: ZLIB, BZIP2, ZIP, nicht komprimiert
 Eigenschaften: MDC, Keyserver no-modify
gpg> adduid

Now enter the other e-mail addy and a relevant comment if you wish. So, we now got a fresh key – but what about the old one(s)? At first, we should use them to sign the new one:

0x220b:~$ gpg --default-key 800e21f5 --sign-key 6C71D217
0x220b:~$ gpg --default-key 7BB7A759 --sign-key 6C71D217

and then finally give everybody access to our new public key by:

0x220b:~$ gpg --keyserver pgp.mit.edu --send-key 6C71D217

 

2. Key Revocation

Okay, now everybody must be able to know that the old keys are not used any longer. This can easily be achieved by first creating a revocation certificate for each of them, then importing that into the own keyring and finally exporting the revoked keys to the internet. Lets do it w/ a small shell skript and gpg2:

#!/bin/bash
for i in 7bb7a759 800e21f5
do
 gpg2 --output revoke.asc --gen-revoke $i
 gpg2 --import revoke.asc 
 gpg2 --keyserver pgp.mit.edu --send-keys $i
done

 

3. More Key Distribution

I also recommend to send everybody you already set up an encrypted communications channel with your new public key as they will be the only ones possibly using the old key material (most OpenPGP clients refuse to use revoked keys for encryption) and as it’s especially them who would need to be informed about any changes.

So, even if anybody interested in establishing a secure communications channel did not yet get your new public key, all that remains to be done is:

gpg --keyserver pgp.mit.edu --recv-keys 6C71D217
gpg --keyserver pgp.mit.edu --refresh-keys

…and don’t forget to attach your own public key if its a first-time contact.

Roundcube: Enigma PGP v2

Meant as an easy guide for novice users, this post will describe how to quickly establish a fully encrypted mail communication between two parties.

 

I assume that the system is up and running and enigma works, which means you have already created your own keypair as described in the 1st part of this howto.

Now, in case you are being uncreative and lazy, here is a hint: What comes next is actually start using it:

  • e-mail Alice
  • sign the mail 
  • attach your public key

Alice can then add your public key from the attachment, attach her own public key and then send a already fully encrypted  reply to you w/o a chance for an eavesdropper to even capture Alice’s (probably unlisted) public key. All of the subsequent messaging will be secured after you import Alice’s key which completes the encryption setup for both sides.

Even if Alice is also a novice user, initially does not have any PGP keypair but is also using roundcubemail  + enigma (or eventually GnuPG, Thunderbird, Enigmail, …), she can just create herself one and attach it. Got the idea?

The argument of being unable to establish a fully encrypted e-mail communication is no longer a valid excuse (and never really has been) b/c it only takes 2 steps per participant: keypair creation, pubkey distribution and as a consequence the import of the other ppl’s pubkey.

A short personal message: Thank you Edward Snowden. You caused a significant growth in the topic of general security awareness which resulted in the empowerment of EFF driven projects like letsencrypt and the rise of (hopefully) secure messengers amongst lots of other things: The so called Snowden-Effect. Be assured: As a final consequence, your statement has been clearly heard and at least partly understood (or turned into a business model) by the masses. Also, let’s not forget about WikiLeaksRapNews,  Phineas Fisher, Equation Group or Shadow Brokers who basically act into the same direction.

TLS Hardening v2: Postfix

So, hows it w/ postfix? Straightforward, but too many options to list them all 🙂 At first, we want to be able to look at some TLS details, in the logs as well as in the header, so we set

smtpd_tls_received_header = yes
smtpd_tls_loglevel = 1
smtp_tls_loglevel = 1

in /etc/postfix/main.cf. Furthermore, if not already set

smtpd_use_tls = yes
smtp_use_tls = yes
smtpd_tls_security_level = may
lmtp_tls_mandatory_ciphers = high
smtp_tls_mandatory_ciphers = high
smtpd_tls_mandatory_ciphers = high
smtp_tls_security_level = may
smtpd_tls_mandatory_exclude_ciphers = MD5, DES, ADH, RC4, PSD, SRP, 3DES, eNULL
smtpd_tls_exclude_ciphers = MD5, DES, ADH, RC4, PSD, SRP, 3DES, eNULL
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
smtpd_tls_protocols = !SSLv2, !SSLv3

also makes sense, but still gives us a C+ rating. Folks at Qualys recommend

tls_export_cipherlist = aNULL:-aNULL:ALL:+RC4:@STRENGTH
tls_high_cipherlist = aNULL:-aNULL:ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH
tls_legacy_public_key_fingerprints = no
tls_low_cipherlist = aNULL:-aNULL:ALL:!EXPORT:+RC4:@STRENGTH
tls_medium_cipherlist = aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH
tls_null_cipherlist = eNULL:!aNULL

but this also gives us C+. Doing more research, we finally get a B – still w/ a self-signed cert – by using

tls_export_cipherlist = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!MEDIUM:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
tls_high_cipherlist = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!MEDIUM:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
tls_legacy_public_key_fingerprints = no
tls_low_cipherlist = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!MEDIUM:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
tls_medium_cipherlist = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!MEDIUM:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
tls_null_cipherlist = eNULL:!aNULL

Together w/ a letsencrypt cert, we finally receive an A grade and full PCI-DSS compliance. Mission accomplished!

P.S.: If you are interested in a complete TLSv1.2 cipherlist, just issue

openssl ciphers TLSv1.2

TLS Hardening v1: Qmail – fail

Recently, a friend notified me about the fact that his server was unable to send mail using PFS to one of my servers. Beginning to deal w/ this – your bet is right, during fullmoon – at first I checkout what i have and where i can tune it.

So for example, we want something like

TLSv1+HIGH:!SSLv2:!MD5

in /var/qmail/control/tls{server,client}ciphers. Afterwards, checking out the available TLSv1 cipher suites results in

Accepted TLSv1 256 bits AES256-SHA
Accepted TLSv1 256 bits CAMELLIA256-SHA
Accepted TLSv1 128 bits AES128-SHA
Accepted TLSv1 128 bits CAMELLIA128-SHA

for ports 25 and 465, which is far from best practice. So no DHE ciphers, b/c of no dh{1024,2048}.pem wich we can easily create by

openssl dhparam -out dh2048.pem 2048

and then set in the /service/qmail-smtpdssl/run file, we find

export KEYFILE=""
export DHFILE=""

which explains why no DHE ciphers were previously available. After having set the values to use our actual key material, we enabled them, thus getting

Accepted TLSv1 256 bits DHE-RSA-AES256-SHA
Accepted TLSv1 256 bits DHE-RSA-CAMELLIA256-SHA
Accepted TLSv1 256 bits AES256-SHA
Accepted TLSv1 256 bits CAMELLIA256-SHA
Accepted TLSv1 128 bits DHE-RSA-AES128-SHA
Accepted TLSv1 128 bits DHE-RSA-SEED-SHA
Accepted TLSv1 128 bits DHE-RSA-CAMELLIA128-SHA
Accepted TLSv1 128 bits AES128-SHA
Accepted TLSv1 128 bits SEED-SHA
Accepted TLSv1 128 bits CAMELLIA128-SHA
Accepted TLSv1 128 bits RC4-SHA
Accepted TLSv1 128 bits RC4-MD5
Accepted TLSv1 112 bits EDH-RSA-DES-CBC3-SHA
Accepted TLSv1 112 bits DES-CBC3-SHA

for port 465 which also has

SSL=1

set in the run file. But, there is still RC4 in there, which we can rule out by adding

export CIPHERS="TLSv1+HIGH:!SSLv2:!MD5"

in the appropriate run file right under the other options mentioned above, resulting in

Accepted TLSv1 256 bits DHE-RSA-AES256-SHA
Accepted TLSv1 256 bits DHE-RSA-CAMELLIA256-SHA
Accepted TLSv1 256 bits ADH-AES256-SHA
Accepted TLSv1 256 bits ADH-CAMELLIA256-SHA
Accepted TLSv1 256 bits AES256-SHA
Accepted TLSv1 256 bits CAMELLIA256-SHA
Accepted TLSv1 128 bits DHE-RSA-AES128-SHA
Accepted TLSv1 128 bits DHE-RSA-CAMELLIA128-SHA
Accepted TLSv1 128 bits ADH-AES128-SHA
Accepted TLSv1 128 bits ADH-CAMELLIA128-SHA
Accepted TLSv1 128 bits AES128-SHA
Accepted TLSv1 128 bits CAMELLIA128-SHA

However, connections using STARTTLS seem to still use non-PFS ciphersuites. The problem lies in the qmail design: If a connection comes in as plaintext – even w/ STARTTLS later on – , the variable SSL=0 is needed in the run file. If it comes in encrypted, SSL=1 is set. It is simply  impossible to set the parameters KEYFILE and DHFILE for qmail-smtpd, so even after STARTTLS, they cannot be used. And if we set SSL=1, STARTTLS will no longer work as the server states that the connection is already encrypted even tho that is not the case ^o.O^

Finally, after two days of effort invested in trying to patch skripts or even source code, I say “goodbye qmail” ! After having used that 20yr old S/W for 12+ years, it is now time to completely switchover to postfix, which i was using on a majority of systems anyways since 8 years.

PHP 7.1

Quite curious about the freshest PHP version and surely inspired by the full moon, I roll out PHP 7.1 + zend op_cache + memcached @ all hosting VMs and integrate it into the ISPC 3.1 customer interface as nicely explained here. I created a skript for easier rollout:

#!/bin/bash
#
# Skript w/ commands taken from http://www.ispconfig.org/blog/install-php-7-1-php-fpm-fastcgi-ispconfig-3-1-debian-8-jessie/
# not fine grained yet as it needs sudo permissions :|
#
echo "Create directories..."
mkdir -p /opt/php-7.1
mkdir /usr/local/src/php7-build
cd /usr/local/src/php7-build
wget http://de1.php.net/get/php-7.1.0.tar.bz2/from/this/mirror -O php-7.1.0.tar.bz2
tar jxf php-7.1.0.tar.bz2
cd php-7.1.0/
apt-get install libfcgi-dev libfcgi0ldbl libjpeg62-turbo-dbg libmcrypt-dev libssl-dev libc-client2007e libc-client2007e-dev libxml2-dev libbz2-dev libcurl4-openssl-dev libjpeg-dev libpng12-dev libfreetype6-dev libkrb5-dev libpq-dev libxml2-dev libxslt1-dev -y
ln -s /usr/lib/libc-client.a /usr/lib/x86_64-linux-gnu/libc-client.a
time ./configure --prefix=/opt/php-7.1 --with-pdo-pgsql --with-zlib-dir --with-freetype-dir --enable-mbstring --with-libxml-dir=/usr --enable-soap --enable-calendar --with-curl --with-mcrypt --with-zlib --with-gd --with-pgsql --disable-rpath --enable-inline-optimization --with-bz2 --with-zlib --enable-sockets --enable-sysvsem --enable-sysvshm --enable-pcntl --enable-mbregex --enable-exif --enable-bcmath --with-mhash --enable-zip --with-pcre-regex --with-pdo-mysql --with-mysqli --with-mysql-sock=/var/run/mysqld/mysqld.sock --with-jpeg-dir=/usr --with-png-dir=/usr --enable-gd-native-ttf --with-openssl --with-fpm-user=www-data --with-fpm-group=www-data --with-libdir=/lib/x86_64-linux-gnu --enable-ftp --with-imap --with-imap-ssl --with-kerberos --with-gettext --with-xmlrpc --with-xsl --enable-opcache --enable-fpm
time make -j2
make install
cp /usr/local/src/php7-build/php-7.1.0/php.ini-production /opt/php-7.1/lib/php.ini
cp /opt/php-7.1/etc/php-fpm.conf.default /opt/php-7.1/etc/php-fpm.conf
cp /opt/php-7.1/etc/php-fpm.d/www.conf.default /opt/php-7.1/etc/php-fpm.d/www.conf

I already tested it w/ the blog you are reading right now running WP 4.6 and 4.7 w/o any issues. Testing w/ an old version of WP 4.1.x seems to not work w/ PHP 7.x as expected and b/c its code is using obsolete PHP 5.x syntax.

So, if you are dealing w/ a freshly installed WP 4.1.x instance, just upgrade first to 4.7, and then select the optional switch to PHP 7.1 for your site to make it work.

PHP 7.1.0 (cgi-fcgi) (built: Dec 13 2016 04:45:07)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.1.0-dev, Copyright (c) 1998-2016 Zend Technologies
with Zend OPcache v7.1.0, Copyright (c) 1999-2016, by Zend Technologies

I get the impression that PHP7  really speeds things up, and would like to hear your comments on that subject as well. Happy testing 🙂

2nd Maintenance 2016

Since Q4/2016 already started, it was time to do some pending upgrades and create fresh backups of important data. This included all the productive VMs for

  • multisite hosting
  • exclusive singlesite hosting
  • e-mail
  • co-working and storage
  • monitoring

resulting in a total of 240GB GPG-encrypted backup data, redundantly stored on 5 different servers.

 

Ethernet Issues v1: Offloads

Recently, I experienced strange symptoms at a network involving very short connection outages resulting in only partial database queries, observed by a competent admin (thnx, thomas!). While doing some further research, the problem – tho under slightly different circumstances –  sounded familiar: Last time at the other site, pretty large downloads were interrupted, w/ content malware scanning on the list of potential culprits, but soon the customer site’s network hardware itself was identified in playing a crucial role. Messages sporadically seen in syslog were e.g.:

e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

or

MASTER conntrack-tools: no dedicated links available!

Last time, it was a Gbit NIC using the e1000e kernel module connected to a D-Link Gbit Switch, this time it was the same NIC connected to Cisco and a HP Gbit Switches.

However, having a closer look into what kind of network negotiation took place and checking my notes from last time, subsequently implementing

ethtool -K eth0 gso off gro off tso

seems to have fully resolved the issue(s) mentioned above so far. What this does is disabling

  • generic segmentation
  • generic receive and
  • tcp segmentation 

offloads @ the NIC itself. On the other hand, maybe this is only an issue w/ HQ switch hardware that triggers the NICs function in the first place.

Either way, above command eliminates the problematic aspects of all this.

 

Roundcube: Enigma PGP v1

What follows, is a short howto to get the Enigma PGP plugin for the Roundcube webmail interface practically up and running for users being new to e-mail encryption.

 

Key Creation

At first, one has to set a username under Settings => Identity. Then, under Settings => Encryption, at least the upper 4 options should be selected.

Afterwards, Settings => PGP Keys should be selected and a new key be generated using the + symbol at the bottom. It is rather important to choose a strong password, as this will also protect the keyfile itself on the server.  You have now created your GPG/PGP keypair (public and private key) .

Now, you can just start using the key as shown in the 2nd part of this howto.

 

Key Import and Export

For each user identity we want to be able to communicate w/ in an encrypted manner, the relevant public key has to be Imported, and we should send every user our Exported public key, preferably over a completely different, secure channel, e.g. Jabber + OTR. (Hint: The Import and Export buttons are found in the upper left corner of the UI in the Settings => PGP Keys context).

If you wonder how you can export an existing public key suitable for the Import mentioned above, you can type e.g.:

gpg --armor --export 800e21f5 > pubkey.asc

If you got the pubring.gpg file, you can do

gpg --no-default-keyring --keyring ./pubring.gpg --export --armor > pubkey.asc

to achieve the same.