Backup 2.0

Since and b/c we are currently on a maintenance- and backup-spree, trouble arose when dealing w/ the gluster distributed-replicated setup involving four rootservers. After ranting a bit about the incompatibility of gluster e.g. in wheezy and jessy, we received a tip (thanks rpw!) and decided to start using git-annex from now on.

Note: To run a really fresh version (5.2), using the neurodebian repo is a good option.

Create Backup Data

As with every backup, it is ideal to have a spare b0x to host all the initial backup data and to function as an initial place to sort and clean-up stuff. As w/ every backup, it is best practice to always keep local backups until we really have a fully functioning, redundant backup. So, in our case, we get most of the backup data via rsync from two other hosts, and some more via scp. It is also a good idea to create a listing of all included files for later comparison, e.g. by doing

tree -ah | tee -a TREE
Initialize local repo

When we have all the data we want in place, we can continue by

git init
Initialized empty Git repository in /home/d1g/ANNEX/.git/
git annex init "NWS4"
init NWS4 ok
(recording state in git...)
git annex direct

Now, files should be added:

git annex add . (takes quite some time to create the initial checksums)
git commit -m "Initial git-annex NWSx REPO"
Initialize remote repo

On all of the remote nodes, the preparation should be something like

mkdir ANNEX
git init
git annex init "NWS3"
git annex direct
Add remote repo
git remote add nws4 ssh://nws4/home/d1g/ANNEX
Synchronize data

Now we can start to synchronize the data in between all involved repos, first copying from the remote location:

git annex sync --content

or, if we want to be more specific:

git annex copy VMB2015 --from-nws4
Keep data up to date
git annex watch

If annex watch is not running, and If now one host sends a new file via

git annex add .
git annex sync --content

we can use nws4 to “spread” it to all the other hosts automatically, if we want – after having added them all as remotes – by issuing the exact same command there.

For common things that could possibly go wrong, this page may contain some useful information.

Malware Deobfuscation

If a PHP based website installation like WordPress starts misbehaving e.g. by sending out lots of typical spam mails, some quick analysis w/ a simple and manual deobfuscation approach may make sense.

1. Correlation

At first, some correlation of what exactly was going on was required. Tailing the webservers logfiles together w/ running ngrep shows a clear connection: When a certain URL is called, a new spike in the mailqueue happens. Also, ngrep data reveals an interesting string:



This looks like some base64 to me. Decoding this results in



which potentially looks like some c&c data. Facts so far: An attacker uploads a php file, and regularly calls that file to send out spam mails.

2. Code Analysis

When checking out the php code of the malicious file – only two scanners @ virustotal detect CPR17F2.Webshell respectively PHP.Packed.11 – it becomes clear that the code is obfuscated not only by base64, but also in some proprietary way. Most of the files content consists of lines like:



Function names are random, and code seems to be stuffed in obfuscated and encoded manners. However, in the beginning of the file, we got some hints, and at the end there is something like a key scheme for all this:


$felhrwy = Array('1'=>'m', '0'=>'g', '3'=>'Y', '2'=>'B', '5'=>'y', '4'=>'M', '7'=>'l', '6'=>'Q', '9'=>'b', '8'=>'7', 'A'=>'A', 'C'=>'2', 'B'=>'v', 'E'=>'j', 'D'=>'w', 'G'=>'0', 'F'=>'F', 'I'=>'O', 'H'=>'H', 'K'=>'k', 'J'=>'1', 'M'=>'T', 'L'=>'U', 'O'=>'x', 'N'=>'q', 'Q'=>'C', 'P'=>'R', 'S'=>'N', 'R'=>'o', 'U'=>'6', 'T'=>'4', 'W'=>'P', 'V'=>'K', 'Y'=>'X', 'X'=>'G', 'Z'=>'8', 'a'=>'p', 'c'=>'S', 'b'=>'n', 'e'=>'L', 'd'=>'3', 'g'=>'E', 'f'=>'r', 'i'=>'i', 'h'=>'I', 'k'=>'5', 'j'=>'t', 'm'=>'h', 'l'=>'z', 'o'=>'9', 'n'=>'e', 'q'=>'f', 'p'=>'Z', 's'=>'s', 'r'=>'u', 'u'=>'W', 't'=>'c', 'w'=>'a', 'v'=>'V', 'y'=>'d', 'x'=>'D', 'z'=>'J');

3. Code Deobfuscation  – partial only

Okay, we got that list, so we can use tools like sed to change the data by the rules of the author. After having compiled a list that looks like



and so on, a small shellscript (that I call intentionally)  is needed to do what we want, containing:


for i in `cat ARRAY `
  echo $i $j $k
  sed $i FILE$j > FILE$k
  j=`expr $j + 1`
  k=`expr $k + 1`

Later on, it becomes clear that this is not the quick and dirty way, as that would rather have been


sed -f ARRAY $1 > $2

Output in both cases becomes a lot clearer, but is still heavily obfuscated, and fiddling around w/ all the textmanipulation utilities is a very abstract thing for sure. Also, the techniques used so far do not deal w/ things like CRLF or “\r\n” and so on.

4. Code Deobfuscation – SUCCESS

It looks a lot more reasonable to use the code that the attacker already gave us to deobfuscate and decode the whole php file. So we take a closer look to the very last function that does all that:


eval(xlvgapr($wufa, $felhrwy));?

In short, this runs the deobfuscated and decoded code directly on the machine the file is executed on. All we gotta do is not run the code, but print it, so all it takes is


print(xlvgapr($wufa, $felhrwy));?>

and a command like


php MODIFIED_inc.php > DECODED.php

The resulting DECODED.php file is ~ 108kb in size (vs. ~ 152kb originally), and seems to heavily borrow code from phpmailer. What we got now, is the source code to some sort of complete framework to send mails featuring things like DKIM as well.

FreeBSD as KVM guest

FreeBSD is a great O/S in itself but let’s just run it as a KVM hypervised instance for testing purposes as described in the following quick introduction.

1. Base Install

At first, we create an image by e.g.

qemu-img create -f raw NWS3-FBSD10 64G

Using VMM, the VM can be setup easily, choosing linux/wheezy as O/S will activate virtio drivers, which is quite important. For performance reasons, the VM should be set to use 4GB of RAM (also to enable zfs preloading) and 4 CPU cores.

FreeBSD will be installed onto the vtblk device, and it will use vtnet as  10Gb ethernet adapter. Benchmarking via iperf shows that it comes quite close to the max:

[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9.68 GBytes 8.32 Gbits/sec

2. S/W Installation

After having installed the O/S, a couple of tweaks have to be made to be able to really run and use the KVM guest. First, I always want my tmux, so I build it via:

cd /usr/ports/sysutils/tmux; setenv BATCH yes; time make install

Installing other S/W from binary packages is as simple as e.g.

pkg install pidgin-otr firefox xorg nmap ettercap trafshow ngrep

3. Building a VESA Kernel for Xorg

The window manager shall startup after reboot, and we still need a VESA kernel for that, so we edit /etc/rc.conf and add:


Now its time to build a new kernel w/ VESA enabled:

cd /usr/src/sys/amd64/conf

Edit VESAKERN and add:

options VESA

Then build and install the kernel:

cd /usr/src
time make buildkernel KERNCONF=VESAKERN
time make installkernel KERNCONF=VESAKERN

Gnome needs procfs, so we add the following line to /etc/fstab:

proc /proc procfs rw 0 0

4. Further Tweaks

Now, it is time to halt the VM, and do a backup if you wish so. I recommend using qemu-img convert (64GB => 8GB) and gpg (8GB > 4.5GB) to accomplish that.

To make the mouse work, it is essential to do the following in VMM:

Add USB Mouse,Remove Tablet, Set VMVGA Graphics

Done! Overall time should be ~ 1hr