NWS0: H/W Upgrade

logo

As we all know, IOWAIT is the Professor Moriarty of good performance. So, I decided to replace the oldest, least powerful rootserver w/ a brandnew one. The following describes a rough overview of the steps involved to conduct a non-parallel upgrade of a productive system, which – at least in my case – succeeded perfectly.

By using virtualization, we gain a modularization of the overall functionality. However, the rootserver itself needs to be a stable, scalable (as in having enough RAM and CPU power) and ideally secure platform. In my case, that also involved server hardening, VPN, a robust firewall ruleset, GPG and monitoring of e.g. netflow and system logs as well as libvirt for better VM maintenance.

The first thing on the todo is planning the productive VM backup, which must be taken right before the actual upgrade to pertain actuality of all data. A while ago, I used to convert the RAW images to QCOW2 first (also as some sort of disk integrity check), then encrypted the resulting file using gpg. However, since the I/O of the old rootserver was rather a pain, I just skipped the conversion and directly encrypted the RAW images (gpg also compresses data) which has proven to work out very well.

What you should ask yourself now is:

  • How big is every VM (and how much data is actually in it)
  • How long does it take to create a backup of it
  • How long does it take to transfer the backup
  • Result: How long does it take for all VMs to be safely backed up

As we are already using git annex for all of our backups thanks to the tip from rpw, we of coz use this to actually store the backups redundantly @ multiple other root servers. All the configuration files are also redundantly backed up so that we are able to conduct a fast rollout.

So, after having done the steps mentioned above, the old server can be securely deleted. Wiping is most secure, but not really an option b/c we are in a hurry to have a productive system again. Reinstalling the old server 4 times w/ a fresh linux works, and the fact that most sensitive data was already stored using GPG allows to take less care @ secure data deletion.

The data center involved worked really fast, and it took less than 30 minutes for them to disconnect the old and connect the new server in their rack, even tho it was around 2 A.M.

When the fresh system was connected, the configuration rollout could take place, and after the backup restoration via git-annex (hint: git annex reinit $OLDUID really made sense here!) or via scp for only the productive VMs at first, we should take the time and adjust the CPU featureset for the VMs, e.g. using virt-manager to meet the new specs, also to overcome some ugly kvm warnings.

Last but not least, here are the times needed for the actual migration steps in my case:

  • Backup of 5 productive VMs and configs + transfer: ~ 3 hrs
  • Implementation of configs @ new server: ~ 30 minutes
  • Transfer and extraction of productive VMs: ~ 1.5 hrs
  • “Production Gap” caused by the whole migration: ~ 5hrs
  • Transfer of all git-annex data (430GB): ~ 3 hrs

In most cases, a good planning and realistic preparation (which clearly involves experience) is really a must, and if you ask me, it is also alot of fun, especially when the new server has ~ 4 times the capacity and performance of the old one while being cheaper at the same time.

Boost your old Laptop

Introduction

Many “old” Laptops from around 2003 seem to have been produced using quality components, as most of them still function properly. However, bottlenecks most certainly appear: Lack of RAM, and sluggish HD performance. The old enemy, IOWAIT, jumps on the stage. Also, the BIOS does not support booting from USB, burning a CD is rather environmentally ill and really not neccessary.

However, having thought about these issues for a bit, a pretty fast, nice and simple rollout including a complete O/S reinstall and full data migration was a matter of only minutes.

In my case, the laptop was manufactured in 2003, equipped w/ a 1.5GHz Celeron, 512MB RAM and a 80GB IDE HDD running Ubuntu 14.04 LTS.

 

Requirements of this howto:

  • old 32bit laptop
  • other laptop running KVM
  • ext. mSATA SSD chassis
  • mSATA SSD
  • Converter IDE 44 Pin > mSATA with 2.5′′ Frame
  • ext. 2.5″ IDE HDD chassis

 

Installation of O/S

After having inserted the mSATA SSD into the external chassis, we can connect that to the laptop running KVM via USB and directly install linux onto the mSATA SSD:

sudo kvm -hda /dev/sdb -cdrom ubuntu-14.04-desktop-i386.iso -m 1024

This is only a matter of a few minutes, and allows to install the i386 architecture onto the mSATA SSD even tho the host laptop’s arch is x86_64.

 

Booting up the “refurbished” Laptop

Now, remove the old IDE HD from the old laptop, put the mSATA SSD into the converter and implant that one where the old IDE HD was located. You will see that the bootup speed is really boosted by factor 5 up to 10.

 

Migrating the old Data

Put the old HDD in the ext. 2.5″ IDE chassis, connect that via USB to the freshly booted computer, and simply copy your home directory (including hidden directories e.g. for firefox or thunderbird data) in my case w/ ~ 23MB/s. This takes less than a minute / GB of data.

 

Some I/O Background

The performance of the old laptop is really astonishing, as especially one problem is eliminated: The read/write access of the O/S to the swap partition no longer hinders the actual execution flow, and subsequently, deadlock situations mostly vanish, e.g. when copying some files while at the same time opening a new terminal to copy configs and install additional software – now no longer a problem.

FreeBSD as KVM guest

freebsd

1. Base Install

At first, we create an image by e.g.

qemu-img create -f raw NWS3-FBSD10 64G

Using VMM, the VM can be setup easily, choosing linux/wheezy as O/S will activate virtio drivers, which is quite important. For performance reasons, the VM should be set to use 4GB of RAM (also to enable zfs preloading) and 4 CPU cores.

FreeBSD will be installed onto the vtblk device, and it will use vtnet as  10Gb ethernet adapter. Benchmarking via iperf shows that it comes quite close to the max:

[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9.68 GBytes 8.32 Gbits/sec

2. S/W Installation

After having installed the O/S, a couple of tweaks have to be made to be able to really run and use the KVM guest. First, I always want my tmux, so I build it via:

cd /usr/ports/sysutils/tmux; setenv BATCH yes; time make install

Installing other S/W from binary packages is as simple as e.g.

pkg install pidgin-otr firefox xorg nmap ettercap trafshow ngrep

3. Building a VESA Kernel for Xorg

The window manager shall startup after reboot, and we still need a VESA kernel for that, so we edit /etc/rc.conf and add:

gdm_enable="YES"
gnome_enable="YES"
allscreens_flags="MODE_280"

Now its time to build a new kernel w/ VESA enabled:

cd /usr/src/sys/amd64/conf
cp GENERIC VESAKERN

Edit VESAKERN and add:

options VESA
options SC_PIXEL_MODE

Then build and install the kernel:

cd /usr/src
time make buildkernel KERNCONF=VESAKERN
time make installkernel KERNCONF=VESAKERN

Gnome needs procfs, so we add the following line to /etc/fstab:

proc /proc procfs rw 0 0

4. Further Tweaks

Now, it is time to halt the VM, and do a backup if you wish so. I recommend using qemu-img convert (64GB => 8GB) and gpg (8GB > 4.5GB) to accomplish that.

To make the mouse work, it is essential to do the following in VMM:

Add USB Mouse,Remove Tablet, Set VMVGA Graphics

Done! Overall time should be ~ 1hr