PHP: for vs foreach

So this week I was asked the question on which was quicker in PHP – A for loop or a foreach loop. It turns out that my assertion that they were both about the same performance was about right. Here are my test results

jason@server:~/php$ php speedtest.php
Starting test: Test for loop data length=4000000
     Mem: 795.63MiB Used
Peak Mem: 795.63MiB Used
[xxxxxxxxx ]
Results for test Test for loop data length=4000000
Min   : 0.6085901260376
Max   : 0.61222791671753
Mean  : 0.6110053062439
StdDev: 0.00015155474344889

     Mem: 795.63MiB Used
Peak Mem: 795.64MiB Used
Starting test: Test foreach loop data length=4000000
     Mem: 795.63MiB Used
Peak Mem: 795.64MiB Used
[xxxxxxxxx ]
Results for test Test foreach loop data length=4000000
Min   : 0.41838312149048
Max   : 0.42730903625488
Mean  : 0.42159442901611
StdDev: 0.0010704358418783

     Mem: 795.63MiB Used
Peak Mem: 795.64MiB Used
On average test 2 is faster than test 1 by 1.4493x

And the code I used to run the test is:

<?php

$data_length = 4000000;//rand(10000000,20000000);
for ($i=0; $i<$data_length; $i++) {
        $data[] = $i;
}


$avg1 = run_test("Test for loop data length={$data_length}", "test_for",&$data);
$avg2 = run_test("Test foreach loop data length={$data_length}", "test_foreach",&$data);

print "On average ";
if ($avg1 < $avg2) {
        print "test 1 is faster than test 2 by ";
        printf("%0.4fx\n", $avg2 / $avg1);
} else {
        print "test 2 is faster than test 1 by ";
        printf("%0.4fx\n", $avg1 / $avg2);
}


function test_for(&$data) {
        $max = count(&$data);
        for($i=0; $i<$max; $i++) {$x=&$data[$i];}
}

function test_foreach(&$data) {
        foreach ($data as &$x) {}
}


function run_test($name, $func, &$data) {
        print "Starting test: $name\n";
        printf("     Mem: %3.2fMiB Used\n", memory_get_usage()/1024/1024);
        printf("Peak Mem: %3.2fMiB Used\n", memory_get_peak_usage()/1024/1024);
        $num_runs = 10;

        for($i=0; $i<$num_runs; $i++) {
                print "[" . str_repeat('x',$i) . str_repeat(' ',$num_runs-$i) . "]\r";
                $run_times[] = time_run($func,&$data);
        }

        print "\nResults for test {$name}\n";
        $min = null;
        $max = null;
        $mean = null;
        foreach ($run_times as $run_time) {
                $mean += $run_time;
                $max = (is_null($max) || $run_time > $max) ? $run_time : $max;
                $min = (is_null($min) || $run_time < $min) ? $run_time : $min;
        }
        $mean /= $num_runs;

        // now work out the std deviation aka confidence
        $std_dev_count = 0;
        foreach ($run_times as $run_time) {
                $dev = $run_time - $mean;
                $std_dev_count = $dev * $dev;
        }
        $std_dev = sqrt( $std_dev_count / ($num_runs-1));

        print "Min   : {$min}\n";
        print "Max   : {$max}\n";
        print "Mean  : {$mean}\n";
        print "StdDev: {$std_dev}\n\n";

        printf("     Mem: %3.2fMiB Used\n", memory_get_usage()/1024/1024);
        printf("Peak Mem: %3.2fMiB Used\n", memory_get_peak_usage()/1024/1024);

        return $mean;
}

function time_run($func,&$data) {
        $start = microtime(1);
        $func(&$data);
        $end = microtime(1);
        return $end - $start;
}

rm: Too Many Files

Ever come across a folder you need to delete but there are too many files in it?

Basically the shell expansion of * attempts to put everything on the commandline – so:

jason@server:~/images/# rm *

turns into

jason@server:~/images/# rm image1.jpg image2.jpg image3.jpg image4.jpg...

and there is a limit (albeit rather large) on the length of a command this can be a pain to try and figure out which files to delete on mass to get rid of the folder.

Fortunately there is some awesome commandline foo that you can do – and here it is:

ls -1 | tr '\n' '\0' | sed 's/ /\\ /' | xargs -0 rm

xargs will append all the file names onto the end of the rm command and run as many as needed to delete all the files. The explanation of this command is:

  1. List all files in the current folder, one per line
  2. Change all newline characters to null characters (better for xargs to split upon)
  3. Escape all the spaces in file names
  4. Finally run the rm command via xargs
  5. we could simplify this a little if we only wanted to remove jpeg images

    find -type f -name \*.jpg -print0 | xargs -0 rm

Why is my Quad Core VPS Running Slowly?

Or how a host schedules CPU cycles.

So I learnt an interesting tidbit of information the other day to do with a VPS and why it had high load and bugger all CPU usage. If you see something similar to this in top:

top - 20:30:53 up 8 days,  6:42,  1 user,  load average: 9.37, 10.81, 9.67
Tasks: 135 total,  12 running, 133 sleeping,   0 stopped,   0 zombie
Cpu(s): 30.8%us,  0.8%sy,  0.0%ni, 67.8%id,  0.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3790268k total,  3621780k used,   168488k free,   350528k buffers
Swap:  1830908k total,    11464k used,  1819444k free,  2753548k cached

over a long period of time, and you have more than 2 CPUs in your VPS – consider dropping back to 2 vCPU’s.

But Why? Surely 4 CPUs is Better than 2!

They certainly are, but when the underlying host gets busy, slots for 2 vCPU hosts get scheduled a lot more often than slots for quad vCPU’s – this is all down to the scheduler.

The case for turning it off and on again

So the other day we started having issues with our mail server. The symptom was the mail queue showing hundreds of emails with a message like “SMTP Server rejected at greeting”. Amavis (the mail scanner / coordinator) was rejecting mail and ClamAV was not working properly. We found that simply restarting the Amavis daemon and flushing the mail queue would resolve the problem for a short time before it would happen again.

The postfix mail queue spikes
The postfix mail queue spikes

Before we managed to resolve the problem, it happened over and over again, with more and more frequency as you can see in the above graph.

The resolution? I restarted the server and waved goodbye to the 200+ day uptime. Because the file system had not been fsck’d in such a long time, it was forced and low and behold there were busted inodes and file system errors. These problems were fixed and since then the mail server has been happily behaving itself. I am also no longer scratching my head as to why load was so high while processing the mail queue and why Amavis was failing!

Have you tried turning it off and then on again?
The IT Crowd – Have you tried turning it off and then on again?

That said, I will not be reaching for the turning it off and then on again approach to resolve all the problems we encounter, as most of them can be fixed quickly if you look through the logs!

Using varnish as a HTTP Router

A Layer 7 Routing Option

So one of the novel uses I have put the Varnish Cache to is a HTTP (Layer 7) Router.

Our Setup:

We have a single IP address that forwards port 80 to a Virtual Machine. This virtual machine runs varnish. We have a whole number of virtual machines that we use for development and need to be accessible from the great wild web. How do we do this?

HTTP Router Setup

The simple solution is to setup multiple backend definitions and do if statements on the req.http.host.

backend int_dev_server_1 {
    .host = "10.1.2.1";
    .port = "80";
}

backend int_dev_server_2 {
    .host = "10.1.2.2";
    .port = "80";
}

sub vcl_recv {
	// ... your normal config stuff

	if (req.http.host ~ "^(.*)dev-server-1.example.com") {
	    set req.backend = int_dev_server_1;
	    return(pipe);
	}
	if (req.http.host ~ "^(.*)dev-server-2.example.com") {
	    set req.backend = int_dev_server_2;
	    return(pipe);
	}

}

Why does windows 8 not accept my password?

I have been testing out the new Windows 8 builds from the start, but my one main problem is that once installed, I could not log back in after rebooting! I wrote this off to there being something wrong alpha and beta quality software and didn’t think more of it.

So when I was installing the release preview last night, I kept closer eye on things and noticed that the password field in the setup screen stops taking input after the 16 character has been entered. I also noted that the password field you use to log in will quite happily take more than the 16 characters that the initial box is limited to, thus giving you a bad password!

This is normally all find and dandy, except when you start using xkcd style passwords that are easy to remember and hard to brute force.

The main issue though is that there is no obvious indication that the password length has been cut off. I mean, I have been using this password with my microsoft services for a while now without issue – why should windows 8 be any different?

when rescan-scsi-bus.sh fails to notice disk size increases

So usually when I am doing online capacity expansions of vmware/raid devices I use a tool called “rescan-scsi-bus.sh” and it works. It detects the size increase on the disk and then I run a resize2fs /dev/sdb or something like that.
Well, I have come across a time when this tool does not work and I need to do things the hard way. Luckily the hard way is rather simple.

  1. Get the SCSI id of the device you want to use. The tool “lsscsi” does this nicely
    # lsscsi
    [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0
    [2:0:0:0] disk VMware Virtual disk 1.0 /dev/sda
    [2:0:1:0] disk VMware Virtual disk 1.0 /dev/sdb
  2. Now comes the hard part, you need to tell the server to rescan this device.
    echo 1 > /sys/bus/scsi/devices/2\:0\:1\:0/rescan
  3. And now you watch dmesg for the expansion message!
    dmesg | grep change
    [4768364.446120] sdb: detected capacity change from 171798691840 to 268435456000
    [4768364.834677] VFS: busy inodes on changed media or resized disk sdb

MD Raid 5 with 2 dropped disks

So today I was happily going about stuff when there was a brown out. nothing turned off so I was happy to continue going about doing whatever I was doing. Until I got these emails

Subject: Fail event on /dev/md1:server
This is an automatically generated mail message from mdadm
running on server

A Fail event had been detected on md device /dev/md1.

It could be related to component device /dev/sdg.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid5 sdi[3] sdh[1](F) sdg[0](F)
2930274304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/1] [__U]

md0 : active raid5 sda[5] sdb[0] sdd[1] sde[3] sdc[4]
7814051840 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]

unused devices: <none>

and

Subject: Fail event on /dev/md1:server

This is an automatically generated mail message from mdadm
running on server

A Fail event had been detected on md device /dev/md1.

It could be related to component device /dev/sdh.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid5 sdi[3] sdh[1](F) sdg[0](F)
2930274304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/1] [__U]

md0 : active raid5 sda[5] sdb[0] sdd[1] sde[3] sdc[4]
7814051840 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]

unused devices: <none>

After the small panic attack that my backups raid array had gone walk went about trying to fix it.

Root Cause Analysis

So It turns out that the power brown out had knocked two of my disks offline for a brief second and when they came back online Ubuntu said “oh hey – new disks!” and promptly decided to give them new names. This in turn caused my /dev/md1 array to go bonkers.

The Fix

Well, I wanted this fixed quick – so I turned my Linux server off and then on again – thinking that the disks will renumber themselves back into what they should be and the MD1 array would reassemble.

I was half right, I still needed the array to assemble and run after the reboot. Here is the command I used to get things going again.

mdadm --assemble /dev/md1 --force --run

and there we are, making an MD array run properly after power failure. I am just lucky that there was no writing going on at that time.

Maybe time for a UPS and some ZFS loving?

Samba veto and .DS_Store OS X files

So i have a samba file share that is accessed by several macs here at work. and since I do not like their .DS_Store files, I have added a couple of lines to the smb.conf to make my live easier! In the [global] section, add these lines:

veto files = /.DS_Store/._.DS_Store/
delete veto files = yes

western power is fun!

*yay* power outage… good enough time to try doing this portable hotspot thing through the android phone. Let’s just say that the vodafone Edge network is laaaaaaaggyy

Reply from 110.173.227.121: bytes=32 time=936ms TTL=47
Reply from 110.173.227.121: bytes=32 time=1664ms TTL=47
Reply from 110.173.227.121: bytes=32 time=3786ms TTL=47
Reply from 110.173.227.121: bytes=32 time=9418ms TTL=47
Reply from 110.173.227.121: bytes=32 time=6857ms TTL=47
Reply from 110.173.227.121: bytes=32 time=5424ms TTL=47
Reply from 110.173.227.121: bytes=32 time=5629ms TTL=47
Reply from 110.173.227.121: bytes=32 time=485ms TTL=47
Reply from 110.173.227.121: bytes=32 time=413ms TTL=47
Request timed out.
Reply from 110.173.227.121: bytes=32 time=2907ms TTL=47
Reply from 110.173.227.121: bytes=32 time=1394ms TTL=47
Reply from 110.173.227.121: bytes=32 time=2046ms TTL=47
Reply from 110.173.227.121: bytes=32 time=1841ms TTL=47
Reply from 110.173.227.121: bytes=32 time=1122ms TTL=47
Reply from 110.173.227.121: bytes=32 time=304ms TTL=47
Reply from 110.173.227.121: bytes=32 time=326ms TTL=47
Reply from 110.173.227.121: bytes=32 time=438ms TTL=47
Reply from 110.173.227.121: bytes=32 time=346ms TTL=47
Reply from 110.173.227.121: bytes=32 time=372ms TTL=47
Reply from 110.173.227.121: bytes=32 time=385ms TTL=47
Reply from 110.173.227.121: bytes=32 time=305ms TTL=47