Hacking a Cisco/Linksys NSS6000

So I was given a Cisco/Linksys NSS6000 to upgrade and root.  Luckily I have was also provisioned with the instructions to root this machine.

Thanks to some hacker types that had already been and done this the process was relatively straight forward.

  1. Create User
  2. Insert USB Key
  3. Backup Configuration onto USB Key
  4. Unmount USB Key
  5. Dive into the tar ball (which is simply /etc) and:
    1. Change the root password in etc/password – I just copied my new users password!
    2. Added the following line to etc/cron.d/root
      */5 * * * * /usr/sbin/dropbear_start.sh
  6. Tar the extracted files
  7. Put the tarball back on the USB drive
  8. Mount it in the NAS and Restore from backup
  9. Profit, Right!?

Well,  nearly. I had a couple of issues:

Incorrect tarball Permissions

So, my first derp was when I tar’d the etc folder back up and well… instead of root owning everything, you get the picture.

What happens is that you get an error like this:

Warning: touch(): Unable to create file /etc/nas/ran_wizard because Permission denied in /www/html/index.php on line 48

And you end up getting into a loop with dialog boxes and never ending redirects to the same page.

The down side of this is that you cannot get to any other pages in the administration to even consider doing a factory reset. Luckily you *can* post to it still

curl --data "p=admin&s=maintenance&restore_all=Restore+ALL+Settings+to+Factory+Defaults" http://admin:admin@

This command will reset the device to factory defaults – you should change the IP address and user/pass to what you need it to be. It obviously uses cURL so you will need that too 🙂

We also tried to overwrite the start of the one disk we had (using dd if=/dev/zero…) in the machine to see if that would work – but alas it did not. We were able to go through the setup wizard again, but ended back at the loop we had before.

SSH Connection Closed

The second problem I had was ssh’ing to the box. We discovered that if we used an older version of openSSH we could connect to it, but newer versions of openSSH would just not connect.

Pro Tip: Putty connects fine 🙂

To be able to connect to older dropbear’s with newer openSSH clients – try this in your ssh_config file

jason@workstation:~$ cat ~/.ssh/config 
	Ciphers aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc

I hope this information saves someone a few hours of frustrations.

The case for turning it off and on again

So the other day we started having issues with our mail server. The symptom was the mail queue showing hundreds of emails with a message like “SMTP Server rejected at greeting”. Amavis (the mail scanner / coordinator) was rejecting mail and ClamAV was not working properly. We found that simply restarting the Amavis daemon and flushing the mail queue would resolve the problem for a short time before it would happen again.

The postfix mail queue spikes
The postfix mail queue spikes

Before we managed to resolve the problem, it happened over and over again, with more and more frequency as you can see in the above graph.

The resolution? I restarted the server and waved goodbye to the 200+ day uptime. Because the file system had not been fsck’d in such a long time, it was forced and low and behold there were busted inodes and file system errors. These problems were fixed and since then the mail server has been happily behaving itself. I am also no longer scratching my head as to why load was so high while processing the mail queue and why Amavis was failing!

Have you tried turning it off and then on again?
The IT Crowd – Have you tried turning it off and then on again?

That said, I will not be reaching for the turning it off and then on again approach to resolve all the problems we encounter, as most of them can be fixed quickly if you look through the logs!

when rescan-scsi-bus.sh fails to notice disk size increases

So usually when I am doing online capacity expansions of vmware/raid devices I use a tool called “rescan-scsi-bus.sh” and it works. It detects the size increase on the disk and then I run a resize2fs /dev/sdb or something like that.
Well, I have come across a time when this tool does not work and I need to do things the hard way. Luckily the hard way is rather simple.

  1. Get the SCSI id of the device you want to use. The tool “lsscsi” does this nicely
    # lsscsi
    [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0
    [2:0:0:0] disk VMware Virtual disk 1.0 /dev/sda
    [2:0:1:0] disk VMware Virtual disk 1.0 /dev/sdb
  2. Now comes the hard part, you need to tell the server to rescan this device.
    echo 1 > /sys/bus/scsi/devices/2\:0\:1\:0/rescan
  3. And now you watch dmesg for the expansion message!
    dmesg | grep change
    [4768364.446120] sdb: detected capacity change from 171798691840 to 268435456000
    [4768364.834677] VFS: busy inodes on changed media or resized disk sdb

iSCSI and windows 7 home premium backups

So my new laptop decided to let me know to that I should setup some backups to protect my data. I was somewhat disappointed when I was not able to backup to a network target so I went in search of an answer.

I found my answer in iSCSI. I setup my linux server as an iSCSI target and fired up the windows 7 iSCSI Initiator, connected it and then formatted my new disk. Windows backup was more than happy to write to a “local” disk for it’s backups.

So how did I do this? well I setup a 50gig lun1.img file and setup my iSCSI target according to this article on setting up iSCSI on ubuntu and then followed this article on technet for the windows side of things.

Problem Solved! This is the first time I have setup iSCSI and it was surprisingly simple