Upgrade Makes Mirabox Unbootable

For some reason (I can't really put a finger on it exactly), I like to keep my servers up-to-date. I know! All those security patches, all those new features-- they're a killer!

In all seriousness, I host several websites on a Mirabox from right behind my TV. I hopped on the plug computing train way back when the SheevaPlug was just a dev kit, and I've been on it ever since. The reasons are pretty simple: reasonable power and capabilities while using less power than my desktop computer when it's off (i.e. phantom power). I mean come on-- the Mirabox in particular has two gigabit ethernet ports and two USB 3 ports. I have a fantastic RAID 1 setup running on that. Anyway, I digress.

The Mirabox comes with a custom flash of Debian Squeeze on it that worked just fine. A few weeks ago I decided to get it up-to-date.

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get dist-upgrade # What was I thinking...

And reboot and... the Mirabox never came back. It just hung when it tried to bring the network interfaces back up. This is the end of the boot log:

<snip>
[ ok ] Setting kernel variables ...done.
[ ok ] Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
[FAIL] startpar: service(s) returned failure: module-init-tools ... failed!
pool #0: pkt_size=1536, buf_size=1632 - 2048 of 2048 buffers added
eth0: started
ADDRCONF(NETDEV_UP): eth0: link is not ready
mlan: module license 'Marvell Proprietary' taints kernel.
Disabling lock debugging due to kernel taint WLAN FW is active
ADDRCONF(NETDEV_UP): uap0: link is not ready
Bluetooth: Core ver 2.15
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.14
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM ver 1.11
WLAN FW already running! Skip FW download WLAN FW is active
ADDRCONF(NETDEV_UP): mlan0: link is not ready

This was, believe it or not, somewhat of a blessing in disguise, since I should NOT have been running the OS off of internal memory. Why, you ask? Because this just happened. There was no way to recover-- I tried booting to different runlevels, just the shell, you name it.

I ended up rebuilding the server from scratch, but on an SD card this time so I could at the very least back it up (via dd) as I made progress.

And you know what the issue was? Check out the sources.list:

deb http://ftp.us.debian.org/debian/ squeeze main contrib non-free
deb http://http.us.debian.org/debian stable main contrib non-free
deb http://security.debian.org squeeze/updates main contrib non-free
deb http://www.backports.org/debian squeeze-backports main contrib non-free

Does anything jump out at you? Perhaps the fact that the second line is actually the same repository URL, first of all. But that's not the bad part: notice the mixture of squeeze and stable repositories. Stable is now Wheezy! Whichever idiot at Globalscale who set this thing up just potentially screwed all of you by mixing releases. That person has wasted nearly two weeks of my time.

The solution is simple: delete the second line. The URL was redundant anyway.

Now you're just using Squeeze!

You might ask: "Well, why not just update to Wheezy?" Because the wheezy repos contain the package updates that cause the Mirabox to no longer boot. Which packages, you ask? One of these five (they're all interdependent... I'm not sure which one breaks it):

  • initscripts
  • netbase
  • rsyslog
  • wpasupplicant
  • ifupdown

You'll notice in the install logs (if you go ahead with that madness) that "netbase breaks ifupdown." I suspect the root cause to be somewhere around there.

If you figure out how to successfully upgrade to Wheezy, let me know! For now, I'm happy with Squeeze.

Comments

Not sure, but maybe debian changed the way to boot services concurrently. Did you had figured out how to solve it?

I just had breaked my debian install - i'll recover the stuff, but… would like to have it current.

I’m thinking in chroot the rootfs and then update inside it, but…

By CQ

Reply

I understand about wanting this stuff current… I totally agree. I’m honestly not sure what changed, and I haven’t figured out a way forward… I still running Squeeze, and it works for now. I’m sorry yours broke, too! Maybe Globalscale will figure it out… I've made sure they're aware. You should email them, too.

By Kyle

Reply

I’ve received a Mirabox from GlobalScale just now with Wheezy on it. If there's a way I can copy the kernel and rootfs off it (nand read? http://elinux.org/DidjUBootFlashingPrimer#nand_read) I can make them available. I’d like to have a copy for my own backup purposes, too. It seems these things are not (yet) available for download anywhere.

By hank

Reply

That’s amazing! First of all, I strongly recommend you install on an SD card instead of using the internal memory (much easier to backup/restore that way). Globalscale gave me a link to the image to use for Squeeze-- you should shoot them an email and ask for the one for Wheezy. I bet they have it somewhere!

Thanks for letting me know! I’ll shoot them an email as well. Check your sources.list and make sure they didn’t make the same mistake!

By Kyle

Reply

Thanks for the advice with the SD card, but out of the box the wheezy is installed on the internal NAND. Any way to move it off? I might ask for an image but I am not putting my hopes up high.

What’s with the sources.list? What can go wrong there?

FYI I’m also trying to re-start this discussion: https://newit.co.uk/forum/index.php?topic=4042.0

By hank

Reply

First of all, regarding the problem with the sources.list, that’s what this blog post was all about! Make sure they didn’t mix named releases with "stable," since "stable" changes when a new version is released and that where problems begin. Make sure that all your repos use the word "wheezy" instead of "stable" and your server should be up and running for a long time.

As for the SD card bit, I had to email them about it and they sent me a guide for how to do it along with a link to a Squeeze kernel and bootloader for the SD card. What I have won’t help you though, since you have the shiny Wheezy kernel. Shoot them an email and say that you got this thing with Wheezy on it and you want to install to the SD card instead of internal NAND. Ask them how.

By Kyle

Reply

Ah yes sorry about the confusion about sources.list. This seems to be fixed in their wheezy, it properly says "wheezy" and not stable or what. I had read somewhere else (https://newit.co.uk/forum/index.php?topic=4042.0) that there was a kernel problem, too, so not just sources.list and some packages that are causing upgrade troubles. Now that I have this here, can I pry the kernel and rootfs out of the nand? I booted via the rescue disk (http://www.newit.co.uk/forum/index.php?topic=3880.0) and copied (with dd) the mtdblock1 and mtdblock2 but can’t mount them. I have emailed their support but would rather not depend on them.

By hank

Reply

got it all figured out. the dd copy of the mtdblock1 is the kernel uImage. the /dev/mtd2 could be mounted as UBIFS and I just tar’d the whole file system off it. the put it on a micro SD according to above rescue disk instructions. after setting the boot args just issue "saveenv" to make it permanent. booting wheezy off micro SD now, all good!

By hank

Reply

Very good! Yeah, the UBIFS took some getting used to for me. Can you post your uImage and filesystem somewhere?

By Kyle

Reply

try this http://rapidshare.com/share/383FE2D9B8616D0504AB809AA5C37649

By hank

Reply

Thanks Hank!

By the way, just for clarification, I don’t think it’s the kernel that was the problem as referenced in the forum post to which you linked. He said he updated and it didn’t boot, so he blamed it on the “kernel.” I specifically blame it on one or more of the five packages I listed above. I had no issues with the kernel images.

Thanks again!

By Kyle

Reply

Interesting. Btw have you tried upgrading the internal WiFi PCIe card? I’d like to have dual band (2.4Ghz + 5Ghz) so I’ve ordered an Intel 6320 and an Intel 6325 to experiment with.

By hank

Reply

I have not. I’m making use of the gigabit myself-- not even using the wifi. I’d love to hear how it works!

By Kyle

Reply

The kernel was apparently compiled w/o the iwlwifi driver and I don’t have a .config to recompile the kernel. /proc/config.gz does not exist. This is a problem even irrespective of wireless driver stuff. Ouch. I hope to get support from Globalscale in the future on how to recompile the kernel (what .config, addnl. modules etc.)

By hank

Reply

Turns out the new box was actually delivered with a dual band wireless module. D’oh. See https://newit.co.uk/forum/index.php/topic,4042.msg11268.html#msg11268 (This means that above image may not work for you though if you have the older wireless module and need to use it.)

Now I just need to retrofit my other older Mirabox to dual band. The 3.12 mainline kernel seems to run well and support Mirabox. https://newit.co.uk/forum/index.php/topic,4036.15.html

All further discussion in the NewIT forum.

By hank

Reply