But wait, there’s more! If you don’t have the hardware and don’t want to build it, or you want to try out OpenLase, or you want to be able to mess around with it on the go, you can now do that. There’s a new OpenGL-based simulator in the OpenLase tree. It works off of the JACK data (so you still need JACK) and it tries to simulate the dynamics of my laser scanner, including brightness effects and some of the physical limitations of the galvos. Here’s a comparison of the simulator vs. the real thing:

I’m aware that documentation on the software is still sorely lacking. Please bear with me while I get my act together and write that up

Ah, the world of computers. Thanks to the wonderful world of bits and bytes, we can experiment with any application, file, driver, or even the core operating system. Rip them apart, change things, put them together, and if it doesn’t work, just try again. At worst, you’ll have to wipe your hard drive and start over. If you somehow manage to destroy a computer purely through bad software, that’s considered a design problem and a true feat to pull off. Just think about it: what other profession or hobby lets you experiment as much as you want and make as many mistakes as you want without having to spend a cent if you do something wrong?

Unfortunately, things have changed. Ever since the advent of embedded devices with upgradable firmware, people have been trying to modify and hack them. These devices are usually a lot less resilient than their bigger, older siblings. Many of the new shiny gadgets that we use every day are internally fragile and a slight software mishap can render them non-functional, a “brick”.

This is a guide for developers and hackers who work on system firmware for embedded devices.

Care About Your Users

The first step towards safe hacking is to develop a deep appreciation towards your users and, especially, their hardware. Most users are clueless and entirely dependent on you to guide them towards a safe result. While everyone releases hacks with no warranty neither express nor implied, that’s just to cover their ass. Remember, users are usually completely lost if they make their devices inoperable, and, unlike you, probably won’t have a backup plan.

One great way to start developing an appreciation for each individual piece of hardware out there is to deeply care for your own. If you’re a hacker with a very low budget, you probably have this already, as you’ll want to keep your device functional for as long as possible to avoid having to spend your hard-earned cash on a new one. If you have to perform emergency repairs on it or flashing, take notice every time you do so. You might have had to spend a few hours wiring up a flasher to your device. An average user probably doesn’t have a chance of being able to do that in a week. And if you have the resources to purchase a few devices for testing purposes, keep in mind that most users don’t have that luxury. It might be tempting to run wild and experiment once you have a recovery plan in place, but remember, every mistake that you make is a mistake that might slip and end up affecting your users instead. If you take great pains you avoid bricking your own hardware, you’ll greatly decrease the chances of a critical mistake making its way into the release version.

If you still decide not to care for your users, make it plainly clear in the product documentation that they are entirely on their own, and that you don’t care about what happens when they run your tool, nor have you make any attempt to make it safe for everyone. They deserve to know.

Understand the System

Before you start working on software that makes permanent changes to a device, you should have a deep enough understanding of its operation. Reverse engineer the boot process. Understand what parts of the firmware depend on what. Know what components are vital for boot, and what recovery modes are available, if any. If you’re the hacker responsible for performing most of the reverse engineering work on the device, you probably already know a good deal about it. If you aren’t, read documentation, try to understand everything, and talk to the person who is. Explain your idea. They will probably have many useful safety tips for you. Work on less intrusive hacks that will deepen your understanding of the system before moving on to riskier hacks that might end up in a brick. Above all, work with other people who also work on that device. Every extra knowledgeable person working on a firmware hack multiplies its chances of being safe.

Program Defensively

Usually, when a program crashes, at worst users get annoyed or lose some data. However, when unstable firmware hacks can mean that devices are irreparably destroyed, entirely different standards apply. Check all error codes. Handle out of memory errors. Make sure there’s enough free disk space. Make sure headers are sane. If you’ve never written a stable app before, one that can gracefully handle most exceptional conditions without crashing or doing the wrong thing, you should seriously reconsider working on critical device firmware hacks until you do so. Learn about what kinds of problems to expect on safe ground first, before you move on to shakier terrain.

One great technique to use is to do as much as possible in advance. Gather all required information about the system, read any required data files, prepare any modifications, and only at the very end actually commit the changes to the device. If anything goes wrong during the preparations, you can just abort the entire operation and be certain that the device is still safe. If you don’t want (or can’t) architect your program like this, you can still tack it on as an underlying layer. Make the low-level functions that perform the actual changes (e.g. write to Flash) actually write to a temporary buffer instead, and bulk write everything at the very end. This also gives you a chance to check the result of the operation virtually, before it’s actually committed. It might even speed up your program as a side effect (bulk writes are faster than scattered ones).

Fail Intelligently

If you’ve followed the prior advice, you’ll have already minimized the amount of code that can fail and cause catastrophic damage to firmware. However, most of the time, there’s always something that might go wrong at just the wrong time. If a critical operation fails, the worst possible thing you can do is panic the application or otherwise halt! Then you’re guaranteed to brick the device. Instead, drop the user into some kind of failsafe mode, shell, or launcher, and direct them to keep the device powered on and seek immediate attention (e.g. on an IRC channel). If there’s a chance of saving the device, even if you have to work together with the user to develop an improvised fix, take it. He or she will be eternally grateful to you.

Sanity Check

Don’t assume anything about the user’s environment. Manufacturers often release dozens of firmware updates, and the number balloons to hundreds or even thousands if you start to consider the possible combinations of hacks that users might have already applied. Profile the system and ensure that everything is sane before you start. If you need to read any data from a user-supplied file or from the network, make sure it is exactly what you expect it to be. You can’t possibly have too many sanity checks.

Cryptographic hash algorithms (such as SHA-1) are a great tool here. Build a database of known-good firmware hashes. Include the hash of the expected result after running your program, so you can check against it before actually writing it out. If you miss an existing firmware that would’ve just worked, that only means you have to add it in and release a new version. If you don’t perform the check and that firmware turns out to be incompatible, you’ve just created a whole class of users that will be bricked by your tool. Blind patching is a recipe for disaster.

You should also make your application check itself, to make sure it hasn’t been corrupted (due to a bad download, bad media, or even bad memory), including any auxiliary files that it needs. Hashes also work great here. You can make this as simple or as complex as you want. You can have the executable check its readonly sections against built-in hashes in memory. Or you can just have a .txt file with hashes of all your files (including the main executable), and check them at runtime before anything else. Sometimes just packing your executable with an executable packer will give you this feature for free (but make sure the packer does, in fact, offer integrity protection).

Protect Users From Themselves

Users will do completely stupid things. It’s not just that they will click on things without understanding what the outcome will be; if you include a big red button that says “Brick Me!”, someone will click it too. That’s why you should at least make it hard for users to destroy their system. Sure, you can just blame them for their own incompetence, but it’s worth covering for the obvious cases. If there’s an option in your hack that will undoubtedly brick a user’s system under a conceivable set of circumstances, check for them and disable the option in that case. There’s no excuse for having a button that deletes critical system firmware, even if it’s marked “delete critical system firmware”. If such a feature makes sense as part of a longer process that will again result in a functional device, automate the process. If there’s a reason why a power user or developer might have a use for such a dangerous option, hide it behind a warning or two and make getting to it more annoying. Your software should pass the cat test: if a cat walks all over the keyboard (or touch panel, or game pad, or Kinect), it shouldn’t be able to cause permanent harm to the system.

This doesn’t mean that you have to try to envision every single possible situation. Users are extremely good at creatively breaking programs. But, at least, make sure they can’t accidentally destroy their systems without putting a moderate amount of effort into it.

Back Up

You should strongly consider offering a back up option to your users, or even automatically backing up critical information. Sometimes, having a backup can mean the difference between a device that can be fixed with a reasonable amount of effort (say, a hardware flasher), and a device that is forever toast and not even the manufacturer could hope to repair. If the amount of critical information is small, it’s worth putting the effort in and making sure it is automatically backed up whenever any dangerous operation is about to happen. Test it, to make sure the correct information is saved. And don’t forget to tell your users that they should keep the backup file in a safe place!

Even if your device is generally “brick-proof”, because it has a ROM bootloader that allows flashing, backing up can still be very important. Many devices store unique per-device data alongside the firmware, and the loss of that information can cause a messy repair process involving lots of manual guesswork, or worse, a device that, though technically alive, will never work again as intended (e.g. if critical calibration data or device private keys are lost). This information is usually very small. Back it up! You never know when a silly mistake will end up scribbling all over it.

Test

Ideally, you’ve put enough effort into making sure your application is safe. However, the unexpected can and does happen, and sometimes you will not have the resources to perform a comprehensive enough test. So gather up a few people that you can trust and who are willing to risk it, and perform a closed test. Do not release a public beta! People are way too impatient, and public betas are essentially synonymous with a release; people will ignore any warnings attached. You’ll want trustworthy people, preferably with the technical knowledge and skill to spot a problem before it is fatal and to have a chance of being able to fix it, if it is. Look for people with hardware experience who can put together some kind of flasher (JTAG, NOR, whatever) if things go terribly wrong.

If your application errors out on some devices, but does not cause any harm, give yourself a pat in the back: congratulations, you’ve saved your tester from a brick. If it doesn’t, and you brick your tester’s device, give yourself a pat in the back: you’ve saved dozens, hundreds, or thousands of potential end-users from a brick.

You should also make sure you’ve covered all bases with your testing. If your device has gone through multiple hardware revisions, especially if those changes are at all related to the firmware of the device (e.g. different flash chip vendors, or an entirely different firmware storage device even), you should test on all of them. If you don’t know, look around and ask. There are plenty of people out there willing to provide you with PCB pictures and chip part numbers that will help you identify any important changes. If you didn’t consider an entire hardware revision and your hack doesn’t work as expected on it, you’re guaranteeing that a huge percentage of your users, potentially thousands or tens of thousands, will brick their devices.

I hope this article convinced you to be careful when you write firmware hacks for embedded devices. If you follow the guidelines in it, you’ll save money, save your users money, and build a reputation for robust and dependable hacks. These are the previously unwritten principles that Team Twiizers followed when we developed the HackMii Installer, and we haven’t heard of a single brick out of 1.2 million installs to date. Ultimately, though, whether you follow them is entirely up to you. Is it worth it? You decide.

If your device is not jailbroken, you’ll have to wait until the new hash is reverse engineered. However, if your device is jailbroken, you’re in luck. As it turns out, the old DBVersion trick once again works to convince those devices to use the previous hash method.

In a nutshell, log in to your device via SSH, edit /System/Library/Lockdown/Checkpoint.xml, find the DBVersion key, change its value from 5 to 4, and finally reboot your device. This has been successfully tested on an iPhone 4, but I assume it will work for the others too.

Caveats regarding the iOS 2.x hash still apply. Specifically, libgpod needs some information to generate the hash. It can gain this information from a prior sync with iTunes, though this probably won’t work unless you sync again after changing DBVersion, and this hasn’t been tested. Alternatively, you can use this page to generate a HashInfo file for your device and manually copy it; this should always work.

First of all, as I’m sure everyone knows by now, I’ve been working on hacking the Kinect and writing open drivers for it. There’s a website for the community and a Git repo with the code, and it’s working fairly nicely by now.

With that out of the way, here’s a project that I’ve been working on on-and-off for the past year or so. I’ve been interested in laser scanning and DIY laser projectors, but I couldn’t find any good open source software to drive them. Specifically, I was interested in the realtime aspect: rendering and showing dynamically generated images and responding to events, not just making and preprocessing laser shows. So I set out to write my own set of software to do real-time rendering. This was the result:

DIY laser projectors commonly use sound cards as DACs. This shifts most of the processing over to the PC, but also lets us get very fine control over the realtime aspects of projection, which is what I want. Thus, my laser projector is based on a bog-standard USB soundcard, modified to pass DC. I’ll probably write a detailed article on the hardware later, but suffice it to say that it’s a galvo kit, a hacked chinese laser pointer, my own laser driver and monitoring circuitry, and some other minor parts. Total cost is about €200, if you play your cards right.

Since we’re converting laser images to audio data, why not just treat the laser data as audio in the first place? After all, laser samples are audio-rate data, and 16-bit multichannel 48kHz fits the requirements for laser projection very well. So that is what I did. OpenLase isn’t really a monolithic framework. Instead, it’s a series of stand-alone applications and chunks built on the excellent JACK audio connection kit, which serves to pipe realtime laser data around the different bits.

On a typical setup, you’d have two processes running on top of JACK. On one hand, there’s the output processor, which is responsible for formatting the idealized laser data to suit the peculiarities of the hardware. This includes things like brightness scaling, the obvious X/Y inversion settings, the final output perspective transform (to fit the screen), and minor filters to try to compensate for hardware imperfections. It also generates a 1kHz square wave on one channel – this is a peculiar safety feature of my laser hardware. I have a microcontroller monitoring this signal, such that if the software hangs or crashes for some reason, the laser shuts down immediately (to avoid having a static dot which would be a serious eye hazard). The OpenLase output processor has a simple Qt GUI that lets you tweak these settings on the fly.

On the other hand, you have whatever picture source you want to use. You can have bare JACK applications, such as two examples: ‘circlescope‘, a circular oscilloscope that takes realtime audio data from a media player, and ‘playilda‘, a bare-bones ILDA file player (ILDA / .ild is the standard file format for laser graphics). The circlescope is particularly good for showing off the real-time aspect (note that the input can come from the laser DAC’s line-in with only the small JACK buffering delay):

However, the other big part of OpenLase is libol, a realtime rendering library loosely modeled on OpenGL which lets you produce 2D and 3D graphics on the fly. This is what I used for the LASE demo above. The demo itself isn’t currently open source (and the code is utterly horrid – I wrote half of it at Euskal Encounter and finished it mere minutes before the deadline), but if there’s demand I might open source it too, just please don’t expect pretty code! However, keep in mind that most features used by the demo (text rendering, 3D, “shaders”, ILDA file loading, etc.) were implemented as part of libol, so you aren’t missing out on much. There are two libol-based examples: some rotating cubes, and Pong.

There’s one part of the demo made it into the OpenLase distribution as a separate example: tools/trace.c. This used to be some tracing code that I used for the metaballs and fire effects (I kind of cheated there, as those are rendered as bitmaps and then traced in realtime into laser vectors). It’s a terribly naive algorithm (check the source out for details), but it worked surprisingly well for certain kinds of video, so I hacked it and tacked on more heuristics in order to attempt to make it work better. It now lives next to tools/playvid.c, which is a simple video player using libavcodec. Here’s what it looks like (improved version (mkv), original YouTube video). More complex videos are hit and miss, but some things turn out surprisingly well for such a silly algorithm.

You can also add filters between the output and the image generator. This is what I did for my Kinect + OpenCV + OpenLase demo, which projects anything projectable by OpenLase onto a moving screen, with a dynamic perspective transform (in this case the perspective transform happens in the filter, not in the output processor). That code currently doesn’t even build with current libfreenect, but again, if someone is interested, drop me a line and I’ll make it work again and publish it.

OpenLase doesn’t have any facilities to patch together these JACK apps. Instead, you should just use existing JACK tools, such as QJackCtl, to connect all the input and output ports together. QJackCtl has a patchbay feature that automagically connects the ports when applications start up, so it’s quite seamless.

Right now there is pretty much no documentation, but I’d like to know if people are interested. If you have (or want to build) this kind of DIY hardware, you run Linux or some other UNIX that can run JACK, and you’re interested in hacking on the code or using it for something, please let me know! Here’s the git repo.

Update: Three more videos, musically themed. A laser visualization of MIDI data (MIDI to laser):

And the other way around, a laser harp (laser to MIDI):

Obligatory demo video:


AsbestOS (a mineral, and meaning “inextinguishable” in Greek) is a bootloader to run PS3 Linux without OtherOS. It runs using the USB GameOS exploit (on PS3 version 3.41) from any compatible device, and any reprogrammable devices currently running the PS3 exploit can be used as long as they have enough free internal or external storage (40kB or so) to hold the loader. It is general enough that it should be useful to boot Linux given any other GameOS exploit in the future. It has been tested to work on the PS3 Slim too.

Currently, it only supports netbooting a kernel and no initrd (mostly due to bootmem limitations). This is enough to run a Linux system booting from an NFS share or from USB storage media. Almost everything that works under OtherOS is working. As additional perks of running as GameOS, you also get access to a seventh SPE (needs a kernel patch to enable) and there is clearly full access to the RSX including 3D support, although we still need to learn a few details about how that works to be able to use it.

AsbestOS is a fully independent open source payload and does not contain any code from the original PSJailbreak payload or derivatives. It is licensed under the GPLv2. Compiling it does not require any SDK tools, and it includes a script to build a fully vanilla GNU toolchain for the PS3.

If you’re interested, check out the git repository. The README file contains information on how to run AsbestOS and how to set up kernels. Currently, ports exist for software USB AVRs (Arduino etc.), iPods, and the reference implementation for devices with a TI OMAP3, but anything currently running PSGroove or similar can be adapted with only a few lines of new code.

For the impatient or lazy folks, here’s a kernel that you can use Update: and a stage1 binary and a stage2 binary. You’ll probably want to change the kernel commandline options to set up your NFS root partition. This will eventually be handled by AsbestOS, but for now, open it up in a hex editor, search for HEXEDIT_THIS, and change the commandline to suit your needs (without changing the total length, of course). Do note that this kernel does not have built-in USB support, so it can only be used for NFS booting (the USB stuff is built as a module).

You can use this filesystem as a starting point. It’s a Gentoo stage3 updated to date and with PS3-specific tools installed. Keep in mind that there’s no Portage tree included, so be sure to either emerge --sync or NFS mount your server’s Portage tree (which is what I do). At the very minimum, you’ll want to edit the following files to configure your NFS and networking settings (or to specify USB device partitions, if you want to go that route – but you need to compile your own kernel then): /etc/fstab, /etc/hosts, /etc/resolv.conf, and quite likely a few others. This filesystem includes kernel modules for the above kernel. The root password is ‘ps3′.

Now, there are two things that mean you “really need to” hit the disc: reading data that isn’t already cached (duh), and the fsync() and fdatasync() calls. The latter are requests by an application to ensure that all of the data written to far has hit the disc, and they cause the disk to spin up in laptop mode.

Unfortunately, Firefox has a habit of randomly issuing fdatasync() calls. It does this as part of the SQLite backend for its various databases (especially places.sqlite), in order to avoid data corruption. Now, data corruption isn’t terribly likely on a laptop with a battery (which is essentially a built-in UPS), so this is a terrible annoyance with little benefit anyway. There has been talk about this problem, but apparently it’s still around (and the toolkit.storage.synchronous about:config property didn’t work for me). Most of the reports appear to claim that this happens while browsing, but I’ve seen it happen periodically even when Firefox is left idle. Maybe it’s caused by some extension or AJAX webapp? Who knows.

Ideally this problem would be fixed in laptop_mode itself, but that doesn’t appear to be the case and I’m not going to poke this part of my kernel just yet, so here’s a simpler userspace solution: we can preload a library into Firefox that intercepts fsync() and fdatasync() and ignores them. Even better, we can make it do that only if we’re in laptop_mode. All it takes is a little C code:

#define _GNU_SOURCE
#include <unistd.h>
#include <dlfcn.h>
#include <sys/stat.h>
#include <fcntl.h>

static int in_laptop_mode(void)
{
	char buf[2];
	int fd;
	int l;
	fd = open("/proc/sys/vm/laptop_mode", O_RDONLY);
	if (fd < 0)
		return 0;
	l = read(fd, buf, 2);
	close(fd);
	if (l == 2 && buf[0] == '0' && buf[1] == '\n')
		return 0;
	else
		return 1;
}

int fsync(int fd)
{
	static int (*_fsync)(int);
	if (!_fsync)
		_fsync = dlsym(RTLD_NEXT, "fsync");
	if (!in_laptop_mode())
		return _fsync(fd);
	else
		return 0;
}

int fdatasync(int fd)
{
	static int (*_fdatasync)(int);
	if (!_fdatasync)
		_fdatasync = dlsym(RTLD_NEXT, "fdatasync");
	if (!in_laptop_mode())
		return _fdatasync(fd);
	else
		return 0;
}

To compile and use it:

$ gcc -O2 -Wall -shared -fPIC -o killsync.so killsync.c -ldl
$ LD_PRELOAD=`pwd`/killsync.so firefox

Et voilà, firefox no longer wakes up the hard drive(s). You can confirm that the fsync/fdatasync calls no longer make it to the kernel by running strace -f -p `pidof firefox`, but only in laptop_mode.

I’ve been typing this blog post with this hack applied and so far HDD spinups have been few, far in between, and fairly short. I still have a few offenders to nail (mostly KDE related), but Firefox was one of the worst ones and it’s now history. PowerTOP says I’m saving 3-4W of power. Huzzah! Now I just need to get a kernel with timer stats built so I can fix my insane wakeups-from-idle per second…

A few notes:

• Keep in mind that if you’re using distro packages for anything I’m going to mention here, you’ll need the development headers too (-dev package or similar).
• When linking to packages below, I’ll link the name to the homepage and the version to a download link or something close to it: if you’re reading this post soon after it was posted, then the versions should be up to date. If it’s been more than a few days, you should check for newer versions.
• You’ll also need cmake to build some things (2.6 or newer); your distro should provide a package for it.
• Most packages will install to /usr/local by default. This should be fine, but you may have to uninstall any older distro packages that live at /usr, and apps may or may not pick up libraries on /usr/local. If you want, you can change the prefix to /usr. You should already know how to do this for autotools-based packages. For CMake-based ones, pass something like -DCMAKE_INSTALL_PREFIX=/usr when calling cmake.
• The libiphone/ifuse page has links to distro repositories with packaged versions of everything here. Yay, you’ve just saved a bunch of time! Do follow the steps below, but you can use these packages instead of building from source.

Possible pitfalls:

• If you’ve had prior versions of usbmuxd/libiphone/etc, make sure they’re gone, especially any udev rules files in /etc/udev/rules.d.
• If you’ve been syncing via SSH, and you log in as root, your permissions might be messed up. Run chown -R mobile /var/mobile/Media on the phone, and make sure you don’t get any permission errors when changing files via an ifuse mount.
• There’s a udev bug that causes usbmuxd not to respond to signals, which means it will never quit if you remove all devices, and it might not detect any new devices after you’ve plugged in the one that caused it to start up. As a workaround, kill -9 it, and run it from the console. A workaround for this bug is also in the usbmuxd Git repo.

Step 1

Install libusb-1.0.3 (or newer). This should be straightforward. If your distro carries it, that’s fine, just make sure it’s 1.0.3 at least (if it isn’t, the repos I mentioned above probably have it).

Step 2

Install usbmuxd-1.0.0. Make sure you read the README file, especially the part about creating a user named usbmux with USB access permissions (does not apply if you’re using distro packages, hopefully). After installing usbmuxd, you should increase the syslog debug level by editing /lib/udev/rules.d/85-usbmuxd.rules and adding -v -v flags to the end of both RUN statements, like this: RUN+="/usr/sbin/usbmuxd -u -U -v -v" (and similarly for the other line). This will get you useful debug information in syslog without flooding you with messages.

Have syslog open (tail -f, xconsole, whatever). Plug in (or replug) your device. You should see something like this:

usb 3-4.3: new high speed USB device using ehci_hcd and address 68
[...]
usbmuxd[12705]: [3] usbmux v1.0.0 starting up
usbmuxd[12707]: [4] Creating socket
usbmuxd[12707]: [3] Successfully dropped privileges to 'usbmux'
usbmuxd[12707]: [4] Initializing USB
usbmuxd[12707]: [4] Found new device with v/p 05ac:1290 at 3-68
usbmuxd[12707]: [4] Using wMaxPacketSize=512 for device 3-68
usbmuxd[12707]: [3] Connecting to new device on location 0x30044 as ID 1
usbmuxd[12707]: [4] 1 device detected
usbmuxd[12707]: [3] Initialization complete
usbmuxd[12707]: [3] Connected to v1.0 device 1 on location 0x30044 with serial number be2975afb30b6db9025f95261b9e0a7041044661

If you don’t see anything related to usbmuxd, either you forgot the -v -v or you need to reload your udev rules: $ sudo udevadm control --reload-rules. As a last resort, try rebooting. If it still doesn’t work, something’s wrong. You can continue by running usbmuxd manually as explained in the README, but you should report a bug.

If you have a jailbroken phone with SSH, try running $ iproxy 2222 22 and then SSH to localhost, port 2222 on another console. You should be able to SSH into your phone.

Unplug your phone. If usbmuxd was started via udev (not manually), it should quit. If it doesn’t, report a bug.

Step 3

Install libplist-0.16 and then libiphone-0.9.4 and ifuse-0.9.4.

Now you should be able to mount your phone by simply running $ ifuse <mountpoint>. You should be able to see the media folder structure and even grab photos and other stuff. If you have a jailbroken phone, you can use the --root option to mount the root filesystem with root privileges (I usually prefer ssh/sshfs for this, though). Do NOT use the --root option to sync music, it will not work and may mess up your permissions.

If you have access to more than one iPhone or iPod Touch, we could use some multiple-device testing! Usbmuxd should happily start up when you plug in the first device, keep running while you plug in any other devices, and stop once all devices have been removed. Usbmuxd will report the device UUIDs (it calls them “serial numbers”) as they are connected (if you’re using -v -v). You should be able to pass the --uuid=<uuid> option to ifuse to mount a specific device. Please test this for us!

Intermission

If you are not used to alpha software, checking out development sources, testing patches, and working with developers, stop here. Now you’ve got all the underlying stuff working, so you’re probably better off waiting for some time until libgpod stabilizes some and things are merged and tested.

If you are a developer willing to help, a very enthusiastic power user, or simply a masochist, feel free to read on. Be warned: pain lies ahead, don’t even think about using the following for day-to-day use at this point in time. From here on there are no distro packages and no nice source tarballs. Experiments and feedback only. This Will Eat Your Music Collection For Dinner And Replace It With Hours And Hours Of <insert terrible music>.

Step 4

Clone teuf’s sandbox Git repository for libgpod, and switch to the iphone30 branch the libgpod git repository at SourceForge, then compile and install it:

$ git clone git://gtkpod.git.sourceforge.net/gitroot/gtkpod/libgpod
$ cd libgpod
$ CFLAGS="-g -O0" sh autogen.sh --prefix=/usr
$ make
$ sudo make install

Step 5

Mount the device and create the iTunes_Control/Device directory. Then, get your UUID (it should be in the syslog from usbmuxd, or you can find it by running $ lsusb -v | grep -i iSerial). It should be 40 characters long. Then, run $ tools/ipod-read-sysinfo-extended <uuid> <mountpoint>. This should generate a file named iTunes_Control/Device/SysInfoExtended. Make sure it’s not empty and whatnot; it should be a large-ish plist (XML file) with a bunch of info.

Step 6

Make sure your device already contains a valid music database created by iTunes. If it doesn’t, drop me a line on IRC and we can try some cool new stuff. New databases can be created, but there might be issues and you need some way of generating the HashInfo file (more on this later).

Step 7

Open an application that uses libgpod to manage iPods. Gtkpod is recommended at this stage, as its what I use to test. Other apps (Rhythmbox, Amarok, etc) theoretically should work. Theoretically, because Amarok 1.4 crashes/hangs pretty badly for me, for example, and I’m not sure who is to blame. I’d like to hear any success or failure stories. Please run your app from a console so you can see any debug messages.

First, load your device. Once the application has accessed the music library, a new file called HashInfo should have popped up in the iTunes_Control/Device directory. If it isn’t there, you’ve got a problem: check stdout for any messages and and preferably ping me on IRC or at least e-mail me your UUID and your iTunes-created iTunes_Control/iTunes/iTunesCDB file (it shouldn’t have been modified yet).

Make some changes (add a song, make a playlist, etc), save, and cross your fingers. With any luck, you’ll get some debug spew, any songs will transfer, the iPhone/iPod will show a pretty “syncing” screen, the database will be written, the “syncing” screen will go away, and you’ll be able to launch the iPod application and see your changes and play any new songs. There will be bugs at this stage, but please do report any issues along with expected behavior.

Step 8

Plug your device into a computer running iTunes (unmount it on your Linux computer first!) and check that it can see the changes made with libgpod. This is very important, because iTunes and the device use different databases: the data managed by iTunes and libgpod is maintained in an iTunesCDB (a compressed version of the old, classic iTunesDB format), but it is converted to a new SQLite database format for the iPod app to use. So both libgpod and iTunes read the iTunesCDB, but they write out the iTunesCDB and the SQLite databases. You should be able to make changes in iTunes and swap your device from iTunes to libgpod and back and see them, etc.

Bugs

Tons. The only things tested to work are managing simple music and working with simple playlists. There’s a known annoying bug where the iTunes special playlists (Ringtones/etc) are converted to normal playlists by libgpod, and then iTunes recreates them if you use it again, creating duplicates each time you switch between using libgpod and iTunes. The DB schema is outdated (3.0 instead of 3.1), so you’ll see a (hopefully short) “Updating” step when you launch the iPod app. Please report issues, but keep in mind that they’re to be expected. This is very very very alpha code. I probably screwed up something on this post even, so chances are you’ll get stuck somewhere. Tough luck, it’s 7am and I need to catch some sleep

Hope you had fun! Keep in mind that you can usually find us on #gtkpod on freenode if you need to report an issue. Comments on this post are fine too, for now.

• The iPhone uses a custom USB communications protocol, connection-oriented, which is totally different from standard mass storage devices
• It provides multiple services over this protocol, and many have to be interacted with to complete sync
• iPhone OS 2.x brought us a new hash (designed to lock users into iTunes) that so far had not been reverse engineered
• iPhone OS 3.x brought us a new SQLite-based database format which, although a lot more Linux-friendly than the old proprietary format, does require a bunch of new code (and the old format still needs to be supported for compatibility).

Well, I’m happy to say that all of this is coming to an end and that Linux will finally get quite complete iPhone/iPod Touch syncing support soon, thanks to the work of many dedicated people. This is what the software stack looks like:

• libusb-1.0 provides an advanced API to access USB devices under Linux, replacing the old libusb-0.1 API
• usbmuxd coordinates application access to the device and talks the specific iPhone/iTouch USB protocol
• libiphone implements the Apple-specific protocols that are tunneled through usbmuxd: it can launch services through lockdown, retrieve device info, send notifications, and access the filesystem via AFC.
• iFuse and gvfs-backend-afc both provide access to AFC to regular Linux apps. iFuse does this by mounting via FUSE, while gvfs-backend-afc is obviously a backend for gVFS.
• libgpod (the library that traditionally has managed music databases for iPods) is being extended to support the new SQLite format, the new hash, and also to talk to libiphone to properly put the device in to and out of sync mode.
• Theoretically, actual music players such as Amarok and Rhythmbox will need none or very few modifications to work.

I originally started work on usbmuxd which I am maintaining, and I’m now helping libgpod compatibility with the new formats. The software stack needs to be working from the bottom up, so I’ve just tagged and released the first Release Candidate for usbmuxd-1.0.0. If you have an iPhone or an iPod Touch and you’d like to use them with Linux in the future, I encourage you to test it out. If your device is jailbroken, you get the immediate benefit of being able to SSH over USB (which is a lot faster than over WiFi!), and if you’re using firmware 2.x or earlier and the hacky sync-over-WiFi-with-sshfs method, you can immediately get a speed boost by doing it over USB.

If you’re feeling adventurous, the development versions of libiphone and iFuse are also quite good. Theoretically, if your usbmuxd is working an you’ve got iFuse installed, you can just type “ifuse <mountpoint>” and instantly have your device mounted. Don’t expect to get libgpod to work at this time without quite some pain though. If you’re a developer and you’re feeling really adventurous though, you can join us on #gtkpod on freenode and help us test and fix the new libgpod support!

Please report any bugs or fixes in usbmuxd either directly to me or to the iphone-linux-dev mailing list. Any issues with libiphone or iFuse should also go to the list.

I certainly wanted to use a feature that I paid for, so I started investigating the BIOS and here’s what I found out.

Update: There’s a 1.21 BIOS floating around that seems to have VT enabled natively. See below.

Under the hood

The InsydeH2O BIOS is no ordinary old-style BIOS. Instead, it’s based around the UEFI platform. This goes way beyond the old BIOS paradigm and turns system firmware into practically its own separate OS, that even runs in full 64-bit mode on 64-bit machines. Unfortunately, they make no effort to expose any of this to the user. The firmware has support for booting EFI executables, there’s an EFI shell, there’s an EFI boot manager… but I haven’t been able to figure out how to access any of this.

If you want to reverse engineer EFI stuff, downloading TianoCore’s EDK2 is a must. It contains source code for a lot of Intel’s framework, which is what most vendors use as a base for their EFI support. A lot of the code is exactly the same as what’s in the Insyde BIOS (read the spec here).

As for the Setup tool, it does indeed have a huge Advanced menu with even more options than your average desktop. There’s also a hidden Power menu. EFI defines a “form browser” protocol and formats for user input, which is what Insyde uses for their setup utility (spec here). I found these tables when disassembling the Setup binary and wrote a little dump utility to turn them into text. The result is a complete dump of the Setup hierarchy, including the Advanced menu, which also includes the offsets in the non-volatile storage corresponding to each setting. Insyde stores this configuration blob into an EFI variable named Setup. Here’s my dump: the first part is the hierarchy, while at the end I added a rough auto-calculated mapping from configuration offsets to setting names (grep for [0xOFFSET in the top section for better context - the format is [0xOFFSET<FIELD_WIDTH>] for all references to the storage blob). You’ll find the tools I used here, if you’re interested, but they’re rough and need quite a bit of manual help too.

I wasn’t able to find out how to enable the hidden menus, other than that their form Subclass is 5 instead of 0 (but I haven’t found what, if anything, checks for this and whether its behavior can be altered). However, manually enabling VT support in the Setup variable is easy enough, now that we have the offset of the VT Enable byte.

Enabling Intel VT

The easiest way to enable the setting as far as I can see is to dump out the entire BIOS, patch the setting into the Setup variable (which is part of the data storage section – we aren’t modifying any actual BIOS code, as this is the equivalent of changing a CMOS setting on other BIOSes), and then flash the resulting image. These laptops use a weird flash-behind-EC hardware solution for which there is no open flasher, so instead we can just use the normal BIOS flashing tool. In short, we’ll flash the existing BIOS back on, but in the process also modify a Setup setting.

FAIR WARNING: This might apply to other similar laptops, or it might not. It might work, it might do nothing, or it might brick your expensive laptop. Even if you own an Aspire 8930G, I take no responsiblity if your laptop dies, turns into an expensive brick, melts into a pool of slag, blows up, flicks you off, develops self-awareness, or becomes Skynet. You have been warned. I have only tested this on an Aspire 8930G with BIOS Version 1.10. If you want to try this on another system or BIOS you should make sure you understand EXACTLY what is going on and are prepared to spot any problems or fix things yourself.

First, dump the exiting BIOS out. It resides at the top of the 32-bit address space, and is 2MB in size. You can use dd to dump it out of /dev/mem:

$ dd if=/dev/mem of=original_bios.fd bs=1024 count=2048 skip=4192256

It is a very good idea to back up this BIOS somewhere safe outside the laptop. Note that it not only contains your existing BIOS code, but also all your settings and manufacturer data (serial number, software license if you run an OEM version of Vista, etc).

Next, run vtenable.py. This will attempt to locate the Setup EFI variable on the non-volatile storage section and patch the VT byte to one.

$ python vtenable.py original_bios.fd vt_bios.fd

You can edit the source code to make other changes to the variable, but make sure you know what you’re doing. It’s worth reiterating that this does not patch your BIOS code. It only makes a setting change, just as if you’d turned on the VT option in the BIOS had it been there. In fact, there are two variables: Setup and Custom, and Setup is the one that changes are committed to when you use the setup utility. Restoring defaults should turn VT back off (untested). It also appears that Custom is probably what the setup defaults are, so changing that should semi-permanently enable VT.

I highly recommend performing a sanity diff between the original and modified images using vbindiff:

$ vbindiff original_bios.fd vt_bios.fd

Only two or three bytes should change: one or two adjacent bytes for the checksum (they should be decremented by one when you look at them as a 16-bit unsigned integer), and the VT enable byte should change from 00 to 01. Right after the checksum bytes you should be able to see the Setup name in UTF-16 (something like S.e.t.u.p.).

Finally, flash vt_bios.fd using the vendor-supplied flash utility. I use the DOS version (FLASHIT.EXE) with FreeDOS and a grub menu option so I don’t need to mess around with external media. Grab a base image here, then you can use mtools to copy the bios into it:

$ bunzip2 freedos_flashit.img.bz2
$ mcopy -i freedos_flashit.img vt_bios.fd ::/vt_bios.fd

To boot it using GRUB, get MEMDISK, part of SYSLINUX, and put something like this in your grub.conf:

title=BIOS Update
root (hd0,0)
kernel (hd0,0)/boot/memdisk
initrd (hd0,0)/boot/freedos_flashit.img

Of course, copy memdisk and the boot image to your boot partition, and change (hd0,0) to your boot (or root) partition everywhere and remove the /boot part if you have a dedicated boot partition.

Once you’re in FreeDOS, just type FLASHIT vt<tab> and be happy that FreeDOS supports tab-completion

Caveat: by doing this, you’re flashing the entire BIOS image. The flash tool makes no attempt to flash only the parts that changed, and the “flash only variables” commandline option seems to have no effect. You’re effectively reflashing your entire BIOS back on, so the usual BIOS flashing caveats apply: don’t turn the power off, etc. This could be accomplished a lot more cleanly if we had drivers for the flash chip / EC, since then we could use the normal EFI variable store procedure to atomically update the variable, which is completely safe.

You can use the MSR Magic tool to check whether VT is indeed enabled on your CPU.

Update: Several people are working on improved, more general tools to perform this hack across a broader range of InsydeH2O-based BIOSes. Read the comments and check them out, they’ve done some very good work.
Update 2: There’s a 8930G 1.21 BIOS version floating around (not on Acer’s site yet) with build date 3/3/2011. This version seems to have VT enabled without any hacking required (I’ve checked the contents and everything seems to be kosher and a real update, not some hack). I haven’t come across an official update file, but I did find a dump made by someone. I’ve manually cleaned it up to remove the variables and serial numbers so it should be identical to an official 1.21 update file (it will keep your current config and serials). You can find it here.

The upshot of this is that a whole new support library will need to be written or large changes in libgpod need to happen to support the new SQLite format. The DBVersion hack also isn’t going to work – either someone needs to reverse engineer the complete hashing algorithm (and maybe that encrypted file stuff), or the MusicLibrary binary on the phone needs to be patched. So, if you’re currently syncing music with free software, you’ll want to hold off on upgrading to 3.0.

Patching the check out of MusicLibrary looks trivial enough, so although it’s not as easy as the DBVersion hack, it isn’t a real showstopper. The hash algorithm looks the same as the old one, or at least quite similar (and just as badly obfuscated). Those encrypted files do scare me a bit though.