Now, there are two things that mean you “really need to” hit the disc: reading data that isn’t already cached (duh), and the fsync() and fdatasync() calls. The latter are requests by an application to ensure that all of the data written to far has hit the disc, and they cause the disk to spin up in laptop mode.

Unfortunately, Firefox has a habit of randomly issuing fdatasync() calls. It does this as part of the SQLite backend for its various databases (especially places.sqlite), in order to avoid data corruption. Now, data corruption isn’t terribly likely on a laptop with a battery (which is essentially a built-in UPS), so this is a terrible annoyance with little benefit anyway. There has been talk about this problem, but apparently it’s still around (and the toolkit.storage.synchronous about:config property didn’t work for me). Most of the reports appear to claim that this happens while browsing, but I’ve seen it happen periodically even when Firefox is left idle. Maybe it’s caused by some extension or AJAX webapp? Who knows.

Ideally this problem would be fixed in laptop_mode itself, but that doesn’t appear to be the case and I’m not going to poke this part of my kernel just yet, so here’s a simpler userspace solution: we can preload a library into Firefox that intercepts fsync() and fdatasync() and ignores them. Even better, we can make it do that only if we’re in laptop_mode. All it takes is a little C code:

#define _GNU_SOURCE
#include <unistd.h>
#include <dlfcn.h>
#include <sys/stat.h>
#include <fcntl.h>

static int in_laptop_mode(void)
{
	char buf[2];
	int fd;
	int l;
	fd = open("/proc/sys/vm/laptop_mode", O_RDONLY);
	if (fd < 0)
		return 0;
	l = read(fd, buf, 2);
	close(fd);
	if (l == 2 && buf[0] == '0' && buf[1] == '\n')
		return 0;
	else
		return 1;
}

int fsync(int fd)
{
	static int (*_fsync)(int);
	if (!_fsync)
		_fsync = dlsym(RTLD_NEXT, "fsync");
	if (!in_laptop_mode())
		return _fsync(fd);
	else
		return 0;
}

int fdatasync(int fd)
{
	static int (*_fdatasync)(int);
	if (!_fdatasync)
		_fdatasync = dlsym(RTLD_NEXT, "fdatasync");
	if (!in_laptop_mode())
		return _fdatasync(fd);
	else
		return 0;
}

To compile and use it:

$ gcc -O2 -Wall -shared -fPIC -o killsync.so killsync.c -ldl
$ LD_PRELOAD=`pwd`/killsync.so firefox

Et voilà, firefox no longer wakes up the hard drive(s). You can confirm that the fsync/fdatasync calls no longer make it to the kernel by running strace -f -p `pidof firefox`, but only in laptop_mode.

I’ve been typing this blog post with this hack applied and so far HDD spinups have been few, far in between, and fairly short. I still have a few offenders to nail (mostly KDE related), but Firefox was one of the worst ones and it’s now history. PowerTOP says I’m saving 3-4W of power. Huzzah! Now I just need to get a kernel with timer stats built so I can fix my insane wakeups-from-idle per second…

A few notes:

• Keep in mind that if you’re using distro packages for anything I’m going to mention here, you’ll need the development headers too (-dev package or similar).
• When linking to packages below, I’ll link the name to the homepage and the version to a download link or something close to it: if you’re reading this post soon after it was posted, then the versions should be up to date. If it’s been more than a few days, you should check for newer versions.
• You’ll also need cmake to build some things (2.6 or newer); your distro should provide a package for it.
• Most packages will install to /usr/local by default. This should be fine, but you may have to uninstall any older distro packages that live at /usr, and apps may or may not pick up libraries on /usr/local. If you want, you can change the prefix to /usr. You should already know how to do this for autotools-based packages. For CMake-based ones, pass something like -DCMAKE_INSTALL_PREFIX=/usr when calling cmake.
• The libiphone/ifuse page has links to distro repositories with packaged versions of everything here. Yay, you’ve just saved a bunch of time! Do follow the steps below, but you can use these packages instead of building from source.

Possible pitfalls:

• If you’ve had prior versions of usbmuxd/libiphone/etc, make sure they’re gone, especially any udev rules files in /etc/udev/rules.d.
• If you’ve been syncing via SSH, and you log in as root, your permissions might be messed up. Run chown -R mobile /var/mobile/Media on the phone, and make sure you don’t get any permission errors when changing files via an ifuse mount.
• There’s a udev bug that causes usbmuxd not to respond to signals, which means it will never quit if you remove all devices, and it might not detect any new devices after you’ve plugged in the one that caused it to start up. As a workaround, kill -9 it, and run it from the console. A workaround for this bug is also in the usbmuxd Git repo.

Step 1

Install libusb-1.0.3 (or newer). This should be straightforward. If your distro carries it, that’s fine, just make sure it’s 1.0.3 at least (if it isn’t, the repos I mentioned above probably have it).

Step 2

Install usbmuxd-1.0.0. Make sure you read the README file, especially the part about creating a user named usbmux with USB access permissions (does not apply if you’re using distro packages, hopefully). After installing usbmuxd, you should increase the syslog debug level by editing /lib/udev/rules.d/85-usbmuxd.rules and adding -v -v flags to the end of both RUN statements, like this: RUN+="/usr/sbin/usbmuxd -u -U -v -v" (and similarly for the other line). This will get you useful debug information in syslog without flooding you with messages.

Have syslog open (tail -f, xconsole, whatever). Plug in (or replug) your device. You should see something like this:

usb 3-4.3: new high speed USB device using ehci_hcd and address 68
[...]
usbmuxd[12705]: [3] usbmux v1.0.0 starting up
usbmuxd[12707]: [4] Creating socket
usbmuxd[12707]: [3] Successfully dropped privileges to 'usbmux'
usbmuxd[12707]: [4] Initializing USB
usbmuxd[12707]: [4] Found new device with v/p 05ac:1290 at 3-68
usbmuxd[12707]: [4] Using wMaxPacketSize=512 for device 3-68
usbmuxd[12707]: [3] Connecting to new device on location 0x30044 as ID 1
usbmuxd[12707]: [4] 1 device detected
usbmuxd[12707]: [3] Initialization complete
usbmuxd[12707]: [3] Connected to v1.0 device 1 on location 0x30044 with serial number be2975afb30b6db9025f95261b9e0a7041044661

If you don’t see anything related to usbmuxd, either you forgot the -v -v or you need to reload your udev rules: $ sudo udevadm control --reload-rules. As a last resort, try rebooting. If it still doesn’t work, something’s wrong. You can continue by running usbmuxd manually as explained in the README, but you should report a bug.

If you have a jailbroken phone with SSH, try running $ iproxy 2222 22 and then SSH to localhost, port 2222 on another console. You should be able to SSH into your phone.

Unplug your phone. If usbmuxd was started via udev (not manually), it should quit. If it doesn’t, report a bug.

Step 3

Install libplist-0.16 and then libiphone-0.9.4 and ifuse-0.9.4.

Now you should be able to mount your phone by simply running $ ifuse <mountpoint>. You should be able to see the media folder structure and even grab photos and other stuff. If you have a jailbroken phone, you can use the --root option to mount the root filesystem with root privileges (I usually prefer ssh/sshfs for this, though). Do NOT use the --root option to sync music, it will not work and may mess up your permissions.

If you have access to more than one iPhone or iPod Touch, we could use some multiple-device testing! Usbmuxd should happily start up when you plug in the first device, keep running while you plug in any other devices, and stop once all devices have been removed. Usbmuxd will report the device UUIDs (it calls them “serial numbers”) as they are connected (if you’re using -v -v). You should be able to pass the --uuid=<uuid> option to ifuse to mount a specific device. Please test this for us!

Intermission

If you are not used to alpha software, checking out development sources, testing patches, and working with developers, stop here. Now you’ve got all the underlying stuff working, so you’re probably better off waiting for some time until libgpod stabilizes some and things are merged and tested.

If you are a developer willing to help, a very enthusiastic power user, or simply a masochist, feel free to read on. Be warned: pain lies ahead, don’t even think about using the following for day-to-day use at this point in time. From here on there are no distro packages and no nice source tarballs. Experiments and feedback only. This Will Eat Your Music Collection For Dinner And Replace It With Hours And Hours Of <insert terrible music>.

Step 4

Clone teuf’s sandbox Git repository for libgpod, and switch to the iphone30 branch the libgpod git repository at SourceForge, then compile and install it:

$ git clone git://gtkpod.git.sourceforge.net/gitroot/gtkpod/libgpod
$ cd libgpod
$ CFLAGS="-g -O0" sh autogen.sh --prefix=/usr
$ make
$ sudo make install

Step 5

Mount the device and create the iTunes_Control/Device directory. Then, get your UUID (it should be in the syslog from usbmuxd, or you can find it by running $ lsusb -v | grep -i iSerial). It should be 40 characters long. Then, run $ tools/ipod-read-sysinfo-extended <uuid> <mountpoint>. This should generate a file named iTunes_Control/Device/SysInfoExtended. Make sure it’s not empty and whatnot; it should be a large-ish plist (XML file) with a bunch of info.

Step 6

Make sure your device already contains a valid music database created by iTunes. If it doesn’t, drop me a line on IRC and we can try some cool new stuff. New databases can be created, but there might be issues and you need some way of generating the HashInfo file (more on this later).

Step 7

Open an application that uses libgpod to manage iPods. Gtkpod is recommended at this stage, as its what I use to test. Other apps (Rhythmbox, Amarok, etc) theoretically should work. Theoretically, because Amarok 1.4 crashes/hangs pretty badly for me, for example, and I’m not sure who is to blame. I’d like to hear any success or failure stories. Please run your app from a console so you can see any debug messages.

First, load your device. Once the application has accessed the music library, a new file called HashInfo should have popped up in the iTunes_Control/Device directory. If it isn’t there, you’ve got a problem: check stdout for any messages and and preferably ping me on IRC or at least e-mail me your UUID and your iTunes-created iTunes_Control/iTunes/iTunesCDB file (it shouldn’t have been modified yet).

Make some changes (add a song, make a playlist, etc), save, and cross your fingers. With any luck, you’ll get some debug spew, any songs will transfer, the iPhone/iPod will show a pretty “syncing” screen, the database will be written, the “syncing” screen will go away, and you’ll be able to launch the iPod application and see your changes and play any new songs. There will be bugs at this stage, but please do report any issues along with expected behavior.

Step 8

Plug your device into a computer running iTunes (unmount it on your Linux computer first!) and check that it can see the changes made with libgpod. This is very important, because iTunes and the device use different databases: the data managed by iTunes and libgpod is maintained in an iTunesCDB (a compressed version of the old, classic iTunesDB format), but it is converted to a new SQLite database format for the iPod app to use. So both libgpod and iTunes read the iTunesCDB, but they write out the iTunesCDB and the SQLite databases. You should be able to make changes in iTunes and swap your device from iTunes to libgpod and back and see them, etc.

Bugs

Tons. The only things tested to work are managing simple music and working with simple playlists. There’s a known annoying bug where the iTunes special playlists (Ringtones/etc) are converted to normal playlists by libgpod, and then iTunes recreates them if you use it again, creating duplicates each time you switch between using libgpod and iTunes. The DB schema is outdated (3.0 instead of 3.1), so you’ll see a (hopefully short) “Updating” step when you launch the iPod app. Please report issues, but keep in mind that they’re to be expected. This is very very very alpha code. I probably screwed up something on this post even, so chances are you’ll get stuck somewhere. Tough luck, it’s 7am and I need to catch some sleep

Hope you had fun! Keep in mind that you can usually find us on #gtkpod on freenode if you need to report an issue. Comments on this post are fine too, for now.

• The iPhone uses a custom USB communications protocol, connection-oriented, which is totally different from standard mass storage devices
• It provides multiple services over this protocol, and many have to be interacted with to complete sync
• iPhone OS 2.x brought us a new hash (designed to lock users into iTunes) that so far had not been reverse engineered
• iPhone OS 3.x brought us a new SQLite-based database format which, although a lot more Linux-friendly than the old proprietary format, does require a bunch of new code (and the old format still needs to be supported for compatibility).

Well, I’m happy to say that all of this is coming to an end and that Linux will finally get quite complete iPhone/iPod Touch syncing support soon, thanks to the work of many dedicated people. This is what the software stack looks like:

• libusb-1.0 provides an advanced API to access USB devices under Linux, replacing the old libusb-0.1 API
• usbmuxd coordinates application access to the device and talks the specific iPhone/iTouch USB protocol
• libiphone implements the Apple-specific protocols that are tunneled through usbmuxd: it can launch services through lockdown, retrieve device info, send notifications, and access the filesystem via AFC.
• iFuse and gvfs-backend-afc both provide access to AFC to regular Linux apps. iFuse does this by mounting via FUSE, while gvfs-backend-afc is obviously a backend for gVFS.
• libgpod (the library that traditionally has managed music databases for iPods) is being extended to support the new SQLite format, the new hash, and also to talk to libiphone to properly put the device in to and out of sync mode.
• Theoretically, actual music players such as Amarok and Rhythmbox will need none or very few modifications to work.

I originally started work on usbmuxd which I am maintaining, and I’m now helping libgpod compatibility with the new formats. The software stack needs to be working from the bottom up, so I’ve just tagged and released the first Release Candidate for usbmuxd-1.0.0. If you have an iPhone or an iPod Touch and you’d like to use them with Linux in the future, I encourage you to test it out. If your device is jailbroken, you get the immediate benefit of being able to SSH over USB (which is a lot faster than over WiFi!), and if you’re using firmware 2.x or earlier and the hacky sync-over-WiFi-with-sshfs method, you can immediately get a speed boost by doing it over USB.

If you’re feeling adventurous, the development versions of libiphone and iFuse are also quite good. Theoretically, if your usbmuxd is working an you’ve got iFuse installed, you can just type “ifuse <mountpoint>” and instantly have your device mounted. Don’t expect to get libgpod to work at this time without quite some pain though. If you’re a developer and you’re feeling really adventurous though, you can join us on #gtkpod on freenode and help us test and fix the new libgpod support!

Please report any bugs or fixes in usbmuxd either directly to me or to the iphone-linux-dev mailing list. Any issues with libiphone or iFuse should also go to the list.

I certainly wanted to use a feature that I paid for, so I started investigating the BIOS and here’s what I found out.

Under the hood

The InsydeH2O BIOS is no ordinary old-style BIOS. Instead, it’s based around the UEFI platform. This goes way beyond the old BIOS paradigm and turns system firmware into practically its own separate OS, that even runs in full 64-bit mode on 64-bit machines. Unfortunately, they make no effort to expose any of this to the user. The firmware has support for booting EFI executables, there’s an EFI shell, there’s an EFI boot manager… but I haven’t been able to figure out how to access any of this.

If you want to reverse engineer EFI stuff, downloading TianoCore’s EDK2 is a must. It contains source code for a lot of Intel’s framework, which is what most vendors use as a base for their EFI support. A lot of the code is exactly the same as what’s in the Insyde BIOS (read the spec here).

As for the Setup tool, it does indeed have a huge Advanced menu with even more options than your average desktop. There’s also a hidden Power menu. EFI defines a “form browser” protocol and formats for user input, which is what Insyde uses for their setup utility (spec here). I found these tables when disassembling the Setup binary and wrote a little dump utility to turn them into text. The result is a complete dump of the Setup hierarchy, including the Advanced menu, which also includes the offsets in the non-volatile storage corresponding to each setting. Insyde stores this configuration blob into an EFI variable named Setup. Here’s my dump: the first part is the hierarchy, while at the end I added a rough auto-calculated mapping from configuration offsets to setting names (grep for [0xOFFSET in the top section for better context - the format is [0xOFFSET<FIELD_WIDTH>] for all references to the storage blob). You’ll find the tools I used here, if you’re interested, but they’re rough and need quite a bit of manual help too.

I wasn’t able to find out how to enable the hidden menus, other than that their form Subclass is 5 instead of 0 (but I haven’t found what, if anything, checks for this and whether its behavior can be altered). However, manually enabling VT support in the Setup variable is easy enough, now that we have the offset of the VT Enable byte.

Enabling Intel VT

The easiest way to enable the setting as far as I can see is to dump out the entire BIOS, patch the setting into the Setup variable (which is part of the data storage section – we aren’t modifying any actual BIOS code, as this is the equivalent of changing a CMOS setting on other BIOSes), and then flash the resulting image. These laptops use a weird flash-behind-EC hardware solution for which there is no open flasher, so instead we can just use the normal BIOS flashing tool. In short, we’ll flash the existing BIOS back on, but in the process also modify a Setup setting.

FAIR WARNING: This might apply to other similar laptops, or it might not. It might work, it might do nothing, or it might brick your expensive laptop. Even if you own an Aspire 8930G, I take no responsiblity if your laptop dies, turns into an expensive brick, melts into a pool of slag, blows up, flicks you off, develops self-awareness, or becomes Skynet. You have been warned. I have only tested this on an Aspire 8930G with BIOS Version 1.10. If you want to try this on another system or BIOS you should make sure you understand EXACTLY what is going on and are prepared to spot any problems or fix things yourself.

First, dump the exiting BIOS out. It resides at the top of the 32-bit address space, and is 2MB in size. You can use dd to dump it out of /dev/mem:

$ dd if=/dev/mem of=original_bios.fd bs=1024 count=2048 skip=4192256

It is a very good idea to back up this BIOS somewhere safe outside the laptop. Note that it not only contains your existing BIOS code, but also all your settings and manufacturer data (serial number, software license if you run an OEM version of Vista, etc).

Next, run vtenable.py. This will attempt to locate the Setup EFI variable on the non-volatile storage section and patch the VT byte to one.

$ python vtenable.py original_bios.fd vt_bios.fd

You can edit the source code to make other changes to the variable, but make sure you know what you’re doing. It’s worth reiterating that this does not patch your BIOS code. It only makes a setting change, just as if you’d turned on the VT option in the BIOS had it been there. In fact, there are two variables: Setup and Custom, and Setup is the one that changes are committed to when you use the setup utility. Restoring defaults should turn VT back off (untested). It also appears that Custom is probably what the setup defaults are, so changing that should semi-permanently enable VT.

I highly recommend performing a sanity diff between the original and modified images using vbindiff:

$ vbindiff original_bios.fd vt_bios.fd

Only two or three bytes should change: one or two adjacent bytes for the checksum (they should be decremented by one when you look at them as a 16-bit unsigned integer), and the VT enable byte should change from 00 to 01. Right after the checksum bytes you should be able to see the Setup name in UTF-16 (something like S.e.t.u.p.).

Finally, flash vt_bios.fd using the vendor-supplied flash utility. I use the DOS version (FLASHIT.EXE) with FreeDOS and a grub menu option so I don’t need to mess around with external media. Grab a base image here, then you can use mtools to copy the bios into it:

$ bunzip2 freedos_flashit.img.bz2
$ mcopy -i freedos_flashit.img vt_bios.fd ::/vt_bios.fd

To boot it using GRUB, get MEMDISK, part of SYSLINUX, and put something like this in your grub.conf:

title=BIOS Update
root (hd0,0)
kernel (hd0,0)/boot/memdisk
initrd (hd0,0)/boot/freedos_flashit.img

Of course, copy memdisk and the boot image to your boot partition, and change (hd0,0) to your boot (or root) partition everywhere and remove the /boot part if you have a dedicated boot partition.

Once you’re in FreeDOS, just type FLASHIT vt<tab> and be happy that FreeDOS supports tab-completion

Caveat: by doing this, you’re flashing the entire BIOS image. The flash tool makes no attempt to flash only the parts that changed, and the “flash only variables” commandline option seems to have no effect. You’re effectively reflashing your entire BIOS back on, so the usual BIOS flashing caveats apply: don’t turn the power off, etc. This could be accomplished a lot more cleanly if we had drivers for the flash chip / EC, since then we could use the normal EFI variable store procedure to atomically update the variable, which is completely safe.

You can use the MSR Magic tool to check whether VT is indeed enabled on your CPU.

Update: Several people are working on improved, more general tools to perform this hack across a broader range of InsydeH2O-based BIOSes. Read the comments and check them out, they’ve done some very good work.

The upshot of this is that a whole new support library will need to be written or large changes in libgpod need to happen to support the new SQLite format. The DBVersion hack also isn’t going to work – either someone needs to reverse engineer the complete hashing algorithm (and maybe that encrypted file stuff), or the MusicLibrary binary on the phone needs to be patched. So, if you’re currently syncing music with free software, you’ll want to hold off on upgrading to 3.0.

Patching the check out of MusicLibrary looks trivial enough, so although it’s not as easy as the DBVersion hack, it isn’t a real showstopper. The hash algorithm looks the same as the old one, or at least quite similar (and just as badly obfuscated). Those encrypted files do scare me a bit though.