iPhone syncing on Linux
Those who watch my git repos may have noticed that I’ve been working on this for a while now. iPhones and iPod Touches have never been properly supported under Linux (especially non-jailbroken devices) because they are just so different from all the previous iPod models. There was a lot of new code to write and Apple hasn’t exactly made it easy either:
- The iPhone uses a custom USB communications protocol, connection-oriented, which is totally different from standard mass storage devices
- It provides multiple services over this protocol, and many have to be interacted with to complete sync
- iPhone OS 2.x brought us a new hash (designed to lock users into iTunes) that so far had not been reverse engineered
- iPhone OS 3.x brought us a new SQLite-based database format which, although a lot more Linux-friendly than the old proprietary format, does require a bunch of new code (and the old format still needs to be supported for compatibility).
Well, I’m happy to say that all of this is coming to an end and that Linux will finally get quite complete iPhone/iPod Touch syncing support soon, thanks to the work of many dedicated people. This is what the software stack looks like:
- libusb-1.0 provides an advanced API to access USB devices under Linux, replacing the old libusb-0.1 API
- usbmuxd coordinates application access to the device and talks the specific iPhone/iTouch USB protocol
- libiphone implements the Apple-specific protocols that are tunneled through usbmuxd: it can launch services through lockdown, retrieve device info, send notifications, and access the filesystem via AFC.
- iFuse and gvfs-backend-afc both provide access to AFC to regular Linux apps. iFuse does this by mounting via FUSE, while gvfs-backend-afc is obviously a backend for gVFS.
- libgpod (the library that traditionally has managed music databases for iPods) is being extended to support the new SQLite format, the new hash, and also to talk to libiphone to properly put the device in to and out of sync mode.
- Theoretically, actual music players such as Amarok and Rhythmbox will need none or very few modifications to work.
I originally started work on usbmuxd which I am maintaining, and I’m now helping libgpod compatibility with the new formats. The software stack needs to be working from the bottom up, so I’ve just tagged and released the first Release Candidate for usbmuxd-1.0.0. If you have an iPhone or an iPod Touch and you’d like to use them with Linux in the future, I encourage you to test it out. If your device is jailbroken, you get the immediate benefit of being able to SSH over USB (which is a lot faster than over WiFi!), and if you’re using firmware 2.x or earlier and the hacky sync-over-WiFi-with-sshfs method, you can immediately get a speed boost by doing it over USB.
If you’re feeling adventurous, the development versions of libiphone and iFuse are also quite good. Theoretically, if your usbmuxd is working an you’ve got iFuse installed, you can just type “ifuse <mountpoint>” and instantly have your device mounted. Don’t expect to get libgpod to work at this time without quite some pain though. If you’re a developer and you’re feeling really adventurous though, you can join us on #gtkpod on freenode and help us test and fix the new libgpod support!
Please report any bugs or fixes in usbmuxd either directly to me or to the iphone-linux-dev mailing list. Any issues with libiphone or iFuse should also go to the list.
October 27th, 2009 at 8:29 pm
It’s not possible from this to create something to add songs in iphone music db from the device?
October 27th, 2009 at 9:14 pm
hi i’m very interesting for this project,
i’m linux and iPhone’s user and “iPhone syncing on linux” is one of my dreams!!
Thisi il my e-mail andrea.barbieri1991 at gmail.com , please conctat me !!
October 27th, 2009 at 9:18 pm
Wow, what a nice surprise!
I actually thought that I’d have to keep using firmware 2.x jailbroken with the hack you released earlier to use the old hash. I had no idea that the new hash had been cracked already and I didn’t know that you had progressed this far either.
I’ll take this opportunity to thank you for your amazing job. I hope I’ll be able to get this to work. Otherwhise I’ll be patient and wait for some noob-friendly tutorials.
October 27th, 2009 at 9:20 pm
Awesome work dude. The usbmuxd thing is a really nice idea. Been following iphone sync status on linux since i got a 2g and now I see you were able to get rid of the new sync protocol on 3.x firmware. Gonna install the new ubuntu, try and give feedback! What about the itunes db hashes, were they reversed?
October 27th, 2009 at 10:40 pm
It works on Ubuntu 9.10 without magic. Also I had to add the user usbmux otherwise it complains about permissions.
October 28th, 2009 at 12:45 am
@maikel:
Keep in mind that the libgpod stuff, although working to some extent, is very very alpha at the moment. However, feel free to try to get the lower layers (usbmuxd, libiphone, ifuse) to work, since those are relatively stable by now.
@knoopx:
The new hash has been reverse engineered enough to bring near full functionality to Linux.
@Rob:
Any idea why it doesn’t work with magic? You can try editing the udev rules file that it installs and adding “-v -v” to both usbmuxd commandlines. Then check syslog when you connect/disconnect a device and see if there’s anything interesting. Also, keep in mind that you may have to reload your udev rules before it works: “sudo udevadm control –reload-rules” or just reboot. If none of this works, please report back or drop me an e-mail so I can try to get it fixed.
October 28th, 2009 at 3:58 am
Is there a way to use the SSH over USB feature of this on a Mac? I tried the tcprelay.py tool but wasn’t sure on how to use it exactly (not that smart when it comes to this stuff).
October 28th, 2009 at 4:10 am
tcprelay.py will work on a Mac and on Windows, as long as you have iTunes installed (it will use Apple’s usbmuxd). Just run ‘python tcprelay.py -t 22:2222′ to forward local port 2222 to the iPhone’s port 22 (SSH).
October 28th, 2009 at 4:16 am
marcan: It was the user permissions thing. syslog was flooded with could not open device messages. Then I realized that the user didn’t have proper permissions. It worked after rebooting.
October 28th, 2009 at 4:18 am
Thanks marcan, worked perfectly!
October 28th, 2009 at 10:32 pm
The link to the mailing list doesn’t like me, or is locked down for some reason :
[quote]
High security alert!!!
You are not permitted to download the file “iphone-linux-dev-mattcolyer.com”.
URL = http://lists.mattcolyer.com/listinfo.cgi/iphone-linux-dev-mattcolyer.com
[/quote]
October 28th, 2009 at 10:42 pm
That sounds like some kind of retarded antivirus or filter which completely ignores the HTTP spec and thinks any URL ending in .com is a DOS executable and likely a virus, instead of looking at the mimetype.
October 28th, 2009 at 10:47 pm
Thanks for the quick reply. I’ll try again from a different location.
October 29th, 2009 at 8:11 am
[...] progetto si chiama “usbmuxd” ed è portato avanti da uno sviluppatore indipendente. Lo scopo è quello di unire una serie di software open source e creare un’applicativo in [...]
October 29th, 2009 at 9:24 am
Works great, I’m impressed.
I’ve got one question though.
I get 1.3M/s tops. Is that expected?
October 29th, 2009 at 12:33 pm
Wow finally someone who has taken on something that Apple apparently chooses to ignore.
I support this project wholeheartedly and would like to make a financial contribution (just a small donation). How do I go about doing that?
October 29th, 2009 at 4:00 pm
@faxm0dem
Is this using SSH or AFC, and which way are you transferring the data?
@eugenevdm
The best way to donate would be to just help test everything and report bugs!
October 30th, 2009 at 11:18 pm
It works for me (Ubuntu 9.10, iPhone V1 OS3.0 & iPhone 3GS OS3.1). Any ideas to make it mount automatically ?
Thanks for the great job ! Hope your will succeed with libgpod !
November 1st, 2009 at 2:27 pm
Hi, i also installed this on karmic (from repo), i am able to mount the root filesystem, but it says i have only 166 mb-s free, so i cant copy a big file to the device.
anyone else experiencing this? (i want to copy to /var/mobile/Library/Downloads)
thanks for your efforts anyway!
November 1st, 2009 at 8:16 pm
First of all, great work. On firmware 2.2.1. I loved syncing my music under Ubuntu. As from firmware 3.0, I’ve had to go back to Windows, using iTunes. This is a step in the right direction towards bringing the iPhone back where it belongs.
Second, I had to use ‘./configure –prefix=/usr’ to configure libusb-1.0.3. Otherwise files couldn’t be found. Now, it shows up in my syslog.
November 7th, 2009 at 5:22 pm
Yes perhaps I am spamming your post but it’s for the good of mankind, friend. I tried some more realistic library editing (at first I just added a song, this time I was fixing broken id3 tags) and the sync appeared to go fine, but it appears the stack has corrupted the database (or something) as the Music app just hangs (at first it sat at Updating Library for a loooong time, after a huge wait I rebooted the iPod) now it shows the artists pane, lets me flick around for about 4 seconds before freezing. I would venture a guess that it has a lot to do with the database schema update.
November 7th, 2009 at 6:47 pm
That sounds serious. Please tar up the iTunes directory and either e-mail it to me or upload it somewhere where I can get it. Also, it would be good if you could report the result of running tests/test-rebuild-db on the mountpoint, including all debug output, and whether that fixes the problem on the iPod/iPhone or it’s still broken. Also, let me know exactly what device/firmware you have and what Git revision of the repo you used.
November 23rd, 2009 at 8:36 am
marcan> Is this using SSH or AFC, and which way are you transferring the data?
I just did a simple test using ssh and cat:
dd if=/dev/zero bs=1M | ssh localhost -p2222 ‘cat >/dev/null’
January 30th, 2010 at 12:26 pm
I am new Linux, and have been trying to sync my Iphone since last year with Linux. I found this post – http://www.ubuntugeek.com/ipod-touch-3g-sync-over-usb-without-jailbraking-in-ubuntu-karmic.html and it worked the very first time. Good Luck.