iPhoto and importing large numbers of images

Summary: Panasonic Lumix DMC-FX01 and Nikon D200 rule, iPhoto sucks, external iPhoto libraries still use gobs of temporary space on startup disk during large imports (need to move /private/var/tmp)

Long version:

It has been a while since I posted, which I will attribute partly to having been on a long, computer-free, vacation back in Australia, visiting my parents, and indulging my wife's passion for wild-flower photography. The latter accounts for the timing, in that spring is not perhaps the best time to visit due to the prospect of rain, but a few weeks in September are the only time that the flowers in Western Australia bloom. Which brings me to the subject of the post ...

We took two cameras. The first was a tiny little Panasonic Lumix DMC-FX01, a great little 6MP snapshot camera that we bought immediately when we saw a friend's, because it is one of the very, very few with a macro (5cm) mode. The second was a recently purchased Nikon D200. My old 35mm Nikon had finally given up the ghost (its tiny little almost 15 year old brain just locked up one day), and on an impulse walking past Adorama on 18th St we decided to get back into the world of "real" photography. Though I have a good set of AF lenses that still work perfectly well with the D200, I also couldn't resist the impulse to replace our old manual macro lens with an AF Micro Nikkor 60mm f2.8, which worked very well for our wildflower photography in the field on this trip. The D200 has more than enough buttons and functions to make a programmer happy and yet in its default mode is easy enough for a normal human being to point and shoot; just be careful not to inadvertently move the auto-focus target away from center with the cursor buttons accidentally, which is easy to do. The best thing for me as an old Nikon user is that the buttons and functions are an incremental improvement on the film-based predecessors and hence fall naturally to hand.

So the cameras worked well and on returning with several thousand photos and a severe case of jet lag, I eagerly proceeded to import them into my wife's computer. Like me, Eleanor has always been a Mac aficionado. In her daily work as an artist, she lives and breathes Photoshop, but for expediency's sake uses iPhoto to manage her personal digital photographs. I am ashamed to say that her machine is not the latest and greatest, something that I will rectify next time she is between contracts and can tolerate some downtime, so it has a bit of a hybrid collection of disk drives to spread things around, which brings me to the primary subject of this post.

Since space on the startup internal hard drive was tight, I had recently moved the iPhoto library from its default location at "~/Pictures/iPhoto Library" to a folder on an external USB attached drive (by copying the library with the Finder, and then holding Option during iPhoto startup which prompts the user to create a new library or to locate one). This was with iPhoto 2.0, since I had not ever bothered to update iPhoto even when updating her system from Panther to Tiger. Anyhow, everything seemed to work just fine that way, in iPhoto's usually inimitable (sluggish) manner.

So back to importing. I plugged in the D200 via USB with its 8GB CF card with several thousand large JPEGs on board (told you I wasn't a serious photographer; no camera raw format for us at this stage). Told iPhoto to begin the import, and since it looked like it was going to take a really long time, left it overnight.

Surprise, surprise, in the morning a) there was a message that disk space was low on the start up disk, b) only 700 or so photos had imported and c) the battery in the D200 was now depleted. No message within iPhoto that anything was wrong though, and in particular no message that despite having started to import 2309 pictures only a small set had succeeded.

Concluding that I was an idiot for not having ordered an external power supply for the camera in the first place (having either not thought about it or assumed that USB power might be available or suffice), I thought that perhaps that was the root cause of the problem, and ordered a Nikon EH-6 AC Adapter ($79.80 from Amazon; at the price there is no excuse for not having one of these). Whilst waiting for it to arrive I was of course unable to resist playing around though.

Let's swap batteries and just try again and pay a little more attention this time; but I first I thought about the matter of running low on disk space on the startup disk. The iPhoto library was definitely on the external drive. There was nothing to speak of in /tmp, before or after quitting iPhoto. Strange, I thought. Assuming that perhaps iPhoto had just been allocating lots of memory to keep track of things, build thumbnails or whatever, and had run out of swap space (wherever that might happen to be stored, something I had not previously considered), I figured a reboot might be in order, and sure enough there was a gigabyte and half or so spare back again after restarting the machine.

Repeated the import process (selecting not to reimport duplicates when the dialog appeared) and a few more were imported, but again, not all, no message from iPhoto to confirm this or explain why, and a shortage of startup space once again. Tried a few other things, like disabling the inactivity sleep in energy saver as well as moving the library from the external USB drive to another internal hard drive with lots of space, but no joy. Old software perhaps, I mused, time for an upgrade, since I had a family pack of iPhoto 6 kicking around that I occasional used on my own machine.

Just to confirm that jet lag impairs decision making, this process did not of course go smoothly, though I did at least check that I had a pre-vacation backup of the the iPhoto Library, which it turned out that I needed ! Somewhere in the process of repeating the import and/or upgrading iPhoto, I managed to completely corrupt the iPhoto database, I don't recall exactly at which point. No amount of rebuilding (hold down command and option whilst starting iPhoto) thumbnails and/or the database succeeded in recovering the database despite the presence of all the photos themselves right there in the various folders in the library. Since my wife had put a considerable effort at sorting her existing (pre-vacation) collection into albums, starting afresh by just re-importing all the photos was not an option, so I restored the old database from the Retrospect backups, started up iPhoto 6 which then wanted to upgrade the library, which it did OK this time, and things seemed more or less back to normal.

Repeated the import from the D200 with exactly the same results - incomplete import, consumption of space on the startup disk, and no helpful message from iPhoto. Each time I repeated the process iPhoto would import a handful more, but never the complete set. I was still running out of battery power, but when my external power supply arrived, nothing changed. Of course the camera remained powered up and mounted as a disk, but the import behavior remained the same.

Hmm; time for some more serious Googling. I could find no specific reference to large imports failing in a similar manner. I did find somewhere though mention of iPhoto using "/private/var/tmp", particularly in relation to such a folder not being cleaned up on restart, though that turned out not to be relevant. Having a more than passing familiarity with "/tmp", but never having heard of "/private/var/tmp", some investigation seemed in order.

Sure enough, when iPhoto begins importing, a folder is created in "/private/var/tmp" and it starts to fill up with hundreds if not thousands of large JPEGs named in the same pattern as the camera file names; it seems that the import process involves copying to this folder prior to copying into the iPhoto library folder. It doesn't seem to be the complete set though, and perhaps there is some queue mechanism with one thread copying from the camera to the temporary folder and another copying them to the library and then removing them.

Whatever, the net result is that a large iPhoto import results in a huge amount of temporary space being consumed on the startup disk, and hence failing when it runs out, even if the iPhoto library is on another disk with plenty of space. Furthermore, though the operating system warns about this (presumably because there is a competition with swap space), iPhoto itself remains silent, which I find particularly dismaying.

It was possible to work around this simply enough, by:

- creating a temporary folder on another drive (e.g., "/Other/private/var/tmp") and giving it the appropriate permissions (as root, using chmod to make it rwxrwxrwxt (+arwx,+t))
- replace the "/private/var/tmp" folder with a symbolic link
(ln -s) to "/Other/private/var/tmp"

Then everything worked fine.

I was a little reluctant to mess with the temporary folder on a running system, and was too lazy to go down to single user mode, but I did quit all other applications and it worked OK; I wasn't game to reboot in this configuration though, in case "/private/var/tmp" is need during boot before the other file systems are mounted, so I put it back to normal before rebooting.

So, in short, I can't say that I am that impressed by iPhoto's robustness or scalability. To be fair, I have not tried the very latest version, which I gather is iPhoto 7 (in iLife 2008), since one has to pay for the upgrade and it has some significant changes in user interface and library organization so may interfere with my wife's workflow.

But I find it hard to excuse a silent failure to import images, since that has the potential to lose a photographer's hard work if they are not very careful to check for this (e.g., by comparing the expected number of images with the actual number in the last imported "roll").

It's probably a bit harsh to go so far as to state that iPhoto sucks, but like most people, it only takes one really bad experience to really put me off a product and this has come close; thankfully no pictures were actually lost though. Last time I was this irritated at a product to the point of publicly eviscerating it was when Retrospect did not keep up with support for the internal DVD writers in the new Mac laptops causing my entire personal backup strategy to go down the toilet. I guess I just have a low tolerance for inconvenience.

As far as my work around is concerned, I would not recommend it, and post it here only to make folks aware of the problem. Messing with system temporary folders as root is obviously not something for the average digital photographer and Mac user to be doing, so I conclude that in general imports of a large number of photographs requires a correspondingly large amount of free space on the startup disk to be completed reliably, regardless of the location of the iPhoto library itself.

David

PS. Note that at no stage did I check the "remove from camera after import" button in iPhoto. I was not willing to risk the camera images being deleted without having been successfully imported.