Using Python to migrate from flickr to WordPress

Saturday 17th Aug 2013

This tutorial is designed to give you a broad overview of how to take all your images from flickr and then setup your own photoblog using WordPress. The core of the process is a library of tools written in python. All the python code is available on my github repository: https://github.com/bencowellthomas/photoblog

Since around 2002 I’ve kept many of my photographs online. A couple of years ago I took down my old fashioned static html photoblog and uploaded the photographs to Flickr. I then created a wordpress photoblog template ready for my newer photographs. However in recent months Flickr feels like a dying platform, the redesign has taken it toward an instagram style site rather than the classic format that felt like a safe and convenient place to archive and display photographs.

So I decided that while I was rebuilding my site as bct.me I’d also completely rework the photoblog templates and keep everything together the way I wanted it and under my control.

I had nearly 1,000 images on Flickr, plus another 200 on my wordpress site. So I needed a way of batch processing the photographs plus making it easy to look after my photoblog in future.

Here’s the plan I came up with:

Download all the images from Flickr and my WordPress site,
Use meta tagging formats such as IPTC, EXIF and XMP to store the information within the images themselves. Meaning if I ever have to do this again I don’t lose my tags and metadata.
Use the WordPress Custom Post types to implement a ‘photo’ post type that can run alongside the usual posts and pages in the database.
Create a custom Taxonomy within WordPress for sets of photographs, plus labelling of images.
Create a Python tool to batch upload all the images to WordPress using the XML-RPC API.

Downloading and tagging images

(Note: Since completing this process I’ve extended my Python tool to download the images from Flickr for you. It should also grab the tags and set data and embed it into the metadata automatically).

Downloading from WordPress was an easy FTP job. For Flickr I used the Flickandshare service, but there’s plenty of bulk downloaders out there.

I then wrote a Python Script to put the images into monthly folders (YYYY_MM) and set about tagging and labelling the images. I used XNview MP, manually going through groups of images and adding tags, copyright information etc. I arbitrarily chose the IPTC fields that best fitted my purposes, as in the end they’re just placeholders to allow to data to be transferred into WordPress.

Luckily most of my images still had the camera EXIF data inside, so I could use the date the image was taken as the date for the blog post. This meant that my posts would appear historically correct rather than appear dated the day of the batch upload.

Creating the post-type and template files in WordPress

I used another useful post on wp tuts+ as my guide to setting up a custom post type to handle photographs. This meant I could keep the data within the same database as my main blog, but create template files specific to the photoblog. I won’t go into detail on the WordPress specific steps as they’re covered in great detail by the wp tuts+ tutorial along with the Smashing Magazine guide to custom post types.

But to summarise this required:

Creating a custom post-type in my functions.php file.
A few custom taxonomies to handle labels, sets and whether the photograph should be included in my favourites page.
Plus I also had to update the rewrite rules to allow for a date archive for the post-type.
I then created a set of template files to display the photographs.
To display each entry I wrote a php function to grab the first image found in a blog post and return it, this was called within the loop in my template files rather than the default wordpress the_content() function.
I used wordpress featured images to display my photographs in archives and lists.
To ensure each post had a featured image I added a function found at wpforce to ensure a featured image is always created when creating a post within my custom post-type.

A Python tool to download and upload photographs

So now I had all my images stored locally, neatly tagged and filed away. I also had a WordPress blog set up ready to receive them. Now I needed a way to post them to the WordPress blog via a batch process.

I began exploring a number of methods and hit upon the following python solution:

Download from flickr using the flickr API via flickrapi by Sybren Stüvel (perversely I implemented this after I’d downloaded my images, but it’s there in my python library should you find it useful)
Parse the files and read the metadata using the pyexiv2 library
Upload and create blog posts using XML-RPC and the WordPress API

The entire library of code is available on github. As I encounter new problems or tools I need I’ll continue to expand it. So far it’s saved me huge amounts of time as with a couple of commands and clicks I can now take a jumble of images and turn them into blog posts. I’ve also added functions to help me deal with organising folders and renaming images.

Using the photoblog library

Once you’ve downloaded the library place it in your site-packages or somewhere else found in your PYTHONPATH. You’ll also need to ensure you have both the pyexiv2 and flickrapi libraries set up on your machine. This blog post covers the process of building pyexiv2 and the flickrapi on OSX.

Before you start rename ‘default_userinfo.py’ as ‘userinfo.py’ and add your flickr username, API key and secret (if you don’t have an API key you get one from the flickr site). Then add your WordPress username, password and URL.

You’ll also have to add one or more structs to userinfo.py to define how your flickr sets, file metadata and WordPress taxonomy are related. For example if you’d like flickr ‘tags’ to be stored in each jpeg file as IPTC ‘keywords’ and then posted via a custom taxonomy ‘labels’ you would add an array element like this:

CUSTOM_TAXONOMYS = [{'name': 'label', 'taxonomy': 'labels', 'metadata': 'Iptc.Application2.Keywords', 'flickr': 'tag'}]

You can add as many of these structs as you like. Add the value ‘None’ for the flickr keyword if you don’t want to associate the taxonomy with flickr (currently there’s only support for flickr tags). For example this is how my photoblog is set up :

CUSTOM_TAXONOMYS = [{'name': 'label', 'taxonomy': 'photo_label', 'metadata': 'Iptc.Application2.Keywords', 'flickr': 'tag'},
{'name': 'set', 'taxonomy': 'photo_set', 'metadata': 'Iptc.Application2.Caption', 'flickr': None },
{'name': 'favourite', 'taxonomy':'photo_favourite', 'metadata': 'Iptc.Application2.SpecialInstructions', 'flickr': None}]

Now load up python and import the library:

import photoblog

Here’s a quick guide to the three main functions:

flickr.download()

Downloads all photographs from a particular Flickr user and embeds flickr meta into the file meta data. Returns a folder name where the images have been saved.

Arguments:
– download folder (path) – folder in which to save images
Optional:
– amount (int) – max amount of images to retrieve. Set to 0 for all (default)

files.organize()

Organises a folder filled with images, movies and thumbnails into dated folders (YYYY_MM). This is especially useful for sorting out images from data cards etc.

Arguments:
– source folder (path) – folder to organise
Optional:
– debug (bool) – will analyse folder without moving images (default = False)

NOTE: Be careful this module will MOVE files, always make a backup first

wordpress.post()

Uploads a folder of images and creates individual blog posts for each, uses file meta data to create wordpress taxonomy. Use the userinfo module to set how you want the match file metadata with WordPress taxonomy.

Arguments:
– source_folder (path) – folder to retrieve images from
Optional Arguments:
– subfolders (bool) – whether to process subfolders (default = True)
– debug (bool) – if set to True will not post to WordPress (default = False)

NOTE: Be careful this module could very quickly post hundreds of images to your blog. Backup your WordPress database BEFORE attempting to post.