The “stuff” we create

I’ve rebuilt my website. This one you’re reading, unless you’re reading this via RSS, in which case, this one.

It’s taken a long time and it’s not noticeably much different. I’ll get on to the how and why, but first, here’s what’s missing, new or changed.

What’s missing

• Posts in the Writing blog no longer have categories. I’m only using tags now (listed here). The old category URLs should redirect. The tags need tidying up, as indicated by this post currently being tagged “Misc”. That might happen one day.

• The RSS feed URLs have changed. The old URLs should be redirected by Feedburner. But I’m amazed Feedburner is still going so if you subscribe to any feeds you might want to update your subscriptions. New links are at the bottom of every page (e.g. the home page).

• The “everything” RSS feed includes less. There’s now only one combined feed (instead of two) and it doesn’t include posts from WhitStillman.org or The Diary of Samuel Pepys’ Site News, or the mothballed Septivium and Overmorgen. Also, it doesn’t include videos I’ve liked on YouTube, although I hope to reintroduce them someday.

• Comments on links have gone. My Pinboard links are no longer grouped by day, which makes more sense — they have individual permalinks and can now be viewed by tag. But this means I lost the handful of comments that people had posted on the old daily link collections (back when commenting worked).

• There are no daily maps. In 2010 I added a map to some of the pages that aggregated things from individual days. I wrote about that here. Those maps are no longer there, although I think they stopped updating some time ago. Also, no Foursquare check-ins appear on the site now, although they also stopped accumulating a while back.

What’s new or changed

Aside from all the underlying technology, more of which below, some things are noticeably different:

• The design and HTML/CSS. The most obvious change, although it was a minor part of the overall work. The general site and page structure is much the same. This structure seemed to work and while re-writing the site’s code I wanted something stable. I’m generally pleased with the design but there are some parts that don’t quite hang together for me, so I’ll continue to tweak it. I continue to find design harder than I continue to expect. I made a set of HTML/CSS components to keep track of what I was doing across the site.

• Old pages have their old designs. Taking a cue from BBC News, whose old articles keep their old designs, my Writing posts now have the HTML they had when published. For example, here are a post each from 2000, 2002, 2004, 2006, 2008 and 2010. On the downside, these sudden changes in appearance can be a bit jarring and the old HTML behaves worse on small screens. But I think it’s important to retain some of the context given by the design and (front-end) technology of the time. These aren’t static archived pages: I’ve fixed a lot of the navigational links and I can still add updates to posts if required. A living archive.

• The daily pages now include tweets I’ve liked. e.g. scroll down on 3 September 2016 or 17 June 2017. The display of these and my own tweets are much improved over the previous site.

• The home page. I liked the main part of the old home page (see an old version here). The items were organised by recency plus a weighting of “importance” (to me). Now the home page is a single column, organised by recency. It’s OK but not ideal. And it’s a bit ugly. I may improve this at some point.

• Archived things have their own site. I had copies of several old sites and pages, all of which have been moved to the new archive.gyford.com.

How I made it

The old version of the site was running on Movable Type (MT) and had been since, I think, the start of 2001. Not a bad run. It had accreted a lot custom PHP code to bring the three MT blogs (Writing, Comments-posted-on-other-sites, and Links) together, pull in other content (tweets, Last.fm scrobbles, Flickr photos, Foursquare check-ins), and manage the lists of things I’ve read.

I wanted to extricate myself from Movable Type entirely and avoid PHP, which I don’t use much these days. My language and framework of choice is python with Django, and standardising on one way of building sites makes life a bit simpler (e.g. The Diary of Samuel Pepys is also Django).

Writing code for a weblog using something like Django isn’t too difficult (relatively) and shouldn’t take too long. But I didn’t want to lose any of the extra bits of my site (tweets, reading, etc). In fact I wanted to make those bits more robust and useful. The previous site didn’t store much useful data about the things it pulled in from other sites. For example, some code grabbed the week’s most-played Last.fm tracks and wrote out a small HTML file which the PHP could include on the appropriate pages. Simple, and it worked fine for years. But that doesn’t save data about my listening in any other form; it’s not reusable in any way.

Being me, and having no budget or time constraints, I was able to wildly over-engineer a new solution to my own more-or-less-satisfaction. After a false start or two, I spent a lot of time making Django Ditto, a standalone set of Django apps that I wrote about in August 2016, for grabbing, storing and displaying tweets, Flickr photos, Pinboard links and Last.fm scrobbles.

I then made Django Spectator, another set of Django apps, for tracking my reading and also (spoiler alert!) visits to movies, plays, exhibitions etc. That was “finished” (it’s never finished) around the middle of 2017. I doubt it’s useful to anyone else but at the time it seemed sensible to make it a standalone project. That was a lot more work than necessary.

Since then I’ve spent much of my free time writing the rest of my site’s new code (which is on GitHub). This stitches together Ditto and Spectator with the weblogs and all the other parts of the site.

My main goal was to lose as little as possible in the move, which I’ve achieved. The list of missing things above is satisfyingly short. All the blog posts and reading data has been imported from the old site’s databases into the Django database. This includes all the comments left on blog posts from when the commenting was still working, and all the old Trackbacks. The comment data is now stored in django-comments so I may bring back the ability to post comments. I liked comments. Trackbacks are unlikely to make a comeback, but are still displayed, even if the blogs they link back to usually no longer survive. Don’t lose data.

All my tweets, liked tweets, photos, links and scrobbles have been imported and stored. Although I don’t do a lot with most of them now, Django Ditto will make it easy to do more. For example, I can give every Flickr photo its own page on my site, with all its associated metadata. Which would be especially useful should something happen to Flickr… I also have access to a satisfying amount of statistical data about 21,945 tweets, 3336 photos, 4563 links and 168,758 scrobbles.

Why?

This is a question I’ve tried not to ask myself too much while writing all this code over the past few years.

I could have imported all my old “Writing” blog posts into WordPress and not worried too much about all the other stuff on the site. That would, by most measures, have been the sensible thing to do.

But I think a personal website should be or, at least, could be, more than just a blog. I’ve always been uneasy about all the “stuff” we create, or data we generate, on third-party sites and how it can all disappear if something should happen to the company, however nice for them. I want it to be easy to have some control over this stuff, and for it to be integrated more closely. And, simply, wouldn’t it be nice to have a single personal site that reflects all the public things you do in other places online?

There are at least two approaches to this, with their own pros and cons:

1. Create things on your own site/service and push them out to Twitter, Flickr, Pinboard, or wherever. You create the “original” and send copies elsewhere. e.g. you could push each of your status updates to Facebook, Twitter and Mastodon.

2. Write or upload your content directly on the third-party services. They probably have good interfaces for this kind of thing. Then you can pull your updates, photos, etc. to your own site. This is what I’ve done.

I still don’t know if this aggregation of personal content is worthwhile. Part of me thinks it’s important that this should be possible for those who want to do it. You should be able to “possess” all your digital creations and display them in one place. And the easier this is, the better. Me hand-crafting my own solution doesn’t make it easier for many people, but I hope that demonstrating what can be done is useful. Even if it shows it’s not something most people want or need…

Because another part of me thinks this is ridiculous. Maybe anything posted on a commercial site should be considered ephemeral — it’s interesting or fun or important for the day or week after creation but after that the item’s worth often decreases dramatically. It’s still important that you should be able to grab it all somehow (like Twitter’s quite good downloadable archives). And it’s still important that someone should be archiving some or all of this material for future generations.

But does anyone need to store and display all this stuff personally and semi-permanently on their own site? Maybe the web, outside of archive.org, doesn’t need to be permanent? Maybe I’m pushing against the grain and making life hard for myself?

I’ve tried not to give those thoughts much space over the past few years. I’ve been doing my best to keep my own public digital things available, and aggregating as many of them as possible, for a long time. I wanted to create as good a representation of my public life online as possible, in a single place of my own. I figured it was worth staying on the bus and seeing this idea through, ignoring any doubts about the idea and whether this was the best use of my time.

It’s a relief to get to this point. Rebuilding the whole thing has been a struggle, taking a lot of time and persistence. I don’t know how long it took, but let’s say “months” of full-time-work-equivalence spread over years, since I first started looking at aggregating tweets, photos, etc. with Django in 2011. It’s only “finished” now because I’ve been able to devote a lot of “work” time to it over the past few months.

Have I learned anything about this kind of work? Let’s look at what I wrote in 2006 when I did most of the initial work on the previous Movable-Type-and-PHP version of the site:

This has been an exercise in how not to tackle a web project, an exercise in which I utterly flouted all the sensible rules of thumb I stick to on any work project. With no deadline, no budget, and a client (me) who kept throwing in features it’s a wonder the site’s finished. I’m trying hard not to be sick of the whole thing already.

This time I convinced myself I was older and wiser because I didn’t plan on adding any new features to the site. But both Ditto and Spectator contain many things I’m not yet using. So these are indeed new features, only features I’ve hidden away from myself so I can spend more time integrating them into the site in the future. I am a dozen years older but not much wiser.