My ‘viral’ moment – A Street Near You & the power of linking First World War data sources

Technology, Uncategorized, 9 Comments

by James

A Street Near You started as an idea to demonstrate the potential of combining and enhancing large datasets focused on the First World War.

Three weeks after its launch on 9 November it had:

  • had 435,000 unique visitors
  • been tweeted over 3,600 times with links to the site (one tweet alone had over 1,000 retweets and 1,800 Likes)
  • nearly 40,000 interactions on *public* Facebook posts including over 10,000 shares (and heaven knows how many on private posts).

Beyond the numbers, the reaction has been truly humbling, with people discovering more about their community, their house, their own ancestors, or simply appreciating the global impact of the conflict in a way they hadn’t before (for some amazing examples see the Twitter Moments shared at the bottom of this post).

It takes data from three principal sources – Commonwealth War Graves Commission, Imperial War Museums Collections, and the IWM Lives of the First World War crowdsourcing project – and plots them on a map, typically showing family addresses associated with the person who served whilst combining records, images and facts to create a richer profile, complete with links off to the sources.  The aim is to “explore the local legacy of the First World War” though I have to admit the motivation relied a lot on an intense frustration that so much has been accomplished within individual projects during the centenary, but very little has seemingly been done to integrate these efforts and truly realise the benefits and potential impact of this work.

How it all happened

Built in a week (though based on two years of experience handling this sort of data), on the Wednesday it didn’t even have a name, on the Thursday the domain name was registered, it went live Friday evening, and by the end of Sunday it had had 187,000 unique users!

This timeline gives a little potted summary of how this all came about and what happened.  I think my little demonstrator worked, but if there’s one main message that I would send out it’s that this wasn’t down to me, it was down to the power of the data and what can be done with it!

Some time in late 2016 – I started investigating the idea of georeferencing First World War related records, exploring how locations stored as text in sources such as the Imperial War Museums collections, the War Memorials Register, and Operation War Diary could be mapped to provide a new way to both visualise and search these records

November 2017 – first publicly shared map based on the area where I live, Ealing in West London

During 2018 – extended georeferencing beyond Ealing into other areas (within Google’s free API limits!)

By November 2018 – ca 600,000 records processed, yielding just under 500,000 locations for approx 410,000 individuals

2 November – started building a map! Based on Bootstrap and Leaflet to make it as quick and simple as possible; worked on it mainly during the evenings (and into the nights!)

7 November – first attempt to deploy to a public web server, just as a sub-directory on my catchingtherain.com shared hosting; deploy failed as the host MySql version didn’t support spatial indexing

8 November – plan B: to get the database to work I needed to port it to my Tsohost hosting (used for culturepics.org ) and just to make things easier needed to register a new domain name (what a blessing that turned out to be!) which meant thinking of a name; having taken the day off work, by mid-morning the name ‘A Street Near You’ was concocted, the .org domain registered, and the site deployed and finally working.

8 November (pm) – after showing it to a colleague and a couple of contacts (the full extent of my pre-launch user testing!) I radically simplified the user interaction

9 November – couldn’t do a lot as I was taking some German visitors to the Imperial War Museum; whilst there I read this tweet from a local school who had been inspired by my 2017 map; even though the site wasn’t really ready I sent them a link, which seems to have been the start of things!

9 November (evening) – further tweaks and fixes whilst watching the Google Analytics stats; what was five people on the site soon became 20, and then 50, and then 100

10 November – woke up to a barely responsive site and over 550 real-time users; put out a tweet …

  • thanks to entirely voluntary help, primarily from James Grimster of Orangeleaf Systems (got to love the Museums digital community!) set up a free Cloudflare account which immediately handled about half the requests from my shared hosting; meanwhile made some optimisations to the site (it was only ever intended as a hack/demonstrator!) but it was pretty obvious this wan’t going to be enough
  • added some temporary messaging to the site to apologise for poor performance
  • started talking with my host Tsohost about options; realised that dedicated hosting was the only way to go, but that was going to be £99+VAT per month; added a PayPal link to the messaging!
  • cancelled the dinner/theatre night out with my wife (ooops!) whilst working with my hosts to do the site migration to the new server; went live about 8pm as I recall
  • got the host to add a further 3GB memory (monthly cost £36+VAT)
  • not even sure what I was doing, but finally got to be about 2am; actually some of this was replying to tweets and emails about ‘missing’ people, the response being that they’d need to add records to the Lives of the First World War site and I would pick them up from there – I know several people have, and that’s a win-win as it means I don’t have to add them manually (or not at all) whilst Lives gets more contributions.

11 November – still 600+ concurrent users on the site and still struggling; decided there wasn’t much more I could do and, given the day and the plans I had made to commemorate the day myself (which I would have forever regretted missing), decided to inject a bit of perspective into things

By the end of 11 November, little more than 48 hours after launching the site, it had seen over 187,000 unique visitors and countless posts on social media. What is especially rewarding is to hear that the site led to significant numbers of visits to the source sites – at one point 30% of all traffic to the Lives of the First World War site was from A Street Near You, which amounted to tens of thousands of new visitors  Whilst at a much lower level, that effect is still continuing.

On 12 November the Guardian published a piece by Suzanne Moore who was inspired by the local names she found when she looked up her own address


The tweets, Facebook posts and emails kept on coming, and I even ended up doing an early morning slot on BBC Radio Manchester!  To give a flavour of the reaction, there are some ‘Moments’ that I compiled on Twitter embedded at the bottom of this post.

Why I think it worked

The near impossible questions!  But in reality it was the data that makes it what it is, so simply giving that some context that users could relate to, and making it easy for them to explore, was all I feel it took to unlock that potential.

The other obvious factor is timing, though with so many centenary activities it was a crowded space. I had no ‘communications plan’ as it was only a demonstrator and it took off entirely of its own accord. What I think worked though is that presenting this in a local context was something people could really relate to, reflecting how the conflict may have been over but the impact remained.

I like to think that small things from both a content and technical perspective helped. There are actually only about 10,000 images across the 410,000 individuals, but making sure they appeared first/on top immediately gives them more prominence. I’ve had a few people comment on the name, which was actually a rushed choice but seems to have captured the essence of the project. Setting up permalinks (every map view and filter has a unique url) and proper social metadata definitely helped, or rather not doing this would I think have hindered. Making sure it worked properly on mobile (yes, I know this is old hat but it’s such a common mistake) was also key, as the stats reveal …

Crowdtangle social media stats for astreetnearyou.orgIt was all about social! It started on social and I only need to look at referrer stats and analytics from the platforms themselves to see how massive a part they played.  Over that first weekend at least 80% of the traffic came from social, with nearly 140,000 visits from Facebook alone (compared with 12,000 from Twitter, which gives some real perspective on the scale of  ‘private’ Facebook posts given that there were over 1,000 tweets with links in this period). Clearly connected to this, over 91% of all visitors were on a mobile or tablet device.

When in need, don’t be afraid to be honest, ask for help, and to learn! Being honest about the site performance issues appeared to illicit a lot of sympathy rather than frustration , and adding the PayPal link to raise funds for the server upgrade has raised  enough to cover enhanced hosting for about 6 months! Communicating updates and issues also led to the sort of technical support that James Grimster gave, and also to the way my host went above and beyond the call of duty to handle server upgrades and migrations late into Saturday evening.

What next?

As well as a one-off success I feel this has firmly proven my original ambition to show just what potential there is in connecting and enriching and representing data in this way. I had originally intended to build in some further datasets, small and large, and already have a long list!

I have already done some enhancements, for example the facility to search by cemetery and just recently a facility to report georeferencing errors. But more than anything the underlying technology needs a radical overhaul to make it scalable. Once that is done though it feels like a relatively simple process to pull in new data sources (perhaps a day’s work for each?) so I am talking with a few people about a possible funding bid. All suggestions on that front are welcome!  Of course in the true spirit of a project like this I am happy to share the data back as well.

At some point I’ll do a more technical post about how it was all built, but for now do feel free to ask questions or make suggestions here, via Twitter or email. Meanwhile, I’ll leave this with a sample of the reactions from Twitter.



9 Comments

  1. Thank you James, for your wonderful website. My family was lucky – my great grandfather came home. (My sister is hoping to publish his letters home from the Front very soon – http://thejollymanletters.co.uk/ )Your excellent work coupled with your honesty compelled me to donate. Long may your site continue to inspire and educate future generations.

    Reply

    1. Hi, I am not sure I fully understand the question, but all the sources are linked within the records, as extracted from the source data

  4. Pingback: Takeaways from the Museums Computer Group #Musetech19 – Write it down, Natalia

  6. Great work, James! How about also linking the profiles of WW1 European soldiers on Europeana 1914-1918? I’m also going to include my great-grandpa’s life story there soon 🙂

    Reply
  8. Pingback: Digital Project Evaluation – Kaylee's Kabob

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.