An interview with the team behind the remarkable crowdsourcing exercise that informed Scotland’s approach to easing lockdown
Around the world we’ve been seeing governments respond to COVID-19 in markedly different ways, but one consistent thread has been heavy state intervention and use of government powers rarely or never seen in peacetime. Governments, both local and national, all of a sudden were making snap decisions on behalf of all their citizens without the standard bureaucratic processes. This was absolutely necessary, but while resources were diverted to the Coronavirus response, consultation and engagement took a back seat.
So it came as a welcome surprise when long-time Delib customers the Scottish Government contacted us at the end of April, asking if one of our platforms could support a conversation with all of Scotland, commissioned by people at the top of the government, about the effects of lockdown on the Scottish people and how to transition out of it.
Also, could it launch within a week?
What followed was an exercise that was unique both for Delib and for the team at the Scottish Government. Hosted on Dialogue, Delib’s platform designed for hosting constructive online conversations, it ran for one week. (It’s closed now, but you can still view the site and all the published submissions.) The engagement was extremely popular, with about 4000 ideas posted and nearly 20000 comments. I was lucky enough to be able to speak with members of the Digital Comms team and the Principal Researcher who worked on the project – about the process behind it, the intricacies of moderating and analysing thousands of comments about a public health crisis, and what happens next. For the purposes of this piece I’ll refer to them collectively as Scottish Government (SG).
This interview is in two parts: this part talks about running the exercise itself; and the second part will discuss its impacts on policy and the wider contexts of the Scottish Government’s approach to public participation in the future. Enjoy.
So let’s start at the beginning. Where did the initial idea come from?
Scottish Government team (SG): There was a commitment in the original Coronavirus Framework for Decision Making that the Scottish Government published saying that we would listen to the people of Scotland, and then the First Minister, Nicola Sturgeon announced it in one of her public briefings.
SG: Yes, so we piped up and said ‘hey, we’re the digital engagement team, let us help you!’
For context, the Scottish Government released the document linked above at the end of April, setting out clearly and concisely how Scotland would decide what steps to take through the Coronavirus crisis. One of its commitments was to meet the World Health Organisation criteria on necessary steps to lift restrictions, and one of these criteria is to effectively engage the people of Scotland in that process.
SG: It was a very unique experience, given that we’re all in lockdown and working from home. There was a series of emails with inclusion from policy, comms, our Open Government team, digital comms, the analysts…and then a submission was drafted which people from all across government got involved with and fed into. At that stage I had had quite a few calls with [Louise Cato, Delib’s Delivery Director] just to make sure the platform could handle the proposal.
Dialogue, while it lends itself well to this style of discussion (the clue being in the product name), isn’t used to handling huge amounts of traffic. There was some significant wrangling done on our end by account managers and developers to beef up its capacity so that it didn’t get overwhelmed and crash. You can read about what precisely took place in Louise’s blog post about this.
And what made you decide to go with Dialogue for this exercise?
SG: Dialogue is something we’ve used before, so there was that comfort in understanding the tool and how to use it. There’s a lot to be said for experience, knowledge, a working relationship with you guys.
SG: There’s the simplicity of it as well – we didn’t want people to be confused or have to read a load of information before they could take part. Dialogue was ideal in that it’s just easy: you come in, you say your piece, and leave.
Crowdsourcing is a simple enough concept, but there’s a good deal of nuance between doing it well and doing it badly. There’s also a bit of a stigma around it – the Boaty McBoatface debacle is a classic example of why some organisations might be reticent to try asking the public for their input. Broadly there are a few things crowdsourcing work needs in order to be successful: a clear question that people can personally relate to; a LOT of promotion; accountability from the organisation leading it. All of these were present for this project. Still, on a topic like Coronavirus where there has been a great deal of misinformation, there’s nothing to say that the team weren’t going to be inundated with conspiracy theories.
We didn’t want people to be confused or have to read a load of information before they could take part. Dialogue is just easy: you come in, you say your piece, and leave.
Why crowdsourcing, rather than for example a consultation?
SG: Crowdsourcing lends itself much more to a proper conversation than a consultation. Lockdown has affected everyone in Scotland in various and unique ways. Part of the commitment to having this conversation is enabling people to share views without the barriers of set questions. Also, because the nature of Dialogue is public and comments are getting posted as they come in, people are able to actually converse with each other rather than submitting a formal consultation response which is published much later in a report.
Scottish Government are no stranger to large-scale engagement exercises: their consultation on same sex marriage, run in 2011, got over 77,000 responses. Consultations with such large response rates are usually analysed by external contractors. With crowdsourcing, however, the process is vastly different: rather than receiving submissions and analysing them once a consultation has closed, this specific exercise in crowdsourcing needed an altogether different approach. Comments needed to be moderated in real time, and with nearly 20,000 in a week, that was no small feat.
Dialogue has a couple of moderation options: pre- and post-moderation. For this project, the team opted for pre-moderation – meaning each comment and idea had to be checked by a person before it was publicly posted on the site. To minimise the delay between a user submission and it displaying publicly, moderators worked in shifts to approve them.
Were you expecting this huge number of responses? How did you manage the workload?
SG: We honestly had no idea what to expect! From a moderation perspective we didn’t know what the volume of response was going to be so we planned for two shifts a day, a morning and evening shift. We had to pull extra resource once the challenge was live and we had a better sense of the pace of the engagement.
SG: We had two analysts on a shift and then more in reserve, and I think after day one we just brought all the reserves into the team, and then brought also a couple more people on to help! We developed thematic tags as we examined the early responses, and then would tag topics as they came in. So even though we didn’t have time to read every single comment or idea we got a broad sense of what was coming in because of the tags, which correlated with what the moderators said they were seeing which was a reassurance. That was largely how we managed it – instead of trying to read every comment or just the ones that were highest rated, we focussed on the threads and ideas that were engaged with the most. Typically the ones that weren’t quite so highly rated were the ones that had a bit more of a polarising impact. For example, people saying ‘open schools now’ or whatever. The ones we found that were most engaged-with were the seeded ideas we’d put in there, just to guide people towards responding to the framework. Consistently those were that ones that tended to attract most commentary.
There’s been a lot of misinformation going round about COVID – people have been burning down 5G towers because they claim that’s what’s spreading the virus. Weren’t you worried you’d get overloaded with conspiracy theorists?
SG: We were actually more worried that nobody would take part! We weren’t generally too worried about getting lots of responses that would be silly or not able to publish because the question that we asked was a sensible one. The promise here was that we wanted to hear the public’s ideas and experiences; that was made very clear in the framing documents and in the way it was articulated in the daily press briefings, which get quite a lot of attention. It was clear that it was a space to share, rather than making a finite decision on something, and I think that was really important.
We didn’t want it to become a platform for people to share conspiracy theories, but people were very sensible…they moderate themselves.
SG: On the other hand, as moderators we’re not experts on what is disinformation and what isn’t. We were not ‘fact checkers’. So really my concern was people posting things like, for example, about 5G, and then us not having the time or the resource to actually know and check if it was accurate. Obviously we wanted it to be a constructive conversation, we wanted people to be able to express their view, but we didn’t want it to just become a platform for people to share conspiracy theories. Which I don’t think it did, to be fair – I think people were very sensible. If someone did say something like that, other commenters would come on and say ‘that’s not true’, so people do moderate themselves.
SG: There were also some very interesting situations because obviously the lockdown measures in England are different from Scotland. Because there were announcements coming out whilst this was running, it had impacts in terms of how people in Scotland interpreted what lockdown meant here.
In the end, despite the subject matter, only a small percentage – about 3% – of ideas submitted to the platform were removed in accordance with the Scottish Government’s moderation policy.
Huge thanks to the team for taking the time after an extremely busy project to talk with me. Part Two of this interview will cover the post-project phase and the wider impacts of this exercise both on policy and on the Scottish Government’s approach to public engagement on the whole. Keep an eye on this blog or sign up to our newsletter to be notified when it’s published.
Read Part II of this interview.
Follow the Scottish Government’s digital engagement team on Twitter and check out their Citizen Space site.
Follow Dani on Twitter.
To find out more about Dialogue, book a free demo and we’ll talk you through it.
If you’d like to get content like this delivered straight to your inbox, sign up to our newsletter, the Practical Democracy Post.
