Evaluation | Science Communication Unit blog

The Academy Zone: Reflective Online Science Communication Training

Science Communication Unit | 11 October 202111 October 2021

This blog post is part of an exhibit for Engage Conference 2021.

“I’m a Scientist, get me out of here” is an online communication activity where school students interact with scientists. Classes of students ask scientists about their work and lives in a live, text-based, fast-paced chatroom, as well as on a question-answer platform which allows for more considered interactions.

Over the 10 years it has run, scientists have often reported that their communication skills have improved through participation in I’m a Scientist. However, this experiential training has never been formalised, and the scientists aren’t always aware of what they are learning. As such, in 2019 a short communication course was designed to run alongside the I’m a Scientist event.

This course comprised of resources, such as short videos and articles, and questions designed to help the scientists reflect on their learning from participation. They were also invited to participate in daily live text-based chats with a science communication expert to help them reflect on their learning, make connections and receive advice.

We evaluated the course using the responses scientists provided throughout the course to analyse their reflections and understanding. Data from 11 scientists, who gave their consent to have their responses analyzed and presented in this work, showed reflections around the themes of raising science capital, providing an inclusive environment for all students to ask questions and engage, and also an inclusive environment for the scientists participating too.

Scientists felt that the I’m a Scientist format worked to help the students lead the conversation to be about what interests them, and also to see the scientists as humans with lives and interests outside of science. This linked nicely to resources about science capital, as linking science to your interests or seeing scientists as sharing your interests can help students see that science could be for them (Archer et al., 2015):

“I like that we are not expected to tell the students about science and being scientists – sometimes it could seem like we are talking at them rather than them being able to find out what they really want to know. In this case, they are asking what they want to know, which probably isn’t what I would have talked to them about if I were to lead the conversation. That has been really eye-opening! “

“Often, in other outreach activities I have done previously, they would still emphasise your role as scientific expert, but not as human being with interests, hobbies, doubts and dreams.”

Scientists felt that the text-based, anonymous format worked to create an equal playing field between the scientists and students, and also among students, giving them the ability to all have their questions answered and preventing barriers for students who may be shy or anxious about asking something:

“Loud students don’t dominate our attention, and quieter, shier students can get their question across as equally as anyone else. Also, with anonymised names in the live chat, I don’t spot patterns of the same people talking to me all the time – the pseudonyms aren’t memorable.”

“I think the text-only format helps in various ways. From the student perspective, it means that participants who might be shy or anxious about speaking in public have an opportunity to ask a question without having to actually vocalise. Students can prepare in advance exactly what they want to write, and the anonymous nature of the interactions frees students to ask anything they like without fear of embarrassing themselves. It also gives students a chance to digest the scientists’ answers at their own pace, and go back over things if they want to. From my perspective, it means that all the questions “look the same”: I won’t end up devoting all my attention to the loudest or most persistent voices in the room.”

There was also evidence that the text-only format made the chats feel more inclusive for the scientists too:

“I think the anonymity of the live chats helps immensely. It allows students to ask whatever they’d like, be it science or not, whilst removing the fear of asking a silly question in front of their classmates or even to you directly. I think being behind a screen/giving text answers works well for the scientists too! Just having a profile photo and not video chat removes a lot of visual biases – I’d be rich if I had a pound for every time someone has told me I don’t ‘look’ like a scientist! But with just the profiles, you ARE a scientist, and everyone was asked questions equally.”

By Dr Hannah Little, Senior Lecturer in Science Communication at UWE Bristol.

References:

Archer, L., Dawson, E., DeWitt, J., Seakins, A., & Wong, B. (2015). “Science capital”: A conceptual, methodological, and empirical argument for extending bourdieusian notions of capital beyond the arts. Journal of research in science teaching, 52(7), 922-948.

Evaluating Europe’s largest project on citizen-inclusive decision making for clean air and carbon management

Science Communication Unit | 14 July 202014 July 2020

With just ten years left to change our carbon intensive lifestyles to mitigate climate chaos, urgent decisions need to be made about how we can reach net zero and clean air. Meanwhile, the Covid19 pandemic and the Black Lives Matter movement show that citizen involvement in science communication has never been more critical to ensure a socially just transition.

At this critical moment, the ClairCity project has reached its final dissemination stage; Europe’s largest ever research project on citizen-led decision making for clean air and carbon reductions. Six cities and regions came together to share engagement methods to involve citizens in policymaking, namely, Amsterdam in The Netherlands, the Aveiro Region of Portugal, Bristol in the UK, the Liguria region of Italy, Ljubljana in Slovenia and Sosnowiec in Poland.

As a result of the project team’s efforts, 818,736 citizens were involved in some form or another. Of these, 8,302 were directly engaged through workshops, events, schools’ activities, mobile games and apps, and even videos, a number which far exceeded the expectations of the team.

Why was this significant? Because these 8,302 citizens all influenced clean air and climate change decisions in their local context.

Over four years, the project partners and Council officers made many collaborations with local community organisations and together with a strong social media presence, the project’s on- and offline presence grew. Through a variety of engagement tools, citizens were able to have their say on what mattered to them regarding transport and home heating and what they would like to see change to enable them to make greener choices. There were also candid discussions on the potential barriers to such changes to not only make these concerns known to decision makers, but to have a deeper understanding of the challenges and trade-offs that need to be made when taking policy decisions.

Equipped with this information, ClairCity was then able to consult policy makers about the policies proposed by citizens and discuss how to operationalise them. As a final step, the top citizen policies were modelled against current policy plans for each case study to assess whether citizens’ demands could affect future emissions and associated health impacts. In nearly every context, citizens were more ambitious than ‘business as usual’, with the exception of Amsterdam where the local government was in fact more ambitious than its citizens.

An evaluation of epic proportion

ClairCity was a fascinating project to evaluate for our SCU team including Dr Laura Fogg-Rogers, Dr Margarida Sardo, Dr Corra Boushel, Sophie Laggan, and David Judge. We have produced a full evaluation report with all of the findings, which you’re welcome to read if you have the time, and a shorter one if you have less time. For those visual learners, there is also a visual evaluation report and our webinar recording. Finally, to satisfy blog readers, we have you covered too! Read on to hear our reflections and key findings.

The aim of the evaluation was to see whether the project had fulfilled its aim of ‘raising awareness of environmental challenges and their solutions through proactive dissemination of the project outcomes’. To do this, we explored the demographics of participants and those less directly involved, examined indicators of awareness, attitudes, knowledge and enjoyment (so called Generic Learning Outcomes) and citizen’s intended behavioural changes. Additionally, we explored differences across countries, demographics and engagement tools, to understand perceptions in different contexts. Data collection was carried out through paper/online questionnaires, including pop-up windows in the game and app, as well as in-depth interviews with staff and social media analysis.

All tools exceeded their targets for audience reach, apart from the App which remained at BETA testing due to technical issues. The Delphi workshops were particularly impressive, with 4887 participants compared with 200 expected, and the ClairCity Skylines game, with 2,800 players worldwide compared with 1500 players expected. Both successes can be largely attributed to the on-the-ground outreach and marketing activities of our case study partners, who made connections with community organisations, produced flyers, spoke on the radio, attended events, and generally made lots of noise to attract people. The cities that spent less time and resources on this groundwork had fewer participants as a result.

Social prowess

Our social media platforms gained a lot of traction over the years, although they were again limited by time and resources available. Our Communication Coordinator in Bristol was able to orchestrate our main sites, resulting in (at the time of writing) 1,392 Twitter followers and 416 Facebook followers, and 36,482 website visitors. Sites managed by our partners – who weren’t dedicated science communicators – had considerably less traffic.

Demographics

Data was collected for age, gender and educational attainment. Given the fact that ClairCity had targeted schools’ engagements, with several team members having direct connections to local schools, in addition to a mobile game, over 40% of participants were aged between 13-24. Working adults occupied around 50% of participants, and over 55+ represented less than 10%. This is quite an impressive finding considering most engagement projects fail to capture the full spectrum of ages.

63% of participants in the study identified as male. The biggest gender differential came from the game, with more than twice as many male players than female, which skewed the gender balance. Alongside this, many stakeholder workshop participants were senior men in regional organisations, which again skewed the gender balance.

Participants were asked about their education level in our workshops. 81% of respondents held a Bachelor’s degree or above. On the other hand, in the game, 79% ranked their level of knowledge on air quality as being low/none. In other words meaning, the game appealed to people with less expertise.

Learning outcomes

Both policy makers and teachers were asked about the usefulness of the tool relevant to them. An overwhelming percentage of policy makers found the policy workshop useful/very useful (95%), compared with a more modest percentage of teachers finding for the schools’ competition (61%). The schools’ activities have since been expanded following this feedback, and our Educator Pack (part one and two) is freely available online, and has been featured in the British Science Association Science Week pack, and through Sustainable Learning.

The majority of participants enjoyed or loved the activities in which they were involved. Both the Delphi and Stakeholder workshops greatly improved participants’ understanding of air quality (88% and 82% more understanding, respectively). 39% of game players left with more understanding, however for 45% their understanding stayed the same. The app mainly left people with the same understanding (47%), or feeling confused (18%).

Perhaps one of the biggest findings was in regards to behaviour change. At least half of all participants in the Delphi workshops, game, schools’ activities and stakeholder intended to change their behaviours as a result of their involvement (58%, 80%, 67% and 79%, respectively).

Upon cross-comparison, it was found that the more participants enjoyed the activity, the more they reported that their understanding of air quality had improved. Similarly, the more participants reported that their understanding had improved, the more they reported that they would change their behaviour. Younger people and those with lower education to start with were more likely to say they would change their behaviour. All of these relationships were highly statistically significant.

Ultimately, the more enjoyable the engagement activities, the more people gain understanding about the issues, and the more likely people are to make a change to their behaviour to reduce air pollution and carbon emissi0ns, and improve the health of our cities.

Reflections on the evaluation process

In future we would recommend other projects take additional time to target women’s groups, or develop ‘tools’ that appeal to women
While efforts were made to reach representivity through undertaking the Delphi process in low socio-economic status neighbourhoods, in hindsight we would have worked harder to amplify under represented voices. Recent Black Lives Matter protests have been a stark reminder of the need to make our work inclusive..
Working on an international project presented issues with translating the website and evaluation forms. More dedicated evaluation time, or expert science communicators in each country, would have helped researchers who were less experienced in social science research methods.
We benefitted from having evaluation embedded from the beginning (rather than an add-on), and as such designed our evaluation methods to work in different contexts and cultures

Most ClairCity staff found engaging with citizens challenging (due to not having experience in this) but highly rewarding. By the end of the project the vast majority stated they have enjoyed engaging with citizens. This was a rich experience in terms of new skills, with our staff reporting to have learned how to pitch their ideas, how to talk to citizens and how important is to listen to people as well.

If you are interested in our experiences, or in benefitting from our reports, please check out our website for a variety of resources and tools to aid future citizen-led decision making on climate change and air pollution.

Sophie Laggan, Communications Officer, ClairCity

Modernising the iconic Royal Institution Christmas Lectures

Science Communication Unit | 8 October 20198 October 2019

A team of Science Communication Unit researchers was selected by the Royal Institution (Ri) to explore ways to continue modernising the Christmas Lectures, an iconic series that has run since 1825. Margarida Sardo, Hannah Little and Laura Fogg Rogers conducted research to explore strengths and opportunities for improving the series, created for children and televised annually for the past 50 years fronted by presenters including David Attenborough and Carl Sagan.

A centrepiece of the national conversation about the place of science in our lives, the lectures were started by scientist Michael Faraday in 1825 and are now designed to be engaging and mind-expanding viewing for people of all ages but particularly children. A series of three on a single topic, the lectures are filmed in London in mid-December every year then broadcast on three consecutive days during the Christmas period. In 2018, biological anthropologist, author and TV presenter Alice Roberts and genetics expert Aoife McLysaght brought the evolutionary story to life in a series called ‘Who Am I?’

Aoife McLysaght and Alice Roberts. Photo credit: Paul Wilkinson Photography

Following interviews with children attending the 2017 and 2018 events, a social media analysis of the 2018 broadcasts and survey of science enthusiasts, researchers found the lectures were cherished by audiences of all ages but format changes could help broaden their appeal among young people, as well as older audiences. They recommended exploring ways that the channel and time of the broadcasts (currently BBC4 at 8pm) could be made more suitable for a younger audience, including cutting the lectures down into short clips for social media to reflect changing viewing habits.

*Photo credit: Paul Wilkinson Photography*

The research team noted that some adult viewers believed the lectures were so focused on a younger audience that they lacked appeal for older viewers. To continue to attract a significant adult audience, they recommended creating a companion lecture aimed at older science fans.

Researchers also found audience members enjoyed the live shows, particularly the engaging, interactive and high-quality demonstrations. TV viewers valued the televised shows and many remarked that watching with relatives had become a family tradition. But some science enthusiasts suggested the Ri needed to re-evaluate its target audience and questioned whether the needs of a live lecture and a TV programme were being confused. Viewers were roundly supportive of female scientists presenting the show, with particular praise reserved for the performance of Alice Roberts and Aoife McLysaght in 2018.

The Christmas Lectures branding includes the prestigious Christmas Lectures broadcast on BBC, as well as live shows, a Schools Conference, the Ri Advent Calendar and “I’m a Scientist… get me out of here” – most of which were covered by the evaluation.

The full evaluation report can be accessed here.

SCU to evaluate Royal Institution Christmas lectures

Science Communication Unit | 3 July 201826 June 2019

The Christmas Lectures are an internationally known landmark of the Science Communication landscape, and we refer to it in our teaching as one of the earliest examples of scientists engaging with the public with institutional backing. Physicist Michael Faraday initiated this series that has run at the Royal Institution of Great Britain (Ri) since 1825, without interruption except World War II. Well known science communicators like David Attenborough and Carl Sagan are among more recent speakers.

It therefore came as a great opportunity to tender for the Ri’s call for an evaluation project, and an honour to be selected to deliver it. A team formed by Margarida Sardo, Laura Fogg-Rogers, Hannah Little and Erik Stengler, supported by the expertise in the wider SCU, has undertaken a close collaboration with the Royal Institution to explore strengths and opportunities for improvement of what has now become a much wider project than the actual lectures delivered at the Ri headquarters around Christmas each year and broadcast for over 50 years, mainly by the BBC. The Christmas Lectures has grown into a project that also includes continued provision of materials and activities for schools, public events such as talks at the Big Bang Fair and a traveling show that reaches out beyond the UK, all of which is also brought into the dimension of social media via different platforms, including YouTube, where the recordings of the Christmas Lectures are actually made available to be enjoyed at any time.

The evaluation by the SCU team will cover all these dimensions of the project and also explore what would attract people who do not yet engage with the Christmas Lectures in one way or another. It will be a great experience to be able to be part of and contribute to the continued success of the Christmas Lectures in these years leading to their 200^th anniversary.

If you have any views and suggestions about the Christmas Lectures, do not hesitate to contact us so we can include them in this exciting evaluation!

Science communicators need to get it: science isn’t fun.

Science Communication Unit | 10 November 201530 January 2018

I am writing on the flight that takes me to a conference in Warsaw, which will nicely draw to a close a research cycle that started shortly after I joined the SCU. As part of my job interview at UWE Bristol almost exactly 4 years ago, I said that one of the things I hoped to share was concerns about the role of “fun” in science communication, and specifically in science centres. These concerns had developed during the decade or so I had spent working in science centres and with the media and I felt they needed to be addressed through academic research alongside practitioners.

I am unable to tell exactly when and where I developed an interest in investigating this currently ubiquitous trend of “fun,” but I found I was not alone in this endeavour. I liaised and collaborated with Guillermo Fernández, an engineer and exhibit designer I knew from the times when he would offer his services to the science centre I worked at. We were later joined by Pere Viladot, a very experienced museum educator and since recently PhD in science education. I was also able to draw into the investigation various undergraduate students in their final year at UWE, through their third year project. I must say, as well as ticking the boxes of integrating research into teaching and providing a unique student experience (in that the students did real original research, as opposed to repeat experiments of well-known phenomena), this has been the part that I enjoyed most. It was simply awesome to see these undergraduates doing research work that was often of better quality than what we see in MSc dissertations, and to share their excitement at seeing their work being accepted and presented at international conferences.

Scientists see their work as fascinating, intriguing, exciting, interesting or important, but not “fun.”

So, quite early on, student Megan Lyons explored the perception of the “fun” in science by people at different stages of their scientific career, showing that with time and experience “fun” becomes less and less of an adjective used to describe their work, and that there are many other words that are preferred in its stead, such as “fascinating, intriguing, exciting, interesting, or important”, to name but a few. We presented this at a conference in Lodz (Poland), where people involved in Childrens’ Universities met to discuss their practice. It was quite an eye opener to encounter such a lack of understanding of what I was saying, and in the lively Q&A session I kept having to repeat again and again what we were warning against (conveying the message that science is fun) and what we weren’t (making science teaching and communication as fun as possible).

In the context of science centres, undergraduate student Hannah Owen had already explored whether science centres are actually addressing and providing opportunities for “dialogue” between science and society. Her analysis of At-Bristol and Techniquest that concluded that indeed they are not, and that the only attempts to do so are through activities and events, not through the exhibitions. This was presented at the 2013 edition of the Science in Public Conference in Nottingham and recently selected for publication in the Pantaneto Forum.

Science centres have tried and failed to understand their own role… and it sends out the wrong message about science.

This provided another example in which to understand how science centres have tried and failed to find a way to understand their own role, focusing on “fun” being just one more of these attempts. Reflecting on this we have been able to identify various “wrong messages” that are conveyed in this way: it sends out the wrong message about science communication (deterring scientists from engaging in it), about science (as being a scientist is definitely not about having fun), about science education (as it seems one has to go out of the classroom to have fun, the conclusion being that class is boring), about science centres (as they become a venue of entertainment only, displacing from them valuable approaches such as inquiry based learning, for which they are an ideal venue), and about children (as it condescendingly assumes they would only engage in things that are fun).

We presented these reflections at the EASST 2014 conference in Torun (Poland) and to Physics teachers at the TPI-2015 (Teaching Physics Innovatively) conference in Budapest, where undergraduate student Jessica Tee complemented the research with the outcomes of her final year project. She confirmed that science centres are indeed a valuable educational resource, thus reinforcing the message that this is not something they should lose at the expense of concentrating on providing “fun”.

Science centres need to return to the long forgotten principles of museology, which is the language they speak and master.

Now in Poland I will present all this to the science centre community, which were the only players of the field to which we had not yet had the occasion to communicate our proposal. We think that instead of “fun,” science centres should focus on what they do best and constitutes their core business, namely exhibitions. To do so, they need to return to the long forgotten principles of museology, which is the language they speak and master, just as movies use the cinematographic language. This means that museologists need to be engaged, something that does not currently happen. In the educational front, science centres could use such well-designed and conceived exhibitions as the “field” for data collection in inquiry based learning. Again, to do so they need a shift in their hiring: this needs highly qualified educators instead of – or at least in addition to – museum explainers that are volunteers, interns or other temporary staff. The alterative is science centres trying to compete with other venues or options in what these others do best (fun, entertainment, multimedia & audiovisual communication etc…), hardly a promising prospect in the long run.

Just two days ago we submitted an article about this to Ecsite’s SPOKES magazine, and another one in more detail is about to appear in the Spanish Journal of Museology. Together with the presentation in Warsaw they constitute a satisfactory culmination of this collaboration. As the plane prepares to land (and I am asked to switch of this tablet), I can’t help enjoying a sense of achievement and starting to look forward to the next research cycle on which I will report in due course…

Erik Stengler is a Senior Lecturer in Science Communication at UWE.
@eriks7

Bristol Bright Night summary

Science Communication Unit | 29 October 201530 January 2018

On Friday 25^th September 2015 Bristol was one of the many European cities celebrating research and researchers. The European Researchers’ Night takes place every year, on the last Friday in September. The event is known locally as Bristol Bright Night and it’s jointly organised by UWE Bristol, the University of Bristol and the Bristol Natural History Consortium.

Running for the second year, Bristol Bright Night offered an array of activities including talks, hands-on, debates, comedy and many others. Margarida Sardo from UWE Science Communication Unit was responsible for evaluating the event. The evaluation was designed to suit different venues and audiences.

For this event the methods used included paper and online questionnaires (using an ipad didn’t work very well!), observations, a feedback wall – which was particularly popular with school children – and post-event interviews with researchers and organisers.

For more photos from the event, go to the BNHC website.

Student opportunities at the Latitude Festival

Science Communication Unit | 13 October 2015

One of the nice things we’re able to do from time to time is offer our Masters students work experience on a public engagement project.

For the last two years (2014 and 2015) Margarida Sardo and I have carried out an evaluation of a strand of activities sponsored by the Wellcome Trust at the annual Latitude Festival.

debs — Debs (2015) handing out feedback cards

The Latitude Festival is a well-known and wide-ranging cultural festival, which includes comedy, music, theatre, literature, poetry, dance and more (think Glastonbury but with less mud and more writers!). The Wellcome Trust events are also hugely varied, including poetry, music and theatre performances, presentations, discussions, dialogues and interactive events.

Margarida designed the evaluation, including snapshot interviews with members of the audiences, informal feedback via comment cards, observations of events and interviews with presenters, while I led the evaluation at the festival. In both years, the students were chiefly responsible for carrying out the audience interviews and looking after the informal feedback, so it was an excellent opportunity to gain an understanding of what is involved in the evaluation of a live event as well as strengthen their communication skills.

Tariq and Tom (2014) sorting out feedback on post-it notes

With around 26 events taking place in half a dozen locations around the three days of the Festival, the help and support of our students was absolutely invaluable in helping to collect as much data as possible. Between them, the 2015 team observed 14 full events, persuaded 45 people to be interviewed and got 192 people to complete a comment card!

louisa — Louisa (2015) doing a snapshot interview

In return for students’ support, we offered a modest payment, subsistence expenses during the Festival and free transport to and from Suffolk. The students also had free tickets to the Festival, which gave them access to most of its 200 or so events. As most of the science events took place during the day, and the big comedy and music headliners were on late at night, the students got to see some really interesting stuff!

You can find our report from the 2014 Festival on the UWE repository. And the hard work of the students is also contributing to two papers that Margarida and I are currently working on.

Ann Grand and Margarida Sardo are research fellows in the Science Communication Unit.

Tag: Evaluation