Subtleties behind the numbers & the enduring appeal of qualitative impact

5 min readDec 18, 2019

My experience conducting Impact Surveys for GoRead.pk in schools across Karachi, Pakistan. By Zahra Parekh, Program Manager at GoRead.pk

It was August 2019, and we were gearing up for our six week pilot Reading Sessions across 11 different schools in Karachi. We had confirmed 26 champion volunteers, now officially referred to as GoReaders, who had been chosen to embark on this journey with us and launch the GoRead.pk program reading stories to over 550 children from Karachi’s under served communities.

As a team, we had worked toether since March/April 2019, making sure we had a robust foundation operationally from finance, marketing to publishing stories; ensured that co-ordination between schools, GoReaders and our team was seamless; and we had also worked with our technology partners to launch our website and mobile App. These wide varieties of tasks and projects to ensure our Program would have a smooth and successful launch were exhilarating (and, at times, frustrating) in their own ways as we pushed through nearing our Program launch in September 2019.

However, the one core part of our program that continued to be the most intriguing and exciting to me, was designing and executing an impact assessment plan that would help measure GoRead.pk’s value — the social impact our program was meant to generate.

We had to design the surveys, plan the implementation across our partner schools and designated classrooms and finally execute it in the most resource efficient manner.

Our main aim in designing impact surveys was twofold:

1. Measuring our output: Pre and Post Test surveys that would help measure whether children (who received Read Aloud Sessions and books to read) (i) learnt new words they had previously not heard before; as well as (ii) whether our stories’ themes, centered on development of specific character traits, had resulted in children learning and understanding these traits and qualities.

2. Efficiency and Ease: Ensuring the survey was (i) resource efficient and relatively easy to execute for us as a team who would be traveling from school to school and class to class implementing this survey in communities that had little or no resources; as well as (ii) making sure it was an engaging activity for the children and that it did not feel like a test so as to make them comfortable when taking part in the survey.

Our Pre Test surveys were conducted before our Pilot Reading Sessions began. In every classroom, we patiently explained the methodology of the surveys to the children and then went through the presentation questionnaire slide by slide projected on either a white sheet scotch taped to the blackboard wall or using the white walls in the school premises. Over 550 children, often unsure of their ability to answer the questions, filled out their answer sheets, curious to know what this activity would result in. Shortly after this, our reading sessions began in the same classrooms we tested and my team and I eagerly waited for 6 weeks to conduct the Post Tests.

It was the beginning of November by the time our pilot of six weeks ended and we geared up for the same activity as we had done, albeit in more favorable weather conditions as the relentless Karachi summer heat had finally dissipated.

As we travelled from school to school again for the Post Test Surveys, we observed intriguing differences this time round. The intangible impact, or as it is commonly known as the qualitative impact, was immediately noticeable and more real or ‘tangible’ during the Surveys than the quantitative data being gathered.

Excitement to participate in the Survey:

During the Pre Test Survey, almost all children assumed that this might be an exam and they questioned us if they would be given marks on their performance. They assumed that it would be similar to a school test. However, when Manal and I entered each classroom again for the Post Test, not only did the students recognize us immediately but they noticed our equipment and were excited to be a part of the survey even before we reminded them what we would be doing. Although our survey was Multiple Choice Questions, the images and questions we used seemed to connect with the children and allowed them to treat it as a game or a fun activity. This slight deviation from their regular school day gave them enough of a push to excitedly take part in the Survey. It was important for us to note that just as our stories were meant to bring a little bit of joy and cheer into their daily lives through reading, our Survey had become a microcosm of this joy.

Confidence in Survey Methodology:

Each Pre Test Survey in almost all classes took 30 minutes to 45 minutes to complete with Grade 3 taking the longest to both understand how to fill up answers in the Answer Sheet provided and go through each question. However, during our Post Tests, our average time to complete the Survey was 15 minutes! It was confidence in their ability to answer the questions as well as their comfort level in the way the Survey was structured.

Vocabulary and word recognition:

In the Pre Test Survey, some of the words in Urdu from our stories were unrecognizable to a lot of the children and a majority would visibly look defeated in trying to guess the correct meaning of the word. However, those same children would excitedly rush to answer that same word in the Post Test Survey. We noticed their eyes light up in recognition of what the word meant and give us and their fellow classmates knowing smiles as they understood how it connected to the story they had become familiar with through our Program.

I could not concretely tell you to what extent their eyes sparkled with excitement and confidence at the sight of a word they could now read and recognize, and I could not give you the percentage of children who were visibly more relaxed and comfortable when taking the survey this time round. However, these subtle changes in their behavior and attitude towards both our survey as well as the recognition that they had truly learnt and expanded their knowledge base through our story read aloud program, was visible, real and so endearing. The excitement was contagious, almost validating our efforts even before the data was collated.

Almost.

We reigned in our excitement and put our logical, M&E hats back on. No assumptions, lets analyze and look at what the data shows.

However, it was clear that the usual ‘subtleties’ behind the numbers did not seem so subtle anymore and it was important for us to try and discuss this expression of impact as well. As we analyze the data, I hope that these nuances in behavior are demonstrated in the quantitative data as well.