Developing a Digital Assessment Tool for Low-Resource Settings

Jigsaw is a London-based research organisation that exists to build evidence for education in low- and middle-income contexts.

In May 2023, the Jigsaw team was asked to help determine the future of Gobee: a digital assessment tool funded by Porticus and developed by War Child Holland and NYU Global TIES, designed to assess the holistic learning outcomes of primary-age students in emergency contexts. Originally developed in response to the need for remote learning assessment during the COVID-19 pandemic, Gobee’s development process had raised some tricky questions. To address these issues, a decision was made to pause development and create a detailed learning agenda.

Our task was to address three key questions relating to the learning agenda:

1. What are the current needs of the education in emergencies (EiE) and wider development sector with regard to assessment?
2. What is Gobee’s potential added value?
3. What are the credible options for sustaining and scaling Gobee?

In this blog post, we highlight how we went about answering these questions, what we discovered and what we recommended based on these findings.

Methods

To respond to the research questions, the Jigsaw team carried out the following activities:

• 18 interviews with educational app developers, EiE and development sector actors, and EdTech funders
• Three focus groups with Gobee development team members
• An analysis of other assessment tools currently being used in the sector
• A review of current assessment practices using Lebanon and Uganda as case studies
• A review of funder profiles

Key findings

What are the current sector needs with regards to assessment?

The findings revealed some significant needs that are not yet being fully addressed within the EiE and wider development sector:

• Tools that measure socio-emotional skills, particularly across multiple contexts
• Tools that measure the learning progress of crisis-affected populations
• Tools that give teachers the information they need to better respond to their students’ learning needs, and which support them to interpret student data
• Low-cost, standardised tools that can gather detailed student learning data for use by NGO or government monitoring and evaluation teams at distance

Cost was identified across the data as a major barrier to the development of these tools. There was also a concern that teachers in emergency contexts may not have the general teacher training, digital literacy or motivation to adopt these tools successfully.

What is Gobee’s potential added value?

According to interview data and our review of comparable tools, eventually combining several of the following features within its offering, will set Gobee apart from other assessment tools used in the sector:

1. Generating granular literacy and numeracy outcomes data
2. Helping teachers to interpret the data as well as just displaying it (e.g. by suggesting actions that teachers can take based on the data)
3. Measuring socio-emotional skills in some form, while also measuring progress in numeracy and literacy
4. Including gamified elements
5. Providing feedback to students themselves
6. Including a facility through which to build bespoke assessments
7. Including strong accessibility features such as text-to-speech and screen magnification
8. Making Gobee’s codes open source

In order to compete with existing offerings, the findings indicate that Gobee must as a minimum:

1. Be able to operate (at least partially) offline and via inexpensive devices
2. Be free to access for the end user
3. Be governed by a robust data protection policy such as GDPR

What are the credible options for sustaining and scaling Gobee?

In terms of ownership, our data indicated a clear need for Gobee to be embedded in a respected organisation, from which point it could be advocated for within the implementation context. There was strong support for War Child to take onward ownership, though interviewees were keen to emphasise the considerable time and resources required to sustain and scale an EdTech tool, so organisational feasibility would have to be considered carefully.

Regarding where and how Gobee could be implemented, several interviewees recommended a short-term focus on implementation in a single, familiar context, where the development team already has presence and contextual knowledge. Within this single context, interviewees considered embedding Gobee within an existing intervention a crucial step. From this point, interviewees suggested that Gobee could be showcased to other NGOs, whose demand for the tool would then convince funders and governments to back it.

In terms of funding, interviews and funder profiles suggested that evidence of impact and sector demand are key prerequisites for supporting a tool. Funders also noted that they were unlikely to fund a tool that is not associated with a wider programme, or one that does not open source codes.  Beyond this, funders noted that high contextual relevance, strong government alignment, a focus on enhancing equity and systems change, and robust evidence of cost-effectiveness were key funding criteria.

Recommendations

Informed by the findings, we recommended that development of the Gobee tool should continue; there are clear gaps that a digital assessment tool could help to address. While acknowledging the high costs associated with EdTech development and the contextual challenges associated with low-resource and emergency settings, our data also led us to make some other specific recommendations:

1. A champion should be identified to ensure maximum uptake: While we recommend that War Child should continue to own and champion the tool, if this is not feasible, then a ‘home’ for Gobee should be found in a parallel implementing organisation. Gobee’s codes should also be made open source in either scenario. However, Gobee should not simply be left without a champion or ‘owner’ in the hope that someone will eventually pick it up from open source developer platforms such as Github; this is unlikely to result in uptake.
2. Gobee development should focus on teachers as its main user group, with the aim of both providing teachers with data and guiding them through how to interpret and apply that data to their practice. We acknowledge that this would also require investment in teacher professional development to ensure that teachers have the skills and confidence to implement the tool successfully.
3. Gobee should maintain its focus on assessing both foundational literacy and numeracy and socio-emotional skills. Gobee currently can only assess numeracy and two socio-emotional competencies (self-efficacy and growth mindset). We recommend that these be finalised and tested in the first instance so as to not try to do all things at once. However, eventually Gobee should add literacy assessments and build out a more comprehensive socio-emotional skills offering.
4. Gobee should be tested and piloted further in Jordan in the immediate future before considering implementation in any other context. Linked to 2 above, this recommendation is strengthened by NYU Global TIES’ recent research into formative assessment practices, which indicates a strong interest in this kind of assessment by teachers in Jordan. Therefore, teachers may be poised to adopt a tool that helps them to assess students more easily and at scale.
5. Gobee should be embedded in a more comprehensive education programme in Jordan rather than trying to fundraise for it as a standalone tool. Embedding the tool in a thoughtful and holistic programme would enhance the holistic learning outcomes for children and facilitate uptake by other implementing agencies and the government.  Additionally, funding could be obtained more easily if Gobee is embedded in education projects in Jordan, rather than as a standalone assessment tool.
6. Evidence should be built throughout the development and deployment process for Gobee. Efficacy research as well as cost effectiveness assessments should be carried out from the start. Both are essential to secure further funding and buy-in from other implementers and governments. Advice on cost capture and outcomes data harmonisation are provided by organisations such as the EdTech Hub, USAID, and Brookings.

Reflections on the Gobee experience

It has been an enormous pleasure to contribute to the Gobee Learning Agenda. We applaud the impressive dedication of all involved in Gobee’s development, especially given the challenges of working in this space. The agenda has highlighted the importance of being willing to pause to critically evaluate the direction of a development process and ensure that the end product stands the best chance of having a truly meaningful impact. We will be following the next steps with great interest and hope to continue to support the process in the future!