SMS Girl Impact Evaluation 2020-2022

Sending Messages about Schooling: Gender Innovation for Re-enrollment and Learning

The baseline survey activity was coordinated by a team of researchers based in Washington DC, while a research firm (RCons) in Lahore, Punjab was contracted to implement the survey. The coordinators of the survey firm had previously been involved in the SABER Service Delivery (SD) survey and were familiar with the details of the survey and had also worked in close coordination with the School Education Department Punjab. The survey firm developed the software for data collection and carried out the enumerator trainings.

RCons (Lahore) carried out the baseline survey activity from 21st August 2020 to 27th October 2020. Data was collected in phases and the second round of the baseline data collection was delayed due to operational issues. The phone-based survey data were collected from a total of 5,898 families, including 5,362 girls’ families and 536 boys’ families. The data was collected across six districts in Punjab, including Attock, Faisalabad, Lahore, Muzaffargarh, Rahimyar, and Sargodha. A team composed of 30 enumerators, 5 quality checkers, and 3 supervisors, collected the phone-based data at 2 calling stations. One team worked exclusively from Faisalabad, and the other team was deployed to Bahawalpur.

Data were collected electronically using a custom-made survey software on android tablets, enabling RCons and the World Bank team to monitor data on a rolling basis. Multiple teams within the World Bank who have worked or are working on Computer-Assisted Telephone Interviewing (CATI) were consulted to develop the baseline survey tool. The questionnaire focused on re-enrolment decisions, child roster, time use of children in the house, parental knowledge about distance learning programs, and other programs that support re-enrollment, household ownership of assets, parental expectations, shocks experienced during COVID, and text literacy of the primary respondent.

Dropped midline (October 2020-January 2021). RCons (Lahore) re-contacted the baseline families by phone (CATI on Android tablets, as in all rounds); 4,215 households completed the interview — 3,872 girls' and 343 boys' families. The round ran in three sub-waves. An initial batch of 1,251 girls' interviews (13-17 October 2020) used version 1 of the questionnaire, whose modules on the government SMS campaign, the Zewar-e-Taleem stipend, and the Taleem Ghar/Tele School remote-learning programs collected real data. The main wave used version 2 — girls 31 October-17 November 2020 (2,621 completed), boys 18-22 November 2020 (343 completed) — which replaced those modules with a roster of the children in the household and covered re-enrollment intentions and status, access to remote learning, and children's time use. A shortened tool then re-contacted the October batch (1,021 of the 1,251) on 4-15 January 2021, during the second school closure, to collect the child roster that version 1 lacked. The data from this round could not be used: it is excluded from the working paper and from the panels. A pilot preceded the round, and a World Bank-contracted field coordinator in Lahore supported training, translation, and quality monitoring throughout — this held for every round.

Round 1 (February-March 2021). Fielded by RCons while schools had temporarily reopened (February 2021); 4,080 girls' and 403 boys' families completed it — 4,483 households in total. The questionnaire measured actual re-enrollment after reopening, school attendance, receipt of government messages, time use (remote learning, other study, tutoring, chores, paid and unpaid work, care work), and parental expectations. Variables from this round carry the suffix _r1.

Round 2 (September-October 2021). Fielded by RCons after schools reopened in phases over August-September 2021 (the instrument asks about re-enrollment since the August reopening); 3,857 households completed it. Two versions of the instrument were prepared, one for children whose schools were open and one for schools still closed. Sections covered consent and call outcomes, re-enrollment decisions and reasons for dropout, take-up of the treatment text and voice messages, child time use, and a child roster. Variables from this round carry the suffix _r2.

Round 3 (December 2021-January 2022). The final round, fielded by RCons like all the others, returned to the girls' sample once schools had permanently reopened. Alongside re-enrollment questions, enumerators administered a short phone-based assessment of Urdu and mathematics to the sampled girls. Data were collected on 2,475 girls; 2,098 completed the full round. Scores are deposited as standardized totals (zurdu_total_r3, zmath_total_r3). Variables from this round carry the suffix _r3.

Caution on variable labels. Variable labels that say "Endline" refer to round 2 (September-October 2021) — for example, reenrolled_r2 is labelled "Re-enrolled at Endline". A few round 3 score labels also carry the word "Endline". The round a variable belongs to is identified by its _r suffix and by the round column of the variable crosswalk, which is authoritative — not by the label text.

More details can be found in the "Nudging at Scale: Evidence from a Government Text Messaging Campaign during School Shutdowns in Punjab, Pakistan" paper, available to download.