Teacher Development Programme In-Service Training Impact Evaluation 2017, Endline Survey

The aim of the sampling design was to define a valid counterfactual (control group) from which comparisons could be made with the treatment group that participates in the Teacher Development Programme (TDP). The control group does not participate in the TDP in-service training but has background characteristics that are, on average, similar to those of the treatment group that does participate in TDP in-service training. The sampling design was based on a quasi-experimental 'constrained' randomisation approach. 'Constrained' is used because certain parameters of the impact evaluation were already fixed. For example, the Local Government Authorities (LGAs) where the TDP was to operate had already been selected by the TDP in agreement with the three states covered by the impact evaluation. In addition, pre-determined groups of schools fulfilling certain criteria (described below) constitute the sampling frame -- this is in contrast to a fully randomised design where one would expect the random drawing of groups (or clusters) of schools from a list of all state primary schools in the region under study. Randomisation was conducted only in allocating groups of schools to 'treatment' or 'control' status.

Sampling frame construction

The intended size of the sampling frame was 1,008 public primary schools eligible for the TDP (504 treatment and 504 control schools) in the three states. This would constitute the target population of eligible schools from which the sample of schools would be drawn for the survey. The sampling frame was constructed through the steps described below.

Step 1: LGAs

In each of the three states, 14 LGAs where the TDP would operate had been predetermined by the TDP in agreement with each state:
-Jigawa: 14 out of 27 LGAs.
-Katsina: 14 out of 34 LGAs.
-Zamfara: 14 out of 14 LGAs.

Step 2: Primary schools

To be eligible for the TDP, a school must have one head teacher and at least three other teachers and at least eight grade 3 pupils. In each of the 14 LGAs in each state, two sets of 12 eligible primary schools were to be selected. Schools within each set were identified according to geographical proximity to facilitate any training and periodic meetings of teachers, and to create a peer network within the locality. The two sets of schools within each LGA were meant to be seleted to be broadly similar. State Education Boards (SUBEBs) were responsible for the selection of the schools and were provided guidelines for how to do this. For example, to take into account the location of schools (rural/urban), school size in terms of pupils enrolled and number of classrooms, condition of school infrastructure, and existence of a school-based management committee (SBMC).

Step 3: Teachers

Before the sampling of schools, the Local Government Education Authority (LGEA) and head teacher from each school in the two sets (see 2. above), were required to identify three teachers in addition to the head teacher, who would potentially receive the TDP support and training. These teachers were identified using the following criteria:
-They teach classes at early grade level (grades 1 to 3); and
-They teach classes in any of the three subjecs of English, mathematics, and science.

Step 4: Random assignment of treatment/control status

After receiving the lists of the sets of eligible schools and teachers from the TDP coordinators, the impact evaluation team randomly assigned one set of schools among every pair of sets in each LGA to TDP treatment, and the other set to control status. This resulted in 42 (14 LGAs x 3 states) sets consisting of 12 schools each, for a total of 504 schools to receive the TDP, and 42 (14 LGAs x 3 states) sets consisting of 12 schools each for a total of 504 schools that would not receive the TDP. The sample treatment and control schools were then selected from these two lists respectively. In the 504 schools that would receive the TDP, all head teachers and the teachers identified in step 3 (see above) would receive TDP support and training, while the head teachers and teachers in the schools on the list of 504 'non-TDP' schools would not.

Sampling stages

Stage 1: Selection of schools

At the first stage, schools were selected using implicit stratification by state, LGA, and treatment/control status. That is, each of the school sets (see step 4 above), was considered a stratum. Four schools were randomly selected from each of these sets. This yielded an intended sample size of 56 (14 LGAs x 4 schools) treatment schools in each state and 56 (14 LGAs x 4 schools) control schools in each state, for an intended 168 treatment and 168 control schools across the three states, and a total intended sample size of 336 schools.

Stage 2a: Selection of teachers

At each selected school:
-The head teacher and the three teachers identified during the construction of the sampling frame (see step 3 above) constituted the sample to be interviewed, with a total intended sample size of 336 head teacher interviews and 1,008 (336 schools x 3 teachers) teacher interviews.
-The three selected teachers as well as the head teachers who teach any primary classes would also be observed while teaching (classroom observations), with a total intended sample size of up to 336 head teacher classroom observations and 1,008 (336 schools x 3 teachers) teacher classroom observations.
-All head teachers and selected teachers would also be adminstered the Teacher Development Needs Assessment (TDNA) in English, mathematics, and science, for a total intended sample size of 1,344 (336 schools x 4 teachers) head teachers and teachers.

Stage 2b: Selection of pupils

At each selected school, eight pupils who started grade 3 in September 2014 and who were being taught English, mathematics, or science by at least one of the selected teachers during that term, would be randomly selected for the pupil learning assessment. The pupils were drawn from a sampling frame consisting of all eligible grade 3 pupils present on the day of the survey. Eligible pupils were those in grade 3 who were being taught by at least one of the selected teachers. The intended pupil sample size was 2,688 (336 schools x 8 pupils).

Survey longitudinal/panel design

The survey uses a panel design, that is, it aims to collect longitudinal data. This means the survey was implemented in the same sample of schools at baseline and endline, and that within these schools, it collected data from the same head teachers, teachers, and pupils over time. However, there was high head teacher, teacher, and pupil sample attrition between the baseline and endline surveys (see 'Sample attrition' section below).

Replacements

1. School replacements

   During the baseline fieldwork, five selected schools were found to be ineligible and one selected school could not be visited because of security concerns, these six schools were not replaced.

During the endline survey there were no school replacements because the survey uses a panel design.

2. Teacher replacements

   There were three different cases of unavailability of selected teachers during the baseline fieldwork:
   -A selected teacher was not present on the day of the survey due to short term absence, and data collectors attempted to re-visit the school at a later date;
   -A selected school was found to be very small with fewer than four eligible teachers (including the head teacher), all teachers were interviewed when possible; and
   -A selected teacher was on long leave, had been transferred, had passed away, or was unidentified, and data collectors would ask the head teacher to name a replacement teacher as per the teacher eligibility criteria (see step 3 above).

During the endline fieldwork there were no teacher replacements because the survey uses a panel design.

3. Pupil replacements

   If one of the eight sampled pupils turned out to not be available, that pupil was randomly replaced with another pupil from the list of eligible pupils. If a selected school had eight or fewer grade 3 pupils present on the day of the baseline survey, all those eligible were selected for the pupil learning assessment.

During the endline fieldwork there were no pupil replacements because the survey uses a panel design.

Sample sizes

The actual sample sizes at baseline were:
-330 schools (intended 336);
-330 head teacher interviews (intended 336);
-2,575 grade 3 pupils (intended 2,688);
-908 teacher interviews (intended 1,008);
-1,070 classroom observations (intended up to 1,344 depending on how many head teachers teach any primary classes); and
-1,158 Teacher Development Needs Assessments (intended 1,344).

The actual sample sizes at endline were:
-330 schools (intended 330 in actual BL sample);
-329 non-panel (new head teachers were interviewed) and 134 panel head teacher interviews (intended 330 in actual BL sample);
-1,566 panel grade 6 pupils (intended 2,575 in actual BL sample);
-447 panel teacher interviews (intended 908 in actual BL sample);
-460 panel and 574 non-panel classroom observations (intended 1,070 in actual BL sample); and
-556 panel and 774 non-panel Teacher Development Needs Assessments (intended 1,158 in actual BL sample).

Sample attrition

There was substantial head teacher, teacher, and pupil sample attrition between baseline and endline. The main reasons for head teacher and teacher attrition were transfer to another school and for pupil attrition dropping out of school.

Sample attrition rates since baseline:
-Schools 0%;
-Head teacher interviews 0.3% non-panel sample and 59% panel sample;
-Pupils 39% (in grade 3 at baseline and in grade 6 at endline);
-Teacher interviews 51%;
-Classroom observations 57%; and
-Teacher Development Needs Assessments 52%.

NOTE: Head teachers that were new at a sample school were interviewed at endline yielding a sample of 329 head teachers (some non-panel and some panel), and a panel sample of 134 head teachers (only head teachers who were head teachers at the same school at baseline and endline).

To reduce teacher and pupil attrition due to temporary absences on the day of the survey, two main steps were taken:
-School revisits: state coordinators and field teams revisited schools in order to conduct missing interviews for teachers and pupils that were unavailable on the original day of visit; and
-Calling pupils from home: for the pupils that lived close to the school but were absent from school on the day of the visit, data collectors worked with the head teachers and teachers to ask pupils to come to the school to take the pupil learning assessment if able to do so.

1. Overall pupil and teacher sample attrition

For the purposes of this impact evaluation, attrition for two units of analysis were of particular relevance: pupils and teachers. To understand attrition dynamics for those units of analysis in more detail, this evaluation assessed which characteristics of pupils and teachers at baseline help explain whether individuals dropped out of the sample or not since baseline. This means that baseline data were used to compare the group of individuals for which data were also collected at endline (the non-attriters) to the group of individuals for which data were not collected at endline (the attriters). The purpose of this analysis is to understand whether estimates of characteristics of pupils or teachers at endline can generally be thought of as being produced on a sample that is comparable to the original group of individuals sampled at baseline. The attrition analysis and statistics are produced taking into account the full sampling structure of the data (weights, clustering, stratification). Test statistics are also corrected fur multiple hypotheses testing given that these tables are comparing many different indicators at the same time.

The baseline variables used to examine sample attrition are of three types:
-Pupil and teacher background characteristics, for example, gender;
-Pupil and teacher outcomes that the TDP seeks to influence, for example, pupil test scores and teacher subject knowledge scores; and
-School characteristics that are likely to be correlated with pupil and teacher behaviour and outcomes, for example, school size.

The results for pupils indicate that pupils who drop out of the sample are mainly older and poorer than pupils who stay in the sample. The results for teachers indicate that teachers who dropped out of the sample since baseline were significantly more likely to have Nigeria Certificate in Education (NCE) qualification or higher, and performed significantly better on the Teacher Development Needs Assessment (TDNA) than teachers who remained in the sample. These results indicate that the evidence for selective attrition is weak among pupils but slightly stronger for teachers. This means that taking overall attrition into account, the sample of pupils is still comparable to the original sample. While estimates generated using the teacher sample needs to be interpreted taking teacher attrition into account, and cannot necessarily be assumed to be representative of the target population of teachers at baseline.

2. Differential sample attrition for treatment and control pupils and teachers

This impact evaluation also examined whether there is differential attrition, which refers to situations where the background characteristics of individuals who drop out between survey rounds differ significantly between the treatment and control groups. This would mean that, after the sample attrition, the two groups are not comparable anymore and that the original assumption of the control group being an appropriate counterfactual to the treatment group for impact identification purposes is no longer correct. The analysis examining key characteristics and outcomes for non-attrited pupils and teachers at baseline across the TDP treatment and control groups indicates that differential attrition is generally not problematic. The estimates suggest that there are no significant differences between the control and treatment pupils who remained in the sample. For teachers there is also very limited indication of differential attrition.

Representativeness

The sample for this impact evaluation was selected to represent the eligible schools in the TDP clusters and corresponding control clusters for the three states. Therefore the sample is not designed to be representative at the state level and should not be treated as such. However, the survey does include schools with a wide range of characteristics, and statistics are likely to be broadly similar to those for the states as a whole.