Teacher Development Programme In-Service Training Component Impact Evaluation 2014, Baseline Survey

(1) Aim of sampling design

The aim of the sampling design was to define a valid counterfactual 'control' group from which comparisons could be made with a 'treatment' group that participate in the TDP. The control group would not participate in the TDP in-service training but would have background characteristics which are, on average, similar to the treatment group that do participate in TDP in-service training.

The sampling design of the IE was based on a quasi-experimental 'constrained randomisation' approach. 'Constrained randomisation' means that certain parameters of the IE were already fixed - for example, the Local Government Areas (LGAs) where the programme operates. In addition, pre-determined groups of schools fulfilling certain criteria (described below) would constitute the sampling frame - this is in contrast to a fully randomised design approach where one might expect the random drawing of groups (or clusters) of schools from a list of all state primary schools in the region under study.

Randomisation was conducted only in allocating groups of schools to 'treatment' or 'control' status.

The sample design was determined to a large extent by practical programme considerations, and also by the available budget.

(2) Construction of sampling frame: Eligible primary schools

The sampling frame was constructed from scratch through the stages described below. The intended size of the frame was 1008 primary schools eligible for the TDP (504 'treatment' schools and 504 'control' schools) and would constitute the target population (or universe) of eligible schools, from which a sample of treatment and control schools would be drawn for the survey.

Stage 1: Selection of LGAs

In each state, 14 LGAs where the programme would operate had already been pre-determined by the TDP as per arrangements with the States.

· Jigawa: 14 out of 27 LGAs
· Katsina: 14 out of 34 LGAs
· Zamfara: 14 out of 14 LGAs

Stage 2: Selection of sets of primary schools

In each of the 14 LGAs in each state, 2 sets of 12 eligible primary schools each were to be selected;

To be eligible for the TDP: (1) each school should have one head teacher and at least another three teachers; (2) each school should have at least 8 grade-3 pupils.

Schools within each set were identified according to geographical proximity in order to facilitate any training and periodic meetings of teachers within each set, and to create a broader peer network within the locality.

It was the intention that the two sets of schools within each LGA would be selected to be broadly similar. State Education Boards (SUBEBs) were responsible for the selection and were provided with guidelines to assist them, such as taking into account the location of the schools (urban/rural), the size of the schools in terms of classrooms and pupils, presence of a School Based Management Committee (SBMC), and state of school infrastructure. In the case of Jigawa, nearly all schools would have had exposure to the also DFID-funded Education Sector Support Programme in Nigeria (ESSPIN). Therefore, care was taken to balance the level of exposure to ESSPIN across the pairs of sets in each LGA.

Stage 3: Selection of eligible teachers

Before the selection of schools which would participate in the TDP or not, the LGEA and head teacher from each school in every set was required to identify three other teachers who would potentially receive TDP support in addition to him/herself, based on the following criteria:

· Classroom teaching at early grade-level (grades 1-3); and
· Classroom teaching in any of the three subjects: English, maths, or science.

Stage 4: Random allocation of treatment/control sets

After receiving lists of school sets and teachers from the TDP coordinators, the IE team randomly assigned one set of schools among every pair of sets to TDP 'treatment' status using a random number generator. The other set would therefore be assigned 'control' status.

This would result in 14 x 3 = 42 'treatment' sets of 12 schools each (504 'treatment' schools in total) and correspondingly 42 'control' sets of 12 schools each (504 'control' schools in total). In 'treatment' schools, all head teachers and identified teachers in the previous stage would receive TDP support.

(3) Drawing of the samples for the baseline survey

Selection of schools

This was performed in one stage, using implicit stratification by state, LGA and treatment/control status. In other words, each set of 12 schools described above was considered a stratum (42 'treatment' sets and 42 'control' sets).

4 schools were randomly selected from each set.

This yielded an intended sample size of 14 x 4 = 56 treatment schools in each state, and correspondingly 56 control schools in each state.

Thus the total intended sample size across all 3 states was 56 x 3 = 168 treatment schools and correspondingly 168 control schools = grand total of 336 schools.

Selection of teachers

At each sampled school, the head teacher and teachers identified during the construction of the sampling frame would be interviewed. Each teacher and head teachers who teach, would also be observed while they taught a lesson. Following the completion of the school survey, the teachers and head teachers (irrespective of whether they teach or not), would be administered a teacher development needs assessment (TDNA) at an examination centre.

Thus the intended numbers of interviews, lesson observations, and TDNAs were as follows:

· Head teacher interviews: 336
· Head teacher lesson observations: up to 336
· Teacher interviews: 336 x 3 = 1008
· Teacher lesson observations: 336 x 3 = 1008
· TDNAs: 336 x 4 = 1344

Selection of pupils

In order to assess pupil learning levels for this baseline survey, eight of all pupils who started grade 3 in September 2014 and who were being taught English, maths or science by at least one 'selected' teacher during that term, would be randomly selected for a combined English, maths and scientific literacy learning assessment.

The pupils would be drawn from a sampling frame consisting of all eligible grade 3 pupils present in school on the day of the survey recorded by data collectors, using a random number generator programmed into their Computer-Assisted Personal Interviewing (CAPI) software.

Thus the intended pupil sample size was 336 x 8 = 2688

Panel component

It is planned that the same teachers and head teachers, and pupils who were surveyed at the baseline will be surveyed again at the endline in June 2018. This will allow measurement of the impact of the TDP on teacher effectiveness, and on pupil learning between grade 3 and grade 6. While the pupils who had recently started grade 3 at the time of the baseline survey in October 2014 were administered a grade 2 level learning assessment, they will be administered a grade 6 level assessment at the endline in 2018 (when they will be in grade 6) that will include a limited number of grade 2 level items to maintain direct comparability with the baseline.

(4) Non-response and replacement strategies

Schools

5 schools were found to not be eligible and were thus removed from the sampling frame. Reasons included:
· No eligible grade 3 pupils
· No teachers who teach grades 1-3 in English, maths, or science
· School found to be an IQTE school, or special school for children with disabilities

Replacements were made for these schools from the same set (stratum) from which they were drawn, albeit now with fewer than 12 schools.

In other cases, replacements were made for schools:
· Which were closed for the duration of the survey team's stay in the LGA; or
· For which there were security concerns.

Teachers

3 different scenarios for unavailability of 'selected' eligible teachers arose (2 and 3 were due to outdated lists which had been used to make the teacher selection):

(1) A 'selected' eligible teacher was not present on the day of the survey due to short-term absence - data collectors attempted to re-visit the school at a later date.

(2) A sampled school was found to be very small and had fewer than 4 (but at least 1) eligible teacher(s) - all were interviewed (if possible).

(3) A 'selected' eligible teacher was on long-term absence, had been transferred elsewhere, had died or was unidentified. After consultation with the programme it was decided that data collectors would ask the head teacher to name a replacement teacher as per the selection criteria.

Lesson observations and TDNAs

Replacements were not possible.

The number of lesson observations/TDNAs would be reduced according to the number of 'selected' eligible teachers on short-term absence who could not be revisited, or where teacher replacements could not be made.

In addition, head teachers/teachers were not always available to have all three instruments (interview, lesson observation and TDNA) administered to them.

The head teachers and all selected teachers from 7 of the sampled schools did not show up at the examination centres for the TDNA.

Pupils

If a school was found to have only 1-8 eligible grade 3 pupils present on the day of the survey, all were selected for assessment on the day.