Education Quality Improvement Programme Impact Evaluation Baseline Survey 2014-2015

The EQUIP-T IE administered eight different questionnaires in the sampled primary schools using Computer-Assisted Personal Interviewing (CAPI) except for the teacher development needs assessment (TDNA) questionnaires, which were administered on paper as these take the form of mock pupil tests in Kiswahili and mathematics.

List of instruments

-HEAD TEACHER INTERVIEW (EQUIP-T IE head teacher (HT) questionnaire): Head teacher background information, qualifications, frequency/type of school planning/management in-service training received, availability of different types of school records, pupil enrolment, school start and closing time, number of streams, use of shifting, school timetable, availability and contents of whole school development plan, school committee, parent-teacher association, existence and types of teacher performance rewards, financing, frequency of staff meetings, district and ward supervision and support to the school, head teacher motivation, school infrastructure, teacher sampling and pupil sampling.

-HEAD COUNT (EQUIP-T IE head count (HC) questionnaire): Teacher attendance by headcount on the day of the survey, teacher punctuality and pupil attendance (by headcount on the day of the survey), and school infrastructure.

-SCHOOL RECORDS (EQUIP-T IE school records (SR) questionnaire): Teacher attendance from school records, pupil attendance and enrolment, school open days.

-PUPIL BACKGROUND (EQUIP-T IE pupil background and learning assessment (PB) questionnaire): Pupil background information, Kiswahili Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) based on standards 1 and 2 national curriculum requirements. Note: The same pupils were assessed in both Kiswahili and mathematics.

-POVERTY SCORECARD (EQUIP-T IE poverty scorecard (PS) questionnaire): Information on household characteristics, household assets, mother's literacy and writing. Note: The respondent was a sampled pupil's parent(s) but if not present the pupil was administered this instrument instead.

-TEACHERS WHO TEACH STANDARD 1-3 KISWAHILI AND/OR MATHEMATICS INTERVIEW (EQUIP-T IE teacher interview (TI) questionnaire): Teacher background information, qualifications, frequency/type of in-service training received, frequency/nature of lesson observation and nature of feedback, frequency/nature of performance appraisal and teacher motivation.

-TEACHERS WHO TEACH STANDARD 1-3 KISWAHILI (EQUIP-T IE teacher development needs assessment (TDNA) questionnaire): Kiswahili subject knowledge assessment (teacher development needs assessment) based on the primary school Kiswahili curriculum standards 1 to 7 but with limited materials from standards 1 and 2.

-TEACHERS WHO TEACH STANDARD 1-3 MATHEMATICS (EQUIP-T IE teacher development needs assessment (TDNA) questionnaire): Mathematics subject knowledge assessment (teacher development needs assessment) based on the primary school mathematics curriculum standards 1-7 but with limited materials from standards 1 and 2.

-TEACHERS WHO TEACH STANDARD 4-7 MATHEMATICS (EQUIP-T IE teacher development needs assessment (TDNA) questionnaire): Mathematics subject knowledge assessment (teacher development needs assessment) based on the primary school mathematics curriculum standards 1-7 but with limited materials from standards 1 and 2.

-LESSON OBSERVATION (EQUIP-T IE lesson observation (LO) questionnaire): Standard 2 Kiswahili and mathematics lesson observations of inclusive behaviour of teachers with respect to pupil gender, key teacher behaviours in the classroom, availability of lesson plan and availability of seating, textbooks, exercise books and pens/pencils during the lesson.

Development of the survey questionnaires

To develop the quantitative survey questionnaires the OPM impact evaluation team collected and reviewed relevant existing survey questionnaires, discussed with national and international experts the different types of questionnaires, visited schools during the November 2013 and January 2014 missions, worked with a national team of experts to develop the pupil assessment and the teacher development needs assessments (TDNA), and conducted three pre-tests of the questionnaires (for more details see section 3.3.2 in OPM. 2015. EQUIP-Tanzania Impact Evaluation. Final Baseline Technical Report, Volume II: Methods and Technical Annexes). In regard to the development of the questionnaires it should be noted that the Reference Group for the EQUIP-T IE was provided with penultimate drafts of the survey questionnaires and its comments guided the questionnaire revisions and finalisation.

Pupil background and learning assessment

The IE administered an early grade reading assessment (EGRA) and an early grade mathematics assessment (EGMA) style questionnaire developed in Tanzania specifically for the IE to standard 3 pupils. These questionnaires capture core standard 1 and 2 curriculum skills in Kiswahili literacy: reading, reading comprehension and listening comprehension and writing, and in mathematics. The learning assessment questionnaires developed for the IE will be used to test standard 3 pupils in 2016 and 2018 as well.

---Testing the same pupils in both subjects---

The IE initially considered assessing different pupils in the two subjects: Kiswahili and mathematics, to avoid pupil fatigue and to ensure that the maximum assessment administration time per pupil would be 45 minutes given the young age of the pupils. However based on feedback from the Reference Group for the Impact Evaluation and pre-test 1 and pre-test 2 findings, the IE design was revised to test the same standard 3 pupil in both subjects while ensuring a feasible test administration time.

---Item difficulty---

Given the relatively young age of the pupils and type of learning assessment, the IE administered the pupil learning assessments one-on-one (one enumerator testing one pupil at a time). Following the learning assessment development in late December 2013, the draft (paper) pupil assessment instruments were pre-tested to ensure that each item was testing the appropriate curriculum level (standards 1 and 2) and to assess whether the difficulty range of items allows for discrimination for a range of pupil performance. For the latter aspect, the pre-test schools were purposively selected to include both high and low performing schools with respect to education performance. After the first pre-test, analysis was conducted to determine item difficulty, functioning and reliability, and some items were discarded based on this analysis. These revised pupil learning assessments were then pre-tested again (using CAPI) and further revised, and were pre-tested a third time and revised further during the enumerator training and pilot, before finalisation.

Poverty scorecard

To collect household characteristics and data to be used to construct poverty measures the OPM impact evaluation team considered two approaches. First, an asset approach where pupils are asked about their home, for example, "Where do you normally get your water from at home?" and responses are used to construct an asset index. Second, a poverty scorecard approach where parents are asked to come to the school and answer questions about household characteristics such as "How many tables does your household own?" (proposed by the Reference Group for the Impact Evaluation). Both approaches were pre-tested by the OPM team and showed discrepancies between the answers provided by pupils and their parents to the same questions. This result combined with the fact that a poverty scorecard with a mapping to poverty likelihoods already existed for Tanzania led the OPM impact evaluation team to select the poverty scorecard approach with parents as the respondents.

Teacher development needs assessment (TDNA)

The teacher development needs assessment (TDNA) measures teachers' primary school subject knowledge in Kiswahili and mathematics. Investigation of suitable questionnaires was guided in part by the possibility that teacher could perceive an assessment covering the pupil curriculum as threatening and partly by the fact that assessments measuring teacher subject knowledge based on the pupil curriculum has not been done in Tanzania in the past. Accordingly, the OPM impact evaluation team decided to use a format that is an anonymous, non-threatening mock pupil test that teachers mark. This type of questionnaire has been used successfully in other countries, for instance Nigeria. The IE baseline TDNA questionnaire is based on the format used by a TDNA developed by Dr David Johnson at Oxford University for DFID-Nigeria. The OPM impact evaluation team worked with the same national team that developed the pupil learning assessments to develop the TDNA. This work was guided by the Test Specification Framework designed by the OPM impact evaluation team and the questionnaires were revised based on the pre-test findings.

---Groups of teachers selected for the TDNA---

The TDNA questionnaires developed for the IE are used to assess subject knowledge of teachers who teach Kiswahili and mathematics to standard 1-3 and of teachers who teach mathematics to standard 4-7. The reason for assessing these groups of teachers that the EQUIP-T programme will provide: Kiswahili in-service teacher training (INSET) for standard 1-3 curriculum levels to standard 1-3 teachers (programme phase 1); mathematics INSET for standard 1-3 curriculum levels to standard 1-3 teachers (programme phase 1); and mathematics INSET for standard 4-7 curriculum levels to standard 4-7 teachers (programme phase 2).

---Item difficulty---

The TDNA questionnaires were pre-tested to ensure that they cover the whole primary school curriculum (only a minimum number of items for the standards 1 and 2 curriculum levels were included as these items are very basic), and that the difficulty range of items allows for discrimination for a range of teacher performance to avoid potential ceiling and floor effects. After the first pre-test, analysis was conducted to analyse item difficulty, functioning and reliability, and some items were replaced based on the analysis. The revised TDNA questionnaires were pre-tested a second time using CAPI, and then further revised, and were then pre-tested a third time, and then further revised during the enumerator training, before finalisation.

Lesson observation

To develop the questionnaire the OPM impact evaluation team reviewed existing lesson observation questionnaires including the work done in Tanzania by UNICEF. There are three choices which have been shown to work well in the Tanzanian context: (i) frequency observation; (ii) timeline analysis; (iii) computerised observation schedule (gives frequency and duration information). Based on observations from the school visits during the November 2013 scoping mission and pre-test 1, the OPM impact evaluation team chose the frequency observation option for the lesson observation questionnaire.

Pre-tests

The first pre-test (January 2014) tested the initial set of draft questionnaires before the start of programming these into CAPI, and took place in eight primary schools in two districts. This pre-test was conducted by a national team consisting of Kiswahili and mathematics specialists from the University of Dar es Salaam, primary school teachers and an expert who had previously developed pupil learning questionnaires for the UWEZO learning assessment and Kiufunza project in Tanzania, accompanied by two OPM impact evaluation team members. The questionnaires were revised based on the pre-test 1 findings. The aims of this pre-test were to:

-Assess whether the difficulty range of the items in the draft pupil learning assessment corresponds to the Tanzania primary 1 and 2 curriculum levels for Kiswahili and mathematics;
-Test whether the draft pupil learning assessment items discriminate adequately among pupils;
-Assess whether the difficulty range of the draft items in the draft TDNAs corresponds to the Tanzania primary school curriculum for Kiswahili and mathematics;
-Test whether the draft TDNA items discriminate adequately among teachers;
-Test appropriate time of administration for the pupil learning assessments and TDNAs; and
-Test accuracy of all questionnaires in terms of precision and clarity of wording, translation, answer categories etc.

The second pre-test (February 2014) was conducted in four schools in one region by two teams consisting of local enumerators, OPM impact evaluation team members and OPM Tanzania survey team members. The aims of this pre-test were to:

-Further test accuracy of the questionnaires in terms of precision and clarity of wording, translation, answer categories etc.;
-Test technical aspects of administering the questionnaires using CAPI;
-Test questionnaires and procedure from an education perspective;
-Test and improve fieldwork protocols and logistics;
-Learn about the administration time of the questionnaires for fieldwork planning purposes:
-Assess adequacy of materials and equipment;
-Learn about potential challenges in the field;
-Gather field experience to improve training delivery;
-Further test appropriate time of administration for the pupil learning assessments; and
-Test whether it is appropriate to administer both assessments (Kiswahili and maths) to the same pupil.

The third and final pre-test was conducted in March 3-5, 2014 by a team of OPM impact evaluation team members, OPM Tanzania survey team members and local enumerators to test the revised CAPI software, gain further insight for the enumerator training and collect additional information to inform the baseline fieldwork planning.

The questionnaires were further revised, in particular translations, during the enumerator training in March, 2014.

The complete questionnaires are provided under External Resources except for the pupil learning assessment and TDNA instruments for which the assessment items (questions) have been removed as these are confidential until the completion of the EQUIP-T Impact Evaluation Baseline Survey in 2018. However, a summary note describing the learning domains assessed by these two instruments is provided under External Resources.