Ceara Teacher Feedback Program Impact Evaluation 2015, Endline Survey
Brazil, 2015
Get Microdata
Reference ID
BRA_2015_CTFPIE-EL_v01_M
Producer(s)
Leandro Costa, Barbara Bruns, Nina Cunha
Collection(s)
Metadata
Created on
Apr 27, 2021
Last modified
Apr 27, 2021
Page views
3784
Downloads
743
Identification
Survey ID Number
BRA_2015_CTFPIE-EL_v01_M
Title
Ceara Teacher Feedback Program Impact Evaluation 2015, Endline Survey
Subtitle
Endline Survey
Country/Economy
| Name | Country code |
|---|---|
| Brazil | BRA |
Abstract
In Brazil, policy makers in the state of Ceara are looking at how providing information to schools about best teaching practices, as well as offering peer learning opportunities, can help boost the performance of less effective teachers. The state government wants to stimulate more interaction among teachers at the school level that will lead to faster and cheaper diffusion of good practices within schools. The Secretariat requested World Bank assistance with the design and implementation of the random assignment experiment during the 2015 school year to measure cost-effectiveness of this approach.
The endline data collection was carried out in October - November 2015. Researchers used "Stallings Classroom Snapshot" instrument to gather information about teachers' use of time, materials and interactive pedagogical practices. The observations were made in 3,121 classrooms of 10th, 11th and 12th grade in randomly selected 300 schools.
The schools were randomly assigned to a treatment group (156 schools) or a control group (136 schools). The treatment was launched in March 2015. The treatment group received detailed feedback on the school's results from the classroom observations, information on teacher performance, self-help materials that included a book, videos and exercises about effective teaching strategies, and a log book and classroom observation templates to record teachers' observations of one other. The control schools received neither feedback nor information. After the treatment, researchers assessed the results of student test scores to determine whether the campaign helped improve classroom learning.
The baseline survey, which is also documented and published in the Microdata Library, was conducted in November 2014.
The endline data collection was carried out in October - November 2015. Researchers used "Stallings Classroom Snapshot" instrument to gather information about teachers' use of time, materials and interactive pedagogical practices. The observations were made in 3,121 classrooms of 10th, 11th and 12th grade in randomly selected 300 schools.
The schools were randomly assigned to a treatment group (156 schools) or a control group (136 schools). The treatment was launched in March 2015. The treatment group received detailed feedback on the school's results from the classroom observations, information on teacher performance, self-help materials that included a book, videos and exercises about effective teaching strategies, and a log book and classroom observation templates to record teachers' observations of one other. The control schools received neither feedback nor information. After the treatment, researchers assessed the results of student test scores to determine whether the campaign helped improve classroom learning.
The baseline survey, which is also documented and published in the Microdata Library, was conducted in November 2014.
Kind of Data
Observation data/ratings [obs]
Unit of Analysis
Classrooms; Schools
Version
Version Description
v01 (October 2016)
Scope
Notes
The scope of the study includes:
- Teachers’ use of instructional time,
- Teachers’ use of materials, including information and communications technology,
- Core pedagogical practices,
- Teachers’ ability to keep students engaged,
- Basic characteristics of observed classes,
- Teacher engagement,
- Principals’ and supervisors’ assessments of the value and impact of the treatment
- Teachers’ use of instructional time,
- Teachers’ use of materials, including information and communications technology,
- Core pedagogical practices,
- Teachers’ ability to keep students engaged,
- Basic characteristics of observed classes,
- Teacher engagement,
- Principals’ and supervisors’ assessments of the value and impact of the treatment
Coverage
Geographic Coverage
The state of Ceara
Producers and sponsors
Primary investigators
| Name | Affiliation |
|---|---|
| Leandro Costa | World Bank |
| Barbara Bruns | Center for Global Development |
| Nina Cunha | Stanford University |
Funding Agency/Sponsor
| Name | Abbreviation |
|---|---|
| Strategic Impact Evaluation Fund | SIEF |
Sampling
Sampling Procedure
Ceará has 573 secondary schools that offer the complete three-year cycle. Of these, a sample of 400 schools was stratified by size, geographic area and quartile of learning results. Researchers randomly assigned the 400 schools into 4 groups, with the first 175 assigned to the treatment group, a second group of 25 assigned to a no-observation group, the next 175 schools assigned to the control group, and the last 25 schools also assigned to the no-observation group.
A late start to the baseline round of classroom observations and a limited budget led to a reduction in the sample to 350 schools (175 treatments and 175 controls), which were selected through simple randomization to keep the sample balance. The team did not observe any classroom in the group of 50 schools that were randomly assigned to a no-observation group of the study, but was able to analyze the students' assessments results afterwards.
A late start to the baseline round of classroom observations and a limited budget led to a reduction in the sample to 350 schools (175 treatments and 175 controls), which were selected through simple randomization to keep the sample balance. The team did not observe any classroom in the group of 50 schools that were randomly assigned to a no-observation group of the study, but was able to analyze the students' assessments results afterwards.
Deviations from the Sample Design
Out of the 350 schools of the randomization, with 175 each planned for treatment and controls, 292 schools were observed in November 2014 and in November 2015. The full initial sample could not be observed due to disruptions in the school calendar in November 2014 (standardized tests and holidays) and a shortage of observers in the Fortaleza district. The 292-school final sample included 156 schools in the treatment group and 136 in the control group. This difference in the attrition of treatment and control schools was due to the data collection firm focused their efforts on making up for the schools of the treatment group that would benefit from the classroom observation and the intervention. As a result, because the loss of schools from the treatment and control groups was uneven, the research team conducted a series of balance checks to test the randomization.
In the treatment sample, the 19 schools that were not observed could not receive the information treatment (benchmarked classroom observation feedback for the teachers in their school). But these schools were given access to the other three components of the program – self-help materials, face-to-face training and coaching, and were observed again at endline. The same schools were observed to obtain the endline data. Matched repeat observations were made in 2,399 classrooms, 75% of those observed at baseline. Variations in the school calendar and logistical issues resulted in 25% of the 2015 observations being conducted in grades and subjects in the school that had not been observed at baseline.
In the treatment sample, the 19 schools that were not observed could not receive the information treatment (benchmarked classroom observation feedback for the teachers in their school). But these schools were given access to the other three components of the program – self-help materials, face-to-face training and coaching, and were observed again at endline. The same schools were observed to obtain the endline data. Matched repeat observations were made in 2,399 classrooms, 75% of those observed at baseline. Variations in the school calendar and logistical issues resulted in 25% of the 2015 observations being conducted in grades and subjects in the school that had not been observed at baseline.
Data Collection
Dates of Data Collection
| Start | End | Cycle |
|---|---|---|
| 2015-10-26 | 2015-11-13 | Endline survey |
Data Collection Mode
Face-to-face [f2f]
Supervision
Each team of observers was coordinated by a supervisor with advanced expertise in the Stallings method. Supervisors conducted at least two observations side by side with each observer to check consistency, and reviewed the coding sheets submitted by observers for inconsistencies. In the case of major inconsistencies, supervisors were responsible for making a repeat visit to the school to conduct new observations.
Data Collection Notes
The Stallings Classroom Snapshot instrument, technically called the Stanford Research Institute Classroom Observation System was developed by Professor Jane Stallings for research on the efficiency and quality of basic education teachers in the United States in the 1970s. The Stallings instrument generates robust quantitative data on the interaction of teachers and students in the classroom, with a high degree of inter-rater reliability (0.8 or higher) among observers with relatively limited training, which makes it suitable for large scale samples in developing country settings. The instrument is language and curriculum-neutral, so results are directly comparable across different types of schools and country contexts, and a growing body of comparative country data from the US and developing countries is available.
The Stallings instrument generates quantitative measures - at the classroom, school, and school system level - of four main variables:
- Teachers’ use of instructional time
- Teachers’ use of materials, including information and communications technology
- Core pedagogical practices
- Teachers’ ability to keep students engaged
All observers were pedagogical coordinators from treatment schools to avoid any contamination of control schools from having someone at the school familiar with the Stallings observation methods and/or the training program.
The Stallings instrument generates quantitative measures - at the classroom, school, and school system level - of four main variables:
- Teachers’ use of instructional time
- Teachers’ use of materials, including information and communications technology
- Core pedagogical practices
- Teachers’ ability to keep students engaged
All observers were pedagogical coordinators from treatment schools to avoid any contamination of control schools from having someone at the school familiar with the Stallings observation methods and/or the training program.
Questionnaires
Questionnaires
1) The Stallings Classroom Snapshot Coding Sheet. The classroom snapshot records the participants, their activities, and the materials being used in the classroom, at ten separate instances throughout a class period.
2) School Principals Questionnaire. The questionnaire gathers information about teachers' engagement in the school, level of teacher training activities, and Principals' and supervisors' assessments of the value and impact of the treatment.
3) Classroom Demographic Sheet. The instrument is used for identification of the school and classroom, and the basic information related to the observed classroom, such as the number of student and the start time of the class.
2) School Principals Questionnaire. The questionnaire gathers information about teachers' engagement in the school, level of teacher training activities, and Principals' and supervisors' assessments of the value and impact of the treatment.
3) Classroom Demographic Sheet. The instrument is used for identification of the school and classroom, and the basic information related to the observed classroom, such as the number of student and the start time of the class.
Access policy
Citation requirements
Use of the dataset must be acknowledged using a citation which would include:
- the Identification of the Primary Investigator
- the title of the survey (including country, acronym and year of implementation)
- the survey reference number
- the source and date of download
Example:
Leandro Costa, World Bank; Barbara Bruns, Center for Global Development; Nina Cunha, Stanford University. Brazil Ceara Teacher Feedback Program Impact Evaluation 2015, Endline Survey. BRA_2015_CTFPIE-EL_v01_M. Dataset downloaded from [URL] on [date].
- the Identification of the Primary Investigator
- the title of the survey (including country, acronym and year of implementation)
- the survey reference number
- the source and date of download
Example:
Leandro Costa, World Bank; Barbara Bruns, Center for Global Development; Nina Cunha, Stanford University. Brazil Ceara Teacher Feedback Program Impact Evaluation 2015, Endline Survey. BRA_2015_CTFPIE-EL_v01_M. Dataset downloaded from [URL] on [date].
Disclaimer and copyrights
Disclaimer
The user of the data acknowledges that the original collector of the data, the authorized distributor of the data, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses.
Metadata production
DDI Document ID
DDI_BRA_2015_CTFPIE-EL_v01_M
Producers
| Name | Abbreviation | Affiliation | Role |
|---|---|---|---|
| Education Global Practice LCR | GED04 - EDU GP LCR | World Bank | Submitted metadata, documents and the dataset through WB Microdata Library Data Deposit |
| Development Data Group | DECDG | World Bank | Study documentation |
Date of Metadata Production
2016-10-17
DDI Document version
v01 (October 2016)