Health Results Based Financing Impact Evaluation 2014
Health Facility Baseline Survey
Impact Evaluation Survey
The Health Results Innovation Trust Fund (HRITF) launched in 2007. It supports results-based financing (RBF) approaches in the health sector to improve maternal and child health around the world. The HRITF is supported by the Governments of Norway through Norad and the United Kingdom through the Department for International Development (DFID). It is administered by The World Bank.
As of September 2016, the total contributions to the trust fund were US$477.6 million equivalent from Norway and the United Kingdom. To date, HRITF has committed US$385.6 million for 35 RBF programs in <a href="http://www.rbfhealth.org//projects">29 countries</a>, linked to US$2.0 billion in financing from IDA-the World Bank's fund for the poorest. In addition to the RBF programs, the trust fund also finances the evaluation of these programs and other supporting activities. Funded by the HRITF, the Tajikistan Health Results Based Financing Impact Evaluation Baseline Survey was conducted in 2014-2015.
The baseline survey was implemented prior to the implementation of PBF in the 7 study treatment districts and a follow-up survey (endline) is planned to take place after three years of project implementation. The survey is largely based on the HRITF instruments that were modified to the Tajik and project context.
The policy objective of the Impact Evaluation (IE) is to build evidence on the impact and cost-effectiveness of the proposed Performance-Based-Financing (PBF) project in Tajikistan. More specifically, the IE would seek to ascertain: (i) the impact and cost-effectiveness of the PBF model implemented in Tajikistan; and (ii) whether PBF is more effective or cost-effective if implemented in conjunction with additional low cost interventions (Collaborative Quality Improvement, Citizen Report Cards). The results from the IE will help informing the MOH on whether PBF should be scaled-up to additional PHC level institutions in other regions.
The Collaborative Quality Improvement intervention responds to policy concerns that performance incentives may not produce the desired improvements if providers lack the necessary competencies, data to inform decisions and knowledge. The Citizen Report Card attempts to improve the effectiveness of PBF by strengthening the 'short route of accountability', i.e., by increasing accountability of health facilities to their local constituents. Since PBF, collaborative quality improvement (CQI), and citizen report cards (CRC) have never been implemented in large scale in Tajikistan, it is to be expected that the results from the IE will be useful for designing national PHC policy in Tajikistan, and that they will also contribute to the larger body of knowledge on these interventions.
The IE employs both difference-in-difference and experimental approaches to identify the impact of the different combinations of interventions. Assignment to PBF was not random. Three districts in the Sughd region and 4 districts in the Khatlon region were selected to implement the program. All Rural Health Centers (RHCs) in these seven districts are covered by the program. Nine additional district (two in Sughd and seven in Khatlon) were selected as control districts. The selection of the control districts was guided by geographical proximity to treatment districts and similarity in terms of number of health facilities and doctors per capita. The districts were also selected such that the number of RHCs in treatment and control groups in each region would be similar.
Within the chosen 16 districts (treatment and control districts), clusters consisting of a RHC and its subsidiary Health Houses were randomly assigned to implement Collaborative Quality Improvement, Citizen Score Cards, or neither of these two interventions. The randomization was blocked by district. In sum, RHCs were assigned into six study arms.
The goal of the Facility-based survey is to measure multiple dimensions of quality of care and collect detailed information on key aspects of facility functioning. Household surveys are primarily used to measure health service coverage at the population level as well as select health outcome indicators measured through anthropometry or tests. The surveys also collect broader data on the health of the households, health seeking behaviors and barriers to use of health services. In addition, PBF and other administrative data would be used to track outcomes over time in the treatment groups 1-3 (the ones receiving performance-based payments). The baseline survey was implemented prior to the implementation of PBF in the 7 study treatment districts and a follow-up survey (endline) is planned to take place after three years of project implementation. The survey is largely based on the HRITF instruments that were modified to the Tajik and project context.
Kind of Data
Sample survey data [ssd]
Unit of Analysis
The scope of the Tajikistan Health Results Based Financing Impact Evaluation 2014 - Health Facility Baseline Survey includes:
Health Facilities: Assessment of key aspects of facility functioning and structural aspects of quality of care such as - facility staffing; infrastructure and equipment; availability of drugs, consumables and supplies; supervision; record keeping and reporting to Health Managenent Information System; Service volumes
Health Workers: Roles, responsibilities and characteristics of the interviewed health worker; staff satisfaction and motivationl technical knowledge on maternal child health and non-communicable diseases.
Patient-Provider Interactions: assessment of adherence to protocols in terms of IMCI and hypertension management.
Patient Exit Interviews: patients' perceived quality of care and satisfaction with the care given; socio-economic background and the general health of the patient.
Criterion based audit: an assessment of whether the content of clinical care delivered is complete and appropriate in light of clinical best practices.
Three districts in the Sughd region and 4 districts in the Khatlon region were selected to implement the program. All Rural Health Centers in these seven districts are covered by the program. Nine additional district (two in Sughd and seven in Khatlon) were selected as control districts. The selection of the control districts was guided by geographical proximity to treatment districts and similarity in terms of number of health facilities and doctors per capita. The districts were also selected such that the number of RHCs in treatment and control groups in each region would be similar.
Producers and sponsors
Damien de Walque
The World Bank - DECHD
The World Bank - GHN03
The World Bank - DECHD
Health Results Innovation Trust Fund
Table 1.4.1 of the survey report provided under the Related Materials tab presents the list of selected districts, their assignment into the PBF treatment and the number of RHCs in each district. As the set of RHCs in each study district were randomly assigned into three study arms, some RHCs were not included in the study when the number of RHCs in a district was not divisible by three. Excluded RHCs were randomly selected with all RHCs having identical probability of being selected. The sample size is of 216 Rural Health centers, 108 in PBF districts and 108 in control districts. Of the 216 RHCs, 66 are in Sughd and 150 are in Khatlon. The sample of facilities will be identical for the baseline and endline surveys.
An abridged Health Facility survey was implemented also in Health Houses. While some Rural Health Centers have one or more subsidiary Health Houses in their catchment areas, other do not have any. One Health House from each RHC with subsidiary HHs was to be included in the sample. Selection was random with each health house within a cluster having identical probability of being chosen. Non-selected health houses were ranked to serve as replacements in case the survey cannot be implemented in the selected HHs. Table 1.4.2 of the survey report presents the number of HHs selected for the sample for each district (that is, the number of RHC that have subsidiary health houses). Of the 216 RHC selected for the sample (after excluding some RHCs when the total number was not divisible by three), 150 have subsidiary HHs. Forty-three HHs were selected of the sample in Sughd and 107 in Khatlon.
In Sughd, all health facilities were assessed according to selected list and no replacement was used. In Khatlon, four health facilities were replaced. The reason for replacement was absence of RHC and referring HH. One HH was added due to replacement of a RHC and the final number of surveyed health facilities is 367. The achieved sample sizes for the direct observations and exit interviews is low because of low number of patients who visited the facilities during the survey days. It was more difficult to achieve the required sample in Khatlon compared to Sughd region. For the criterion-based audit, the field teams encountered difficulties in finding the medical records to review. The average number of staff members present at the HHs resulted in the low number of health providers interviewed.
Tables 2.4.1 and 2.4.2 in the survey report (provided under Related Materials) show the targeted and achieved sample sizes for each questionnaire.
A complete health facility survey was conducted in RHCs, whereas for health houses a shorter survey was implemented. A challenging and important goal of the facility-based survey is to collect different measures of quality of care in the health facilities.
Form F1: Health Facility Assessment: The facility assessment module seeks to collect data on key aspects of facility functioning and structural aspects of quality of care. The respondent for this module were the individuals in charge of the health facility at the time when the survey team visits the health facility. The main themes to be covered by the facility assessment include:
· Facility staffing, including the staffing complement of the facility, staff on duty at the time of the survey team's visit and staff present at the time of the survey team's visit
· Facility infrastructure and equipment
· Availability of drugs, consumables and supplies at the health facility
· Record keeping and reporting to the Health Management Information System
· Service volumes
Form F2: Health Worker Questionnaire: A random sample of 4 health workers was to be taken at each of the Rural Health Centers and Health Houses included in the sample. Eligible health workers include doctors, nurses, midwife/auxiliary midwife, and any other health worker providing MCH or NCD care. In facilities with less than 4 health workers on their staff roster, all eligible health workers were to be interviewed.
The main themes to be covered by this module include:
· Role, responsibilities and characteristics of the interviewed health worker
· Staff satisfaction and motivation
· Technical knowledge on MCH and NCDs. Knowledge was assessed through the use of provider vignettes on MCH and NCD protocols and diagnosis.
Direct Observation of Patient-Provider Interactions: The goal of the direct observations is to assess adherence to protocols in terms of IMCI and hypertension management. At each Rural Health Center, up to 5 children under-five and up to 5 adults over 40 years who are potential candidates for hypertension identification/management services was to be selected. A member of the survey team observed consultations using a structured format to note whether key desired actions were carried out. In the case of patients under five, the instrument focuses on whether IMCI protocols are followed. For adults over 40 years, the instrument focuses on whether MoH and international protocols are followed. The direct observations were implemented only in RHCs.
Form F4: Patient Exit Interviews: The same set of patients who were selected for the direct observations of patient-provider interactions were also selected for exit interviews. If the patient is a child, the child's caregiver was interviewed. The exit interviews collected data on the patients' perceived quality of care and satisfaction with the care given. Additional information was collected on socio-economic background and the general health of the patient. Like the direct observations, the exit interviews were only administered in RHCs.
Criterion Based Audit: A target sample of 5 under-five and 5 adult (40+ years) medical records was selected using systematic random sampling methodology at each Rural Health Center to assess whether the content of clinical care delivered is complete and appropriate in light of clinical best practices. Each indicator in the criterion-based clinical care audit is scored through a review of patient records or other facility logs using a structured format. The criterion-based clinical audit focused on IMCI protocols (for under-fives) and hypertension screening and management (for adults over 40 years).
Dates of Data Collection
Data Collection Notes
The Committee on Ethics of the Ministry of Health and Social Protection reviewed the study design, fieldwork protocols and the instruments and granted ethical clearance for the study on October 24th 2014. The Health facility survey was conducted in two phases, according to the PBF treatment status of the districts. As the baseline was to be implemented prior to the launch of the program, districts that were assigned to the PBF program were prioritized and the health facility surveys in those seven districts were implemented from November to December 2014. The survey in the other nine districts took place in January to March 2015. Field teams for the health facility survey were composed of a supervisor and two enumerators, with each team covering a district or two. All recruited enumerators had some medical background. For the first phase, a seven-day training was held in Dushanbe. For the second phase two 5-day training were held in each of the regions. Each training included a day of piloting the survey instruments by the field teams, using the tablets.
The World Bank
Is signing of a confidentiality declaration required?
Confidentiality declaration text
Before being granted access to the dataset, all users have to formally agree:
1. To make no copies of any files or portions of files to which s/he is granted access except those authorized by the data depositor.
2. Not to use any technique in an attempt to learn the identity of any person, establishment, or sampling unit not identified on public use data files.
3. To hold in strictest confidence the identification of any establishment or individual that may be inadvertently revealed in any documents or discussion, or analysis. Such inadvertent identification revealed in her/his analysis will be immediately brought to the attention of the data depositor.
Licensed datasets, accessible under conditions
Use of the dataset must be acknowledged using a citation which would include:
the Identification of the Primary Investigator
the title of the survey (including country, acronym and year of implementation)
the survey reference number
the source and date of download
Shapira, Gil., Damien de Walque. 2014. Tajikistan Health Results Based Financing Impact Evaluation 2014, Health Facility Baseline Survey (HRBFIE-FBL 2014). Ref. TJK_2014_HRBFIE-FBL_v01_M.The World Bank. Dataset downloaded from [URL] on [date].