Health Results Based Financing Impact Evaluation 2015
Tajikistan, 2015
Damien de Walque, Aneesa Arur, Gil Shapira
Identification
IDNO
TJK_2015_HRBFIE-HBL_v01_M
Title
Health Results Based Financing Impact Evaluation 2015
Subtitle
Household Baseline Survey
Countries
| Name | Code |
|---|---|
| Tajikistan | TJK |
Study notes
The policy objective of the Impact Evaluation (IE) is to build evidence on the impact and cost-effectiveness of the proposed Performance-Based-Financing (PBF) project in Tajikistan. More specifically, the IE would seek to ascertain: (i) the impact and cost-effectiveness of the PBF model implemented in Tajikistan; and (ii) whether PBF is more effective or cost-effective if implemented in conjunction with additional low cost interventions (Collaborative Quality Improvement, Citizen Report Cards). The results from the IE will help informing the MOH on whether PBF should be scaled-up to additional PHC level institutions in other regions.
The Collaborative Quality Improvement intervention responds to policy concerns that performance incentives may not produce the desired improvements if providers lack the necessary competencies, data to inform decisions and knowledge. The Citizen Report Card attempts to improve the effectiveness of PBF by strengthening the 'short route of accountability', i.e., by increasing accountability of health facilities to their local constituents. Since PBF, collaborative quality improvement (CQI), and citizen report cards (CRC) have never been implemented in large scale in Tajikistan, it is to be expected that the results from the IE will be useful for designing national PHC policy in Tajikistan, and that they will also contribute to the larger body of knowledge on these interventions.
The IE employs both difference-in-difference and experimental approaches to identify the impact of the different combinations of interventions. Assignment to PBF was not random. Three districts in the Sughd region and 4 districts in the Khatlon region were selected to implement the program. All Rural Health Centers (RHCs) in these seven districts are covered by the program. Nine additional district (two in Sughd and seven in Khatlon) were selected as control districts. The selection of the control districts was guided by geographical proximity to treatment districts and similarity in terms of number of health facilities and doctors per capita. The districts were also selected such that the number of RHCs in treatment and control groups in each region would be similar.
Within the chosen 16 districts (treatment and control districts), clusters consisting of a RHC and its subsidiary Health Houses were randomly assigned to implement Collaborative Quality Improvement, Citizen Score Cards, or neither of these two interventions. The randomization was blocked by district. In sum, RHCs were assigned into six study arms.
The goal of the Facility-based survey is to measure multiple dimensions of quality of care and collect detailed information on key aspects of facility functioning.Household surveys are primarily used to measure health service coverage at the population level as well as select health outcome indicators measured through anthropometry or tests. The surveys also collect broader data on the health of the households, health seeking behaviors and barriers to use of health services. In addition, PBF and other administrative data would be used to track outcomes over time in the treatment groups 1-3 (the ones receiving performance-based payments). The baseline survey was implemented prior to the implementation of PBF in the 7 study treatment districts and a follow-up survey (endline) is planned to take place after three years of project implementation. The survey is largely based on the HRITF instruments that were modified to the Tajik and project context.
The Collaborative Quality Improvement intervention responds to policy concerns that performance incentives may not produce the desired improvements if providers lack the necessary competencies, data to inform decisions and knowledge. The Citizen Report Card attempts to improve the effectiveness of PBF by strengthening the 'short route of accountability', i.e., by increasing accountability of health facilities to their local constituents. Since PBF, collaborative quality improvement (CQI), and citizen report cards (CRC) have never been implemented in large scale in Tajikistan, it is to be expected that the results from the IE will be useful for designing national PHC policy in Tajikistan, and that they will also contribute to the larger body of knowledge on these interventions.
The IE employs both difference-in-difference and experimental approaches to identify the impact of the different combinations of interventions. Assignment to PBF was not random. Three districts in the Sughd region and 4 districts in the Khatlon region were selected to implement the program. All Rural Health Centers (RHCs) in these seven districts are covered by the program. Nine additional district (two in Sughd and seven in Khatlon) were selected as control districts. The selection of the control districts was guided by geographical proximity to treatment districts and similarity in terms of number of health facilities and doctors per capita. The districts were also selected such that the number of RHCs in treatment and control groups in each region would be similar.
Within the chosen 16 districts (treatment and control districts), clusters consisting of a RHC and its subsidiary Health Houses were randomly assigned to implement Collaborative Quality Improvement, Citizen Score Cards, or neither of these two interventions. The randomization was blocked by district. In sum, RHCs were assigned into six study arms.
The goal of the Facility-based survey is to measure multiple dimensions of quality of care and collect detailed information on key aspects of facility functioning.Household surveys are primarily used to measure health service coverage at the population level as well as select health outcome indicators measured through anthropometry or tests. The surveys also collect broader data on the health of the households, health seeking behaviors and barriers to use of health services. In addition, PBF and other administrative data would be used to track outcomes over time in the treatment groups 1-3 (the ones receiving performance-based payments). The baseline survey was implemented prior to the implementation of PBF in the 7 study treatment districts and a follow-up survey (endline) is planned to take place after three years of project implementation. The survey is largely based on the HRITF instruments that were modified to the Tajik and project context.
Kind of data
Sample survey data [ssd]
Coverage
Geographic coverage
Three districts in the Sughd region and 4 districts in the Khatlon region were selected to implement the program. All Rural Health Centers in these seven districts are covered by the program. Nine additional district (two in Sughd and seven in Khatlon) were selected as control districts. The selection of the control districts was guided by geographical proximity to treatment districts and similarity in terms of number of health facilities and doctors per capita. The districts were also selected such that the number of RHCs in treatment and control groups in each region would be similar.
Geographic unit
District
Unit of analysis
Household
Producers and sponsors
Authoring entity
| Name | Affiliation |
|---|---|
| Damien de Walque | The World Bank - DECHD |
| Aneesa Arur | The World Bank - GHN03 |
| Gil Shapira | The World Bank - DECHD |
Funding agencies
| Name | Abbreviation |
|---|---|
| Health Results Innovation Trust Fund | HRITF |
Sampling
Sampling procedure
The evaluation relies on two samples of households. As the primary focus of the PBF intervention is on Maternal and Child Health (MCH) services, the main household sample is of households with women who experienced a recent pregnancy. This sample, however, would not be appropriate to study the impact on the coverage of services related to Non Communicable Diseases (NCD). Therefore, a secondary sample consists of households with individuals over the age of 40. The household samples are clustered according to the catchment area of each Rural Health Center (and its affiliated health houses).
To estimate the needed sample size of households per cluster for the households with recent pregnancies, the research team used data from a household survey collected by the Swiss Tropical and Public Health Institute in the project regions in 2012 for a study conducted to inform the design of the PBF project. The outcome chosen for this analysis is the completion of at least 4 antenatal consultations during a pregnancy, one of the PDO level results indicators of the project. 65.5% of women reported at least 4 consultations during their last pregnancy. The intra-cluster correlation, after controlling for rayon of residence, is 0.052. Assuming a t-test significance level of 0.05 and power of 0.8, a sample of 20 households per cluster could detect an effect size of six percentage points in the diff-in-diff analysis comparing the PBF and control districts. This power calculation is conservative, given that it ignores the Difference-in-Difference and matching strategies that could increase power. Under the same set assumptions, the cluster size of 20 households per RHC would also allow to detect an effect of eight percentage points when employing the experimental design to compare the outcomes with and without the CSC and CQI interventions.
The resulting targeted primary household sample size is of 4,320 households, with twenty in each of the 216 clusters in the six study arms. To be eligible for inclusion in the household survey sample, households must have at least one woman aged 15-49 years who has had a child in the preceding 3 years. The same villages will be covered for both the baseline and follow up survey and eligibility will be determined at each round by a listing exercise.
For budgetary reasons, the impact on NCD outcomes will only be measured in treatment group 1 (PBF only) and the control group. To estimate the needed sample size of households per cluster for these outcomes, we use a nationally representative data that includes blood pressure measurement. The outcome chosen for this analysis is an indicator of high blood pressure. The data show that prevalence of high blood pressure increases significantly between ages 30 and 40. However, because of the relative low amount of individuals above the age of 40, the following calculations are performed using the sample of individuals above the age of 30 and combine both men and women. About 40% of individuals in this age group have high blood pressure. The intra-cluster correlation, after controlling for rayon of residence, is 0.03. Assuming a t-test significance level of 0.05 and power of 0.8, a sample size of 22 households per cluster will allow detection of nine percentage points reduction.
The resulting targeted sample size for the secondary household sample is 1,584 households, with 22 in each of 72 clusters in two of the six study arms. Eligibility for this sample is determined by the presence of an individual over the age of 40 in the household. Eligibility for the two different samples is determined by a common listing of households in selected villages. Households that satisfy both eligibility criteria can be randomly selected to count towards the sample size requirements for both.
A two-stage cluster sampling methodology was employed to identify the random samples. First, villages were randomly selected out of a list of the villages served by each facility. The list was obtained from the MoH. RHCs have either a single or multiple villages in their catchment areas while HHs typically serve a single village. If a RHC has at least one affiliated HH, then two villages were selected. One of the village was directly served by the RHC while the other included in the sub-catchment area of the HH. In each village, 100 households were listed. In case the village had more than 100 households, a random walk method was used to select the target number. A short questionnaire was conducted at each household to determine households' eligibility for the two samples. From all eligible households, the target sample for each catchment area was selected. In catchment areas in which two villages were included in the sample, half of the households were to be selected from each village. Sampling for the follow-up survey will follow the same procedures.
To estimate the needed sample size of households per cluster for the households with recent pregnancies, the research team used data from a household survey collected by the Swiss Tropical and Public Health Institute in the project regions in 2012 for a study conducted to inform the design of the PBF project. The outcome chosen for this analysis is the completion of at least 4 antenatal consultations during a pregnancy, one of the PDO level results indicators of the project. 65.5% of women reported at least 4 consultations during their last pregnancy. The intra-cluster correlation, after controlling for rayon of residence, is 0.052. Assuming a t-test significance level of 0.05 and power of 0.8, a sample of 20 households per cluster could detect an effect size of six percentage points in the diff-in-diff analysis comparing the PBF and control districts. This power calculation is conservative, given that it ignores the Difference-in-Difference and matching strategies that could increase power. Under the same set assumptions, the cluster size of 20 households per RHC would also allow to detect an effect of eight percentage points when employing the experimental design to compare the outcomes with and without the CSC and CQI interventions.
The resulting targeted primary household sample size is of 4,320 households, with twenty in each of the 216 clusters in the six study arms. To be eligible for inclusion in the household survey sample, households must have at least one woman aged 15-49 years who has had a child in the preceding 3 years. The same villages will be covered for both the baseline and follow up survey and eligibility will be determined at each round by a listing exercise.
For budgetary reasons, the impact on NCD outcomes will only be measured in treatment group 1 (PBF only) and the control group. To estimate the needed sample size of households per cluster for these outcomes, we use a nationally representative data that includes blood pressure measurement. The outcome chosen for this analysis is an indicator of high blood pressure. The data show that prevalence of high blood pressure increases significantly between ages 30 and 40. However, because of the relative low amount of individuals above the age of 40, the following calculations are performed using the sample of individuals above the age of 30 and combine both men and women. About 40% of individuals in this age group have high blood pressure. The intra-cluster correlation, after controlling for rayon of residence, is 0.03. Assuming a t-test significance level of 0.05 and power of 0.8, a sample size of 22 households per cluster will allow detection of nine percentage points reduction.
The resulting targeted sample size for the secondary household sample is 1,584 households, with 22 in each of 72 clusters in two of the six study arms. Eligibility for this sample is determined by the presence of an individual over the age of 40 in the household. Eligibility for the two different samples is determined by a common listing of households in selected villages. Households that satisfy both eligibility criteria can be randomly selected to count towards the sample size requirements for both.
A two-stage cluster sampling methodology was employed to identify the random samples. First, villages were randomly selected out of a list of the villages served by each facility. The list was obtained from the MoH. RHCs have either a single or multiple villages in their catchment areas while HHs typically serve a single village. If a RHC has at least one affiliated HH, then two villages were selected. One of the village was directly served by the RHC while the other included in the sub-catchment area of the HH. In each village, 100 households were listed. In case the village had more than 100 households, a random walk method was used to select the target number. A short questionnaire was conducted at each household to determine households' eligibility for the two samples. From all eligible households, the target sample for each catchment area was selected. In catchment areas in which two villages were included in the sample, half of the households were to be selected from each village. Sampling for the follow-up survey will follow the same procedures.
Deviations from sample design
For the household sample, there were two main deviations from the suggested protocol. 1) All eligible household members were interviewed regardless of the selected sample and 2) whereas the listing exercise followed the suggested protocol and sampled for households with a pregnancy within past 3 years, the questionnaire was programmed to interview women with a pregnancy within 2 years for the maternal health section. These two deviations caused the final sample to have more than the target number of adults and less than the target number of women with a recent pregnancy. About 8% of all visited households were replaced. In most cases, households were replaced as a household member refused to be interviewed. Other reasons include change of residence (temporary during summer), and some households were not meeting the eligibility criteria.
Weighting
The household samples are clustered according to the catchment area of each Rural Health Center (and its affiliated health houses). Equal number of households were selected for the survey in each catchment area. Sample weights were created to ensure the sample is representative of the selected districts in Sughd and Khatlon provinces. Section 2.5 of the survey report (provided under the Related Materials tab) presents estimates of the catchment population and explains the calculation of sample weights used for the analysis.
Data Collection
Dates of collection
| Start | End |
|---|---|
| 2015-04 | 2015-07 |
Mode of data collection
Computer Assisted Personal Interview [capi]
Data collection supervision
The Household survey was conducted between April and July 2015. Field teams were organized in two regional groups. Four supervisors and 28 enumerators were recruited in the Sughd region while seven supervisors and 62 enumerators were recruited in Khatlon. All team members had either medical background or prior experience in working on medical research projects. Each supervisor oversaw work in one or two districts. The size of the teams varied by district as the sample sizes were different among districts. A ten-day training was conducted in each region prior to the launch of the field work. Each training included a day of piloting the survey instruments by the field teams, using the tablets.
Data collection was launched after trainings were completed. The regional manager was assigned for each interviewer's team to assist them; provide logistical support and material; monitor interviewers' activities during the data collection process; assess the quality of interviewers' work and the quality of data from the completed questionnaire; provide feedback to interviewers on quality assurance and methodology requirements. On the first day of data collection, each survey supervisor went to the representatives of local government. They informed about the goals and objectives of the survey and presented a support letter.
Each team was directed to the selected household in related districts. On the first day, the interviewers went to the health facilities in each district and introduced the team, presented the list of selected households. With the assistance of a representative from health facility, the selected households were identified and interviews were introduced to the household members. The household selections were determined by the instructions and the survey methodology. All selected households included either women, who had recent birth (since January 2012) or adults over the age of 40. The majority of households had both requirements: women with recent birth and adults over 40 years old. At the end of each interview, the households were given small remuneration (soaps, detergents powder) for the time they spend for an interview.
Data collection was launched after trainings were completed. The regional manager was assigned for each interviewer's team to assist them; provide logistical support and material; monitor interviewers' activities during the data collection process; assess the quality of interviewers' work and the quality of data from the completed questionnaire; provide feedback to interviewers on quality assurance and methodology requirements. On the first day of data collection, each survey supervisor went to the representatives of local government. They informed about the goals and objectives of the survey and presented a support letter.
Each team was directed to the selected household in related districts. On the first day, the interviewers went to the health facilities in each district and introduced the team, presented the list of selected households. With the assistance of a representative from health facility, the selected households were identified and interviews were introduced to the household members. The household selections were determined by the instructions and the survey methodology. All selected households included either women, who had recent birth (since January 2012) or adults over the age of 40. The majority of households had both requirements: women with recent birth and adults over 40 years old. At the end of each interview, the households were given small remuneration (soaps, detergents powder) for the time they spend for an interview.
Questionnaires
The HRITF survey instruments were adapted to the Tajik context. Additional modules were developed to correspond to the project focus on NCD outcomes on top of the MCH outcomes. Representatives of the MoH, Zerkalo and members of the WB team conducted several field visits to health facilities to inform the adaptation of the survey instruments. The instruments were pretested three time between August and October 2014, prior to the training of the field team. A consultant with public health and medical expertise was hired to develop the clinical instruments. Household questionnaires were translated into Tajik while health facility questionnaires were translated into Tajik and Russian.
The household survey is composed of three questionnaires: main household questionnaire, a female and child questionnaire, and a questionnaire for adults over 40 years. The main household questionnaire was implemented in all households. According to the sampling strategy, separate samples were to be selected for a recent pregnancy and adults over 40. However, the over-40s questionnaire was implemented to eligible respondents in the entire household survey sample as the questionnaires were programmed such that an interviewer could not skip a section if there was an eligible household member. As a result, in the final sample we end up with more than the target number of over- 40s.
Main household questionnaire: The main respondent for the household-level questionnaire is the head of household and/or spouse, although a few modules were administered to each member of the household. The respondent could ask for support from other household members on specific questions regarding the household. The household questionnaire focused on the following topics:
• Socio-demographic characteristics: household composition and the age, marital status, employment and education level of all household members
• Income, transfers, assets and housing
• Consumption of food and other items
• Migration of household members in and out of the country
• Mortality
• Utilization of health care
• Blood pressure measurements for all adults over 18 years
Female questionnaire. The female questionnaire was administered to female household member(s) 15-49 years old. The topics covered by the questionnaire were:
• General health status
• Pregnancy history, reproductive health and utilization of family planning methods
• Antenatal, delivery and postnatal care received during recent pregnancies/births
• Vaccination of children under 5 years
• Anthropometric measures of the children under 5 years
Adult over 40 questionnaire: The respondent(s) for this questionnaire are household members above the age of 40 years. The topics of focus for the questionnaire were:
• General health status
• Health-related behaviors such as physical activity, smoking and alcohol consumption
• Health care seeking
• High cholesterol and other health conditions
The household survey is composed of three questionnaires: main household questionnaire, a female and child questionnaire, and a questionnaire for adults over 40 years. The main household questionnaire was implemented in all households. According to the sampling strategy, separate samples were to be selected for a recent pregnancy and adults over 40. However, the over-40s questionnaire was implemented to eligible respondents in the entire household survey sample as the questionnaires were programmed such that an interviewer could not skip a section if there was an eligible household member. As a result, in the final sample we end up with more than the target number of over- 40s.
Main household questionnaire: The main respondent for the household-level questionnaire is the head of household and/or spouse, although a few modules were administered to each member of the household. The respondent could ask for support from other household members on specific questions regarding the household. The household questionnaire focused on the following topics:
• Socio-demographic characteristics: household composition and the age, marital status, employment and education level of all household members
• Income, transfers, assets and housing
• Consumption of food and other items
• Migration of household members in and out of the country
• Mortality
• Utilization of health care
• Blood pressure measurements for all adults over 18 years
Female questionnaire. The female questionnaire was administered to female household member(s) 15-49 years old. The topics covered by the questionnaire were:
• General health status
• Pregnancy history, reproductive health and utilization of family planning methods
• Antenatal, delivery and postnatal care received during recent pregnancies/births
• Vaccination of children under 5 years
• Anthropometric measures of the children under 5 years
Adult over 40 questionnaire: The respondent(s) for this questionnaire are household members above the age of 40 years. The topics of focus for the questionnaire were:
• General health status
• Health-related behaviors such as physical activity, smoking and alcohol consumption
• Health care seeking
• High cholesterol and other health conditions
Data collector(s)
| Name |
|---|
| Zerkalo |
Data access
Access authorities
| Name | Affiliation | |
|---|---|---|
| Gil Shapira | The World Bank | gshapira@worldbank.org |
metadata.study_desc.data_access.dataset_use.conf_dec
Before being granted access to the dataset, all users have to formally agree:
1. To make no copies of any files or portions of files to which s/he is granted access except those authorized by the data depositor.
2. Not to use any technique in an attempt to learn the identity of any person, establishment, or sampling unit not identified on public use data files.
3. To hold in strictest confidence the identification of any establishment or individual that may be inadvertently revealed in any documents or discussion, or analysis. Such inadvertent identification revealed in her/his analysis will be immediately brought to the attention of the data depositor.
Access conditions
- Licensed datasets, accessible under conditions
Citation requirements
Use of the dataset must be acknowledged using a citation which would include:
- the Identification of the Primary Investigator
- the title of the survey (including country, acronym and year of implementation)
- the survey reference number
- the source and date of download
Shapira, Gil., Damien de Walque. 2015. Tajikistan Health Results Based Financing Impact Evaluation 2015, Household Baseline Survey (HRBFIE-HBL 2015). Ref. TJK_2015_HRBFIE-HBL_v01_M.The World Bank. Dataset downloaded from [URL] on [date].
- the Identification of the Primary Investigator
- the title of the survey (including country, acronym and year of implementation)
- the survey reference number
- the source and date of download
Shapira, Gil., Damien de Walque. 2015. Tajikistan Health Results Based Financing Impact Evaluation 2015, Household Baseline Survey (HRBFIE-HBL 2015). Ref. TJK_2015_HRBFIE-HBL_v01_M.The World Bank. Dataset downloaded from [URL] on [date].
Metadata production
IDNo
DDI_TJK_2015_HRBFIE-HBL_v01_M_WB
Producers
| Name | Abbreviation | Affiliation | Role |
|---|---|---|---|
| Development Economics Data Group | DECDG | The World Bank | Documentation of the DDI |
Production date
2017-03-20
Version
Version 01 (March 2017)