Community Performance-Based Financing Impact Evaluation 2013, Health Providers Follow Up Survey
Rwanda, 2013 - 2014
Get MicrodataIdentification
RWA_2013_HRBF-HP_v01_M
Community Performance-Based Financing Impact Evaluation 2013
Health Providers Follow Up Survey
| Name | Country code |
|---|---|
| Rwanda | RWA |
Impact Evaluation
The Baseline survey for The Community Performance-Based Financing Impact Evaluation (Health Providers) was conducted in Q1-Q2 2010
The Follow up survey was conducted from Q4 2013 to Q2 2014. Data collection consisted of household, Community Health Worker (CHW) and CHW cooperative surveys. A health facility assessment was conducted at the follow up survey.
In 2010 the 'second generation' of PBF in Rwanda, the Community Performance-Based Financing (CPBF) program, was initiated to tackle the remaining issue of low utilization of health services by mothers and their children. The Community PBF implemented the following three interventions: (i) demand-side in-kind incentives for women, (ii) financial rewards for community health worker (CHW) cooperatives, and (iii) combined demand-side and CHW rewards. The three CPBF interventions were introduced in October 2010 in randomly selected sectors. The CPBF program is evaluated by a prospective, randomized impact evaluation (IE). The IE evaluates the causal effect of the CPBF interventions on maternal and child health outputs and outcomes.Analysis of the IE data that assessed the program impact compares outcomes in sectors implementing the three different interventions and a comparison group of sectors not implementing any of the interventions. The IE relies on baseline (Q1-Q2 2010) and follow up (Q4 2013-Q2 2014) data collection, consisting of household, CHW and CHW cooperative surveys. A health facility assessment was conducted at the follow up survey.
The endline survey for the impact evaluation was fielded from November 2013 to June 2014. The University of Rwanda College of Medicine and Health Sciences School of Public Health (UR-CMHS-SPH) managed all activities related to data collection and entry. The baseline survey questionnaires were adapted to account for the modification to the sample described in the previous chapter. In addition, modules asking about the experience with the CPBF program have been added. Because the baseline survey did not include the health facility assessment, the relevant questionnaires were created by adapting the Health Results Innovation Trust Fund (HRITF) samples. Apart from the health facility assessment that was conducted in French, all other interviews were conducted in Kinyarwanda.
Interviews with the CHWs and presidents of the cooperatives were conducted at the health centers and conducted jointly with the health facility assessment. The in-charge of each health center and the center's head of community health were informed two weeks prior to the visit. The CHWs in charge of MNH in the sample villages were asked to be present at the health center at the day of the visit. The head of community health was in charge of contacting the CHWs. Each survey team was composed of 3 enumerators led by a team leader. During a one-day visit, a team fielded the health facility questionnaire with the in-charge of the health center or the deputy, the CHW cooperative questionnaire with the cooperative's president in addition to interviews with providers, patients and CHWs. 2220 CHWs were successfully interviewed in endline out of the target of 2376. This is a response rate of 93%. A response analysis was performed to test whether assignment to a specific treatment arm affected the propensity to comply with the endline survey. The results indicate that response is not significantly correlated with the introduced interventions.
The endline survey also included a health facility assessment for health centers containing interviews of facility in-charges and health providers as well as exit interviews with patients visiting the health center.
Sample survey data [ssd]
Community health workers; cooperatives of community health workers; health centers
Scope
<b>Community Health Worker Survey</b>
The individual community health worker survey was administered to the CHWs in charge of maternal and neonatal health and included 9 sections:
- General Information: This section collects information on the CHWs’ socio-demographic background, the community which they serve and general information on their experience as a CHW such as their recruitment and number of team members in the village.
- CHW Payments: This section collects information on compensation received by the CHWs, whether monetary or in-kind.
- Training and Services: This section collects information on the training received by the CHWs and the services they provided in the 3 months prior to the survey.
- Supervision: This section collects information on the nature of supervision given to the CHWs.
- CHW Resources: This section collects information on the support given to CHWs by health centers, NGOs, the communities, and other CHWs. It also covers the supplies available to the worker and asks about the challenges faced by the CHWs in providing health services to their communities.
- CHW Health Knowledge: This section complements the Health Knowledge section in the household survey and collects data on the CHWS’ knowledge of specific health-related categories, including hand washing, water, sanitation, pregnancy danger signs, child nutrition, vaccination, contraception, tuberculosis, malaria and child illness.
- CHW Satisfaction: This section collects data on the level of satisfaction of the CHWs with different aspects of their role.
- CHW Motivation: This section collects data on the motivation of CHWs.
- Program Knowledge and Attitude: This section collects information on the CHWs’ knowledge of the different components of the CPBF program and their attitude towards it.
<b>Community Health Worker Cooperative Questionnaire</b>
The community health workers cooperative questionnaire was administered to the president of the cooperative and includes 7 sections:
- General Information: This section collects data on location of the cooperative, its infrastructure, its legal status and finances. It also covers the presence of national health protocols.
- Administration and Management: This section collects data on the relationship of the cooperative with the PBF sector steering committee, development of an annual work plan, budget, and supervision.
- General Human Resources: This section collects information about recruitment, training, dismissal and resignation of cooperative members.
- General HMIS: This section collects data on the population in the catchment area covered by the cooperative, completion of activity report and availability of different registries.
- CHW Payments: This section collects data on payments to the cooperative, both monetary or in-kind. It also asks about investments in income generating activities.
- CHW Resources: This section asks about support the cooperatives receive from health centers, government, NGOs and the communities. It also asks about available supplies and challenges faced by the cooperative.
- Income Generating Activities: This section collects information on the income generating activities of the cooperatives as well as the related equipment or livestock they own for this purpose.
Health Facility Assessment
<b>Health Facility Main Questionnaire</b>
The Health Facility Main questionnaire was administered to the head or deputy head of the health facility and any other person designated by the head or deputy head to provide information for the different sections. It includes 7 sections:
- General Information: This section collects general information on the infrastructure of the facility, hours of operation and sources of funding.
- Administration and Management: This section asks about the management of the facility in terms of health committee membership, development of an annual work plan, detection of health priorities in the catchment area, and the supervision of staff.
- Staff Roster: This section lists all staff members of the health center and collects information on their qualifications, roles, experience and wage. It also records the presence of each staff member in the day of the survey and the type of the services provided.
- Laboratory: This section collects information on laboratory tests performed within the facility.
- Services: This section asks about the different maternal and child health services provided by the health center.
- HMIS: This section collects data on the population in the catchment area covered by the health center and about presence of various reports.
- Health Services Utilization: This section collects data on utilization of various services provided by the health center.
- Direct Observations: This section is collected through direct observations by the enumerator. The enumerator lists the existence of different elements of the facility infrastructure (e.g. waiting room) and presence national protocols.
- Equipment (Direct Observations): In this section the enumerator lists the presence of different medical equipment.
- Drugs and Vaccinations storage and availability (Direct Observations): In this section, the enumerator lists the presence and the means of storage for a set of commodities.
- Community Performance-Base Financing and CHWs: This section collects data on the health center’s director’s awareness and attitude towards the CPBF program and community health workers.
<b>Health Worker Questionnaire</b>
The Health Worker questionnaire was administered to the provider of antenatal care and the provider of child curative care. All sections were asked to both providers except Section 6 which was specific to the service provided on the day of the interview (antenatal or child curative care).
- General Information: This section collects information on the socio-demographics characteristics of the health providers, work experience, the role at the health facility and the type of services provided on the day of the interview.
- Training: This section collects information on the in-service training received and additional training needed.
- Roles and Responsibilities: This section collects data on the provider’s working load and the presence at work during the last month.
- Salary: This section collects information on the provider’s salary and the regularity of the payment.
- Other remuneration and benefits: This section collects data on any other compensation received by the provider.
- Knowledge vignettes: This section measures the provider’s knowledge of and competency in antenatal or child curative care services.
<b>Exit Interview</b>
The Exit interview was administered to the patients who have been received by a health provider interviewed for the health provider questionnaire. All sections were asked to both types of patients except section 2 which was specific to the type of service received on the day of the interview (antenatal or child curative care).
- Identification: This section collects information related to socio-demographics characteristics of the patient and the type of service received that day.
- Treatment and counseling: This section collects information on the content of care received by the patient during the visit.
- Health expenditure: This section collects information related to transportation cost to the health facility, out-of-pocket payment and health insurance coverage.
- Patient Satisfaction: This section collects data related to the quality of care as perceived by the users of the health facility.
- Knowledge about community health workers’ activities: This section assessed if the patient know about any community health worker in the village, if the patient had received any service from the CHW, the perception of the role of CHW in the community and the quality of care received from CHW.
Coverage
198 sectors (sub districts) in 19 districts in 4 provinces
Producers and sponsors
| Name | Affiliation |
|---|---|
| Gil Shapira | DECHD |
| Ina Kalisa | University of Rwanda College of Medicine and Health Sciences school of Public Health |
| Name | Affiliation |
|---|---|
| Jeanine Condo | University of Rwanda College of Medicine and Health Sciences school of Public Health |
| James Humuza | University of Rwanda College of Medicine and Health Sciences school of Public Health |
| Vedaste Ndahindwa | University of Rwanda College of Medicine and Health Sciences school of Public Health |
| Name |
|---|
| Health Results Innovation Trust Fund |
Sampling
The sampling strategy required several stages.
Stage 1: Identify 200 eligible sectors and randomly assign 50 to each of the four study arms
In order to randomly assign 50 sectors to each of the four study arms, the evaluation team used available data from the MoH Community Health Desk to meet the following objectives:
- Forcibly exclude 30 Vision Umurenge Program (VUP) sectors. These 30 sectors were not included in the random assignment, as they were implementing the demand-side intervention prior to the launch of the study. For this reason, these 30 sectors are considered phase 0 of the CPBF project and are not included in the impact evaluation.
- Exclude all sectors with no health center. Due to operational complexity for the demand-side intervention, the sample of sectors was restricted to those sectors with at least one health center/CHW cooperative.
- Minimize geographic disbursement in order to minimize costs. The 18 districts from the 2006-2008 health center level PBF impact evaluation were included in order to reduce costs and use the data collected during that first impact evaluation on those districts in the future. An additional district was required after randomization in order to meet sample size requirements. The Gicumbi district met the required number of sectors and was geographically close to the Northern districts included in the sample. This resulted in a final sample of 223 sectors. In order to randomly assign 50 sectors to each study arm, the team blocked the sectors by a) district and b) poverty ranking. Within each block, the sectors were randomly assigned a number 1, 2, 3 or 4. In study arms 1, 2 and 3, there were incomplete blocks, and replacement sectors were needed for those study arms. Once the sectors were randomly assigned to each study arm, the evaluation team conducted difference in means tests between each treatment group and the comparison group, as well as the combination of all treatment groups and the comparison group, on the following variables:
• VUP poverty ranking (to validate the blocking)
• The sector’s health center is classified as “Public” or “Agrée” (Government or non-for-profit faith-based facilities)
• Sector Population: Male/Female
There were no statistical differences between the treatment and control groups based on these variables.
Stage 2: Identify one CHW cooperative per sector
On average, each sector has one health center with a corresponding CHW cooperative. The CHW cooperative leader was interviewed.
Stage 3: Randomly select 12 villages per sector
Ex-ante power calculations demonstrated that for each sector, the team required 12 households. The team first needed to identify the 12 villages that households would be selected from for each sector. Using administrative data, three cells were randomly selected for each sector from the total sample of cells. For each of the total 600 cells, four villages were randomly selected from the total number of villages, resulting in 12 villages per sector and a total number of 2400 households.
Community Health Workers
The research team decided to only interview the community health worker in charge of maternal and neonatal heath in each village in the endline survey to release resources for the doubling of the household survey. It is important to note that the number of CHWs included in the baseline survey was not based on a power analysis. In addition, most of the targeted indicators are related to the work of these community health workers.
Community Health Cooperatives
The endline survey was planned to conduct a cooperative survey in each of the cooperatives covered by the baseline survey.
Health Facilities
Unlike the baseline survey, the endline survey covered also the health facilities associated with each community health workers cooperative. The health facility survey included:
• A health facility assessment completed by interviewing the head or deputy head of the health center.
• A health worker survey: in each health center, the research team selected for interview two health workers providing the target services on the day of the survey (one for antenatal care and one for child curative care services). A set of vignettes were administered to the provider to measure their practical knowledge on the specific service provided.
Patient exit interviews
At the end of the visit with the above mentioned providers, six patients in total per health center were interviewed on the day of the survey (three patients who received antenatal and three patients who received child care services) to assess the competence of the providers and the quality of care received.
According to the study design, the survey should have covered 200 sectors, 50 for each study arm. Of the 200 originally selected sectors, 12 did not meet the criteria of having a health center with an active CHW cooperative. While replacement sectors were assigned to the 3 treatment groups, there was no replacement sector assigned for the control group. Therefore, one sector of the control group has been dropped from the sample without being replaced. Another sector assigned to the control group has been wrongly coded in the data and subsequently also dropped from the sample. As a result, the final sample covered 198 sectors.
2220 CHWs were successfully interviewed in endline out of the target of 2376. This is a response rate of 93%.
Survey instrument
- Community Health Worker
- Community Health Workers Cooperative
- Health Center Assessment
- Health Provider
- Exit Interview with patients of health centers
Apart from the health facility assessment that was conducted in French, all other interviews were conducted in Kinyarwanda.
Data collection
| Start | End | Cycle |
|---|---|---|
| 2013-11 | 2014-06 | Endline |
| Name |
|---|
| The University of Rwanda College of Medicine and Health Sciences School of Public Health |
Interviews with the CHWs and presidents of the cooperatives were conducted at the health centers and conducted jointly with the health facility assessment. The in-charge of each health center and the center's head of community health were informed two weeks prior to the visit. The CHWs in charge of MNH in the sample villages were asked to be present at the health center at the day of the visit. The head of community health was in charge of contacting the CHWs. Each survey team was composed of 3 enumerators led by a team leader. During a one-day visit, a team fielded the health facility questionnaire with the in-charge of the health center or the deputy, the CHW cooperative questionnaire with the cooperative's president in addition to interviews with providers, patients and CHWs.
The long duration of the data collection was due to the tracking of baseline women who moved out of the baseline district and not due to the health provider survey.
Data appraisal
A response analysis was performed to test whether assignment to a specific treatment arm affected the propensity to comply with the endline survey. The results indicate that response is not significantly correlated with the introduced interventions.
Data Access
| Name | Affiliation | |
|---|---|---|
| Gil Shapira | DECHD | gshapira@worldbank.org |
| Is signing of a confidentiality declaration required? | Confidentiality declaration text |
|---|---|
| yes | Before being granted access to the dataset, all users have to formally agree: 1. To make no copies of any files or portions of files to which s/he is granted access except those authorized by the data depositor. 2. Not to use any technique in an attempt to learn the identity of any person, establishment, or sampling unit not identified on public use data files. 3. To hold in strictest confidence the identification of any establishment or individual that may be inadvertently revealed in any documents or discussion, or analysis. Such inadvertent identification revealed in her/his analysis will be immediately brought to the attention of the data depositor |
- Licensed datasets, accessible under conditions
Use of the dataset must be acknowledged using a citation which would include:
- the Identification of the Primary Investigator
- the title of the survey (including country, acronym and year of implementation)
- the survey reference number
- the source and date of download
Example:
World Bank, DECRG: Human Development. Rwanda Community Performance-Based Financing Impact Evaluation 2013, Health Providers Follow Up Survey (HRBF-HP), Ref. RWA_2013_HRBF-HP_v01_M. Dataset downloaded from [url] on [date]
Contacts
| Name | Affiliation | |
|---|---|---|
| Gil Shapira | DECHD | gshapira@worldbank.org |
Metadata production
DDI_RWA_2013_HBRF-HP_v01_M_WB
| Name | Affiliation | Role |
|---|---|---|
| Development Economics Data Group | The World Bank | Documentation of the DDI |
2016-07-05
Metadata version
Version 01 (July 2016)