Farmer Training and Development 2008-2011
Honduras, 2008 - 2011
National Opinion Research Center (NORC)
Identification
IDNO
HND_2008_MCC-FTD_v01_M
Title
Farmer Training and Development 2008-2011
Countries
| Name | Code |
|---|---|
| Honduras | HND |
Study notes
The evaluation design for this activity changed over the course of the evaluation due to problems faced during implementation. In its original conception, the independent evaluator, NORC, and MCA-Honduras planned to use a randomized experimental design involving randomized assignment of communities (aldeas) to treatment. Following a series of implementation problems, the final approach used was an econometric model that relied on a model-based approach to impact evaluation.
As raised in the evaluator report, one key assumption is that the causal models are correct. This is based on the assumption that all important unobserved variables affecting selection, such as proven ex-ante ability to grow horticultural crops, are time invariant (i.e., are constant between the two survey rounds).
Assumptions of the econometric model are:
1. The stable unit treatment value assumption (SUTVA, no macro effects assumption, partial equilibrium assumption) is made. This means that the effect (potential outcomes) on one individual are not affected by potential changes in the treatment exposure of other individuals. This implies, for example, that the program is not so large that the outcomes are correlated (e.g., that farmers would produce such a large amount of horticultural crops that the market would collapse).
2. The causal models are correct. The key assumption here is that all important unobserved variables affecting selection are time invariant (i.e., are constant between the two survey rounds).
3. The program intervention represents a “forced change” in (experimental control of) the agricultural system in Honduras.
4. The half of the country treated before this evaluation began is similar to the half yet to be treated, with respect to relationships among the important causal variables represented in the causal model underlying the statistical analysis.
The exposure period was 12-36 months.
Results from the final evaluation report include: the Model-based approach estimated net income change from horticultural crops is on average USD 600 higher for program participants than for nonparticipants. Input expenditures on these crops increased far more than they did for basic crops, implying a higher level of activity in cultivation of high value crops among program farmers. The results suggest a corresponding decline among program farmers in income from basic crops, as might be expected with changing crop mix; however, this decline is not statistically significant.
However, the program also did not appear to have had a positive effect on the proportion of farmers growing horticultural crops. This could well be because the implementer primarily chose as program participants farmers who showed a proven ability to grow horticultural crops. It is likely that increments in income from horticultural crops came from increased production among farmers already growing horticultural crops and not from farmers who switched over for the first time.
Even though there was an increase in income from horticultural crops, the evaluator did not find a corresponding statistically significant increase in net household income or household expenditures/consumption, as might have been expected.
As raised in the evaluator report, one key assumption is that the causal models are correct. This is based on the assumption that all important unobserved variables affecting selection, such as proven ex-ante ability to grow horticultural crops, are time invariant (i.e., are constant between the two survey rounds).
Assumptions of the econometric model are:
1. The stable unit treatment value assumption (SUTVA, no macro effects assumption, partial equilibrium assumption) is made. This means that the effect (potential outcomes) on one individual are not affected by potential changes in the treatment exposure of other individuals. This implies, for example, that the program is not so large that the outcomes are correlated (e.g., that farmers would produce such a large amount of horticultural crops that the market would collapse).
2. The causal models are correct. The key assumption here is that all important unobserved variables affecting selection are time invariant (i.e., are constant between the two survey rounds).
3. The program intervention represents a “forced change” in (experimental control of) the agricultural system in Honduras.
4. The half of the country treated before this evaluation began is similar to the half yet to be treated, with respect to relationships among the important causal variables represented in the causal model underlying the statistical analysis.
The exposure period was 12-36 months.
Results from the final evaluation report include: the Model-based approach estimated net income change from horticultural crops is on average USD 600 higher for program participants than for nonparticipants. Input expenditures on these crops increased far more than they did for basic crops, implying a higher level of activity in cultivation of high value crops among program farmers. The results suggest a corresponding decline among program farmers in income from basic crops, as might be expected with changing crop mix; however, this decline is not statistically significant.
However, the program also did not appear to have had a positive effect on the proportion of farmers growing horticultural crops. This could well be because the implementer primarily chose as program participants farmers who showed a proven ability to grow horticultural crops. It is likely that increments in income from horticultural crops came from increased production among farmers already growing horticultural crops and not from farmers who switched over for the first time.
Even though there was an increase in income from horticultural crops, the evaluator did not find a corresponding statistically significant increase in net household income or household expenditures/consumption, as might have been expected.
Kind of data
Sample survey data [ssd]
Version
Version
Anonymized dataset for public distribution
Scope
Keywords
| metadata.study_desc.study_info.keywords.keyword |
|---|
| Agriculture |
| Farmer Training |
| MCC First Five |
Coverage
Geographic coverage
Nationwide in Honduras, with the exception of Gracias a Dios Department, national parks and tourist areas (Islas de la Bahia Department).
Unit of analysis
Sampling unit was aldea and household; unti of analysis was the household. Data was collected at the level of individual HH members, but aggregated to the HH level for analysis.
Universe
All aldeas in Honduras except Gracias a Dios Department, national parks and tourist areas (Islas de la Bahia Department), as well as any aldea that implementer had already entered
Producers and sponsors
Authoring entity
| Name | Affiliation |
|---|---|
| National Opinion Research Center (NORC) | University of Chicago |
Funding agencies
| Name | Abbreviation |
|---|---|
| Millennium Challenge Corporation | MCC |
Sampling
Sampling procedure
A two-stage survey design was used, in which a first-stage sample of 203 aldeas (villages) was selected, and a second-stage sample of households was selected from each sample aldea. The total number of aldeas in the sample frame (from the GIS, also from Census) was 3,675. After deleting aldeas in Islas de la Bahia and Gracias a Dios departments, those having 100% of caserios in protected status, and those already processed by Fintrac, the sample frame was reduced to 1,822 aldeas. These are the primary sampling units for the survey.
The sample sizes that were decided on were 113 treatment aldeas and 90 control aldeas, with an expected sample size of 9 program farmers and 20 other households in treatment aldeas, and 9 potential treatment farmers and 20 other farmers in control aldeas, for a total sample size of 203 aldeas and (expected) 203 x 29 = 5887 households in each survey round. This sample is constructed by selecting a sample of 113 matched pairs (226 units in all), randomly dividing them into treatment and comparison aldeas, and dropping 23 of the comparison aldeas (resulting in the desired sample size of 113 treatment and 90 comparison aldeas).
The sample sizes that were decided on were 113 treatment aldeas and 90 control aldeas, with an expected sample size of 9 program farmers and 20 other households in treatment aldeas, and 9 potential treatment farmers and 20 other farmers in control aldeas, for a total sample size of 203 aldeas and (expected) 203 x 29 = 5887 households in each survey round. This sample is constructed by selecting a sample of 113 matched pairs (226 units in all), randomly dividing them into treatment and comparison aldeas, and dropping 23 of the comparison aldeas (resulting in the desired sample size of 113 treatment and 90 comparison aldeas).
Deviations from sample design
It was not possible to implement the original design, for a number of reasons (documented in the Final Report). The final sample (for the first survey round) consisted of the responding part of the original (experimental) design (3,981 households) and an additional sample of 545 clients (households) from the program implementer's client list (who entered the program at the same time as the program farmers of the original design). The final responding sample size for the first round survey was 4,526 households (farmers). The design was a panel design in which it was attempted to reinterview, in the second survey round, every household that had been interviewed in the first survey round. The number of households interviewed in the second round was 2,736, for a total of 7,262 household interviews in both survey rounds.
Response rate
The sample sizes for the two survey rounds were 4,533 in the first round, with 4,533 responding, and 3,063 in the second round, with 2,736 responding.
Weighting
The probability of selection for each sample aldea is included in the file RecodedExtract.xls. The "base" survey weights are equal to 1/prob.
Data Collection
Dates of collection
| Start | End |
|---|---|
| 2008 | 2011 |
Data collection supervision
The data collection for each round of the baseline, as well as for the endline, was completed by 5-person field teams during 30 day data collection periods. Three senior technical supervisors oversaw each data collection effort and monitored progress on the ground during the entire data collection period. NORC provided the study sample for each round, along with any available geo-coding and contact information. INE used this information to organize the national data collection in the most cost-efficient manner possible, depending on the geographic dispersion of the cases.
INE required that interviewers review and code any completed interviews and provide them to the editor by the end of each working day. The editor reviewed the completed questionnaire within one working day and, if necessary, discussed questions or problems with the interviewer and the supervisor. This rapid review permitted the interview staff to return to a household if data retrieval or verification were required. Since an average of just 2 to 3 days was spent in each zone, it was critical that these reviews be conducted promptly so updates could be made before the team left the zone. Completed questionnaires were reviewed by supervisors and if complete, returned in regular shipments to the Central Office in Tegucigalpa for receipting and processing.
To assure standards of quality in the field, INE used evaluation forms to assess the performance of supervisors, interviewers and team editors (críticos) during each round of data collection. These instruments, which were administered by direct the supervisor for each of the aforementioned groups, collected information on a range of tasks performed by each group. The data gathered using these forms was used to respond quickly and efficiently to any issue that was identified in the field.
INE required that interviewers review and code any completed interviews and provide them to the editor by the end of each working day. The editor reviewed the completed questionnaire within one working day and, if necessary, discussed questions or problems with the interviewer and the supervisor. This rapid review permitted the interview staff to return to a household if data retrieval or verification were required. Since an average of just 2 to 3 days was spent in each zone, it was critical that these reviews be conducted promptly so updates could be made before the team left the zone. Completed questionnaires were reviewed by supervisors and if complete, returned in regular shipments to the Central Office in Tegucigalpa for receipting and processing.
To assure standards of quality in the field, INE used evaluation forms to assess the performance of supervisors, interviewers and team editors (críticos) during each round of data collection. These instruments, which were administered by direct the supervisor for each of the aforementioned groups, collected information on a range of tasks performed by each group. The data gathered using these forms was used to respond quickly and efficiently to any issue that was identified in the field.
Questionnaires
Household
Data collector(s)
| Name | Abbreviation | Affiliation |
|---|---|---|
| Honduran National Institute of Statistics (Instituto Nacional de Estadistica) | INE | Data collection firm |
Data Processing
Data editing
Once the “raw” survey data were available from INE, they were prepared for analysis by the ESA Consultores, the Honduras subcontractor. This cleaning and aggregation process is documented in detail in a series of Stata command (.do) files, Do*FTDAImpact.do (where “*” represents digits 1-11).
metadata.study_desc.method.data_collection.method_notes
For each round of data collection INE trained a team of 15 to 20 data entry clerks and two supervisors. INE would conduct 5 day-training of data entry staff prior to the start of data entry. Staff were expected to complete the data entry of 20 surveys per day during an 8 hour work day for the first week and then increase to as many as 25 per day as they became more familiar with the instrument.
They performed data entry using an in-house program, which was developed and tested by INE programmers and approved by MCA and NORC prior to the start of data collection. INE protocols require 100% double data entry. To ensure quality and detect any data entry errors, we required that each questionnaire be data entered twice, using different clerks for each of the two entries. Then, supervisors performed a reconciliation of all data entries to identify and correct any errors that were identified. The data entry program was designed to conduct consistency checks and perform a series of validation measures automatically. The next step in processing was to conduct a number of additional consistency and error checks. INE then generated frequencies and crosstabs in SPSS for validation. The data were delivered to the client within 6 – 8 weeks of the end of data collection in the field.
They performed data entry using an in-house program, which was developed and tested by INE programmers and approved by MCA and NORC prior to the start of data collection. INE protocols require 100% double data entry. To ensure quality and detect any data entry errors, we required that each questionnaire be data entered twice, using different clerks for each of the two entries. Then, supervisors performed a reconciliation of all data entries to identify and correct any errors that were identified. The data entry program was designed to conduct consistency checks and perform a series of validation measures automatically. The next step in processing was to conduct a number of additional consistency and error checks. INE then generated frequencies and crosstabs in SPSS for validation. The data were delivered to the client within 6 – 8 weeks of the end of data collection in the field.
Data Appraisal
metadata.study_desc.method.analysis_info.sampling_error_estimates
Standard errors are reported for all impact estimates presented in the final report. Standard errors were estimated using the "bootstrap" (resampling) procedure.
Data access
Citation requirements
Caldwell, Joseph, Fidel Ordoñez, Michael Reynolds, Varuni Dayaratna, John Felkner, Impact Evaluation of the Millennium Challenge Corporation Farmer Training and Development Activity in Honduras: Merged Baseline and Endline Household Survey Data, NORC at the University of Chicago, November 15, 2013.
Data collection locations
Millennium Challenge Corporation
Original archive where collection stored
Millennium Challenge Corporation
http://data.mcc.gov/evaluations/index.php/catalog/52
Cost: None
http://data.mcc.gov/evaluations/index.php/catalog/52
Cost: None
Contacts
Contact(s)
| Name | Affiliation | |
|---|---|---|
| Monitoring & Evaluation | MCC | impact-eval@mcc.gov |
Metadata production
IDNo
DDI_HND_2008_MCC-FTD_v01_M
Producers
| Name | Abbreviation | Affiliation | Role |
|---|---|---|---|
| NORC | NORC | University of Chicago | Review of Metadata |
Production date
2014-03-05
Version
Version 1 (Original 2014-03-05)
Version 2.0 (April 2015). Edited version based on Version 01 (DDI-MCC-HND-IE-AG-2014-v1) that was done by Millennium Challenge Corporation.
Version 2.0 (April 2015). Edited version based on Version 01 (DDI-MCC-HND-IE-AG-2014-v1) that was done by Millennium Challenge Corporation.