The evaluation uses three methodologies to rigorously evaluate the causal impact of the program on outcomes. The first is a difference-in-difference methodology, whereby the project roads are matched to a set of similar comparison roads where no intervention has taken place. These comparison roads are chosen from a number of potential candidates using a propensity score matching technique. The difference-in-difference analysis thus compares traffic counts as well as socioeconomic outcomes for residents of communities located near the project roads to those of residents of communities located near the comparison roads. Secondly, the evaluation incorporates a continuous treatment approach. Project impact is modeled in a dose-response framework, so that communities nearer the project roads are assumed to experience greater impacts than those more distant. Finally, the evaluation estimates a matched difference-indifference model, using propensity score matching to improve the comparability between the treatment and comparison groups. Combining these three approaches allows for results from each to be compared in order to ensure a robust set of findings that is not dependent on the assumptions of one particular modeling approach.
Kind of data
Sample survey data [ssd]
Anonymized dataset for public distribution
The Samtskhe-Javakheti region
Unit of analysis
To collect the data, enumerators travelled to each settlement and worked with local authorities to identify a small group of individuals who were identified as knowledgeable about conditions in the settlement.
Producers and sponsors
National Opinion Research Center (NORC)
University of Chicago
Millennium Challenge Corporation
The sample for the first round used the 2002 Census to identify a sampling frame of 732 settlements around either the project or comparison roads, of which 690 were surveyed. The sample size was increased for the second and third rounds, which conducted surveys in all settlements that met at least one of the following criteria: settlements along the SJ Road; settlements along comparison roads where traffic counts are conducted; settlements included in the Integrated Household Survey (IHHS) that the evaluation uses to evaluate household-level outcomes, and any other village that was included in the baseline. The second and third rounds each included 960 settlements.
Our approach to selecting the comparison roads uses the technique of Propensity Score Matching (PSM) to identify eight comparison road segments to be included in the analysis. The comparison roads were selected from an inventory of 117 road segments for which data on a variety of characteristics was available from RDMED, the Georgian government roads agency. Our application of PSM in this case is to estimate a logistic regression model of the probability that a road is part of the treatment group as a function of observable characteristics. We then calculate the predicted probability (or propensity score) that a road segment is part of the treatment group for each of the eight treatment roads and 117 potential comparison roads using these estimated regression coefficients. Finally, each of the eight treatment roads is matched to a single comparison road for which the propensity score is the closest in value from among the 117 potential comparison roads.
Dates of collection
Data collection supervision
Miranda Berishvili, Giorgi Giorgadze, Sophia Chiburdanize, and Dea Tsartsidze
National Opinion Research Center
University of Chicago
National Opinion Research Center. Samtskhe-Javakheti Roads Activity Impact Evaluation Final Report. 2013.