Social Assistance Grants for Empowerment Programme 2013, Evaluation Follow-Up Survey

<b> Overview </b>
In order to deliver the quantitative impact evaluation study design, a three-round longitudinal household panel survey was conducted. The quantitative survey was implemented in 398 clusters across 48 sub-counties in eight programme districts. The evaluation was conducted in eight of the 14 districts, in agreement with the programme. This was in order to minimise the operational burden of the evaluation on the programme, due to the requirement that the evaluation randomly assign targeting mechanisms between sub-counties within evaluation districts. Consequently, it was agreed to exclude from the evaluation the six 'new' districts that were created from the original districts when theadministrative boundaries were redrawn in 2010.

<b> Selection of Sub-counties </b>
Evaluation sub-counties were randomly selected from a list of sub-counties provided by the Uganda 2002 census. This list had to be adjusted to incorporate the 2010 sub-county boundary changes, with the populations of the new sub-counties provided by SAGE. The sample frame was thus comprised of the 74 sub-counties (as defined by the old administrative boundaries) in the eight programme evaluation districts, minus six that were excluded from selection for the evaluation.

Prior to selection, this list of 68 sub-counties was first randomly divided into two lists, one from which SCGsub-counties were drawn and one from which VFSG sub-counties were drawn. This random allocation of treatment was done to ensure a similar spread of sub-counties in both SCG and VFSG lists, allowing for rigorous comparison across the two targeting methodologies. The 24 SCG and 24 VFSG sub-counties to be covered by the evaluation were then randomly selected from the SCG and VFSG sub-county lists respectively. Sub-counties were selected using probability proportional to size (PPS).

The sampling of evaluation sub-counties had to account for the fact that in Karamoja only the SCG targeting mechanism was to be applied. To avoid sub-counties in the Karamoja region being over-represented in the SCG sub-county list, the list of VFSG sub-counties was not restricted to exclude those in the Karamoja region. Instead those Karamoja sub-counties that were randomly allocated to the VFSG sub-county list were then excluded, with the 24 VFSG evaluation sub-counties randomly selected from the restricted sub-county list.

<b> Selection of Evaluation Primary Sampling Units </b>
Within selected evaluation sub-counties a number of primary sampling units (PSUs) or clusters were drawn. The precise number of clusters depended on balancing a number of different factors: whether the unit was practically viable for use as a cluster for survey implementation; the population density of treatment and comparison households per cluster at the specified bandwidth; the number of clusters required at the specified bandwidth in order to achieve the proposed household sample size; and the number of clusters that it was financially viable to survey.

400 clusters (200 SCG; 200 VFSG) were randomly selected from across the 48 evaluation sub-counties, where the unit of cluster was the village, and using PPS based on the number of households within the bandwidth in each PSU. Due to the use of PPS and the relatively large size of a few villages compared to all the other villages, one SCG community was selected twice. Furthermore, during fieldwork it was found that two SCG communities in the sample frame that had been selected were in fact one community. This means that the final number of SCG communities is 198 and not 200, meaning that the final cluster sample comprised 398 discrete villages.

<b> Sampling of Evaluation Households </b>
From each of the 398 sampled villages, five treatment and five comparison group households were randomly selected for interview; with the exception of the two clusters that were sampled twice, from which 10 treatment and 10 comparison households were selected. In cases where insufficient treatment or comparison households were present within a particular village, the sample was re-distributed according to the following protocol:

• For low density villages that contain between six and nine evaluation households (i.e. treatment or comparison households within the evaluation bandwidth), replacements were taken from other sampled villages within the same sub-county. This was done by randomly selecting replacement households from the full list of households living in sampled evaluation villages in the same sub-county, that had not already been sampled.
• In order to minimise the negative effect of the redistribution of sampled households between clusters on the logistics of the fieldwork, we restricted the total number of households to be interviewed within a particular village to a maximum of 12 households.
• Extremely low density villages containing less than six households within the bandwidth in total (either treatment or comparison) were dropped from the sample frame. Analysis of the most recent available SAGE MIS data from the six pre-pilot sub-counties shows that this represents only a very small proportionof beneficiaries and villages.

<b> Selection of Control Communities </b>
A sample of 100 control communities was also surveyed in order to measure impact on a selection of community-level outcomes. The control communities survey did not include a household survey. The control communities were identified using matching techniques, which match treatment and control communities using characteristics drawn from the 2002 Uganda Census. The control communities are located across six control districts, chosen using the same rationale as was used to select the 14 pilot programme districts, to obtain maximum comparability. The six control districts selected were: Nakasongola in the Central region; Kamuli and Buyende in the Eastern region; Pader and Agago in the Northern region; and Kamwenge in the Western region.