Social Assistance Grants for Empowerment Programme 2013
Evaluation Follow-Up Survey
<b> Objectives of the Project </b>
The data files described in this documentation correspond to a household sample survey carried out in three rounds (baseline in 2012, follow up 1 in 2013 and follow up 2 in 2014) with the objective of evaluating the impact of the Uganda Social Assistance Grants for Empowerment (SAGE) programme in 14 pilot districts across the Eastern, Central, Western and Northen districts in Uganda.
The impact evaluation has the following objectives:
- Assess the impact and operational effectiveness of the SAGE pilot programme.
- Compare the relative performance of the two targeting methodologies used in the pilot.
The evaluation assesses SAGE against its main objectives of empowering recipient households through:
- Reducing material deprivation
- Increasing economic security
- Reducing social exclusion
- Increasing access to services.
<b> What is the SAGE Programme?</b>
The SAGE programme transfer is currently worth UGX 25,000 per month and is paid every two months. This transfer is provided using electronic transfers. The two targeting methodologies being implemented are as follows:
- Vulnerable Family Support Grant (VFSG) - This employs a composite index based on demographic indicators of vulnerability to determine eligbility.
- Senior Citizen Grant (SCG) - This uses age to determine eligibility.
Under the VFSG, if they are present in a beneficiary household, adult women are selected by the programme to be the physical recipient of transfers. In the case of SCG, the transfer is given to the specific older person enrolled.
Anonymous datasets for public distribution
The data covers the 14 pilot districts selected for the SAGE programme pilot districts. These districts were as follows:
Kiboga, Katwaki, Kaberamaido, Kyenjojo, Nebbi, Apac, Moroto, Nakapiripirit, Kole, Zombo, Amudat, Napak, Kyegegwa and Kyankwanzi.
However, for the evaluation the original 8 districts were selected (Apac, Kaberamaido, Katakwi, Kiboga, Kyenjojo, Moroto, Nakapiripirit and Nebbi).
Unit of analysis
The basic unit of analysis is at the household level. However, some datasets contain individual level data, whilst some data is collected at the community level.
The quantitative survey was implemented in 398 clusters across 48 sub-counties in eight programme districts (Apac, Kaberamaido, Katakwi, Kiboga, Kyenjojo, Moroto, Nakapiripirit and Nebbi). The two targeting mechanisms (SCG and VFSG) were randomly assigned evenly between the 48 sub-counties, with the exception of the Karamoja region in which only the SCG targeting mechanism was employed. The SAGE programme implemented the targeting process in evaluation areas where selected recipients received the transfer, but only after they were surveyed at baseline by the evaluation teams.
The households in the evaluation areas that were selected for the programme are referred to as the treatment group. The comparison households that were selected were those households that fell just shy of the selection thresholds. For SCG the threshold is 65 years of age, dropping to 60 years of age in the Karamoja region. For VFSG eligibility depends on a household's labour capacity and dependency (LCD) score, with the threshold score for eligibility varying by region. Households who fell just below the relevant SCG/VFSG threshold are referred to as the comparison group.
Producers and sponsors
Oxford Policy Management Ltd.
Oxford Policy Management Ltd.
Research Guide Africa
Department For International Development
UN Children's Fund
<b> Overview </b>
In order to deliver the quantitative impact evaluation study design, a three-round longitudinal household panel survey was conducted. The quantitative survey was implemented in 398 clusters across 48 sub-counties in eight programme districts. The evaluation was conducted in eight of the 14 districts, in agreement with the programme. This was in order to minimise the operational burden of the evaluation on the programme, due to the requirement that the evaluation randomly assign targeting mechanisms between sub-counties within evaluation districts. Consequently, it was agreed to exclude from the evaluation the six 'new' districts that were created from the original districts when theadministrative boundaries were redrawn in 2010.
<b> Selection of Sub-counties </b>
Evaluation sub-counties were randomly selected from a list of sub-counties provided by the Uganda 2002 census. This list had to be adjusted to incorporate the 2010 sub-county boundary changes, with the populations of the new sub-counties provided by SAGE. The sample frame was thus comprised of the 74 sub-counties (as defined by the old administrative boundaries) in the eight programme evaluation districts, minus six that were excluded from selection for the evaluation.
Prior to selection, this list of 68 sub-counties was first randomly divided into two lists, one from which SCGsub-counties were drawn and one from which VFSG sub-counties were drawn. This random allocation of treatment was done to ensure a similar spread of sub-counties in both SCG and VFSG lists, allowing for rigorous comparison across the two targeting methodologies. The 24 SCG and 24 VFSG sub-counties to be covered by the evaluation were then randomly selected from the SCG and VFSG sub-county lists respectively. Sub-counties were selected using probability proportional to size (PPS).
The sampling of evaluation sub-counties had to account for the fact that in Karamoja only the SCG targeting mechanism was to be applied. To avoid sub-counties in the Karamoja region being over-represented in the SCG sub-county list, the list of VFSG sub-counties was not restricted to exclude those in the Karamoja region. Instead those Karamoja sub-counties that were randomly allocated to the VFSG sub-county list were then excluded, with the 24 VFSG evaluation sub-counties randomly selected from the restricted sub-county list.
<b> Selection of Evaluation Primary Sampling Units </b>
Within selected evaluation sub-counties a number of primary sampling units (PSUs) or clusters were drawn. The precise number of clusters depended on balancing a number of different factors: whether the unit was practically viable for use as a cluster for survey implementation; the population density of treatment and comparison households per cluster at the specified bandwidth; the number of clusters required at the specified bandwidth in order to achieve the proposed household sample size; and the number of clusters that it was financially viable to survey.
400 clusters (200 SCG; 200 VFSG) were randomly selected from across the 48 evaluation sub-counties, where the unit of cluster was the village, and using PPS based on the number of households within the bandwidth in each PSU. Due to the use of PPS and the relatively large size of a few villages compared to all the other villages, one SCG community was selected twice. Furthermore, during fieldwork it was found that two SCG communities in the sample frame that had been selected were in fact one community. This means that the final number of SCG communities is 198 and not 200, meaning that the final cluster sample comprised 398 discrete villages.
<b> Sampling of Evaluation Households </b>
From each of the 398 sampled villages, five treatment and five comparison group households were randomly selected for interview; with the exception of the two clusters that were sampled twice, from which 10 treatment and 10 comparison households were selected. In cases where insufficient treatment or comparison households were present within a particular village, the sample was re-distributed according to the following protocol:
- For low density villages that contain between six and nine evaluation households (i.e. treatment or comparison households within the evaluation bandwidth), replacements were taken from other sampled villages within the same sub-county. This was done by randomly selecting replacement households from the full list of households living in sampled evaluation villages in the same sub-county, that had not already been sampled.
- In order to minimise the negative effect of the redistribution of sampled households between clusters on the logistics of the fieldwork, we restricted the total number of households to be interviewed within a particular village to a maximum of 12 households.
- Extremely low density villages containing less than six households within the bandwidth in total (either treatment or comparison) were dropped from the sample frame. Analysis of the most recent available SAGE MIS data from the six pre-pilot sub-counties shows that this represents only a very small proportionof beneficiaries and villages.
<b> Selection of Control Communities </b>
A sample of 100 control communities was also surveyed in order to measure impact on a selection of community-level outcomes. The control communities survey did not include a household survey. The control communities were identified using matching techniques, which match treatment and control communities using characteristics drawn from the 2002 Uganda Census. The control communities are located across six control districts, chosen using the same rationale as was used to select the 14 pilot programme districts, to obtain maximum comparability. The six control districts selected were: Nakasongola in the Central region; Kamuli and Buyende in the Eastern region; Pader and Agago in the Northern region; and Kamwenge in the Western region.
Weights were given by the inverse of the probability of being selected. The household's probability of selection was broken down into two component parts:
1) the probability of selection of the PSU; and
2) the probability of being selected into treatment and comparison groups from the list of all possible SAGE eligible and non-eligible households within the specified bandwidths in that PSU.
In the calculation of the survey weights we ignored the probability associated with the selection of the evaluation sub-counties. Doing so reduces the variance of the final weights, thereby reducing the variance of point estimates and increasing the likelihood of detecting impact should the SAGE programme impact key outcome indicators. Furthermore, 48 out of a total 68 sub-counties have been included in the evaluation, meaning that the evaluation sample of sub-counties is already very representative of the total pilot population of sub-counties.
We defined the two component probabilities as follows:
<b> P1 </b>: Probability of a PSU being selected. PSUs were randomly selected using the PPS techniques separately for SCG and VFSG areas, drawn from a sample frame of all PSUs within evaluation sub-counties.
<b> P1 </b> = Number of households in bandwidth in PSU/total number of households in badwidth in evaluation sub-counties
<b> P2 </b>: Probability of being selected from the full list of treatment or comparison group households within a PSU (depending on whether household was a treatment or comparison household)
<b> P2 </b> = Number of sampled treatment or control households in PSU/total number of treatment or control households in PSU
The final probability of a household being selected for the SAGE baseline survey was calculated by combing <b> P1 </b> & <b> P2 </b>, as follows:
Probability of selection = <b> P1 x P2 </b>.
Thus, the final analytical weights applied to each household were constructed by taking the inverse of the probability of selection.
Dates of collection
First Follow-up data collection
Mode of data collection
Data collection supervision
There were a total of 8 team, each with a supervisor. There was also a roving quality assurance team that circulated between the teams, which provided support and monitoring. Additionally, international staff did some fieldword support/supervision, particularly at the start of each survey round.
In each round there was a household questionnaire which was administered to each household and collected the household and individual level information. This contained modules on household characteristics, education, health, activities of the household members, migration, housing conditions, empowerment, assets, land and livestock, subjective poverty, saving and borrowing, formal/informal transfers, hunger scale, consumption, operational and targeting, anthropometry. In addition a community questionnaire was conducted in each community. This includes modules on communities access to facilities, wages, local goods and prices.
Research Guide Africa
Ipsos Synovate Uganda
Once data had been entered, the data was exported and sent to OPM. This data was checked in STATA, which produced a list of queries which were responded to by going back to field teams and households. Data was then revised and sent back to OPM to be checked once again. This whole process was repeated iteratively until all queries that could be resolved were.
The questionnaire was conducted on paper. This data was then double entered in Nairobi by RGA using CSPRO.
Public use files, accessible to all after registration
Merttens, Oxford Policy Management. Evaluation of the Uganda Social Assistance Grants for Empowerment (SAGE) Programme: Impact after two years of programme operations 2012-2014. Ref. UGA_2013_SAGE-ML_v01_M. Dataset downloaded from [URL] on [date].
Disclaimer and copyrights
The user of the data acknowledges that the original collector of the data, the authorized distributor of the data, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses.