It is important for ECD programmes to know whether they are making a difference to children’s development. The Early Learning Outcomes Measure (ELOM) is often used for this purpose. Children are measured early in the programme and then at a later point, and the growth in ELOM scores over this period is then examined. An improvement in ELOM scores can lead one to assume that the programme is making a difference. Unfortunately, this would be an incorrect conclusion because in addition to the programme hopefully making a difference, the children are ageing between baseline and endline assessments. And as ELOM performance improves with age, it is not possible to know whether the difference in scores is simply due to children maturing or to the effects of the programme. 

A key challenge then, is to separate out the effects of children’s maturation from the effects of the programme.  This is best addressed by comparing the amount of change in ELOM scores over time in children attending a programme (an intervention group), with those who are not enrolled in any programme or in a different programme (a comparison group).  Optimally one would randomly allocate children to each group prior to the investigation (a Randomised Controlled Trial design), but this is rarely feasible – particularly for small-scale programmes run by NGOs. An alternative is the Quasi-Experimental design (QED) in which random samples of children attending programmes are compared with those who are not (or attend other programmes). An example is the Early Learning Programme Outcomes (ELPO) study which compared the effects of three playgroup and two centre-based models on children’s development as measured by their ELOM scores. When we use this design, we can to a large extent separate out the effects of ageing from the effects of the programme itself on change in ELOM scores.

When ECD organisations do not have an opportunity to compare two groups (it is an expensive exercise), they cannot separate out maturation and programme effects. That is why when Innovation Edge provides partner ECD programmes with ELOM reports on their children’s progress over time, we have always pointed out the limitations of attributing the observed changes to the programme when there is no comparison group.

While the aim should always be to compare intervention and comparison groups, recent research conducted by Servaas Van den Berg (2021)[1] has very helpfully provided us with a way of distinguishing between maturation and programme effects when there is no comparison group. We outline this approach below.

Separating maturation and programme effects without a comparison group

Van den Berg analysed large ELOM survey datasets and the ELPO study housed with DataFirst to compare outcomes expected from maturation alone with the amount of change that could be attributable to the ELPO study programmes. His analysis showed that ELOM Total scores increase by about 1.02 ELOM Total score points per month due to maturation. We can now calculate the difference in ELOM Total score points between the change expected due to children’s natural maturation over the course of a programme from the change in scores observed from baseline to endline. If the difference is greater than zero, that is the contribution of the programme.

In Table 1 we use van der Berg’s analysis of data from the five ELPO study programmes, to provide an example of how we can separate out the contributions of maturation and the children’s early learning programmes to change in ELOM Total scores from baseline and endline (a period of eight months).

Table 1: Comparison of maturation effects relative to ELPO study programme outcomes

Mean ELOM Total Scores
a) Average ELOM Total Baseline scores40.031.625.949.435.2
b) Average ELOM Total Endline scores51.246.143.763.955.9
c) ELOM Total Change over 8 months (b-a)11.214.517.814.520.7
d) Maturation Effect: ExpectedELOM Total Endline score due to maturation alone over 8 months: = a)+8.16*48.6139.7634.0657.5643.36
e) Programme Effect: Difference between ELOM Total Change (c) and Maturation Effect (d) = c) – d)3.046.349.646.3412.54
+Prog. = Programme. * We expect a change due to maturation alone of 1.02 ELOM Total score points per month; In the ELPO study, the interval between baseline and endline assessments was 8 months; therefore  8.16 points was added to the baseline scores to obtain the Expected Endline score due to ageing alone (the maturation effect).

Table 1 shows that using the estimate of a gain due to the child’s maturation (ageing) (row d) alone of 1.02 ELOM Total score points per month, all the ELPO programmes contributed gains independent of maturation effects (row d) of between 3.04 and 12.54 ELOM Total score points over 8 months (row e).

In sum, when you do not have a comparison group, the difference between the Maturation Effect (d above) and the Programme Effect (e above) can be calculated and used as illustrated in Table 1. This procedure is not as scientifically rigorous as results from the designs discussed earlier but is acceptable for practical purposes.

A final word: it is very important to note that this is a separate exercise to establishing the extent to which programmes enable children to achieve the ELOM Standard. Children may improve more than expected by their maturation over the course of the programme but may or may not achieve the ELOM standard at the endline assessment. That would depend on how close their scores were to the standard for their age at the start of the programme and the extent of change in their ELOM Total scores.

[1] Van den Berg, S. (2021). Estimating the impact of five early childhood development programmes against a counterfactual. ECD Working Paper Series. Ilifa Labantwana & Resep. No. ECD WP 001/2021.