Op-Ed: Science vs West: When experts buy bad science

The stakes are too high for forests and communities.

Robust science is built on thorough investigation and rich, diverse debate. Without these fundamental ingredients, we have little hope of achieving an equitable path towards ending deforestation or fighting climate change.

We welcome Calyx’s perspective on our analysis of REDD+ baselines, which explores critical questions about the REDD+ mechanism that should help guide its future as we work to scale it effectively and with the urgency demanded by the climate and biodiversity crises. What is concerning however, is the apparent un-tested ‘buy-in’ of West et al’s report by Calyx, and many others, without the robust scrutiny we would expect of a community of experts who are dedicated to having legitimately strong scientific foundations for our work.

We recognise that West et al’s paper has been peer reviewed and printed in one of the most prestigious journals on the planet. While we, as a community of experts, should be able to rely on that as a strong indication of the quality of the report, it is inherent to the culture and advancement of science to undertake critical reviews of published work. Our review indicates that this work has serious problems that fundamentally call into question the results.

West et al’s findings are based on serious methodological and analytical flaws, as well as data inaccuracies, miscalculations, and a misunderstanding of the fundamental drivers of forest loss and how REDD+ carbon crediting works. When that is accepted as robust science without proper scrutiny, everyone loses. But communities on the frontlines of the climate crisis lose the most.

Everland is preparing a detailed analysis of West et al at the individual project level, which we will publish in due course. However, in the meantime, we feel it is important to point out a few critical errors we have already identified in West et al’s approach.

5 Fundamental Flaws in West et al’s analysis

1. Basic mathematical error and over-simplified methodology: Inaccurate reporting of project baselines by the authors resulted in their use of significantly higher baseline deforestation rates for analysis than the projects actually operate under and generate avoided emissions in relation to. In one example, West et al appear to have made a basic error in their calculations that resulted in a significant overstatement of cumulative deforestation: While the actual cumulative baseline (2009-2018) reported in the Project Design Document for Madre de Dios REDD+ Project (ID 844) is 21,982.8ha, West et al mistakenly added together the cumulative baseline deforestation numbers for each year – resulting in an inaccurately reported value of a value of 125,501ha. Furthermore, in estimating the project emissions reductions, rather than using the verified values, the authors calculated the emissions reductions themselves using an over-simplified method. This resulted in an overestimation of the actual amount of emission reductions the majority of projects have been credited with. On average, this oversimplification created an overestimation of verified emission reductions of nearly 1Mt of CO2e per project on average, with a total overestimation of >17.6 million tCO2e. Combined with the overstatement of project baselines, this significantly exaggerates the climate impacts that REDD+ projects are actually credited for, undercutting the validity of West et al’s conclusions.
2. Omission of highly material factors: The authors used a number of variables to identify potential ‘control’ areas that are statistically “similar” to a project area. However, they failed to include four key predictors of deforestation risk; road networks, navigable rivers, population density and proximity to previous deforestation. The authors also failed to account for other critical factors in predicting deforestation rates, comparing projects established in areas that would otherwise be 100% logging concessions to areas with no logging concessions. In another example, two project areas in West et al’s research were located in areas with 0% protected area coverage, but were matched with control areas where more than half the land was covered by legal protection. Deforestation in protected areas is typically much lower than in adjacent unprotected forests. These are highly material omissions that necessarily skew the data towards incorrect conclusions. No doubt this is precisely why the VCS standard requires projects include key indicators that have been missed by West et al.
3. Misunderstanding of deforestation risk: Furthermore, the authors failed to weight their variables based on the relative risk they pose to deforestation within the context of a specific project. In other words, they failed to account for the specific factors driving deforestation for a given project. As a result, in the majority of cases the project is surrounded by more extreme deforestation than the selected control areas, meaning they’re at greater risk of imminent forest loss. Failure to understand the specific local context underlying deforestation risk is a failure to recognise the foundational baseline scenarios and theory of change underlying REDD+ interventions.
4. Inadequate data: West et al’s analysis uses open-source geospatial data (Hansen et al 2013) in selecting control sites, validating their model and estimating verified emission reductions. This dataset is known to be highly uncertain in some key areas, including  Sub-Saharan Africa (error: ±79%) and the Democratic Republic of the Congo (error: ±65%) (Tyukavina et al. 2015). Projects are required to use far superior geospatial imagery and on-the-ground verification for baseline setting and monitoring, and account for data error where it exceeds 15% by reducing the number of credits it can issue. Using inadequate data creates an apples-to-oranges comparison that cannot be relied on as robust science. And no discerning buyer would consider this to be robust due diligence.
5. Lack of transparency of control areas: The authors did not map or otherwise indicate where the selected control areas were in relation to the project areas, providing no way to evaluate the extent to which they are located within socio-economic and biophysical regions that are similar to the project areas. This is required by the VCS standard when project developers select and propose a Reference Area as a control as part of the project validation, which is then independently audited. Without this level of transparency, it is impossible to independently verify the reasonableness of the control sites, which is the underpinning of the study’s methodology and findings.

The flaws in West et al’s approach are so fundamental we do not believe that this study can rightly be classified as ‘robust science’. As an industry, we have a duty of care to ensure we adapt and evolve our standards and methodologies in line with robust, best-in-class scientific findings. This is critical to ensure REDD+ projects continue to work as an effective tool against deforestation.

And that means we have to apply the appropriate level of rigor in evaluating which scientific findings should shape the future of REDD+, and which should not. If we get this decision wrong the implications for communities on the front lines of the climate crisis and our ability to end deforestation globally are both real and disastrous.

(Everland will publish a detailed analysis of West et al in due course.)