1/11/2017

40.3 million slaves? Four reasons to question the new Global Estimates of Modern Slavery

 

Four years ago the Walk Free Foundation (Walk Free) published the first ever Global Slavery Index (GSI), with the goal of spurring the international community to action by quantifying human exploitation. For an issue to exist, Bill Gates apparently advised Walk Free’s founder Andrew Forrest, it must be measurable. And if something can be measured, so can the world’s progress be towards eliminating it.

This first attempt, published in 2013, drew harsh methodological criticism. The data underlying the index was much too flimsy for the foundation of a global ranking. In the attempt to attract attention, critics argued, Walk Free had prioritised shocking headline figures over due diligence. Walk Free acknowledged these criticisms but stuck to its guns by broadening the GSI empirical basis for the 2014 and 2016 editions. However, the key weakness remained: surveys about the prevalence of modern slavery existed only for a minority of countries around the world, and the GSI simply extrapolated from the ones it examined to the rest of the globe.

2017 marks a new departure for the Global Slavery Index, and a huge boost in legitimacy for Walk Free. With its advocacy, it has attracted global attention to the plight of millions of people around the world. The GSI in particular has proven hugely effective in focusing attention on the issue. So for the first time this year, its index is fused with data from the International Labour Organisation (ILO). With additional support from the International Organisation for Migration, Walk Free and the ILO issued a new joint report in September: the Global Estimates of Modern Slavery (GEMS).

The stamp of approval from these two UN-associated institutions places the GEMS at the centre of planning around the sustainable development goals (SDGs) – the global master plan to promote human well-being and sustainability of the planet’s ecosystems – as it directly relates to target 8.7 on the elimination of forced labour, modern slavery and child labour. According to the ILO, “[the] 2017 Global Estimate of Modern Slavery will provide benchmark figures against which progress of global efforts to eradicate modern slavery can be measured”. This is remarkable. Within four years the work of Walk Free, despite continuing and vocal reservations over its quality, has become an official policy tool of the global agenda.

This elevated status of the GEMS invites debate and scrutiny on a new level. As the report notes, “[to] be effective, policies and programmes must be grounded in the best possible understanding of the root causes of modern slavery at both the national and global levels" (p. 15). I couldn’t agree more, and therein lies the problem. We must hold tools like indices and indicators to the highest standards because they are intentionally designed to shape the behaviour of governments, international organisations, and citizens around the world. GEMS will act as a benchmark for the future evolution of modern slavery, so the scope for continually updating the methodology has shrunk – otherwise figures over time would be impossible to compare.

So as long as significant data problems persist – and important ones do – it remains a deficient yardstick for progress. If GEMS measurements are skewed, the policy prescriptions based on them will be skewed as well. The report’s authors should expect governments to take notice of its findings and to consider policies that will make them look better – so if the measurements themselves are off, policies may equally veer in an undesirable direction. The stakes in the struggle against exploitation are too high to tolerate misleading conclusions.

Limited source data and extrapolation

One of the most serious criticisms of the early reports centred on the global extrapolation from random sample surveys that existed for only 19 countries. How do we know that findings for some South East Asian countries hold true for others? The short answer is of course that we don’t. We can make informed guesses, based for example on the economic profile of countries. But those are guesses, not more.

The current GEMS features survey data for roughly a quarter of all countries (48 out of approximately 200). It leaves unclear how global and regional estimates were generated from that amount of data. The methodological annex in the report itself offers no detail, and the separate methodology guide promised on the Alliance 8.7 website is still “coming soon” at the time of writing – several weeks after the report itself had spread around the world.

But even without a methodology paper there is reason to treat the results with caution, as the GEMS itself repeatedly suggests that its underlying data may be rather weak. It states, for example, that data on "[forced] labour imposed by state authorities was derived from validated sources and systematic review of comments from the ILO supervisory bodies with regard to ILO Conventions on forced labour" (p. 11-12). That is vague, to say the least. It remains unclear what these sources are, how they were validated, and by whom. After all, even governments themselves – officially committed to the fight against forced labour – will be loath to incriminate themselves. At a time when getting the issue on the agenda was the main goal, that may have been good enough. With GEMS’s direct link to SDG 8.7, it no longer is.

Equally, the report offers caveats about the coverage of some parts of the world: "The regional figures are important but should be interpreted with care, bearing in mind critical gaps and limitations of the data. This is especially the case in Central Asia and the Arab States, where few surveys have been conducted despite numerous reports of forced labour and forced marriages occurring. Far more research and survey work is required at the national level to provide a more comprehensive picture".

On the one hand, such honesty is laudable. On the other hand, it should set off readers’ alarm bells. Despite acknowledging “critical gaps and limitations in the data” for the Arab States, as the grouping is called, the report nevertheless lists figures for the prevalence of forced marriages and forced labour in them. These figures, in turn, feed into the global estimates, thereby tainting the accuracy of the headline numbers as well.

Reports with as much political weight as the GEMS should not be papering over data gaps, playing fast and loose with extrapolation, or adding bad data to good simply because nothing better was available. Would it not have been appropriate simply to leave countries or regions with “critical gaps and limitations of the data” blank on the map? It would have made the map – and the global estimate – incomplete, but it would have more accurately reflected what was believed to be known. And how critical are these data gaps anyway? At present, there is no way for me as a reader to find out.

There is a bigger problem, however. In the reporting of statistics more generally, disclaimers about data quality and other warnings to data users quickly get lost. Activists, politicians and journalists are interested in ranks, headline figures, and neat maps, not in the ‘ifs’ and ‘buts’ of data collection. When an organisation publishes a snappy report, it should be aware that the figures contained therein will lead a life of their own, without health warnings attached.

Search “global estimates modern slavery” on Twitter and – unsurprisingly – you will mostly find the 40 million aggregate number. (Even the future, 280-character limit on the platform does not allow much more nuance than that.) As it stands, it is not clear that the authors are themselves convinced that the published figures are solid enough to stand on their own, without all the qualifiers. It is thus fair to ask how responsible it is to publish them – in our world of instant and unstoppable digital propagation – especially now that they are linked to policy through the SDGs.  

This is an extract of an OpenDemocracy report. Read the full report here.

Republished under CC BY-NC 4.0. Image: Randomthoughtstome.

Comments