A recent poll conducted by Thomson Reuters Labs is in the news right now. The poll asked 7 questions of 548 alleged experts on how they perceive 193 countries with respect to women’s health care, sexual violence, non-sexual violence, discrimination, cultural practices (by ‘western’ standards) and human trafficking. It was conducted ‘online and via email and telephonic interviews’. The answers to these questions were weighted and the results published. The results have been selectively presented in various e-papers in ways that push whatever narrative that particular e-paper’s editors want to push. So this is not a commentary on the latter. This article looks at the poll’s methodology (available here) and points out why, although the results are telling us important things about biases in Reuters Labs, they are most definitely not telling us any of the things Reuters claim they are!How were these ‘experts’ picked?

The poll allegedly asked 548 ‘experts in women’s issues’ to answer questions, often over the phone. These are people in funding agencies, universities, health camps, policy, NGOs, journalism and other forms of journalism (social commentators).

This was not an open poll that anyone could answer (even within these agencies) and these experts were presumably selected based on whether folks at Thomson Reuters knew them or had heard of them.

This is a very strict no-no in data sampling because people one has heard of (and remembers) are usually part of the same echo chamber, so their opinions will usually agree with yours and generally be worthless as unbiased data.

However, since only 548 people were asked out of a world population of more than 7 billion, which is the opinion of less than 0.000008% of the people on the planet, it is rather worthless anyway.

But assuming the selection process was logical and careful thought had gone into it (even though we cannot fathom how), we come to the next question:

Language?

We are talking about global perception here. So going by the number of speakers of each language (links here and here), the global poll ought to have been conducted in the 5 most spoken languages in the world, which are Mandarin, Spanish, English, Hindi/Urdu and Arabic, in that order.

Clearly, restricting the language of the questions to English or including any of the other European languages would result in a bias towards people around the world who are familiar with the ‘western world’ (studied / worked/ lived there, live in former colonies that still glorify the colonial times, get invited to conferences there, etc.) and who would therefore be biased about it, most probably favourably so.

This brings us to the next question:

Equally distributed in all continents?

The poll methodology claims the experts are equally distributed across all the continents. However, people are not equally distributed over all continents. Has this been factored into the methodology? From the description, this seems unlikely.

On the other hand, asking more people from Asia, would naturally result in a bias towards Asia and might explain the inordinately high number of Asian countries that figure in that list.

More importantly, would an English speaking expert (in women’s issues) in say the Democratic Republic of Congo (formerly Zaire - English isn’t an official language), be more likely to be a well-read, Professor of gynaecology or a part-time, American missionary filling a gap year between school and college? This is not clear at all.

Data points

To come back to the main problem with this poll though, since it is a global perception index, the method of data collection ought to have been considered carefully.

The first question to ask is why the perception of 548 people (my engineering class usually had about 600 students in every lecture), is news?

If these were 548 people from the 548 top companies in the world, it might significantly affect trade. If they were 548 leaders of the 193 countries considered for the poll, it might affect policies around the world. If they were 548 of the top engineers in the world, it might significantly affect technology solutions that are put out this year. 548 top medical & pharmaceutical professionals polled – significantly change healthcare solutions and so on

But who are 548 ‘experts on women’s issues’ and why is their opinion of greater value than the opinions of the people listed in the previous paragraph (and everyone else for that matter)? This is again, not clear.

Here are some links to some actual statistics on the issues discussed in the poll’s questions:

WHO data on health care here and data on maternal mortality rates here (which correspond fairly closely with each other). These are mostly the poorest countries of the world that most of us would struggle to find on the map.

Data on sexual violence here (amongst ‘western countries’, Sweden, Australia, Belgium and the USA feature in the top 10. Of these, only USA features in this poll as well, in 3^rd place, clearly showing that more than anything else, not nearly enough people were asked)

Data on human trafficking here (opens a pdf. Worst offenders are countries such as Comoros, Eritrea, Burundi, Mauritania…)

Data on non-sexual violence here and here. Here again, most of the countries in the top 10 are relatively unknown in the ‘western’ / ‘westernised’ parts of the English-speaking world population, which might explain why they don’t feature much in this poll.

There is another big difficulty with having such a small data set and so many possible answers (193 possible answers to each question, 548 questionnaires). This is the difficulty in finding any one country that has the maximum “votes” to any question at all! Even assuming those polled actually knew the names of all 193 countries and could correctly place each on a map, even then, based on which continent they were from, the answers would have differed greatly (this is why an actual serious poll has sample sizes in the hundreds of thousands). To get over this difficulty, the pollsters have made the next big data-faux-pas!

Weighted answers? Really?

The first  question in the poll asks “Which are the five most dangerous country in the world for women?” and this gets the highest weightage in the poll (so all the other answers have way less influence on the end result).

Since the ‘experts in women’s issues’ are presumably not all experts in geography as well, they in all likelihood picked countries that they have visited / heard of a lot, but don’t themselves live in. For most of us, home is safe. We speak the local language (or know how to circumvent speaking it safely), we have things to do and we are usually happy we live here and not in distant country ‘XYZ’ where apparently crime rates are high according to newspaper reports that we read for entertainment.

Now for this author, living in India, that country ‘XYZ’ is of course, USA. One can easily imagine that for an expert living in the USA, that country is likely to be India. It isn’t El Salvador or Belize because this author will have to google to figure out what continent they are on and who is going to do that much research when answering a quick random poll (‘this will only take a minute of your time’), that too telephonically?

India on your mind & confirmation bias

In fact, anyone who has travelled to a country like India where many people speak English to you but then turn away and speak a series of different languages to each other, would have a hard time fighting the rising but perfectly normal ‘are they talking about me?’ panic. As a woman who has travelled alone in a bunch of non-English speaking countries, this author is quite familiar with the fear that things could suddenly go south while one is blissfully unaware of the danger because one doesn’t understand what is going on.

So, assuming that a significant number of these 548 ‘experts in women’s issues’ have travelled to India and Pakistan (looks great on the CV and on facebook) while another significant number are actually from India but now live elsewhere (so your home country is now country ‘XYZ’), they would naturally (and in all good faith) pick India without realising that every ‘Indian men are rapists’ article that they come across and believe, is in fact a classic example of the good old confirmation bias doing it’s work. Essentially, a lot of people are likely to have picked India over Comoros or Burundi because they have heard about India and noticed it a lot more than they have Comoros or Burundi.

Once again, these are standard pitfalls that any survey deals with and the main way they deal with it, is by asking very many people so that all these biases even out. They also recognise that if you don’t have enough data points, the correct thing to do is to go out and collect more data while the wrong thing to do is to give weightage to meaningless questions and send out newspaper articles on the strength of that.

To summarise and drive the point home, the lack of data points makes the poll completely irrelevant. In the famous words of Wolfgang Pauli “That is not only not right; it is not even wrong”!

And speaking of perception, the global perception of journalists is already rather low. In fact for most people, there is no difference between a mainstream media journo and a copy writer for an ad agency. Polls such as this and the extremely biased misreporting that follows, do nothing to dispel that image. Maybe that is the first perception that ought to be indexed via a statistically significant global poll?