Poll vault: Television rating points and the US polls
November 10, 2020 6:00 AM
The US election, and the TV rating system highlight the profound methodological deficiencies that exist in the design and execution of statistical sampling
For example, the UK has its Broadcasting Audience Rating Board (BARB) that uses and combines two independent data sources to deliver reach of program audiences across multiple screens.
By Soumya Kanti Ghosh & Bikramjit Chaudhuri What is common between television rating points (TRP) ratings & the US elections? Well, both highlight the apparent fault lines about the statistical system in India and the US. Rating systems exist the world over, though sampling methodologies differ. In a similar vein, predicting election outcomes is not an exact science. The problem arises when both such TRPs and poll polls point to a certain direction and turn out to be wrong; it’s worth asking what caused the error.
For example, the UK has its Broadcasting Audience Rating Board (BARB) that uses and combines two independent data sources to deliver reach of program audiences across multiple screens. The US is dominated by Nielsen, that covers 80,000 households, but with smaller families, the total number of viewers tracked is much less than two lakhs. India’s BARC has set up 44,000 bar-o-meters in selected television households to calibrate the choices of some two lakh viewers because family sizes are larger.
In fact, under the TAM rating system that dominated the television scene prior to BARC’s arrival in 2015, the number of people-meters was only 8,000 for all of India.
A criticism against BARC is that 44,000 meters can hardly claim to represent the views of nearly 20 crore television households. True, though all sample surveys have similar limitations. The Retail measurement, consumer panel, readership, employment, and the opinion poll, exit polls surveys the world over bear ample testimony to that and India is no isolation.
BARC had been tasked by the government to install 55,000 meters by 2021, while the Telecom Regulatory Authority of India (TRAI) had recommended an increase to 60,000 now and 100,000 by the end of 2022. One may still argue that a few thousand additional meters may not make much of a difference, but sampling is about just ensuring the right representations of the population, based on sound statistical principles and honest execution.
In India, such sample households are selected through an ‘establishment survey’ of 3 lakh TV households that BARC conducts every few years. Design of this establishment survey is claimed to be based on scientific principles on agreed socio-economic and other important criteria, and the selection processes are claimed to be completely randomised. Besides, one-third of selected households are expected to be on rotation every year to avoid vested interests from accruing.
To understand how many of the 44,000 meters may be spiked to tamper with that TV segment’s TRP, let us see how viewership patterns emerge from different segments of television.
As an illustration, let’s look at only the ‘English news’ genre. The “English news” constitutes a very minuscule audience—on an average only 0.02% of all the 80 crore or so television viewers of India. Based on one of the latest estimates, English news TV received some 1.2 crore impressions, which is 0.03% of the total TV impressions but corners 1.2% of television revenues, because this segment has greater purchasing power. To compare numbers, we find that Hindi film channels scoop 10% of TV revenues and Hindi news channels get some 6%.
Thus, purely statistically “English News” genre needs a much smaller number of viewers to display a TRP swing in favour of one channel. That exactly is the vulnerability of such an instrument, and it is paramount that the users and consumers of these reports are highly cognizant of that.
In a similar vein, predicting election outcomes is not an exact science, and every poll has a margin of error. In 2016, most polls showed Hillary Clinton clinching the presidency. In 2020, the general sentiments expressed in all the polls leading to this presidential election were that Democrats would have a cakewalk and a landslide and sweeping victory.
In 2016, undecided voters decided to cast their ballots in larger numbers in favour of Republicans as the race ended, and pollsters did not make a crucial adjustment in their calculations: white male voters minus a college degree. This time, the theory going around has been a ‘silent’ Republican voter who is shy of opening to pollsters but enthusiastically turns up at the polling booth on voting day. This has been a decisive factor that can help explain the wide gap between poll predictions and actual outcomes, though the direction of actual outcome remains the same with Democrats winning the US elections.
The US presidential re-election, and the rating system are live examples pointing to profound methodological deficiencies that still exist in the design and execution of statistical sampling.
For India, it is reassuring to see that the government has promptly appointed a high-level four-member committee to dive deep into the details and come up with concrete suggestions and recommendations for resurrections.
Group chief economic advisor, SBI & senior VP, Datamatics Global Services. Views are personal