Survey says: Not all pre-election polls are reliable

IMAGE: Daniella Marie Agacer

MANILA, Philippines—In about two months from now, at least 67.5 million registered voters are expected to cast their votes to determine the country’s next leaders.

Most, if not all, candidates for the national and local elections have been pushing their campaigns to woo voters and hopefully gain more supporters who will shade the circle beside their names on the ballots.

Some hopefuls have also upped their public appearances by attending televised debates, while others opted to pass on these opportunities.

Still, for some of the candidates as well as the considerable number of the voting public, one way people measure candidates’ performance in their races was through survey results—which visualize the gaps between candidates based on survey respondents’ preferences.

Based on results of different surveys for the May 2022 elections, certain candidates lead the race—among the presidential aspirants, candidate Ferdinand “Bongbong” Marcos Jr. has been leading several surveys, followed by Vice President Leni Robredo and the other candidates vying for the top government seat.

A recently released survey also showed the same trend or results, with Marcos leading the race, while Robredo ranked second, Manila Mayor Francisco Isko Moreno Domagoso in third place, Sen. Manny Pacquiao in fourth and Sen. Panfilo Lacson in fifth.

However, for the longest time, questions have also been asked about election surveys—most questions focusing on the credibility, reliability, and accuracy of these.

READ: The complex role of surveys, public opinion in PH elections

Unreliable surveys

Last month, the University of the Philippines (UP) School of Statistics warned the public against private and unreliable presidential surveys, amid the release of several ‘kalye (street) surveys’ online, which are statistically unreliable, according to experts.

These surveys, according to the statement, are usually presented with unclear methodologies.

“We, the faculty members of the University of the Philippines School of Statistics, speak and lend our collective voice to call out the pervasive abuse of survey methodologies by some entities doing their own brand of research, and promoting them in social media,” faculty members of the UP School of Statistics said.

“We have been observing these practices, including surveys not based on random sampling, such as videos of “kalye surveys” being done by some vloggers and suspicious online surveys, especially on Twitter and Facebook,” they added.

“We are dismayed by their cavalier disregard for the principles of data gathering.”

READ: Be critical of ‘kalye surveys,’ statisticians warn public

The faculty members likewise urged the public to become more critical of these kinds of surveys and survey results being published online as the elections draw near.

GRAPHIC: Ed Lustan

They also listed some questions, which should be addressed by election surveys, to make sure that the data used in the surveys are reliable.

These questions were:

• How was the sample selected?
• Which sectors of the population, if any, are being represented?
• What events surround the period of data gathering?
• When facing an interviewer, was the respondent interviewed in a neutral yet professional tone?
• What are the control mechanisms implemented to ensure the accuracy of the protocols in data collection?

“These are only some indications of survey operations with proper implementation and controls,” the UP faculty members’ statement read.

UPSS Faculty Statement on the Proliferation of Surveys with Unclear Methodologies pic.twitter.com/QZ1x1g2peY

— UP School of Statistics (@UPDStat) February 21, 2022

Transparency, public availability

The faculty members added that they also “witness PR companies, private individuals, and even some media organizations publishing results of surveys with unclear methodologies.”

A recently released non-commissioned presidential, vice-presidential, and senatorial survey by a fairly known private polling research and consulting firm was able to address some of the questions previously raised by the faculty members of the UP School of Statistics.

However, full reports of the election surveys—which include the detailed methodology used by the firm—were nowhere to be found on the polling firms’ website.

Survey results were mostly posted on social media, while releases were made available only to mainstream media.

While a staffer or member of the polling firm is able to answer some questions posted on social media regarding some details on the methodology of the firm’s survey, only a handful could read the responses and the rest of those who are curious about the method notes must search for a specific social media post before getting details.

According to experts, details about election surveys—such as the methodologies used—should be made accessible to the public.

“[I]t is imperative for survey groups to be transparent about their method and sampling design and to discuss their findings based on what their sampling protocol can adequately deliver,” said the UP Department of Communication Research in a statement of solidarity with the UP School of Statistics.

“We caution against practices that appear to generalize the results to the population, yet fail to conform to scientific standards of representative sampling,” the statement read.

The UP Department of Communication Research also said that as social science researchers, they value research dissemination which details the survey process and properly interprets the survey results.

“Hence, it is essential for survey reports and the parties that disseminate them to clearly and sufficiently disclose the profile of the survey group; the funding source and any potential for conflict of interest; the survey method employed; the target population, sampling design, sample size, sample distribution, and response rate; the timeframe of data collection; the questionnaire-phrasing of the items being reported; and the evidence-based interpretation of results,” it added.

The UP Department of Communication Research released a statement of solidarity with the UP School of Statistics regarding the proliferation of surveys with unclear methodologies being promoted on social media.

See photo for the full text of the statement. pic.twitter.com/RnDQ9a8XwL

— UP Diliman (@Official_UPD) February 22, 2022

Reputable institutions, such as the Social Weather Stations (SWS) and Pulse Asia, which are led by some of the country’s top social scientists, are known for their election-related surveys.

Full reports of their election surveys are usually posted or published on their websites.

The documents include technical details of their surveys or a detailed methodology applied for the survey—including the sample size, questions asked, and the crucial events that occurred before or when the survey was conducted.

In the United States (US), Pew Research Center—a “nonpartisan fact tank” which conducts election polls and surveys—detailed the survey methodology it follows on its website, where it could be accessed by the public.

READ: U.S. Survey Methodology

“Our hallmarks: a rigorous approach to methodological quality, complete transparency as to our methods, and a commitment to exploring and evaluating ongoing developments in data collection,” Pew Research Center stated.

According to Dr. Jose Ramon Albert, a statistician and senior research fellow of the Philippine Institute for Development Studies (PIDS), in an article based on his presentation at the 20th Jaime V. Ongpin Memorial Lecture on Public Service in Business and Government in October last year, companies conducting polls should be transparent.

“All firms conducting polls in the Philippines should be transparent about their methods and funding, not only because that is what is mandated by law, but also because transparency (or the lack of it) can help (or destroy) the credibility of your organizations, your surveys, and ultimately, statistics,” Albert wrote.

Hits and misses of past surveys

In past Philippine elections—including in 2010, 2013, 2016 and 2019—some companies other than SWS and Pulse Asia conducted their own pre-election surveys.

These included surveys done by Taylor Nelson Sofres (TNS), StratPolls, Laylo, The Center, Argus, D’Strafford and non-profit research group Ibon Foundation.

Some media organizations have also conducted their own surveys in the past, including those by the Manila Broadcasting Co. (MBC) and its flagship station dzRH in 2016, The Standard and Manila Standard Today in 2016 and 2010 and Radio Mindanao Network (RMN) in 2016.

Most, if not all, of these had been published online by some media organization—but not as much as surveys conducted by the SWS and Pulse Asia.

In terms of accuracy of past pre-election surveys, almost all conducted by various polling firms have had hits and misses in predicting who will be elected after the people have cast their votes.

For example, during the 2019 midterm elections, a survey carried out by Pulse Asia from March 6-9 showed the following top five senatorial candidates—based on responses by 1,800 respondents.

• Villar, Cynthia (NP): 55.9 percent preference
• Poe, Grace (IND): 47.7 percent preference
• Cayetano, Pia (IND): 45 percent preference
• Go, Christopher “Bong Go” (PDP Laban): 42 percent preference
• Bong Revilla, Ramon Jr. (Lakas): 39.5 percent preference

On the other hand, the SWS survey conducted from Sept. 15-23 yielded the following results based on preferences of 1,500 respondents nationwide:

• Villar, Cynthia (NP): 53 percent preference
• Poe, Grace (IND): 43 percent preference
• Cayetano, Pia (NP): 43 percent preference
• Pementel, Aquilino “Koko” (PDP Laban): 33 percent preference
• Lapid, Lito (PDP Laban): 33 percent preference

Results of the 2019 elections, according to data from the Commission on Elections (Comelec), showed that among the final top five senators out of the 12 who successfully secured their government seats were:

• Villar, Cynthia (NP): 25,283,727 votes or 53.47 percent
• Poe, Grace (IND): 22,029,788 votes or 46.58 percent
• Go, Christopher “Bong Go” (PDP Laban): 20,657,702 votes or 43.68 percent
• Cayetano, Pia (NP): 19,789,019 votes, 41.84 percent
• Dela Rosa, Bato (PDP Laban): 19,004,225 votes or 40.18 percent

Comparing data from election results to pre-election surveys showed that both survey results were able to correctly project at least three or four candidates who would be among the top five senators in the 2019 polls.

GRAPHIC: Ed Lustan

For Pulse Asia, four out of five candidates on its top five list of senators were among the top five senators during the election period—Villar, Poe, Cayetano, Go.

Bong Revilla, who was fifth in the survey’s list, ranked 11th out of 12 after the final counting of votes. He had a total of 14,624,445 votes or 30.92 percent.

The survey results by the polling firms, however, were able to correctly project only the final ranking of two candidates on their list.

The SWS survey had three candidates on the list who were on the top five with the most votes, according to Comelec data. These were Villar, Poe and Cayetano.

Pimentel, who ranked fourth or fifth in the survey, was 10th on the final list of senators, with 14,624,445 total votes or 31.01 percent. Lapid, who was also fourth or fifth on the survey results, was seventh on the Comelec list, with 16,965,464 total votes or 35.87 percent.

Similar to survey results by Pulse Asia, the final SWS survey for the 2019 elections was able to correctly project the final ranking of two candidates on its list.

2016 projections, results

In 2016, official tally from the Comelec showed that Davao City Mayor Rodrigo Duterte won the presidential race, with 16,601,997 (39.02 percent) total votes—despite ranking second or fifth in pre-election surveys.

The official Comelec tally for the 2016 presidential race was:

• Duterte, Rodrigo (PDP LABAN): 16,601,997 (39.02%)
• Roxas, Mar (LP): 9,978,175 (23.45%)
• Poe, Grace (IND): 9,100,991 (21.39%)
• Binay, Jejomar (UNA): 5,416,140 (12.73%)
• Defensor Santiago, Miriam (PRP): 1,455,532 (3.42%)

(44,549,848 voters who actually voted. 54,363,844 registered voters or 81.95 voter turnout)

Survey results from Pulse Asia (conducted from Apr. 26-29, 2016) and SWS (conducted from May 1-3, 2016) showed the exact same sequence or ranking of candidates.

Surveys conducted by other polling or private firms yielded slightly different sequences on their lists:

D’Strafford (May 1-5, 2016)

• Roxas, Mar (LP): 30.50%
• Duterte, Rodrigo (PDP LABAN): 25.40%
• Poe, Grace (IND): 22.50%
• Binay, Jejomar (UNA): 16.60%
• Defensor Santiago, Miriam (PRP): 3%

(Sample size: 4,000 ±1.2%)

Argus (Apr. 28-30, 2016 )

• Duterte, Rodrigo (PDP LABAN): 31%
• Poe, Grace (IND): 27%
• Roxas, Mar (LP): 19%
• Binay, Jejomar (UNA): 19%
• Defensor Santiago, Miriam (PRP): 4%

(Sample size: 6,000 ±1.9%)

Standard (Apr. 25-May 1, 2016)

• Duterte, Rodrigo (PDP LABAN): 32%
• Poe, Grace (IND): 25%
• Roxas, Mar (LP): 22%
• Binay, Jejomar (UNA): 15%
• Defensor Santiago, Miriam (PRP): 2%

(Sample size: 3,000 ±1.8%)

The Center (Apr.23-29, 2016)

• Duterte, Rodrigo (PDP LABAN): 29%
• Poe, Grace (IND): 25%
• Binay, Jejomar (UNA): 23%
• Roxas, Mar (LP): 21%
• Defensor Santiago, Miriam (PRP): 2%

(Sample size: 1,800 ±2.5%)

MBC-DZRH (Apr. 2, 2016)

• Duterte, Rodrigo (PDP LABAN): 36%
• Poe, Grace (IND): 28.20%
• Binay, Jejomar (UNA): 15.30%
• Roxas, Mar (LP): 15.20%
• Defensor Santiago, Miriam (PRP): 3.50%

(Sample size: 7,490 ±1.13%)

RMN (Jan. 5-14, 2016)

• Poe, Grace (IND): 28.59%
• Duterte, Rodrigo (PDP LABAN): 27.86%
• Binay, Jejomar (UNA): 21.49%
• Roxas, Mar (LP): 15.12%
• Defensor Santiago, Miriam (PRP): 5.14%

(Sample size: 3,578 ±2.5%)

Don’t accept results as they are

The faculty members of the UP School of Statistics also advised the public to not immediately accept survey results as they are.

They reminded the public and media that surveys “reveal facts, beliefs, sentiments, and opinions based on a representation of the population.”

“However, the quality of any inference cannot rise above the quality of the methodology it is based upon,” the group warned. “Sample selection strategies can generate bias in the data collected in the surveys.”

“Biased methodologies will only give biased results. Thus, we call on the public to be critical of surveys in this respect. Do not immediately accept survey results as they are,” the faculty members said.

In street-level surveys, for example, the interviewer “can purposely affect the answer of the respondents.”

GRAPHIC: Ed Lustan

They also said that people should be mindful of whether the sample (a portion of the population being interviewed) was of appropriate size based on, but not limited to, a target margin of error, level of confidence, and nature of the target population.

Also to be examined were how the sample was selected and which sectors of the population were being represented in the poll, they added.

Pre-election surveys help show who are the frontrunners early in the race. The current results, however, could still change as many more events might take place months before election day.

“Voter sentiments are hardly constant, but survey results, which can be input to campaign strategies, often converge about a month before voting,” Mahar Mangahas, SWS president, wrote in an article published by the East Asia Forum.

“Still, the 2022 election outcome is far from set in stone. More drama will arise in the months to come, making any forecast challenging, if not impossible,” he added.

TSB