3:00 P.M. EST
Real Audio of Briefing
MR. MACINNESS: You've seen his bio, so I won't repeat it, but today we're going to be pretty informal. Brian is going to give us a presentation and then we'll talk a bit about polling and then we'll just have a discussion and take questions and we'll probably do a transcript, but really not a formal briefing.
MR. KATULIS: Great. Thank you, Duncan. Can you hear me?
MR. MACINNES: (Inaudible.)
MR. KATULIS: Yeah, good.
MR. MACINNES: And the handouts have been out already?
MR. KATULIS: Yeah, it's a pleasure and -- yeah, there are some hand-outs. I'm actually going to make this interactive, because I think the best way to talk about polling is to give concrete examples. Just a little bit more on my background. I've done public opinion research in about two dozen countries around the world. I worked over the past six years first in American political polling and I'll talk a little bit about that because it's a different animal and then I'll talk about some of the challenges people experience, pollsters experience overseas, particularly in developing countries. I've done a lot of research projects in places like Egypt, Iraq, Morocco, Bangladesh and I'd be happy to talk a little bit about that in the Q&A.
Working with research organizations on the ground in those countries, of course, I think it's a very interesting topic and it's a timely topic. Everybody's talking about polls. If you look at the news this morning and you saw the audiotape of Usama bin Laden, he actually cites polls and he says that -- he criticizes the Bush Administration for not listening to his own people in the polls. So people are looking at polls and following polls and people like -- everybody, you know, in Washington looks at polls all the time.
So what I'm going to do is spend maybe 15, 20 minutes in my presentation going through some of these handouts, some of the points, I think related to polling. But before I do that, I just think many of you are consumers of polls as journalists and I think they're really important tools. They help you understand broad trends in societies. I think we all have to realize especially in Washington when you're reporting from Washington, is that we live in a bubble. We live in a very elite bubble, disconnected from our populations. And this is part of what polling helps us do; it helps politicians in Washington step out of this elite bubble. Outside of this country, development organizations have begun to use polling to understand how people view things like democracy, freedom, a whole host of things, I think that the State Department and President Bush has been trying to promote.
But what I'd like to say here in polls and focus groups are really useful for giving you a depth of the complexity of understanding societal opinions, but you have to be very careful about how you use them and that's going to be the thrust of my talk is how you actually critically examine the value of some of these numbers, some of these polls that are tossed around and what are some of the pitfalls that you as journalists should look out for.
I think polling is very important, but it's not a savior, nor is it a substitute for good investigative journalism. You should see it as a supplementary piece of information when you're doing your reporting and you're from the United States and then when -- back home and other things. You should take particular caution in looking at the polls in places that have low telephone penetration rates, where you have to do face-to-face sampling and this is the sort of thing that we did in Bangladesh and what happens in Iraq, in some of the measures that are out there, in part, because of the difficulty of doing face-to-face sampling, it's more labor intensive, you actually have to get out -- people out in difficult conditions and things like that.
In places like Western Europe and the United States, polling is a little bit more evolved. It's not a complete science, but it's a bit more precise, in part, because of the infrastructure that's here, but also because of the decades of experience and we've got enough information to basically develop more complex hypothesis of how do we do a random sample and things like that. But by no means is it perfect here. If you recall from our 2004 and 2002 elections, there are certain problems with things like exit polls and maybe in the Q&A we can talk about that and their emerging problems for opinion researchers here in the United States, people not responding, people relying on cell phones and things like that.
Let me get in my talk. First, what constitutes a good poll? A good poll locates people who are fairly representative of the population we're interested in. It asks them fair, understandable and useful questions and I'm going to go through some of that. It calculates the results fairly and it reports the results in a way that people can understand and value the judgment. So the polling theory, and I don't want to make this to be a statistics class, but it's basically based on the law of large numbers, which means that if you take a sampling of people from a population, the conclusions that you draw from that sample would apply for the whole population within a certain margin of error and I'll talk a little bit about that.
But if you take anything away from this talk this afternoon, in terms of how to evaluate polls, I call them the "two R's" of polls. Polls should be first a random sample and let me explain what that means and I think it's really important and it's something, particularly when you look at some of the polls coming out of the Middle East, you have to be careful about. But a random sample of the process by which respondents, the people who answer the survey, are chosen to be part of this survey. And I think it's really important because this goes to the science of polling. Oftentimes, you'll see polls on CNN or Al Jazeera where they say: We've got this hot issue, call in, tell us your views or go on the internet and click on "tell us your views." That is not a scientific poll. That is not a random sample. Why? Because there's a thing called self-selection bias. The people who are part of that sample that make up those numbers: (a) are viewers of that particular audience; (b) are hyper motivated. They're more motivated than most to actually express their views on a particular issue.
So when you're writing a story and I think this is very basic maybe for some of you, but I think it's an important point that when a news outlet does something like this and asks and invites people to participate and express their views, it's quite a different measure from the types of polls that we get when you're looking at a political race here in the United States or other types of polls. This self-selection bias I think is really important to think about because in a sample when you develop a random sample, when you're trying to randomly get individuals so that your sample reflects the population as a whole. In scientific polls, the pollsters use a specific method for picking respondents. In unscientific polls, the person picks himself to participate and I think that's an important point and there's a lot of nuance involved when you get into some of the details.
The second "R", in addition to having a random sample, is a representative sample and here I want to talk about three things. One is the size of the sample. And generally when you look at polls in the United States, there's the standard size. You've got 1,000 sample polls, sometimes it goes up to 1,200 or so. A thousand American adults is representative of 180 million, 185 million adults -- however many we have now in the adults population -- in part, because of this theory of the law of large numbers. If you select -- if each respondent in your survey had an equal chance of getting into your survey as opposed -- compared to the people in your population, then you can reasonably say within a margin of error, 3 to 4 percent, that the results that you get in your survey actually reflect what most Americans think. Oftentimes when you look at polls in certain states or overseas, you get surveys that are 350 or 400 sample. And I think you've got to take a careful look at that because the lower the sample, the higher your margin of error. And when you get a result, it's less reliable, it's within a broader range and we can talk about that.
The other thing that I think a lot of analysts don't look at is when you have a sample survey that's 800 for the whole sample, when you look at sub-samples -- men versus women, educated or Democrats versus Republican here in the United States, those margins of error actually increase for those sub-samples and oftentimes people forget about that. When you're looking at segments of the population.
A second point when talking about representative sample, are the demographic characteristics of the sample. And here, if you take a look at the Impressions of America survey (http://www.aaiusa.org/PDF/Impressions_of_America04.pdf), done by Zogby International in 2004, I think this is just a cautionary tale of how you might look at some of these products that are produced and I think Zogby International does a tremendous amount of work and I think it's tremendously valuable. But when you look at the results and you read the results here in the executive summary and then page three, for instance, when you look at the attitudes, Arab attitudes towards the U.S. as the first table here and it says "Morocco," one would be led to believe that this means -- this is what all Moroccans think of the United States. And there's nothing in here, when you go through the whole survey and even go through the methodology towards the end, that tells you exactly who are we talking about.
I was working at a think-tank and they asked me to actually do a critical evaluation of this and we got the SPFS files, which are the files that code the data for each of the interviews and things like this. And when we go -- we went through this for most of these countries, in the case of Morocco, most of the interviews were conducted just in Rabat and a couple of other large cities. And when you look at the demographics of Morocco, where you had such a high rural population, you're discounting and you're not including, I think, a tremendously important and perhaps different perspective and this is something that we've seen in Bangladesh and so many other places and probably you've experienced this, too.
When you think about the perceptual gaps between an urban center or, for instance, you're living in Washington now and you see and perhaps when you go outside of Washington, there are quite different views on a whole range of issues, from abortion to Iraq, to a whole set of other issues. So essentially in this survey, what is portrayed as a nationwide representative sample, is not. It's when you go into the data files and oftentimes researchers don't make it easy for you to go into those data files. And I think it's really important to be critically aware in places where polling is new. And this is not to fault Zogby International because I think they're doing a tremendous job in trying to help Americans and others understand how people in these countries view the U.S. and a whole range of issues.
And in some of these countries, it's tremendously difficult because of civil liberties and political rights constraints to actually do the type of sampling that's available. But that's another thing, the demographic characteristics should reflect the population as a whole. When you get into the details, if you get a survey, it's important to look at, for instance, the gender divide. Sometimes this is reported very easily and sometimes pollsters don't (inaudible) their data. They will have a sample that has 60 percent men and 40 percent women when the population overall is close to 50-50. Now of course, and my wife and I don't agree on all things, so we can't assume when the sample (inaudible) we can't assume that the women have similar views to men. When I was in Iraq in 2003, I remember I met one of the pollsters out there and I was asking him, "Well, what percent of your sample are men and women" and he said, "Well, 90 percent are men and 10 percent are women. And I said, "Well, what are you going to do about that to correct for that?" And he said, "To correct for what?" And his hypothesis was that women didn't have substantially different views than their husbands and brothers and fathers and things like this. And I was saying, "Well, that's an empirical question that you could actually test." And this is -- the kinds of things that you have to be skeptical about when you're looking at the representatives of the sample.
A third point, I think that's really important to look at are -- actually, I'm going to move on here to go a little bit quicker. To another handout here, I've got a handout on analysis on the shift to analysis, we'll move quickly. This handout here I think is really important to take a look at and this is based on polling that was done in the United States. And the issue of question wording and the types of things that can actually skew your results. And here are a couple of polls that were conducted last summer on Iraq. And I want to highlight the poll -- result is on the left and it's on the right. The one on the left actually says that 61 percent of Americans want to bring most troops home from Iraq and the one on the left said 60 percent wants to keep the troops in Iraq. And I think this is an important issue and I have some time series data and hopefully we'll talk a little bit about Iraq. But I want to get into details because of these particular questions. Why the different results? Because of the how the questions were worded. And this is something you have to be very, very careful about and critical about and it only, I think, over time you can develop a sense, if you have an eye for detail about what's different. You know, can somebody tell me what's different here about these two questions. The first one says: "Do you favor keeping a large number of U.S. troops?" which is quite different than the one on the right, which is "Should the U.S. keep troops in Iraq?"
A large number might actually skew the response in one way and I think it does. Another difference here is that keeping the troops in Iraq until the situation has stabilized versus a stable government. And then a third difference here is bringing the troops home at the end of next year, versus bring them home immediately. So this is all to say on a divisive issue, particularly on Iraq, but on any issue, how the pollster frames the question is tremendously important and you've got to be careful. And I think most unbiased news organizations or news organizations that seek to try to find some sort of truth to where do people stand on issues like Iraq, try to keep their questions as unbiased as possible and consistent over time and I think that it's really important for folks to consider. Particularly on an issue like Iraq, I think a lot of numbers are tossed around a bit carelessly without people taking a look at, well, what were the issues actually tested and what were the kinds of questions that were asked.
Another type of question I'd like to highlight back to the Impressions of America, the 2004 Zogby survey (http://www.aaiusa.org/PDF/Impressions_of_America04.pdf). If you look at page seventeen, for instance, and this -- you know, again I think there are certain questions that are extremely useful. There are other questions that -- I don't know what it tells you, because when you get down to it, the issue of polling and why people do polling, you want to have the instrument, the poll tells you something you already don’t know. You know, (inaudible) insight, something that you don't know, a hypothesis that you don't have already.
And when you look at questions like the question that appears on page seventeen, actually going back to page four, the favorable versus unfavorable attitudes towards the United States. When you look at the results, for Egypt, for instance, it says 98 percent of Egyptians -- and again, this is when you look at the sample of just Egyptians in Cairo who are mostly college-educated, 98 percent had a negative or an unfavorable view of the United States. Well, I probably couldn't -- you know, I didn't need to take a poll to understand that. And I think the key question is why and how intensely they are and here's one challenge I think in some of the polls that are done when offered on a two-point scale. It often presents people with a false choice. And a better way to ask questions like this is: How do you view, approve or disapprove, of the United States or are you favorable or unfavorable and ask an intensity follow-up: Are you strongly unfavorable or somewhat unfavorable. It's an important nuance that I think when you look at a real good poll, it tells you, it helps you understand that there is a lot of hatred and all of that. And then moving beyond that, asking questions of why. Why do you feel this way, as opposed to the basic metrics of do you approve or disapprove and all of that and I'll get into that when we talk about qualitative research.
Another issue I want to talk about, when you look at this Iraq time period trends and I think this is really an important issue because sometimes people and journalists analyze a single poll and I think they try to draw many conclusions from one instance in time. And here on (http://fpc.state.gov/documents/organization/59776.doc ) -- if you can move to page four and five here and highlight a CBS News polls, which I think is really important. It's a basic, what I call a "clean question": Do you approve or disapprove of the way George Bush is handling the situation in Iraq? It's hard to find any kind of bias in that kind of question -- do you approve or disapprove -- they didn't ask an intensity question. But is it a positive or negative thing.
And I think when you look at pages four and five, the important thing I think journalists need to do is not only look at the snapshots in time, but look at the trends over time. And I think this helps tell part of the story. When you look at pages four and five on the approval and disapproval ratings, for instance, on George Bush on Iraq, I think it tells you an important story of what actually drives American perception. And here, this is my own analysis, but I think people can have different analyses when you look at the numbers.
When you look at the numbers up till March of 2003, prewar, you can see that about half of Americans approved of where George Bush stood on Iraq. And then there was a tremendous leap of 20 points in favor of what we were doing in Iraq in the spring of 2003 and this was part -- you know, the statues coming down and all of that. But as you go through the data, as the bad news came out, and there wasn't much progress through the fall of 2003, the numbers declined. And then you see an uptick in mid-December, why, because of the capture of Saddam Hussein.
And then since then, since 2003 has been a slow and steady decline in the support in terms of approval and disapprove for President Bush on Iraq. And I think this is an important analytical point in people that are looking at the President's victory strategy in Iraq and trying to understand how this is impacting American perception. There have been slight upticks in his approval, as a result of these communications. But it's events that have driven the numbers -- events that people see coming from Iraq and I think it's an important part of the story to evaluate how well is the Administration doing in terms of a changing opinion here in the United States. And you've got to look at it over time. You don't get the full story if you just look at one poll. You have to look at the trends and see what the trends are telling you.
A couple of last ones before I open it up for questions. A question of question order and this is one thing that I'll -- if you look at the Democracy Corps, Carville and Greenberg frequency questionnaire (http://fpc.state.gov/documents/organization/59774.pdf ). This is a poll that we've done with a firm that I worked for that was working for -- in part, for John Kerry at the time in the race. And this is -- if you take a look at this in instruction, I'm going to go through this in a bit more detail. This is more of a political type message polling and this is for internal consumption, though released publicly. And here I think it's really important to take a look at. This type of poll is a registered voter, likely voter sample and this is something to take a look at as you're going into 2006 here. Whether your polls are general adult sample in a particular state, or if it includes likely voters because there's a substantial view -- difference in views in terms of who people are going to vote for or where they stand on the issues and all of that. General population, general adult population versus registered and motivated voters. And it's a tremendously important point for campaigns who do targeted media and ad campaigns and all of that.
The polls that the campaign do here in the United States and in most countries around the world throw out people who say, there's only a 50-50 chance that I may vote in this next election -- completely throw out people who say they're not registered to vote because it doesn't make economic sense according to the economics that drive politics, fortunately or unfortunately, here.
Basically, there's a strategic decision that is made in these polls to -- in the message-type polls to say we're not interested in hearing what you have to say, even if you're kind of -- you think you might show up. There's a 50-50 chance you might show up. Why? Because these polls have a different aim and objective. And when you look at -- and you look through the structure of it, it starts out with screener questionnaires basically asking how motivated are you to vote? Are you registered to vote and determining if they are fairly motivated according to past performance and statistics. And if they are, we keep you in the sample.
Then we ask a couple of questions about right track, wrong track. Everybody sees the right track, wrong track numbers. That's a basic barometric. To understand is the mood of the country positive or negative, what's driving it? Then they ask about priorities. Then they ask early on questions about how do you perceive the President and leading figures, before you get into more specific questions about certain issues and things like that. And this is important because sometimes polls ask questions, for instance, about the economy upfront and then ask about the President's job approval rating. When they do that, particularly at a time when the economy is not doing well, the President's job approval rating is invariably lower.
When they do the reverse, when the President's job approval rating question is asked before the economy, the job approval rating is higher and this is, I think, an important thing when you're looking through, somebody gives you a number, you can say can I see the questionnaire? I'd like to see the full questionnaire to see the order in which the questions were asked because the questions that are asked early on actually have an impact on the questions that are asked later. That's why it needs -- these political polls you ask the questions that are top of mind, right track, wrong track, what are your leading concerns? Then ask a vote question. Invariably, I think it's an important aspect of any campaign asking early on who you're going to vote for. And in this case, if you look on the poll, you know, this had George Bush over John Kerry three points up on page four. The structure of this is to get a sense, top of mind, where people are at the beginning of the poll. Then you spend another 15, 20 minutes asking people's perceptions about who's the stronger leader, how do they stand on the issues and all of that.
And towards the end when you get, for instance, towards the end of this particular poll, the Democracy Corps poll on page 11 and 12, you get into extensive message testing batteries and this is, I think, an interesting thing and an interesting trick of the trade where a series of messages which are being cooked up by the media campaign and advisors are tested with different words and different formulations to see which ones work better with again, a sample, this very specific sample of registered voters likely to show up and all of that. And then at the end of the poll, before they do the demographics, they do a re-vote to see -- after we've given you all of these messages -- and it's messages on our own candidates and messages on the other candidates and attacks and all of this -- who is moved? Because invariably, you'll see that people's perceptions, even in the context of a 20-25 minute telephone interview after listening to different statements about George Bush vs. John Kerry, they moved.
Certain people don't move, but there are other people that are in that soft middle that move and this is -- we develop metrics in terms of slicing up the electorate and try to make it very scientific in terms of which messages work best and all of that. That's all to say and back to the point of order of questions and how the questions are worded. And I think it's really, really important -- maybe not as relevant to you as journalists -- but to understand a little bit about the message testing that goes on and it happens even in the context outside a campaign.
I can guarantee you when the President of the United States makes a major speech, though he says he doesn't listen to polls and all of this, they do. Every leader does and there are reasons why certain words are chosen, like "freedom" and other things, because they're important and they express our values, but they also have a political -- a domestic political importance. And a lot actually -- and I think of my opinion actually -- too much is poll tested in terms of formulating policy and thinking through all of those issues.
Before I close up, I think there's a little bit -- and take your questions -- there's one thing I think is important to mention that goes beyond polling and that's focus groups and it's something that I've been involved with, too, in addition to the polling. But a focus group is qualitative research generally used by campaigns and others to test ads, to see how people -- certain segments of the population react. It's a group generally of eight to ten people for about an hour and half long.
Overseas focus groups have been used increasingly and I've been involved in some of these activities. To understand why people feel a certain way, because a poll can tell you what percent approves or disapproves of a certain thing and often present a false choice because people, I think, are very complex and have different views and very nuanced views and a focus group is aimed at actually guiding a group of people in a certain demographic, to understand more deeply why they feel that way. You can do that in a poll, by asking an open-ended question, as opposed to do you approve or disapprove, why do you feel this way or what is, you know, what's the top of mind. It's harder to actually code those responses and get a legitimate response and coherent response from 1,000 individuals when you ask an open-ended question, just procedurally because it's hard for all of the different interviewers who are doing this to capture all of the different nuances of, for instance, a question like "Why are women doing better off in this particular country," you get a whole range of answers in an open-ended question. And in a quantitive setting, it's hard to get a number from that and hard to get some understanding. But in a focus group setting, you're listening to people, seeing how they react to questions, the good place actually tests questionnaires before they go out into the field, those are the quantitative measures. So there's a serious need and a utility to do focus group qualitative research to supplement some of the numbers that are out there.
I'm going to close because that's about more than twenty minutes, I think, but I'd love to take any questions you have. It was a basic overview and I'm sure you have some questions.
QUESTION: Thank you. Sonia Schott, Radio Valera (inaudible). Latin America is going through this year through many electoral process and I would like to know or to hear some comment from you, how reliable should be polls and what do you think about these quick (inaudible) what are doing after the polls sometimes to prove the results or to have an idea about the results of a poll? Thank you.
MR. KATULIS: I think the polls should be -- your first question is how reliable should they be? I think they should be very reliable, but there's always a margin of error in any country. It's increasing countries that have less of a history of polling, for instance. Or maybe I didn't understand your question.
QUESTION: Yes. It is because sometimes there is like an abuse of such polls and probably the polls say something and the reality goes to (inaudible), so that's why I will like to know how reliable -- there is a limit of reliability of the polls? How reliable are -- can we consider --
MR. KATULIS: Well, it depends on a -- (inaudible) polls in Latin America. I mean, it depends on a specific country and a specific place. I mean, Venezuela and other places, if there've been a lot of polls and it's extremely hard and let's go back to the point where I started from in terms of getting Iran a representative sample. And as I understand it, and I've never done polling -- excuse me -- in Venezuela, but I've worked in Bolivia and Colombia and other places. And it's extremely hard just logistically to get enough of -- a representative of enough sample. Oftentimes you get polls that are reported -- again, this is what the Venezuelan public thinks. But when you go to the researchers and you go to the research organizations and I think it's really important for journalists to do this, particularly in places like Venezuela and others, can we understand what percent of your sample was urban versus rural -- what was it men versus women? Education, what was the percent that were high school educated versus college educated? Because oftentimes, you get a poll -- and this is not to denigrate the research that's done in some of these places, but you get a poll (inaudible) college educated. And what you should do and the polling firms should provide all of this, this often does not happen. But what you should do is take a look at those demographics and see. If it says 60 percent college educated, but what you know from the census, the Venezuelan census, of the adult population and I think it's important to look at it that it's only 20 percent, then you know that the poll is skewed and you make the choice as a journalist as to whether to report it or not. I'm of the mind that it's bad to throw out any kind of sample.
Yeah -- no I really think if a research organization is unbiased and that's another thing I think journalists should take a look at and we all have our own biases and different research firms work for different candidates and all of that. But if a research firm is with a news organization that tries to play it up the middle that is not affiliated with any political organization, then I still think you can't take their word for it in places that -- where polling is new, where you don't have a plethora of research. I still don't think you can take Gallup's word for it in the United States. I think you need to take a skeptical look to the extent that it's possible to look at those two questions of the two R's. I mean, it's very basic. But what is the procedure by which the respondents were in the survey selected randomly. And then second and more importantly, is it representative of the population as a whole. So I would be just, based on the little bit that I know of polling in certain countries in South America, it's a new -- there's a lot of work to be done.
And then your second question, I'm sorry, was on the --
QUESTION: On the quick counts.
MR. KATULIS: -- there's the quick counts. Yeah. It's similar. And in fact, you know, I mentioned exit polling here in the United States and there are problems emerging methodologically with all of these issues. I think quick counts are intended to serve as a check and it all goes back to the details. We have to go through specifically how were the samples selected, who was selected and what are we reporting. And if you recall from 2004, in the election here, I was getting earlier results from the exit poll -- it was different from quick count -- but exit polls that were saying, you know, in Pennsylvania, John Kerry was winning 75 percent to 20 percent. And it turned out that those numbers were coming mostly first from urban areas and of course, it was skewed -- skewed towards all of that.
So the quick count -- I think the intentions are generally good, if they're done by -- again, it goes back to who's doing it. I think as a journalist, you have to look and think about who is doing the research, because I think that actually can lead to some sort of motive. It can help you understand that if you think a question is worded in a certain way and that it might be biased and you also think that this polling organization which is working with this party has a certain issue to push, then you could be more -- not skeptical, but critical in your analysis of separating the wheat from the chaff because -- and understand. But at the end of the day I think if you don't think it's so overly biased, so as to try to skew opinions about the race or a particular issue, if you qualify it. If you add in this is, you know, mostly an urban sample of people in Caracas who were interviewed in this period of time, you know, mention the time and it seems to be skewed towards college educated. And if it has some insights, you know, after putting all those qualifications -- put it in your report. I tend to think, you know, I don't know if it's newsworthy, but I tend to think any kind of data is good data to try to -- especially in places where I've worked in the Middle East. I feel like we're the blind men touching part of the elephant and more data is better to help us -- even if it's slightly biased or not totally random and representative.
QUESTION: I have a question. My name is Deepak Arora (inaudible) National Herald (inaudible.) I'd thank you for the insight. It was very interesting to hear you. And I wish it was a little longer than what (inaudible.) Now my question is that I wonder if you've done any survey on India? Because in India normally, like the last election -- general elections we had there, none of the surveys went right -- none of the surveys. And they (inaudible) over a period of time. What do you have to say about such things and how do you actually -- like you mentioned that we should look into several aspects of who the (inaudible) has done (inaudible) research, educated (inaudible) and you could do it men versus women, rural versus urban areas. But does newspapers have space to actually give all this information? I don't think so. So what do you have to say to this?
MR. KATULIS: Well, newspapers probably don't have the space to give all of that information. But I think as a journalist, if you're going to report on a poll, you've got a responsibility to try to at least understand a little bit more of how representative it may or may not be. On India, I don't know specifically about the issues in polling in India. I have not done polling in India. I've done some polling in Bangladesh for a couple of development organizations. And I think it's really, really difficult in places like this that don't have a long history of polling because we basically in the United States have been doing pollings for the last 50 or 60 years at least, legitimate kind of nationwide representative samples.
And the experience over time of having more data actually helps you understand when there are mistakes, you have serious researchers trying to figure out why was the sample is off by 15 or 20 points. And they have an answer to that question. Then they move it and improve it. And in a place like the United States, actually we have challenges emerging because people aren't increasingly using their home telephone numbers.
It's difficult, you know, to get people to stay on for a 30-minute survey and all of this. And it's always, you know, a moving target to try to improve the quality of the polls. I don't know the specifics of what's going on in India, but I imagine part of it is just a learning curve and a very steep learning curve of trying to get these polls right, understanding your sample in a population which is growing tremendously and in trying to understand how to weight the different subgroups in your sample to make sure that it's representative of the whole. I'd have to look at some of the specific polls. But then to your second point, I just think no, newspapers don't necessarily have to report all the different details of a poll, but if they think they're that biased, if after taking the additional fifteen, twenty minutes to call the pollster on some of these questions and say: How did you actually develop the sample, is it representative, you know, does it reflect the population? And if you think not, and you think it's so skewed, then I think you absolutely don't report it. I mean, that would be my opinion.
QUESTION: Hi. This is Siu-Wai Cheung. I'm with the Hong Kong newspaper Ta Kung Pao. I have two questions -- two sides. The first part is in some countries, there are people who are doing the polls, while the public are not so accustomed to taking the polls. Do you find a difference from the industrialized countries where people are used to getting the polls and answering questions and from the other side where the developing countries, people are not very used to answering the questions? And what are the results could be, what the difference will be? And the other side is -- the other question is are there many differences on the polls which are done toward certain commercial purpose and the other side is toward political purposes. Are these polls all the same or whether the commercial purpose polls are more reliable or the political side more reliable. Thank you.
MR. KATULIS: Great. Those are two great questions. First, on the differences between developing and developed countries, there are absolutely tremendous challenges in places like Iraq, like Bangladesh -- I mean, obviously, in Iraq. But it's really, really, really difficult in places, particularly, that have lived under repression where people haven't lived in a culture where it was easy for them to speak their mind, to get certain types of people involved in your sample. And when we did research in Iraq in 2003, it was particularly difficult in Shia areas to get women involved in the sample, in part, because of how you actually when you do a face-to-face survey and you knock at the door and oftentimes, there's a procedure of how you get whether it's a man or a woman to randomly participate when you knock at that door. It was difficult and it's difficult in certain -- just culturally, it's difficult, I think for people to -- certain folks to respond in different conservative societies. So there's that issue.
I think another interesting facet in certain -- in developing countries where I've done some research, and again, when I've done this research, it's always in cooperation with partners. So the research is only as good as the partners on the ground who speak the language and who actually have the field interviewers and all of that. Specifically in Bangladesh, we had the opposite problem where we had people when we were doing the face-to-face surveys, we had people in the community coming up to the interviewer and the interviewee saying, can I take part in the survey because, in part, a lot of people, I guess, nobody ever asked them their opinions before -- from their government and other things. And there was this impulse and you see this in other places, too, where people want to take part in the survey and they want their opinion to be included. So I think it varies, but I think there are certain cultural constraints. And I think the more that it's done, kind of like what I was saying about India. And the more that it's done procedurally, you understand what methodologies work best. I think certain westerners when they try to do research and other societies don't adapt their rigid methodologies to take into account these cultural factors. And I think there's a call for creativity for some of the -- among some of the most -- you know, the best researchers at Harvard and Princeton. I think they need to be a little bit less rigid, in terms of how they apply their sampling methodologies to different societies.
The second question, I think, is really important I think is -- when I was working at Greenberg Research full time, we did political polling, we did commercial polling for BP, a number of large corporations and the types of polling, I think, are really different and I have my own personal opinions on which is more accurate and which is better. But they all serve different purposes. I mean, news organizations tend to try to provide, at least in the United States and talking here, unbiased information, depending on which news organizations you're talking about, but unbiased polling results. Those who work for political candidates oftentimes when they're -- again, they're trying to test messages and understand what the temperature of the electorate is and all of that.
Sometimes you have unscrupulous -- and here's the term you should keep in mind is push-polling. You have pollsters who are affiliated with campaigns who are basically trying to push a message about their own candidate or trying to push a negative message about their opponent and this happens in state races to this day. And actually, when it does happen and people hear about it, you have industry groups that complain about it and journalists point this out and I think as journalists, you should really be aware of this push polling, which is basically presenting a questionnaire to a sample of individuals in an attempt to actually either feed false information, to change perceptions in some sort of way, to get people to think about the opponents in a negative way or to push a certain message about your own candidate and then to get a result that basically says we're leading this race, you know, by 65, 70 percent. And sometimes you get journalists who, again, don't do the homework of looking through how are the questionnaires set up.
You know, if somebody in the campaign says to you, we've got a poll that says my candidate's up and we're up 30 points, you should say, I'd like to see the questionnaire and the full questionnaire and so that you can critically evaluate whether the questions had any wording bias in it, had any order bias and all of this and take a look at it and then you can either say, well, this is not legitimate or it is.
The commercial polls, I have -- I think you know, that's a constantly moving target. I don't know if you've been called. I've been called a number of times. I've been on both the writing end of these surveys and the receiving end. And I'm skeptical, for instance, when we did surveys of brands of gasoline. There are certain things that I think commercial research is useful for and, in particular, the qualitative research, when you're testing people's attitudes towards a different ad or a different product and all of that, I think it's more effective. But when you get people on a phone survey for 30 or 40 minutes and ask them about brands of gasoline, I just think -- I question sometimes the results of some of the polls that are asked: Do you prefer BP vs. Shell vs. Exxon Mobil, in part, because when I've gone through some of these surveys myself, and we've seen this, too, it's random. People don't really have a serious impression about one brand versus another. I think there's more detailed polls that are done for different types of corporations that -- where branding is a serious issue and it's more scientific. But again, that's towards their own end of increasing their market share. So I hope that answers your question.
QUESTION: Thank you. My name is Arshad Mahmud and I represent the Protham ALO in Bangladesh. First of all, when were they in Bangladesh?
MR. KATULIS: I was in Bangladesh in 2003 and 2004 and we just completed a survey in 2005 for the International Republican Institute.
QUESTION: Oh, okay. Anyway, my question, as a foreign correspondent -- where you work in Washington and trying to report on the United States, the polling is an important aspect. As you rightly pointed out, that it all depends on how you frame the questions, how you put the questions. And to what extent, we get so confused at times, especially this (inaudible) indicates that, you know, you have got various kinds of results on the same issue. Who should we trust? Who should we, you know, rely on when we are trying to report objectively? What is your take on that? I mean, Harris, PEW, Kellogg, I mean, who do you suggest that -- do you have any --
MR. KATULIS: Well, I think most of the major news organizations, particularly those that work in cooperation with others, when you see a CNN/USA Today Gallup Poll, when you see Washington Post, ABC News, when it's in cooperation and it's outside of a conglomerate, I think that's a little bit more legitimate than when you see, say, one organization like Fox. I mean -- and that's -- and not to say that their polls are any less legitimate, but I think what you have to do and I think the main point is to take all of the polls that are out there and there's a really good site called www.pollingreport.com that actually does a conglomeration of all of the polls and --
QUESTION: Pollingreport.com?
MR. KATULIS: Pollingreport.com and it's free for a nationwide (inaudible) -- through when I went through the time series in the CBS news results on the approval rates for President Bush on Iraq, that was just an example of which was confirmed, if you go through the other sheets and there's several other polls in there, of a trend on that basic situation when you ask it in a similar way. So I think part of the answer to your question is when available, when there are several multiple samples available, you try to compare them one against the other. And to the extent that you have time to do that and you're interested in it, look at the work -- how the question is worded.
Here in the United States, you don't have to worry so much about the sampling issues. I mean, what you should look for is a little bit more basic than what you look for when evaluating polls from places like developing countries, not because it's a lot better, it's just a little bit easier. What you should look for is general adult sample versus registered vote sample versus likely voter sample, particularly when you're going through an election period. And you have to be careful. I mean, I remember we worked on some research about Latino voters in the United States and you have to be careful because there were different types of samples. There was one pollster who was doing a sample of Latinos. There are a lot of Latinos who aren't citizens, first of all, that live in the United States who are residents. There are a lot of Latinos who are citizens, but aren't registered to vote and all of that. And what we were doing in the context of the campaign was trying to figure out who were the Latinos who were registered to vote -- that were citizens registered to vote and all of that and likely to vote and show up. And there were substantial differences in terms of issue orders.
You know, for Latino -- Latino-American population on the whole, immigration, for instance, was a big issue. But when you actually have a narrower sample of likely voters, immigration was less of an important issue, it wasn't unimportant. But education and some other things were really important. So when you look at the polls in the United States, those three tiers -- general public, kind of general adult public versus registered voter, versus likely voter -- and the differences between registered and likely voter aren't going to be terribly too much. You know, it gets into the fine (inaudible) of a couple of points generally and it all goes back to how does the pollster define likely voter.
Usually, it's on a four-point scale: are you certain to show up, are you very likely, somewhat likely. Sometimes it's on a one to ten scale. It's those who, you know, say, ten means I'm certain to show up. But those kinds of things, I think you need to look at here in the United States and evaluate, but take all of them in and try to critically evaluate them yourselves and take a little bit more time, rather than just saying here's a number. I heard some guy at a think tank or something like that, saying, rather than putting that in your report, if you have the time, take the time to look at it, because I think it's really interesting.
QUESTION: Okay. Thank you Ben Bangoura, Guineanews in Guinea. Still back in the United States, do you think polling really matters? I mean, can it influence the outcome of an election? Can they influence the agenda of -- the political agenda? If yes, do you have any specific example? If not, why?
MR. KATULIS: It's a great question -- a couple of questions there. I absolutely think they do influence the results of elections and there are several reasons why. I mean, in particular, you've seen an increase in the number of polls that are done after debate and maybe you've seen this if you were here in the elections last year. But there were a lot of polls that were done overnight of people who watched the debate. Why? Because the candidates could then go out and say, 70 percent said that I won this debate and this perception that, you know, I'm the winner in this particular test in part of a campaign may have some sort of spillover effect in terms of influencing those winnable voters. I mean, I think it's important, I think, to note particularly in the United States that the impact may be -- there's a bit of nuance here. I think it may be a little bit less strong than in other places, in part, because of the red state, blue state divide. I think they're serious.
There are certain voters who are Republican and there's just no way in hell they're going to vote for a Democratic candidate, even if they, you know, support the NRA and whatever, you know, where they're really conservative. And the same goes on the other side. So there's a soft squishy middle. I still think that it matters a heck of a lot because when news organizations report it and you just look at the behaviors of people in campaigns, when a poll comes out in favor of our candidate, you say, "Oh, look, we're up," you know, and you tout it and you do a press conference and all of that. When a poll comes out that says you're behind, your candidate always says, "The only poll that matters is the poll on Election Day," (laughter) and they discount it and all of that. You know, and that's -- those are the two kind of standard talking points.
I think it does influence and shape electoral politics for better and for worse. It actually does shape and influence how we discuss issues, dare I would say, for the worst. I just think -- you know, I think we need to have leaders -- this is my own personal opinion -- who are less followers of what a poll might say and more of -- you know, "I stand for this and that," and use a poll to have -- help them communicate words and ideas more effectively, as opposed to take the temperature of where people are at
You know, I think in a democratic society, we need leaders that are responsive, but I don't think polls should drive policy. I think there are a lot of policy issues that require serious leadership, where leaders are bold enough to say, "Okay, this may not be terribly popular, but it's important, you know, for us to do that," and I think you have some leadership doing that even now.
QUESTION: (Inaudible.)
MR. KATULIS: That last one? I mean, there are a lot of issues. I mean --
QUESTION: (Inaudible) and stuff like that.
MR. KATULIS: Oh, in terms of how the poll --
QUESTION: (Inaudible) outcome of the political agenda, a specific example --
MR. KATULIS: Well, I mean, you can look at anything that happens here in Washington. (Laughter.) I'm sorry to be cynical, but there's a reason why Frank -- John Murtha criticized the President, and it's in part because he believes -- he really believes, you know, in his critique and his alternative on Iraq, but he also saw the numbers. You know, if the numbers were in a different way, I mean, politicians are where they are because of what they do and who they listen to. And part of what they listen to are the numbers. And what I'm saying is sometimes the polls are abused. I mean, it's not -- sometimes the leaders should be more leaders, is what I'm saying, and they should use the polls to help them communicate a little bit more effectively but stand for things, you know, rather than go this way and that with the wind.
QUESTION: Can I (inaudible)?
MR. KATULIS: I don't know. (Laughter.)
QUESTION: Hasan Hazar, Turkiye Daily. What's your average budget for nationwide scientific poll?
MR. KATULIS: How much does it cost?
QUESTION: Yeah.
MR. KATULIS: In the United States?
QUESTION: Yes.
MR. KATULIS: If you're talking general public, I mean, it depends, be it different sample frames. But general public, it's anywhere from $40- to $60,000. That is what it costs you. When you do more business type surveys with opinion formers or people that are harder to get on the phone, it gets more expensive, for instance, when you narrow your sample frame. When you have a frame that's basically we're trying to understand what adult population feels about these issues, it's a lot cheaper than somebody who's likely a registered voter. Why? Because it takes longer to get the likely and registered voters on the phone, according to proper sampling procedures. So the costs go up, you know, to 70- or 80,000, generally, I mean, that's what private firms are paying. You know, people need -- and there are costs of people who are doing the calls and it's actually a very difficult thing to do.
When I was in graduate school about eight years ago, we actually did a poll from start to finish, including writing the thing, doing the sampling frame and calling it, and it was such a painful experience. In New York -- we were doing it in New York State, actually, in college, in graduate school in New Jersey.
So there are all these costs that I think a lot of people think you write some questions and then it goes out and then the numbers come back and then that's it. No, there are a lot of steps in between.
QUESTION: How much business, poll business, poll industry in the United States?
MR. KATULIS: I don't know what the exact numbers are, but in terms of -- I'll tell you a little bit about campaigns. It's miniscule in terms of the amount of money that's spent on a campaign in the United States. Why? Because the vast majority of money is spent on media, and this is an issue that I think -- my own personal view -- cuts to kind of what drives our politics and all of that.
Generally, on both sides of the aisle, Republicans and Democrats, they want to spend as little as possible on research. Even though that sounds like a lot of money in terms of how much a poll costs, when you talk about an ad buy for a 30-second or a 60-second ad in a major media market, I mean, there's no comparison. Much more money is spent on communicating the message as opposed to testing a message. There's a lot of polling that goes on. I know a lot of pollsters who say not enough goes on because of their own business interests and all of that. But where campaigns are, they make, I think, the calculated decision to focus more on communicating the message out.
I honestly don't know the overall numbers but it's miniscule when you compare it to the kind of things -- the commercials and all of that.
QUESTION: Hi, my name is Hiro Sugita. I'm with the Kyodo News, Japanese news agency.
MR. KATULIS: Hi.
QUESTION: I'd like to know a little bit more about exit polls. First of all, the accuracy of the exit poll and (inaudible) something of the polls -- I mean, which polling station you are sending the interviewers. And also, do you have any agreement or some restriction, some ruling, that you have to hold the outcome or result of the poll before some part of the country still -- the polling station are still open?
MR. KATULIS: The answer to the second question is no and the answer to your first question, I think a lot of researchers are struggling with right now, to be honest with you, because the methodologies are out of date. Even when all of the samples came in, in 2000 and 2004, and this led to severe embarrassment, I think, among a lot of news organizations, to the extent that the exit polling data isn't fully available from, I think, the last cycle. So the sampling methodologies that were used were quite similar to what one uses in terms of a nationwide poll, and generally the exit polls are done state by state so the sampling frames were similar and you look at basically where the polling sites are located, making sure that each polling site has a random chance of being included and all of that.
I think there are some issues as to who is approached and how they're approached and who's willing to respond. There's a response rate issue and I think a level of comfort, for whatever reason, that's changed over the last couple of years when people come out of an exit -- out of a polling site, and don't know why that is and I think a lot of researchers don't know why that is. And I don't know how it works in other countries, but in the United States this is the big question that a lot of people are looking at.
And the question is why because the exit poll isn't only important for the day of the election. The exit poll is used -- has been used in terms of historical data for weighting your sample. And I didn't get into weighting your samples, but basically when you do a poll, when I was working in 2002, 2001-2002, and preparing for the election cycles, we would look at historical trends in a particular state, for instance, based on exit polls, which, you know, which people voted which way on certain issues, to try to make sure that our sample was as accurate as possible in reflecting the electorate as we thought it might be. It's always a projection and it's based on turnout models.
And right now, a lot of that is in a black box. A lot of researchers, I think, are working hard and I don't know that they have an answer to it because we basically had two presidential cycles where the exit polls just weren't useful. And they're trying to figure out why and I think it's related to a lot of different dynamics in a changing American society.
Do you want to go to New York?
MR. MACINNES: There was some voluntary holding of polling --
MR. KATULIS: Oh, yes. Some have done that but it's not a --
MR. MACINNES: It's not a legal requirement but, instead, volunteer.
MR. KATULIS: Mm-hmm. Shall we go to New York?
QUESTION: Deepak Arora (inaudible) from National Herald India again. Thank you. I hope that I will be able to see you again sometime in the future in India, which is one of the largest -- along with the U.S. is the largest and the biggest democracy in the world. As a foreign journalist, my question is that we often read about President Bush ratings (inaudible), like if the rates keep going up, keep going down. And it actually (inaudible) verified. What difference does it make actually, I mean, of course, as a foreign journalist does it make a difference to us -- to the President of U.S. That's it. (Inaudible) ratings actually been here (inaudible).
MR. KATULIS: What is his rating? What do his ratings actually mean; is that your second question? Yeah, his ratings, I mean, generally, there are two types of ratings. One is job approval rating and then there's one that's a personal approval rating, and the numbers get tossed around and they're a little bit different. But it's basically a top of mind measurement asked early in most polls of do you approve or disapprove of how President Bush is doing in his job. Do these numbers matter? For most Americans, it doesn't matter. For most Americans, Americans don't care about politics greatly, you know. And I think when you compare it with those of us inside of Washington, a lot of people, a former colleague of mine, Paul Begala, I think it's a great phrase, says, "Most people are rationally ignorant about politics." And it's something that folks inside the beltway, including myself, have to remind ourselves that they could care less.
They have a very different top of mind impression about politics in the way that, for instance, my wife, you know, could care less about the batting averages of the Washington Nationals and Baltimore Orioles, and I'm into it. You know, I like to look at it. And much in the same way, I'm into looking at politics and polls. Most Americans don't care.
Polls and the President's approval rating actually matters at an elite level, and I think it gets into the dialogue and some of your colleagues, American journalists, actually report it all the time and it matters for people, especially now with the President's approval ratings hovering around, you know, the high 30s or low 40s, which is not great. And that matters in that elite dialogue and that positioning on politics and whether or not somebody is viewed as a lame duck and whether or not they have the credibility to actually have some sort of mandate. A poll is viewed, for better and for worse, as whether or not this leader actually has the standing to drive some things home. So I think when a President has lower approval ratings like you see right now, you see different dynamics happening in his party, and in the Democratic Party you see people being a little bit more vocal than they might have been. And again, it goes back to, I think, the unfortunate consequence of people listening too much to the polls and not listening to the substance of different arguments and all of that. But I think it matters at that very elite level. For most Americans, they probably don't pay much attention to where he's at.
MR. MACINNES: Thank you, Brian. I think if you have other questions, you can come up afterwards, but let's break up. Thank you very much.
MR. KATULIS: Great. Okay, great. All right. Thanks for having me.
