by: David Wigder
This week, Jupiter Research released the results of their latest online survey: 38% of online teens are “concerned about the environment.” Interestingly, MediaPost reported that JWT’s March 2007 survey indicated a much higher response by online teens: “more than 80% of American teenagers are concerned about the environment and the role of the United States that is causing pollution”.
At first glance, it seems that these two companies have published dramatically dissimilar responses to a similar audience. (“Green Teens”, Jupiter Research, July 13, 2007; “Advertisers: Teens Value Environment, Buy From Socially Responsible Companies“, MediaPost, March 20, 2007)
If that were the case then how did “concern for the environment” by American teens fall so dramatically in four months? How do marketers and strategists interpret seemingly disparate results from a seemingly similar online audiences?
It is presumed that both surveys generated statistically significant results, though likely with different levels of confidence as JWT surveyed 767 teens to Jupiter’s 2,091. Moreover, Jupiter polled teens ages 13-17, while JWT polled teens 13-19. But, it is reasonable to assume that this does not account for the dramatically different survey responses as the mean age for teens polled by JWT was 14.6 years old.
The simple answer may be that survey questions and the reporting of survey results can sometimes be misleading. JWT’s survey provides a representative case study. Here is the timeline:
1) In mid-March JWT and RelightNY, an organization trying to “educate and inspire” people to take action to protect the environment, released a survey in which it reported that “Nationwide, 79% of teens are bothered by the fact that America has emerged as the world’s leading pollution source.” (“Ten Stats about Teens and the Environment”, March, 2007; Advertisers: Teens Value Environment, Buy From Socially Responsible Companies)
One interpretation of this statement is that the US has only recently become (“emerged” as) the leading polluter in the world. As it is worded, the question introduces a bias by implying that the US has changed its status and is now the world’s worst polluter. This is likely to have influenced more teens to respond affirmatively to the question than if the question had simply asked teens if it “bothered them that the America was the world’s leading pollution source.”
2) On March 19, PRNewswire published a story entitled “JWT Survey Reveals Four of Five U.S. Teens Share Concern for the Environment”. In the body of the text, PRNewswire clarified the survey results (and their headline) stating that “Over 80% of American teens are bothered by the fact that the U.S. represents one of the world’s leading sources of pollution”.
By rewording the question in the article title as “concern for the environment”, PRNewswire effectively changed how readers interpret the survey results (and especially for those who only scanned headlines). Only below in the copy did PRNewswire clarify that the question was referring to the US as a major polluter. Moreover, note the subtle but effective changes in wording of the original survey results by PRNewswire that makes the high level of response by teens all the more dramatic: “Over 80%” vs. “79%” and “represents one of the world’s leading sources of pollution” vs. “the world’s leading pollution source”.
3) On March 20, MediaPost filed its story on JWT’s survey. In this article, JWT writes that its “online study discovered that more than 80% of American teenagers are concerned about the environment and the role of the United States in causing pollution”.
This rephrasing of the original survey question can be interpreted in several ways, albeit incorrectly in each case. First, it implies that the original question included the phrase “concern for the environment” which it did not. Perhaps MediaPost incorrectly repurposes wording from PRNewswire’s story the previous day.
Second, it shifts the emphasis of the question towards “concern for the environment” (because it is read first in sequence) rather than what was actually stated in the original question regarding pollution. Finally, it decouples “concern for the environment” and “the role of the United States in causing pollution,” implying that they are two separate survey responses and that teens responded equally to each of them.
Again, you see subtle rephrasing of survey questions or perhaps again incorrect repurposing of PRNewswire’s story. MediaPost state that “more than 80%” of teens responded affirmatively to the survey question vs. “79%”.
The lessons are clear: surveys and the reporting of survey results can be misleading. Marketers should be weary of very high (and low) responses to questions (or any result that does not pass the gut check), as they are often a sign that the question was leading or unclear to respondents.
Moreover, marketers should always go back to the original source to review questions and responses first-hand. Try to understand in what context the question was asked and how it could have been interpreted by respondents. Finally, understand at what confidence level are the results valid.
In a world where environmentalists have been accused of inflating statistics to bolster their case, it is an imperative that this data is gathered, reported and interpreted accurately. Our understanding of consumer attitudes on the environment depends on it.
Original post: http://marketinggreen.wordpress.com/2007/07/19/correctly-sizing-up-%e2%80%9cconcern-for-the-environment%e2%80%9d-by-american-teens/
