Looks cool – but looks can be deceiving!
In this article, called XBOX Visa Case, the
benefit of Visa’s sponsorship of the XBOX live tournament is discussed. The
data analyzed is reported in the text and represented through histograms. The
histograms are interwoven with other tournament displays down the side of the
article suggesting that they are not the focal point for readers. Rather, the
represented data seems meant to offer the numerical support for the stated
benefits of Visa’s sponsorship. Also, they lend bright colors to the article. I
do think the histogram approach is appropriate for highlighting the findings,
however, there are some issues with the displays as will be discussed below. Specifically,
I am going to focus on the areas where histograms are used in addition to the
text.
1. The section “Visa’s Success With XBOX LIVE Tournament Sponsorship” talks about the attitude towards Visa. It appears that data was collected from two groups, those in the age group from 18-34 and those in the age group from 35-49. From the text, we might assume that all respondents were mostly male. It is not clear how many respondents there were, nor how data from respondents outside of these ranges were handled. This brings me to some questions: Does minimalizing other age groups matter? Perhaps the rationale was that people under 18 cannot get a Visa, but what about 50 and over?
Additionally, the way attitudes were
measured could be problematic. It appears that all respondents could choose
whether they agreed or not to each type of attitude (i.e., cool, fund,
innovative/cutting edge, for someone like me). An initial problem for me is
that these terms do not necessarily represent “attitudes” for me. To further
look into this, I went to Wikipedia and found this statement about the
measurement of attitudes “An attitude can be defined as a positive or negative
evaluation of people, objects, event, activities, ideas, or just about anything
in your environment, but there is debate about precise definitions” (located at
http://en.wikipedia.org/wiki/Attitude_%28psychology%29,
accessed 09/17/2012). As such, it is not clear to me that “fun” is an
appropriate attitude to assess (but you may think otherwise!). But looking
beyond that point it does not surprise me that the numbers are so close (around
70-80%) for all items. Indeed, wouldn’t it seem likely that someone who said
Visa was Cool would also say they were Fun, Innovative, and For Someone Like
Me? Also, it makes me think of a bigger similarity in terms of people in these
age groups; is it that the 35-49 year olds have the same attitudes/mentality as
the younger crowd? Interesting!
2.
The section “Cementing Brand Loyalty” surveyed American adults that are current
card holders, past card holders, and those that have never had a card. Two
surveys were administered. The first was given before the tournament and the
second was given after the tournament. When I look at the “attitudes” of
respondents from pre and post surveys, there do seem to be differences.
Moreover, it is reported in the graphic that a capital letter is used to
indicate significant differences. Interestingly they use a confidence level of
90% which is less rigorous than typical (usually researchers use 95% confidence
levels) so why the relaxation of this test? This is where I start to wonder if
they are trying to make us believe there are differences when there really
aren’t (especially given the sample sizes being large). Speaking of sample
sizes, I note that 304 people filled out the pre-survey and 485 people filled
out the post survey. This raises a red flag for me because it seems that people
may have filled out one or the other – and indeed, because more people filled
out the post-survey it is obvious that some of them did not fill out the
pre-survey. What does this mean for the “change” in attitudes? Really, there
should be some assurance that the same people were surveyed before and after in
order for these results to be meaningful.
3. Next to the final section called “Tournament
Sponsorship is a Winning Proposition” there is a histogram showing the
“Perception of Visa Knowing Brand Sponsored Madden Tournament.” The sample used
for this histogram appear to be (based on the sample size) the same sample used
to complete the post survey for assessing brand loyalty. Overall, the
measurement scale to assess people’s perception of Visa sponsorship seemed the
most appropriate, valid, and reliable of all the scales used. This is because
the five-point scale where the anchors are opposites with seemingly equal
gradations between each middle label exists.
Great start! However, it is confusing as to how the data was compared.
It seems that within each category there was a test of whether the proportion
of respondents on that label differed. Weird. This might be meaningful but we
have no idea how many people are represented in each stratification of the
sample. For example, in the “Very Positive” bucket, 68% of current card holders
chose this answer. But 68% of what total sample size? This may or may not be important but worth
knowing more (just wait till we look at the equations for the proportion tests),
to consider it in more depth check out the book How to Lie with Statistics which discusses how small samples may not be
convincing so if sample size is not explicitly given – be careful.
Two other issues arise from this final
graphic. My first issue is with the capital letters. What do they mean and why
are they different/the same? Another issue is with the percentages. Why don’t
they add up to 100%? Specifically, for current card holders 68%+19%+13%+1% =
101%; for Past card holders 39%+35%+25% = 99%; for never card holders
46%+29%+22%+3% = 100%. Even if the reason is rounding, there should be some
acknowledgement of this fact.
In summary, there are many issues with
this article that concern me. What’s more is that this document is promoting
Visa by showcasing attitudes and perceptions about Visa’s sponsorship of the
XBOX Live tournament. While it may be that legitimate findings were made that
should promote Visa in the minds of customers new and old; the number of
oversights in this article make me skeptical about the ethical use of data.
Admittedly this article alone isn’t going to make me cancel my Visa – but it
makes me (and hopefully you too!) more alert about these types of practices.
Reference
Microsoft Corporation (2008). XBOX Visa
Case. Accessed online on 09/24/2012 at http://advertising.microsoft.com/research/xbox-visa.
