20 August 2009

Your Research Is Wrong

Here's why the research stats you constantly pull out are flawed.

Chatting yesterday, myself and two mates realised that the three of us had all signed up to participate on a number of market research websites. This included online surveys, attending focus groups, taste tests and being sent free products. Remuneration has taken place in the way of money, vouchers and of course free stuff.

It doesn't matter that all three of us are studying marketing which is something you're meant to declare at the start of every survey. It also doesn't matter that I'm now at a point where I know how to manipulate a survey to ensure I fit the criteria and am therefore selected. It also doesn't matter that once passed being selected, I know how to answer the questions to make the survey end as fast as possible and receive payment.

My other two mates were the same, and I imagine we're not the only three people doing it. In fact as we sat there, they were fabricating a story to report back in order to be sent another free bottle of alcohol.

I'm a poor uni student, that's my excuse anyway. Now I haven't written this post for you to lecture me on how I shouldn't be doing blah blah immoral blah blah ethics, but rather so you realise your expensive research is probably not that accurate. Thanks for the free shit though.

10 comments:

  1. Hahaha so true. I can just imagine...

    "So, why do you smoke despite knowing about the health risks?"
    Uni student pauses, takes a drag from his free cigarette. "Because I want to DIE. Next. Do I get another cigarette?"

    Later, on Mad Men...

    ReplyDelete
  2. Care to share the websites you use for the other Uni students reading this?

    ReplyDelete
  3. @ James Rose

    I'll DM you on Twitter.

    ReplyDelete
  4. As much as i hate agreeing with sweeping generalisations based on the actions of one person and their close friends (can you tell i used to be a reasearcher?), I basically agree.

    So what's the solution?

    Before anyone suggests it, a new age of self-confessed gurus with no experience or backing to their crazed theories is not a solution.

    To be moderately serious, how about researchers forget about sample sizes and stat validity and just ask people who give a crap questions that they want to answer...and gee maybe even feed the results back to them (not useless charts, genuine client actions) so they know someone actually listened (assuming they did that is).

    just a thought...

    ReplyDelete
  5. @ Scott

    Interesting comment, particularly given your background!

    A really good point. On the few times I've done something that interests me, or given a product that I'm actually passionate about, I will put in the effort to do it properly if I know it will make a difference. I'm passionate about the product and brand, so why wouldn't I?

    ReplyDelete
  6. Exactly why qualitative data should never be considered a luxury "add on" to quantitative data like online surveys. It takes more thought and time to collect but that's exactly why it's more useful - you can usually weed out the Zac Martins. ;-)

    ReplyDelete
  7. Firstly Zac, it's always tempting to make insightful conclusions based on a sample group of one (or, in this case, three).
    But whenever any client, colleague, or random begins telling me what their wife/sister/daughter/friend thought of an idea/campaign/product, I zone out.

    Yes, you're not the only ones screwing with research. But this is why these companies use large pools of people, to reduce the variance to become as close to negligible as possible.

    Having said that, I'm not saying I wholeheartedly endorse this sort of research, but I wouldn't overestimate the impact you're having on it. ;)

    ReplyDelete
  8. @ Nic Hodges

    Unfortunately all I have are insights into my experience, but maybe someone should do some research into it... ;]

    ReplyDelete
  9. Well if you've done a marketing degree lately (at least at Monash - go team) the lecturers will have told you much the same as what Zac has confirmed here. People participate in focus groups etc. for the money and free food/booze.

    @ Scott

    I definitely agree, the best way to find out about your target markets is to find a dedicated and passionate group of individuals.

    Spend a bit of time finding and engaging these people. The best thing is, you shouldn't need to incentivise them because these people want to participate! (Malcolm Gladwell talks about 'Mavens' in his book The Tipping Point, these are the sort of people you want to try to find).

    There's an advertising agency that did something similar to this recently by engaging marketing uni students to do research for them. The uni students got heaps of practical experience out of it and the agency got awesome results. Win-win.

    ReplyDelete

The views expressed herein are my own and do not necessarily reflect those of my employer. Also ponies are evil.
Pigs Don't Fly © Copyright Zac Martin