Daddy told me there'd be days like these

The last two days have really kicked my ass at work.

A lot has gone on in the Market Research industry over the last 10 years but the majority of it centers on the Internet as a Data Collection methodology. I'm very lucky to have led my company "operationally" through the migration phase of this change and therefore have a unique viewpoint at how it should be done and how it should be positioned.

Any and all work though, that's been done can be shit-canned with one or two industry rag articles. Case in point, Mr. Jack Neff's article from yesterday's AdvertisingAge.

Mr. Neff and I sat at the same table in September of 2006 at an IIR Industry event where 25 of today's Market Research leaders gathered at a round table to discuss Respondent Cooperation. Weirdly, the topic segued to quality of the online respondent pool for market research. Although a few made statements that said, online shouldn't be the ONLY thing under investigation, since today's culture of needing information quickly and cheaply have driven research online, people stayed riveted.

Don't get me wrong, throughout our march to the Internet, we had plenty of horror stories, so I'm not saying that there's isn't a problem. However it's a problem that more exposed than any problem that could have occurred during the move from mailout surveys to f2f, central location testing to the phone. Information and thought just moved SLOWER then and the sampling frames were more available and frankly, market research was a small business back then.

What Mr. Neff doesn't do is recognize that changes have been made already in the Industry. Any conference I've been at since that Sept '06 one, have had special focus on the online quality. So heralding the ARF for the creation of a Council is far too late in the process if you ask me. Additionally, I'm not sure of Mr. Neff's background in Market Research, but when I sat there and watched the P&G presentation, I could only ask myself "What supplier told P&G that that study should have been done online"?. I know that hindsight is 20/20, but seriously, I know that if one of my Client Service folks came to me and wanted me to quote a project similar to that one using online Data Collection, I'd have a serious discussion with them about why it was WRONG. No one in their right might calls out P&G in a public way because they are always the 800 pound gorilla, but they also said that they had done hundreds of side-by-sides, most of which were fine. So in a public way, they used a potential outlier to set the course of future policy.

My last rant about this article is the incessant re-use of the comScore quote. If I see it at one more industry event, I'm truly going to rush the stage and scream fraud at it. It's constantly being used as an intro to either defend other methods, or as a sales technique by online panel companies to scare you to use their panel. At least in THIS article, Mr. Neff states that comScore themselves (Gian Fulgoni, specifically) said that the original press release and research was an attempt to try to get MR companies to buy their own panel that they had created.

Folks, if you read this current article you have no idea that that quote is over three years old. It was either 2003 or 2004 when comScore was invited to my company to present their options. a) I implore comScore to update their data for god's sake and b) I invite them to also present details and findings behind the audience that they draw that sample from. They can only quote that information from the people who willingly (or unwillingly) install their "researchware" on their computers. Researchware is truly AdWare/SpyWare but used for honest research purposes.

When they were last quoting numbers, this panel only represented less than a million households/people. In a country where 302,000,000 people exist, and 70% of those are on the Internet, their universe of sample represents roughly .47% of the online population in this country. I'd like to see the correlation between those who download research ware and those who are in online panels. The quote is used (time and time again) for shock value. What Mr. Neff doesn't finish off by saying is that Gian Fulgoni said it was "to drive business to buy their sample offering, and that in the end, it didn't work".

So for the last two days, I've spent the majority of my time on the phone with Client Service teams, clients, and panel partners re-answering all of the questions that Mr. Neff's September 2006 article brought up again. Although I'm more than happy to do this as we are in the service business and our clients deserve all of the answers and rationale behind our decision to do research on the web, the muck-rakers will continue to exist who don't know all of the facts and details.

As an industry, frankly, we do very little to set industry standards in the United States. By far, our European counterparts go far and above the rest of the world in setting global "seals of approval" that MR companies need to comply with (and are audited against) in order to maintain the level of business that they do. We've needed that for a long time in the US, but I think the spirit or attitude of "free market" gets in the way. So good on ya' to the ARF and to IIR for keeping this in the front of everyone's mind, but let's seriously set those industry standards that can be audited and certified versus just verbal posturing to keep the clients at bay!!