• Marketing Mag Website

  • Marketing Blogs

headshot

by Katie Harris

on Feb 23

Fool's gold?

Advertise with us

Social media monitors (SMMs) trawl the web to find mentions of your brand, or whatever it is that you're interested in monitoring. There are many SMM products and services available: some free, some that you pay for.

Here’s a very basic example: www.whostalkin.com

If you type in the name of a brand or topic of interest, you’ll get an idea of the kind of information SMMs return.

Depending on the level of sophistication built into the SMM you use, you can refine your search with key words, run analytics etc.

There’s a lot of hype around SMMs. Not surprising really. The idea of getting feedback on the cyber-buzz around your brand, product or service is timely and sounds quite marvelous!

Kind of. Until you think about it a bit more. Which I have. And wearing my qualitative researcher’s hat, SMMs fail in two important ways:

  1. Sample definition
  2. Sentiment

Sample definition

What constitutes a SMM sample?

In a nutshell, a SMM sample comprises the searchable/findable content sourced from various online channels. That's as precise as you can get really. The truth is, you just can't know who's represented (or not) within that content.

For example, SMMs can’t identify and screen out marketers who may be posting from domains that haven't been identified as such. This means that in many cases, SMMs can’t distinguish between content generated by marketing folk and content generated by non-marketing folk.

And let's face it, quite a lot (most?) of the brand chatter out there is actually generated, nurtured and sent bouncing around the interwebs by marketing folk. People like us. The kind of people we try very hard to screen out of market research samples.

It’s also worth noting that SMMs can’t automatically, distinguish between content generated by core customers, infrequent customers or non-customers. This means that all customer/brand relationship variations are automatically given the same share of voice and weight in the analysis.

Another factor to consider is that the sample will be skewed. And while a sample skew, in itself, is not necessarily a problem, it's certainly a problem when you don’t know how it's skewed. Which is the case here.

Without being able to define the sample, and without knowing how the sample is skewed, there's no foundation or context for meaningful content analysis.

Sentiment

Sentiment is the very essence of what we're trying to understand through market research. And this is something that SMMs don't gauge very well.

Although automated sentiment analysis is often sold with the SMM package, there are two things about it that trouble me:

  1. Accuracy
  2. Specificity

Accuracy

There seems to be considerable scope for error in the labelling.

For example, how would automated sentiment analysis label a statement such as "F&*#ing brilliant!"?

Depending on the context, this statement could be:

  1. Dripping with irony
  2. An exclamation of genuine excitement and joy
  3. A description of a high-wattage light bulb

So, would it be labelled as negative, positive or neutral?

Notably, some SMMs claim to be contextually savvy, and that they can identify positive, negative or neutral sentiment with 90% accuracy (is 90% good enough?).

BUT…

Specificity

BUT (note caps), even with 90% accuracy, these labels are still seriously wanting. They don't provide me with information that's of much use – if any – because they're too vague.

It’s the finer points of 'sentiment'; the despair, frustration, excitement, boredom, curiosity etc. underlying the positive or negative sentiment labels that I’m interested in. This is the level of sentiment I need if I’m to understand what’s going on with any effect. And to get to this level of sentiment, I really need to dig a bit deeper.

Where’s the gold?

I need to dig deeper, but where do I begin? Back to my earlier point about SMM sample definition and skews; I don't know where the real gold (vs. fool's gold) lies.

Without spending the time and effort to sort through each and every 'buzzversation' (easily in the thousands), I can't distinguish between content of import and that of little consequence. I just don't know where to drill deeper in a meaningful, robust kind of way.

So it's virtually back to square one.

Itching

SMMs are an exciting idea and I'm itching to find a way to use them.

From a PR or customer service point of view, I imagine they're worth their weight in (real) gold.

But in my qualitative market research business, I can't (yet) use them with either confidence, or pragmatic effect.

Sentiment aside, the sample scope/limitations and the unknown skews preclude the output from forming anything approximating a solid foundation for analysis.

Bit of an issue for me.

Related Articles

6 Comments

Editor | Kate
  • Wrote on 26 Feb, at 04:37PM
Hi all,

Apparently there were some problems with people trying to post comments in this thread. Just tesing if this works.

Kate
jennibeattie
  • Wrote on 26 Feb, at 04:43PM
Thank you for raising this timely issue concerning many PR and
marketing managers - the value of SMM.

I agree that the issue of ‘sentiment’ is the missing link and without
it buzz monitoring or SMM’s cannot add the value that our clients
require.

To address this need we have created a SMM that also contains the
important ‘sentiment’ component. In addition this sentiment component
addresses the issue of ‘contextualisation’ - ensuring the results are
valid.

As you mentioned and I wholeheartedly agree it’s the finer points of
sentiment; the despair, frustration, excitement, boredom, curiosity
etc. underlying the positive or negative sentiment labels that I’m
interested in.

To explore these unsolicited consumer comments identified by the SMM
we use an exploratory Market Research Online Communities (MROCS).
These are bespoke Online Communities set up for clients where
participants are identified from our Online Panel to ensure they fit
the key demographic and psychographic data. We then explore some of
the dominant themes (both negative and positive) that were raised in
the SMM. The interactive nature of the MROC platform along with an
experienced moderator ensures we get closer to those consumer truths
that we seek.

Your concern about needing to go into “each and every buzzversation
(easily in the thousands) is certainly valid however reputable SMMs
will have functionality to help you identify those key influencers
within the sector you are interested in. This then helps you drill
deeper in the areas that really count.

Finally, SMMs are not the panacea that some may have imagined but
coupled with a valid ‘sentiment’ component and exploratory MROCs we
have found the combination makes for a powerful tool.

Cheers,

Jenni Beattie
Digital Director
The Digital Edge
http://www.thedigitaledge.com.au

Further Information:

Info on SMM
http://www.youropinion.com.au/TLE/QuantWebCrawl.pdf

Info on MROCs
http://www.thedigitaledge.com.au/the-digital-edge-blog/2008/11/17/have-you-heard-the-acronym-mroc.aspx
http://digitalministry.com/AU/articles/664/Online+Community+Management+The+Fundamentals+apply/1
ZebraBites
  • Wrote on 2 Mar, at 10:38AM
Jenni

I think you completely missed my point about sample issues.

In most cases, the online panel you say would make up your online community just can’t constitute a meaningful or representative sample. And that’s even if you look beyond the demographics.

There are so many problems and issues with online panel samples that it seems like a seriously questionable place to start recruiting for an online community.

Given these factors, I believe your methodology is fundamentally flawed.

P.S While I think it’s great when people contribute to the discussion, I don’t know if my blog is an appropriate place for simply promoting your company’s products. If you want to advertise, there’s a section in Marketing Mag called Blow Your Own Horn.
jennibeattie
  • Wrote on 3 Mar, at 02:40PM
Hi Kate,

Thankyou for the opportunity to further explain our methodology. We take our research integrity and
the investment we have made to build it very seriously.

Regarding your point about online panel samples having problems and issues I disagree.
On-line is now a legitimate channel for representative research and this is why:


• Almost 85% internet penetration 18-64 year old Australians -soon to be higher than fixed telephone penetration.


• Australia has the highest proportion of on-line research in the world (1/3 of all research and over half of all quantitative expenditure done on-line) ESOMAR 2007


• The Australian Market Research industry (AMSRO and AMSRS) is taking a stance on Panel quality. The QSOAP standard is designed to ensure any research buyer feels comfortable about the sample of respondents their research agency uses to collect information - and thats why we have sought this accreditation for our panel of almost 60 000 Australians http://www.youropinion.com.au

We take great pride in our nationally recognised community that is managed in house and managed with extreme care and used for research purposes only.

We believe that we know a lot more about the people who take part in our MROCs (market research online communities) before they take part in the discussion, than your average qualitative researcher knows about the people who turn up to their group rooms. We have far more stringent processes to ensure they are who they claim to be.

As with all approaches however there is always room for improvement. I would value any suggestions you have on where you would go to recruit for Online Communities participants to increase validity.

Finally, rather than being self promotional I trust my comments have helped inform the discussion.

Cheers,
ZebraBites
  • Wrote on 3 Mar, at 08:37PM
Hi Jenni

Thank you for your comments. It’s an interesting discussion.

There's a compelling article on page 13 of this issue of Marketing Magazine on this very topic: 'Just how accurate are online surveys?' by Dr Brent Coker from the University of Melbourne.

It’s a well-considered point of view on some of the problems with online panels, particularly the issue of incentives and motivations.

I think the fact that AMSRO and AMSRS have taken such a position on panel quality strongly suggests that there are, indeed, issues and problems with online panels.

In answer to your question on where to recruit for an online community; assuming 85% internet penetration translates into 85% internet conversance (albeit unlikely, and a key issue in itself), then offline would probably be a reasonably good place to recruit for online communities.

Cheers
Katie
Ben Werzinger
  • Wrote on 22 Jan, at 01:28AM
Katie - great post as usual! Agree 100%. Even if sentiment is 100% accurate, not knowing who it represents is very troubling.

Cheers - Ben

  • Both the above fields are required for unauthenticated comments. * indicates a required field.


    *

    * This email will not be available or visible to anyone.
    (optional)