Can We Measure The Quality of a Healthcare Conference Twitter Stream?

Can We Measure The Quality of a Healthcare Conference Twitter Stream?

We’ve recently looked at a couple of record-breaking healthcare conferences (one here in the States and the other in Europe) as well as the value of social media for a healthcare conference specifically. The first two mentioned blogs looked at tweet volume, and the third used participant numbers as a metric. Tweet volume is a fun and very easy-to-use comparison metric, but unfortunately it doesn’t really give us much insight into the quality of the conversations taking place.

Measuring healthcare conference tweet quality by quantitative means

If tweet volume as a metric fails to measure quality, are there other quantitative metrics that can? Are there ways to measure this other than reading through all tweets and making a subjective judgement? I’m sure glad we have several excellent curators providing summarized blog posts about the conference and a short list of carefully selected tweets from the conference stream. Human curation can never be replaced.

Healthcare Conference Engagement

Let’s compare some metrics that may tell us something about the social media engagement of healthcare conference participants. We’re going to look at three very interesting conferences that recently took place: #himss12, #TEDxMaastricht and #TEDMED.

Can we measure how many attendees at the healthcare conference were socially active?

Obviously, there is no way for us to measure who was actually physically present and who was attending virtually. Unfortunately (at least for this study), less than 1% of tweets have a geo tag associated with it.

What is a good metric value? It really depends on the type of the conference. #himss12 is not just presentations and sharing of ideas; it’s also partly a very large trade/exhibition event from which one should expect less healthcare social media engagement. #TEDxMaastricht and #TEDMED are both pure idea- and knowledge-sharing conferences, highly conducive for social media sharing.

we should expect the social participation of a conference to exceed the physical participation as a new norm.

I would like to suggest that a 1:1 ratio of physical and social participants should be considered a success for such conferences. As the rapid growth of healthcare social media continues, we should expect the social participation of a conference to exceed the physical participation as a new norm.
healthcare conference social active metric

Engagement distribution in a healthcare conference

We’ve looked at tweet volumes and participation ratios, but what about the distribution of the conversation? Volume and participation metrics can still give us the wrong story if the distribution is highly skewed. The assumption we would like to make here is that conversations that are more equally distributed among the participants are of higher quality compared to a tweet stream that is highly skewed by a few participants with a very high tweet volume. This assumption may not hold true, since the top tweeters by volume may indeed provide much value in all their sharing.

Average Tweets per Participant and Median

One very simple metric is the average tweets per participants, which is easily skewed by the top tweeters. The median, the numeric value separating the higher half of tweets per participants from the lower half, can in many cases give a more accurate picture with less “bias”.
healthcare conference participation average and median

Social Engagement Distribution Quintiles

The average and median are two very simple metrics loved for their simplicity (a single number to compare), but they very often don’t tell the real story. For a better impression of the engagement distribution, we’re going to visualize the percentiles of participant tweets, more specifically the quintiles. By comparing each 20% block of all the participants ranked by tweet volume, we can perhaps more easily compare the equality of conversation distribution across conferences.

healthcare conference participation distribution bar chart and trendline
From the colums above, we can quickly observe that the tweet volume of the top 20% tweeters of the #himss12 and #TEDMED conferences is proportionaly much larger than the next next 20%. There is clearly less difference for the #TEDxMaastricht conferene. This insight becomes a little more clear when we look at the trendlines. Our assumption is that more equally distributed tweet volume is better; in other words a flatter trendline is more desirable. We can visually see that the green line (#TEDxMaastricht) is flatter. That can also be confirmed by observing that the slope of equation behind the trendline is less steep (-2.664x) compared to the other trendline equations.

Pie Chart Visualization of Healthcare Conference Participation Distribution

The best visual for comparing tweet distribution is perhaps good old pie charts. In these pie charts we can see the proportion of tweets from each quintile in the context of the total number of tweets from the whole healthcare conference. Again, we can observe that #TEDxMaastricht had a more equally distributed conversation which may signify a broader participation and perhaps higher quality with less outliers.

healthcare conference participation distribution pie chart

Audun Utengen, MBA - @audvin

Job to be done: Connect the dots in healthcare social media. Co-founder of @symplur.

5 Responses to “Can We Measure The Quality of a Healthcare Conference Twitter Stream?”

  1. @blogbrevity

    Angela Dunn - @blogbrevity

    Very interesting analysis, Audun!

    It would also be interesting to analyze content, the number of RTs as a ratio, and the reach of the Twitter handles. TEDMED is a bit of an outlier because they had many physical locations spread throughout the country that had free access to a livestream, but would have been considered virtual.

    If we are looking for quality and the actual spread of ideas, I would also like to see all the tweets eliminated from the analysis that said, “Get this conference to trend now!” :-)

    Reply
    • @audvin

      Audun Utengen - @audvin

      Thanks so much Angela :)

      Those are excellent ideas. In some ways RTs can be seen as “citations”, so we can use similar citation rankings as in published journal articles etc. I will definitely take a look at that.

      I also have some other ideas for slicing these datasets in hope to discover more insight. Keep the ideas coming!

      Audun

      Reply
  2. @ShimCode

    Steve Sisko - @ShimCode

    I think it would be hard to every measure true social media value without some sort of adjustment for “qualitative reach” and “qualitative consumption.” By qualitative reach and consumption I mean some sort of adjustment for the “content of the tweet” in relation to the “topic knowledge level” and “perceived authority” of those who are doing the tweeting and those reading the tweets.

    “Content of the tweet” – What is the tweet about?

    “Topic knowledge level” – How pertinent, germane, “useful,” etc. is the tweet topic to sender and receiver?

    “Perceived authority level” – What is the level of skill, experience and “authority” of the sender and/or receiver?

    For instance, I consider myself somewhat knowledgeable about ICD-10 and healthcare IT systems integration. I believe many people follow me because I share that kind of information and have been involved in healthcare IT for a long time. In terms of my followers (my reach), I ‘prune’ XXX, MLM and other worthless-type of followers in an attempt to cleanse my overall “follower quality” – although I’m beginning to question the value of this pruning effort. :’)

    So if I’m at HIMSS12 and I tweet something about ICD-10 then I think the ‘quality’ of that social media interaction – for my followers – should be measured higher than if I’m tweeting tips on how to knit socks from the HIMSS exhibit floor.

    Conversely, if I’m a Vegas hooker cruising the halls and tweeting about ICD-10 to her Secret Service followers, then the value of that social media interaction as it relates to the ICD-10 topic would likewise score lower; regardless of any latent interest said followers may have about ICD-10. And if I, ShimCode, were to tweet about great places to eat in Cartagena Columbia from the HIMSS12 exhibit floor, the value of that social media interaction is essentially worthless – at least to my followers at that point in time.

    So I think it will be very hard to ever truly measure social media value on quantity alone.

    :’)

    Reply
    • @audvin

      Audun Utengen - @audvin

      Steve, thanks for the excellent insight you’ve shared with us.

      You’ve described a dream scenario :) If we would have the data points you mentioned, we would be able to quantify much more accurately the “quality” of the twitter stream.

      Two of the data points I believe we could be able to get hand of; “Perceived authority level”, and “Topic knowledge level”. Given a large enough database, which we have, this can be extracted.
      However, what I actually perceive as a difficult is to extract the topic of an individual tweet. In so many tweets, without the context, it often impossible to know the true topic of the tweet. Often times, the most valuable tweets, have no topical keywords in the at all.

      Challenges to be solved :)

      Reply
  3. @MedicalMarcom

    Joe Hage - @MedicalMarcom

    Nicely done, Audun.

    I wonder if there is something prescriptive here so social participants can get a sense *before* the event which will be best to follow.

    P.S. I could not find a way to enlarge the text as I read. Perhaps you might consider a larger default font?

    Reply

Leave a Reply