I’m a Data Stream Believer

I’m a data stream believer.

Let me explain what that means.

I see data arriving in a stream: I’ll either process and use it, or I let it pass me by. I don’t bother storing it.

Storing data is a popular paradigm (the “data ocean”) but I believe that people often fail to really understand that the value of data isn’t fixed – it only really has value in context and time.

For example if I tweet “hot cross buns going fast. come to the kitchen now” that data point only has value for a short period of time: a few minutes later, if you care about your buns being hot, and a little longer if you’re not so bothered. In any case, in an hour there will be no hot cross buns to speak of and the value of that data point has now decayed to zero.

To take advantage of the data stream, I need to listen to it and be watching out for key events. Events like those hot cross buns being available. To do that, I’ll either set up an alert for “hot cross buns” or I’ll make sure I follow the right twitter account. In both cases I have thought ahead and set up a listener.

I am now acting as a data stream believer. I’ve decided what I’m looking out for and set myself up with an alert for when it occurs. When it occurs I’ll then take action.

To me, this “data stream believer” should be our dominant strategy when working with modern data – particularly big and social data.

The alternative data ocean strategy is a lot harder to make work. Data ocean thinking goes a bit like this  – “I don’t know what data I really want right now but I am sure in the future it’ll be valuable. What we’ll do is store all the data and then when we need it we can query it. By storing everything we’ll never be caught short.”

I think there a couple of issues with this train of thought:

  1. Data storage costs just keep going up. Over time you’ll always need more storage as more data is created and stored. You’re not recouping the cost of storage anywhere so it’s just a black hole of cost. Admittedly data storage costs are going down but there’s still a cost somewhere.
  2. You aren’t using the data you do have. This is more important than cost – data oceans leads to woolly thinking and sloppy processes. You’re storing something but you’re not doing anything with it. It’s just sitting there gathering virtual dust. Even if its just cognitive cost (at the back of your mind you know that data is there) it’s still a waste of your attention.

Let’s make this real, with a consumer example of data stream versus data ocean thinking – digital photo storage.

Family 1 opts for a data ocean approach. Every photo they ever take is stored on the cloud. They never look at their old photos. Scanning through them is a boring chore as there are multiple copies of the same snap, a bunch of out of focus ones and some scanned expense receipts in there too.

Family 2 opts for a data stream approach. Every year they review the photos they took from the last year and choose 50 to go in that year’s family album. They get the family album printed, share it in the living room and never look at the cloud folder ever again.

Which family do you think is getting the most out of their data (in this case family photos)?

So we can see – a data stream believer takes the data stream and processes or discards it. Value is added immediately – and often it is pushed straight back into the stream.

It is with this philosophy how Rise works – data points are  pulled from the stream on each player, processed into scores and pushed back to the stream as a release of the leaderboard. This then feeds other people’s data stream activities – how did I do last week – time to optimise for next week.

The data stream creates a faster, more vibrant, feedback loop and uses data well.

How about you? Are you ready to become a data stream believer?



3 answers to Gamification and Big Data questions

Gamification and Big Data often go hand in hand. It’s worth asking a few key questions as to why and how:

1. Why is gamification a critical component of any data strategy?

Because gamification provides the context and reason for data sharing between player (e.g. consumer) and manager (e.g. company). i.e. loyalty programs are primitive examples of gamification – you exchange your data (shopping at the supermarket) in return for rewards (redeemable points).

2. “Manage Data like a stream not an ocean.” What does that mean in a Rise context?
With big data you cannot and should not be storing everything in the hope that it will one day be valuable (the data ocean / data lake approach) – this is a waste of resources. Data isn’t that valuable, it is tricky and expensive to store well and you’ll probably never use it. Instead you should be planning ahead, processing the stream as it happens and act at the time, using it to kick start other processes.
For example, Rise processes 90 days of social data into online scores for sales reps wanting to become better at social selling. Each week we create a ‘release’ which summarises the previous week. In so doing we keep a permanent history of their aggregate scores but delete the raw data.
3. Are data brokers like Rise going to be increasingly needed?
Yes, as data privacy becomes an increasingly complex area between employees, the company you work for and the apps you use – there is a key role for data brokers like Rise who protect and support the data interests of all parties. For example in social selling programs, we allow employees to share Linkedin data with their employer on number of new connections per month but do not allow the employer to see who those connections are.