By now it’s well known that making effective use of data is a competitive advantage. But how advanced is your
organization at making use of data?

Over the last few years I have spoken to many organizations on this topic.
One thing I’ve learned is that all of them go through the same learning process in putting their data to work.
From Silicon Valley giants to industry companies in Asia and government entities in Europe, all go through the
same main evolutionary stages. I call these the big data maturity levels.

Level 0: Latent

Data is produced by the normal course of operations of the organization, but is not systematically used to
make decisions.

Example: A movie streaming service is logging each movie viewing event with information about what is viewed,
and by whom.

Level 1: Analysis

Data is used by humans to make decisions. Analysts extract information from the data, such as graphs and figures
showing statistics, which is used by humans to inform their decision making.

Example: A movie streaming service uses logs to produce lists of the most viewed movies broken down by user
attributes. Editors use these to create curated movie recommendations to important segments of users.

Relevant technologies at this level include traditional data warehouses, data analytics platforms such as
Splunk and Elastic Search, and big data query engines
such as Spark.

Level 2: Learning

Data is used to learn and compute the decisions that will be needed to achieve a given objective. Rather
than making each decision directly from the data, humans take a step back from the details of the data and
instead formulate objectives and set up a situation where the system can learn the decisions that achieve
them directly from the data. The offline system both learn which decisions to make and computes the right
decisions for use in the future.

Example: A movie streaming service uses machine learning to periodically compute lists of movie
recommendations for each user segment.

Relevant technologies at this level include machine learning tools such as
TensorFlow and PyTorch,
machine learning platforms such as Michelangelo,
and tooling for offline processing and machine learning at scale such as Hadoop.

Level 3: Acting

Data is used to make decisions in real time. Rather than pre-computing decisions offline, decisions are
made at the moment they are needed. This makes it possible to take all relevant information into account
and base decisions on up-to-date information about the world.

Example: A movie streaming service computes recommended movies for each particular user at the point when
they access the service.

Relevant technologies: Some times it is possible to make decisions by considering a single data point.
In those cases model serving tools such as TensorFlow Serving,
or stream processing tools such as
Storm
and Flink may be used. In general — as in the movie streaming example –
multiple data items are needed to
make each decision, which can is achieved using a big data serving engine such as
Vespa.

Conclusion

Taking a step back and reflecting on the maturity level of your organization (or team — organizations
don’t always evolve in synchronicity) can be helpful in understanding the current type of challenges you
face, what kinds of technologies you should consider, and what’s needed to move to the next level in
your organization.

I hope this post has been helpful in this — it’s the first post in a series exploring this topic. In
the next posts, I’ll take a look at the forces that pushes the worlds most advanced organizations to
move to maturity level 3, the benefits they see from making this move, and why this has traditionally
been so hard to pull off.