There’s a lot of chatter around Big Data and for good reason, too. That’s because Big Data promises to fundamentally change the way a lot of business is done.

Actually, it’s more than a promise: smart companies in New Zealand today are already putting Big Data to work to improve the way they do business.

Whether it is providing insights into how they interact with and meet the needs of their customers, delivering services better or anticipating needs and wants, Big Data can provide the opportunity to not merely meet customer needs, but to delight and amaze.

But first, what is Big Data? The semantic definition is obvious and necessary, but not sufficient. Big Data is indeed ‘just’ a whole lot of information. A whole, whole lot.

And that means a more detailed definition quickly becomes necessary, since Big Data means not only volumes of information, but also types. Structured, unstructured, machine-generated, human-generated.

It also means data which cannot be contained in ‘traditionally’ defined repositories: it is a collection of large and complex data sets which cannot be managed or analysed with traditional database management tools or analysis technologies.

Big Data therefore means the ability to use this information; Big Data analytics goes a step further to describe the techniques to use it.

Perhaps most importantly, Big Data is a relatively new concept, too – so the definition itself is subject to change as is the practical application of techniques and use-cases. We’ll revisit the definition at the close of this article.

The source of Big Data

Where does this information come from? For starters, there are some seven billion people on the planet.

As we increasingly move to a digitally-connected society, our smartphones, computers, tablets and other personal devices enable us to create and store information at astounding rates.

In a single day, humanity creates 70 times more data than the information contained in the US Library of Congress.

There’s a lot more to it. The number of people is dwarfed by the number of connected devices which create and transmit data from one point to another, creating repositories of information which contain the potential to glean value. Machine-to-machine communication generates data at unprecedented rates.

There are billions upon billions of sensors ranging from devices as large and complex as the satellites which orbit the earth to as tiny and simple as the RFID tags which track the movements of consumer goods.

This data holds the promise of delivering the ability to sense, measure, track, understand and know more.

Back to business in New Zealand

But is Big Data something happening in a galaxy far, far away? Far, far from it. As you might appreciate, Big Data surrounds us on a daily basis.

Each of us participates actively in its creation, as we use social media, do our jobs, play our roles in the connected society.

The businesses which are likely to benefit first from the emergence of Big Data tools tend to be those which service large numbers of customers.

Into this broad definition fall telecoms, electricity, retail and financial services. The sorts of organisations with which you are familiar in your daily life and which, when you engage with them, can have you feeling like you are just a number.

One such practical application is that of an electricity retailer. With the ability to gather information at a highly granular level, across its supply chain (from generation, to transmission, to metered supply right down to an individual domestic consumer equipped with a smart meter), this company can provide an off-peak service to heat that customer’s spa pool in the early hours of the morning.

That sounds simple, but in the absence of accurate information, providing this sort of service in the past was the preserve of factories and other large electricity consumers.

That’s because the cost of gathering and managing the information only made the service practicable at scale.

Today, it can be done for one domestic consumer – who is one of tens of thousands of customers who can benefit from the same highly targeted service level. It is this ability to delight that differentiates one electricity retailer from another.

Additional practical applications

Other companies are using Big Data analysis to identify when customers are about to jump ship. By analysing volumes of information from sources which can include emails, web chats and telephone conversations, key words can trigger alerts that signal when a customer is unhappy.

Acquisition is harder (and more expensive) than retention, so when trouble is detected, measures can be taken.

This is related to the concept of sentiment analysis, which provides brutally honest insight into what people are actually saying about your company and product on social media platforms.

Knowing how your company is perceived provides the ability to do something about it.

As the techniques and solutions which are available to take advantage of Big Data are made more accessible, so the impacts of how it is used are likely to be felt across all industries.

For example, it is being used to detect fraud, analyse customer satisfaction and enhance risk analysis. It is being used to gain better insights into our humanity as individuals – our wants, needs and desires.

But perhaps most important, Big Data is not being used to exploit our humanity, but to improve it. The companies which do so successfully will leave – and are leaving – those behind who do not.

By Arron Patterson CTO, EMC New Zealand