To the uninitiated, the term dark data may evoke images of mysterious stores of data tucked away in an obscure location—hopefully, distant enough from your company’s hard drives so that they are no threat to online security. The reality however, is that there’s nothing ominous at all about dark data, it’s closer to you than you think, and more importantly, it can even bring value to your business when used right.

In this article, we define what dark data is and where it comes from, discuss ways to harness the value that it holds, and understand the challenges that may come with its management.

Dark Data: Definition

Research firm Gartner defines Dark Data as “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.” In other words, if you have tons of data in storage that remains unstructured, unclassified, unquantified, underutilised, or completely unused, then you have dark data.

Dark data is generated from systems, devices, and interactions, but because of the lack of tools and resources for proper analytics, it remains hidden in data repositories. Common sources of dark data include system log files, geolocation data, video footage, customer call records, financial statements, raw survey data, email exchanges, previous employee profiles, and others.

The State of Dark Data

Considering that data is now considered the new oil and thus, a powerful asset for any enterprise, it’s surprising that much of it is left to waste. According to a Splunk-sponsored research by TRUE Global Intelligence, a third of the 1,300 survey respondents considered 75% of their data dark. Other similar surveys out there show the same thing—a big chunk of data is largely mismanaged and remains untapped, and organisations are missing out on the business value that may be derived out of this.

There are many reasons why companies are not making the most of dark data. For one, the data they hold may already be too old or too dated to be of any relevance. Dark data also includes the unknown—that which businesses don’t even know to exist in the first place. And then, of course, the sheer volume of data being collected everyday can simply be overwhelming. Organisations don’t know where and how they begin in making sense of unstructured and unclassified data.

Granted, there are analytics tools that can help with efficient data collection and analytics. New technologies focusing on business intelligence are becoming more accessible to the average business, bringing together structured and unstructured data sets so that they produce invaluable insights. You just have to get things in motion.

Making Dark Data Work for You

One of the biggest benefits you can get with dark data utilisation is gaining a deeper understanding of your products and services, and the impact that these have on customers. While processing dark data would be different for every company, you can start with these steps:

1. Ask the right questions. Consider what data you really need for informed decision-making because knowing what to look for points you in the right direction. For instance, if better customer service is a priority, then take a closer look at your customer call records and helpdesk data.
2. Do some ‘housekeeping’. Perform an audit of your database and find out what type of information is meaningless and/or redundant. If there’s indeed data that you’re not making use of, take steps to dispose of it. Offloading non-value data saves you storage and reduces your liability in the event of a cybersecurity breach.
3. Pick the right tools and talent. You can’t mine without digging tools, and this goes for dark data too. Identify the data sets that you want to look further into, and invest in apps that work best for processing this data type. Recruiting data professionals should also be a big boost in effectively managing dark data. 

Dark Data Challenges

Just as there are many opportunities, challenges are also present in dealing with dark data. Knowing what these make you better prepared for harnessing the potential of dark data.

1. Making data anonymous. Online security and privacy concerns could surface when you’re dealing with data that involves personal information. One way to address this issue is to anonymise customer data by removing names, account numbers, or any other identifying information that points to a specific person.
2. Keeping data from getting ‘darker’. Roughly 2.5 quintillion bytes of content are created daily. Even without doing any math, it’s easy to see that more and more data will go into storage everyday. Evaluate the types of data you’re accumulating and stop collecting that you won’t need. Storing data costs money, after all.
3. Ensuring data security. Storing data for long periods could compromise sensitive data such as proprietary information, clients’ financial records, employee personal data, and more. Processing dark data helps you identify which you can keep and remove, and compels you to establish data protection and online security measures for important information.

Don’t Be Left in the “Dark”

Discover the value that dark data holds by identifying how data can best help your business grow. However, make sure that all efforts towards insightful analytics are governed by data governance and cybersecurity best practices.

If you think you could use some assistance in finding the right tools and people for dark data management, talk to an IT managed services provider today.