Dark Data & Ways to Bring it into the Light for Strategic Outcomes & Values

At Illumination Works, we talk a lot with clients about business intelligence and data-driven decisions. In fact, most organizations consider data to be a very valuable asset.

Unfortunately, the challenge many organizations are facing is that this valuable asset (data) is not creating value or strategic advantage.

According to one estimate, 7.5 septillion (7,500,000,000,000,000,000,000) gigabytes of data are generated worldwide each day, and 55% of it has gone unused. Many organizations have allowed their data to go dark. So the question organizations now face has more to do with how to use their data to increase the return on investment (ROI) on the their data collection efforts.

What is Dark Data?

According to Gartner, dark data is “the information assets organizations collect, process, and store during regular business activities, but generally fail to use for other purposes (e.g., analytics, business relationships, and direct monetizing).” Examples of dark data include customer call records, log files, financial statements, raw survey data, previous employee information, email correspondences, geolocation data, notes, presentations, and old documents. Organizations are not leveraging dark data for various reasons:

Data is too old to be relevant or provide value
Information comes in a format that is not accessible by existing tools
Lack of expertise or tools to collate and analyze data from disparate systems
Data is not high quality (e.g., fragmented or incomplete)
Short supply of data science expertise makes it hard to process unstructured data at scale
Organizations do not know where all of their information is stored

Why Should You Care About Dark Data?

Most of these blocks can be overcome with a bit of work; however, an organization needs to be motivated to spend time and effort to utilize this unused data. There are a few reasons why one should get a handle on the dark data floating around an organization.

Unlock Opportunities

A company can gain valuable insights into customers and uncover hidden opportunities. For example, server logs can show website visitor behaviors, and customer call records can reveal consumer sentiments. Mobile geolocation data can offer insights into traffic patterns, and social media data can show trends and behavior patterns.

Stay Ahead of Competitors

Using the right data and insights can help organizations create products, services, and marketing materials to distinguish itself from the competition. On the other hand, if competitors are investing in data analytics and your organization is not, you could be missing out on opportunities to capture new trends and grow revenue.

Improve Customer Experience

Records of customer interactions are a treasure trove of insights into how prospects and customers perceive your brand, products, and services. They offer insights into the human element of business operations. You can leverage the information to improve customer experience, increase retention rates, and drive sales.

Avoid Legal and Regulatory Issues

Without a handle on what data you have and where it lives, you could land in regulatory hot waters – especially if the information involves data such as personal identifiable information (PII), financial data, payment information, and health data. Analyzing dark data from communications can also help you identify potential compliance issues and take preemptive actions.

Minimize Intelligence Risks

Without a comprehensive view of your data landscape, you could risk leaking proprietary or sensitive data on business operations, product design, financial status, intellectual properties, and more.

What Can You Do About Dark Data?

To make the most of dark data, organizations needs employees or external experts with the right knowledge and skillsets. Executives consider two alternatives to address this lack of expertise: 76% consider training current employees in data science, and 70% think hiring more data experts can help them bridge the gap. Other approaches include:

The use of software solutions to make existing data more accessible and usable for non-technical employees
Increase training to raise awareness about the value of data within your organization and provide funding for data projects
Categorize your data so you can apply the appropriate strategies to get the most value from your data

Two broad categories of dark data include structured data and unstructured data. Unstructured data often consists of textual data hidden in notes, comments, and customer reviews.

Structured Data

Structured data is organized in a formatted repository (e.g., a database) to make the data addressable. Various techniques are used to extract insights from structured data:

Use lexical analyses to turn a string of characters into identifiable entities
Leverage machine learning (ML) to seek out identifiers
Use algorithms to track consumer sentiments (e.g., by assigning numeric ratings)
Leverage predictive analytics to identify trends and improve forecast accuracy

Unstructured and Text Data

Unstructured data is often qualitative information that is not easily arranged according to a predetermined data model or in a pre-defined manner. Text and multimedia, such as email messages, business documents, presentations, webpages, photos, videos, and audio files, are the most common unstructured content.

The noisy nature of textual data makes it especially challenging to organize and analyze raw data. Yet, as much as 90% of all digital data is unstructured, locking away critical insights in various locations and formats. A few approaches to manage and extract insights from unstructured data include:

Set your goals for data sorting to identify useful data
Organize content to make the data accessible and searchable
Clean the data to remove irrelevant information and slice the data into manageable pieces
Analyze the data with artificial intelligence (AI)-powered tools, such as ML and natural language processing (NLP) software
Generate charts, reports, and interactive dashboards to visualize the information

Tools for Processing Dark Data

Sorting and analyzing dark data to extract insights is a multi-step process that requires engineering and data science expertise, along with various tools. A minimal software toolkit will support these three key steps:

Data discovery: Gain visibility into your organization’s data landscape and identify useful information for further analysis
Data classification: Identify the value of a dataset, how it can be useful, security concerns, and more. Use automated tools and AI algorithms (e.g., ML, NLP) to organize large amounts of unstructured data into relevant categories
Data quality management: Implement a policy-based data quality management procedure to facilitate decisions on how to clean each dataset to maximize its value and/or minimize storage cost. Tools that can help with this process include video and sound analytics, computer vision, ML, and advanced pattern recognition software

Using Dark Data to Inform Human-Centered Problem-Solving

Fast-evolving business dynamics provoke new and burning questions. Many of the answers are locked within underutilized data. Meanwhile, companies are losing revenue, missing out on opportunities, and running into compliance issues due to a lack of visibility and insights into business operations and customer sentiment.

Dark data can reveal valuable insights into customer behaviors and consumer trends. You can gain insights to support everything from website design and social media marketing to operational efficiencies and product development.

While data analytics tools have made leaps and bounds to help us organize and process information, companies are often missing one critical piece of the puzzle: Rooting the insights and advanced solutions in real-life observations through a human-centered approach.

Here at Illumination Works, we bring the human element into our data interactions with data and machines. We offer custom services to help companies address their dark data effectively through AI-driven technology and derive insights to support fruitful interactions.

If you would like to learn more about how Illumination Works can help you uncover your dark data and bring it into the light for strategic outcomes and value, contact us at [email protected] and reference this blog is the subject line.

About the Author

Rob Keefer, PhD, is the Director of Research and Innovation at Illumination Works. He has 20+ years of experience solving complex problems. In his research and innovation role at ILW, Rob works to develop solutions that promote the user’s perspective through his emphasis on human-machine teaming applications of data, artificial intelligence and machine learning algorithms, and visualization.

Machine Learning

Data Science

Data Engineering