Rapid Data Ingestion to Speed Analytics
Customer Challenge
The Air Force has the need to load data of varying formats faster to enable on-demand data science, advanced analytics, and decision making.
Innovative Solution
ILW extended our proven commercial Data Framework for Air Force use by adding easily configurable sources and targets and removing Hadoop-specific dependencies. This simple framework allows analysts to create source data definition files and use the provided ingestion templates to generate fully executable scripts to ingest data.
Benefits/Outcomes
- Usable by non-developers with minimal database knowledge
- Automated and metadata driven to provide fast and flexible capability
- Extendable and scalable to support complex data sets and data governance
- Light weight and codeless with customizable end points (source, targets, data structure types) for various end-use purposes
- Framework templates and repeatable processes
Business Value
- Accelerate data ingestion tasks from months to weeks for faster analytics/decision making
- Minimize/redirect engineering support by enabling business analysts to perform ingestion tasks
Toolbox
- Open-source Python solution utilizing DoD compatible libraries
- Graphical user interface for use by subject matter experts to define rules
