Building Data Warehouses is still a painful endeavour, they take too long, cost too much and they are too hard to change after deployment. There are many activities around building a Data Warehouse, that make this process labour intensive and time-consuming, such as, requirements gathering and analysis, source data analysis, source-target mapping, data transformation logic, Extract, Transform, Load (ETL) processes, data analysis, design and Load. On the other hand, building a Data Warehouse it’s not easy, it requires integrating several pieces of information that normally are in different systems that can be in different formats and volumes, across the organization.
Most Data Warehouses follow a waterfall system development lifecycle (SDLC) that takes too long and it’s too inflexible to quickly adapt to business changes and needs. Due to this situation, at some point in the project life cycle there will be issues related to resources, project team will be working after hours to keep up with project timeline. Normally the response to this situation is adding more resources to the project, but this is not a productive way of addressing this issue, “adding manpower to a late software project makes it later”, while the project team invests time and effort explaining the project context and training new team members, there is a reduction in productivity and the consequence is projects that falls behind schedule.
Another issue when using waterfall system development for Data Warehouse projects concerns business requirements. Business users struggle in defining, up-front business requirements, without seeing and understand the data first. This may lead to projects that in the end, doesn´t meet organizational needs. Waterfall development doesn´t allow trial and error, exploration and data discovery to rapidly create business insights.
Data warehousing development process in a classic architecture, takes too long, cost too much, it’s not easy to build and it’s hard to change.
Data warehouse automation (DWA) tools can help to address the limitations of waterfall and traditional approaches for building Data Warehouse, turning the data warehouse development from a time-consuming effort into an agile one, with gains in efficiency, effectiveness and agility in data warehousing processes.
by Paula de Oliveira Passio Consulting co-founder and Managing Partner
______
References:
Brooks, F. (1975). The Mythical Ma-month: Essays on Software Engineering. Addison-Wesley.
Ekerson, W. (2015, September 1). Data Warehouse Automation Tools: Product Categories and Positioning. Retrieved 1 29, 2019, from www.tdwi.org
Evelson, B., & Angel, N. (2016, August 16). Forrester. Retrieved December 19, 2018, from https://www.forrester.com
Rahman, N., & Rutz, D. (2015). Building Data Warehouses Using Automation. International Jornal of Intelligent Information Technologies, 1-22.
Wells, D. (2014). Relieving the Pain of the BI Back Room with Data Warehouse Automation.