Thursday, January 16, 2014

Traditional Data Warehouse Design

Traditional Data Warehouse Design:  Before Massive Parallel Processing (MPP) was available to the masses, a data warehouse was typically housed in a single Server.  It maybe clustered with another server for fail over proposes.  This design is good for your first steps in data warehousing, but is limited to the volume of data it can service.   SQL Server 2012 database size limit is about 500 TB.   But saying that you are still limited by the number of CPUs and the amount of memory your service has which limits your performance.    Its not a scalable solution in a world which the volume of data is accelerating exponentially and the demand for low level adhoc discovery analysis is on the rise.  

Features It Should Have:
  • Conformed Dimension Definitions
  • Conformed Measure Definitions
  • Conformed Atomic Level Fact Definitions
  • Managed Dimensional Data
  • Documented Data SLAs going In and Out of the Warehouse
  • Documented Data Lineage
  • Uniform ETL Pipeline Process Management
  • Uniform handling late arriving data
  • Uniform Error Handling
  • Uniform Variance Checks
  • Uniform Security Policies and Management
  • Shared Codebase and Framework
  • Shared Environment
  • Uniform backup and replication of data
  • Uniform Data Life Cycle Management:  Data Archival and Deletion as it ages 

  • Keep in mind that most businesses are in a business information warfare.  More than likely you are competing in a shrinking market as more and more businesses are competing for the same customers.  So any leverage you get is critical to your success.  If you don't leverage big data then your competition will.

    No comments: