Technology Decision Factors

When making a decision about what technology to use for a project, product, or service architecture there are many factors to consider that will impact the total cost in ownership for a company.

Skilled Workforce

Risk: Availability, Salary Cost, Training Cost, Churn Cost

Commentary: The skilled workforce factor sometimes gets forgotten in the hunt for a technology.  If a technology is highly specialized it will decrease chances of hiring someone with the appropriate skill level to deal with the technology.  It will also increase the cost of training and/or salary.  There can be a high churn rate for the skilled resource as they can get head hunted by other companies or they become concerned about their career as the skill set exercised is too narrow for their long term  career goals.  Skill sets must span both Development, Test, and Operations as well as training for end-users of the technology if necessary.

Technology Support

Risk: Code Stability, Documentation Quality, Tech Support, Market Survival, Backwards Compatibility

Commentary: If a technology is bleeding edge or produced by a one man band, it can suffer from poor documentation, poor technical support, instabilities, and may not survive its incubation period.   It is also important to have the assurance that the company producing the technology are committed to backward compatibility so you don't lose your coding investment.

Operational Cost 

Risk: Deployment and Provisioning Costs, Maintenance Costs, Stability, Tech Support, Hot Patch Support

Commentary: The day to day operational costs of technology is very important to take into consideration.  On-Site hardware costs, installation costs, backup costs, failover costs, global data and processing distribution costs, tuning costs, hardware maintenance costs, Software upgrade costs, hot patch costs, production trouble shooting costs, and operation staff costs is a lot to calculate.  On-Site may have a much higher cost of ownership then subscribing to cloud services.   Cloud Services provide a much simpler and predictable cost model and usually provide elastic provisioning to grow or shrink to your needs.  Something to really contemplate when selecting your technology platform.

Security Cost 

Risk: Customer Safety, Business Safety

Commentary: Data and processes in the hands of the wrong people can cause irreversible damage to a customer or the business.   Personal Identification Information (PII) and Business Critical Information must be secured and not taken lightly.  Is the usage of the technology managing such information?  Can the Technology tightly secure the information?   Can it keep up with latest security features and patches to protect against security breaches?  Can it detect security breaches?  Same questions for any sensitive processes.

Business Functional Requirements

Risk: Functionality,  Elastic Scalability, Reliability, Availability 

Commentary: I placed this last because these are the most obvious factors to consider.  Does the technology provide the proper level of functionality to support the business?   Can it scale with the company as its needs grow and shrink through out time?  Its is stable and reliable?  Can it deal with the 99.9, 99.99, 99.999 uptime availability requirements?   What is the Mean Time To Recovery (MTTR)?


Bottom Line: Remember that you get what you pay for, but don't pay for more than what is warranted.  And in a competitive world, the total cost of ownership of your enterprise technology balanced with the need to enable your company to compete is critical to get right.


Please See the following docs for the cost benefits for cloud technology solutions.
Cloud Economics
Cloud Platforms for Business Owners
Cloud Optimization - A Framework for making business decisions about cloud computing

Commentary: The Case of the Missing Enterprise Data Warehouse

Symptom: You have 100 reports that are unreliably generated for you and the reported numbers conflict from report to report.

Root Cause:  Your company has no Enterprise Wide Data Warehouse.

How Did that Happen?   Here is the Typical Evolving Scenario: 
You or your management commissions someone to provide a specific report.  That report was quickly put together and placed on a server under someone's office desk.  Rinse and Repeat...   Eventually you realize you have a series of reports that give conflicting numbers.  So you have someone investigate why and you get the following answer:  You have 100 Reports coming from 10 Reporting servers under 10 different office desks across 3 office sites.   There are 20 different Extraction, Transformation, and Loading Pipelines all with a different codebase and with different logic.  Looking even deeper you may find that the ETL Pipeline's error handling, late arriving data processes, cleaning processes, data enrichment processes, the data conformation processes, and data variance checks all vary in design or just plain absent.   Then the real bad news hits you:  What you thought were the same measures in different reports are not the same.  They technically have different  meanings even though the have the same name.

No Data Warehouse Diagram:

Hidden Costs in Not Having a Enterprise Wide Data Warehouse:

• Increased Development Costs for maintaining multiple and fragmented codebases doing same thing.
• Increased Operations and Development Costs in troubleshooting late or missing reports and erroneous numbers.
• Increased Business Risks in data landing in the hands of the wrong people due to your report servers not being properly secured.  This results in these servers being hacked or just plain walking out the door.
• Decreased Business Opportunities and Optimization caused by relying on in accurate and inconsistent reports.

Solution:  Build a Enterprise Wide Data Warehouse by providing the following.

• Conformed Dimension Definitions 
• Conformed Measure Definitions
• Conformed Atomic Level Fact Definitions
• Managed Dimensional Data
• Documented Data SLAs going In and Out of the Warehouse
• Documented Data Lineage
• Symmetrical ETL Pipeline Process Management
• Uniform handling late arriving data
• Uniform Error Handling
• Uniform Variance Checks
• Uniform Security Policies and Management
• Shared Codebase and Framework
• Shared Environment
• Uniform backup and replication of data
• Uniform Data Life Cycle Management:  Data Archival and Deletion as it ages

Traditional Data Warehouse Diagram:

Baseline Conceptual Models: Warehouse Shard Configuration Model

Warehouse Shard Configuration Model:  Data is partitioned or distributed across compute nodes.  This information is critical for bringing processes to the data located on the each compute node and executing it.  This model is dependent on Pipeline Configuration Model.

Please see (Baseline Conceptual Models Commentary) for further details on what conceptual models are to be used for.

Baseline Conceptual Models: Pipeline Configuration Model

 

Pipeline Configuration Model:  Pipelines in a data warehouse will have many scheduled jobs that process data from raw to atomic fact to a final report.  Some processes are a dependent one or more datasets and can output one or more datasets.  Were these processes are executed is dependent on where the data is located.   This model is dependent on Data & Schema Model and Warehouse Shared Configuration Model.

Types of Pipelines:  There are different pipelines for different stages of the data.

• 3rd Party Dataset Pipelines are extracting, minor transformation, and loading processes that pull data from external companies.
• Raw Log Pipelines are extracting, minor transformation, and loading processes that pull data provided from other systems within the company.  Examples are Website Logs, Sales or Purchase Orders, and Inventory Movement.
• Master Data Domains Pipelines are extracting, minor transformation, and loading processes that pull from a Master Data Service or collected from across multiple company systems. 
• Dimension Pipelines are transformation processes to transform Master Data Domains into dimensions.
• Staging Pipelines are transformation processes that cleanse, enhance, or conform data.
• Atomic Fact Pipelines  are transformation processes that transform logs into atomic facts.
• Aggregated Fact Pipelines  are transformation processes that transform atomic into aggregate facts.   
• Report Pipelines  are transformation processes for transforming atomic or aggregate facts into reports. 

Note:  Pipeline Data entity provides information about the input and output data going into or out of a process and is important for in tracking course grain data lineage.

Please see (Baseline Conceptual Models Commentary) for further details on what conceptual models are to be used for.

Baseline Conceptual Models: Data Service Level Agreement (SLA) Model

Data Service Level Agreement (SLA) Model:  When sourcing data it is a good idea for the data provider and the downstream customer to document and agree upon the specifics of the data that is to be provided and the scheduled window of time it is expected for the data to be available for download.  This helps provide two things:  Set expectations properly between the data provider and the downstream customer and second it enables the ability for the data provider to see who is impacted by delays in the pipeline and communicate to the downstream customers accordingly.  This model is dependent on Data Catalog & Schema Model and the Party Model.

Data SLA:  SLA's should be predetermined by the data provider based on the average time it takes for the pipeline to process the necessary data and make it available at a drop point plus some buffer time for the outliers.  A customer then agrees upon the SLA or make a request for a modified SLA.  Modifying the SLA may require new development or tuning of the pipeline.

Data SLA Landing Locations:  SLA's also includes one or more drop locations on a specific reoccurring schedule.

Please see (Baseline Conceptual Models Commentary) for further details on what conceptual models are to be used for.

Baseline Conceptual Models: Data Catalog and Schema Model

Data Catalog & Schema Model:   In a data driven pipeline for a data warehouse the data definitions stored in a meta data repository and is used to speed up pipeline development as well as enforce design conformity.  Entities are organized into a category hierarchy for simple exploratory navigation.   Throughout the live cycle of an entity can be revised several times.  Each revision of the entity will have one or more attributes.   Attributes are first predefined independently from any entity.  Then each entity will bind to the appropriate predefined attributes.  This enforces a solid naming convention and definitions throughout the pipelines in the data warehouse.

Entity Types:  In a data warehouse data can be classified into different types to signify the value it brings to the warehouse.

• 3rd Party Datasets are upstream data that are provided by external companies.
• Raw Logs are upstream data that are provided by other systems within the company.  Examples are Website Logs, Sales or Purchase Orders, and Inventory Movement.
• Master Data Domains are data that came from a Master Data Service or collected across multiple company systems.  This kind of data is used to build dimensions.
• Dimensions are data elements that enable filtering, grouping and labeling facts (measures).
• Staging Datasets are data that has been cleansed, enhanced, or conformed and ready for further processing.
• Atomic Facts are the lowest level of grain for a set of measures.   Measures can be filtered, grouped, and labeled by a predefined set of dimensions
• Aggregated Facts are an summarized set of measures that have been summed up from an atomic fact.  Used for business snapshot purposes like end month or end of year reporting or can be used for performance tuning.   
• Reports are usually simple facts (sometimes non-pivotable) that the end user uses as a statement of measure on key performance (KPI) for the business. 

Risk Assessment:  In an enterprise data set there can be information the could cause damage to the business or customer if released to the outside world.   Each attribute should be assessed in order to evaluate the risks for the company.

Data Variance Check Definitions:  An attribute to an entity may have a

Attribute Types:  Each attribute in a entity has a specific meaning in a data warehouse.  

• Dimension Keys are surrogate key that is used to make to a dimension. 
• Business Keys are values used to lookup the dimension key using in the key mapping phase of an ETL.  These are found in the upstream source streams.
• Measures Attributes are numeric values representing a count or output of an equation.
• Regular Attributes are general values used for pass-through processes or user consumption.

Data Lineage:  Tracking data lineage from one data source to another within the warehouse is a critical to track provenance of the data and enables faster diagnostics when data goes bad.  This a fine grain data lineage feature.

Please see (Baseline Conceptual Models Commentary) for further details on what conceptual models are to be used for.

Baseline Conceptual Models: Repair Order Model

Repair Order Model:  A repair order can be for a customer or can be an internal repair order for specific number of non-functional inventory.  A repair assessment is performed to determine what needs to be repaired.  If the repair is not under warranty then a repair quote is made and given to the customer.  If the customer agrees to the repair quote a repair sales order is created for "Parts and Labor" incurred by the repair.    This model is dependent on Product Service Request Model, Inventory Control Model, Sales Order Model, Repair Assessment Model, and Repair Quote Model.

Please see (Baseline Conceptual Models Commentary) for further details on what conceptual models are to be used for.

Baseline Conceptual Models: Repair Quote Model

Repair Quote Model:  Before repairs not covered by warranty can be performed a quote is provided to the customer to agree upon before the repair is performed.   The repair quote includes "Parts and Labor".  This model is dependent on Repair Order Model, Product Offer and Price Model.

Note:  Products are goods and services.  Labor falls under service.

Please see (Baseline Conceptual Models Commentary) for further details on what conceptual models are to be used for. 

Baseline Conceptual Models: Repair Assessment Model

Repair Assessment Model:  Before a repair can be performed on a part a series of tests will need to be performed.  These tests are predefined per part along with a set of predefined set of results that provide suggested repair and labor solutions.  This model depends on Repair Order Model and Part Bill of Material (BOM) Model.

Please see (Baseline Conceptual Models Commentary) for further details on what conceptual models are to be used for.  

Baseline Conceptual Models: Technical Support Request Model

Technical Support Request Model:   Once it is determined that technical support is required a request is made and the customer is patched through or a technician may need to be dispatched.    There can be many Technical Support Requests relating to a Service Request.   This model is dependent on Service Request Model, Party and Account Model, and Location Model.

Please see (Baseline Conceptual Models Commentary) for further details on what conceptual models are to be used for.