Data Management

  • Secured Data Capture, Storage, & Movement.
  • Accurate, Integral Relational Data Warehousing.
  • High-Volume, High-Velocity, High-Variety Data Lake.
  • In-Memory, Streaming Data Processing.

Data management solutions are deployed to implement robust, sustainable, high-quality, 'data supply chains' for provisioning individuals and business applications with timely trusted data. They are designed with sufficient agility to successfully adapt and scale in accordance with needs of a changing and evolving business environment.

Data management solutions are designed and constructed in accordance with the organization’s enterprise data architecture policies, standards, and blueprints. Assuring consistent implementation and usage of organization’s data assets across the enterprise.

Upon confirming the data requirements to service the business’s informational needs, data management professionals reference data architecture master data, systems of record, operational, and other standards for provisioning the data. Promoting financial and operational data management efficiencies through consistent data utilization across the organization

Upon confirming the scope of the required data, the nature in which the data is expected to be used is determined in preparing the data management solution. Considerations include whether the data will be captured in predefined data structures or in its native form, acceptable data quality thresholds, latency tolerances, and availability requirements.

Conceptual data structures scope, identify and map the key business data entities required to support the planned data management solution. The conceptual data models are elaborated into Logical data structures which refine the business data entities into more granular detail, populated with the data fields required to support the data processing needs of the solution.

The logical data structures are eventually converted into physical data structures in preparation for implementation in the target data management technology which can include: relational databases, Data Lakes, file stores, No-SQL databases, and Big Data repositories. At time of being deployed into the target data environment, the physical data structures are tuned to ensure optimal throughput and response-time data processing performance.

The implemented data management environments have to be protected against both intentional and unintentional harm. This is achieved through centralized data access controls, high-availability infrastructure, and implementing data backup and recovery procedures.   

​​Relational solutions (schema on write) require the data structures to be defined and implemented prior to the data being captured in the data store. Relational data solutions are developed as either normalized Third Normal Form (TNF) or denormalized Star-Schema data structures for deployment within SMP or MPP infrastructure. Relational solutions are developed and maintained by IT data management specialists.

SMP environments are vertically scalable infrastructure, which can be upgraded with additional storage, memory, and CPU processing to accommodate increasing relational data processing demands. SMP infrastructure is implemented to deploy relational technologies such as Microsoft SQL-Server, Azure SQL Database, and AWS Aurora for supporting transaction processing and lower data volume business intelligence environments.

MPP technologies are horizontally scalable infrastructure which allow further storage, memory, and CPU processing capacity to be added with the addition of further physical servers. Allowing extremely large data volumes to be efficiently processed by spreading the workload across the family of servers. MPP relational technologies include Azure SQL Data Warehouse and AWS Redshift

No-SQL data management solutions provide greater agility compared to a relational in being able to enhance and extend deployed data structures. Contrary to the natural rigidity associated with creating and enhancing relational solutions, No-SQL data structures can be flexed and deployed almost instantly. Providing a highly efficient means of supporting rapidly evolving data management environments such as digital product catalogs of on-line digital transaction processing applications. No-SQL data management technologies include: Document, Column-Family, Key-Store, and Graph databases.

Data Lakes and Big Data stores allow to capture and manage extremely large volumes of varying non-deterministic data arriving at high-velocity. The data is stored in its native form and structured at time of reading by the consumer (schema on read). Instead of relational tables and columns, the data is structured as hierarchies of folders and files. The data is logically partitioned into 'landing', 'curation', and 'publication' zones to manage its storage and flow from reception to consumption.

In-memory data management is a niche technology specifically designed for supporting high-performance near real-time data processing. All data is cached and processed in memory to avoid disk related latencies. Resulting in exceptionally fast real-time data processing throughput. Both Microsoft Azure and AWS offer cloud hosted in-memory data stores such as Redis to capture and store data as it arrives for supporting real-time analytics with technologies such as Apache Storm.

    © 2020 Data2Intelligence Inc.