Data Management

  • Secured Data Capture, Storage, & Movement.
  • Accurate, Integral Relational Data Warehousing.
  • High-Volume, High-Velocity, High-Variety Data Lake.
  • In-Memory, Streaming Data Processing.

Data management practices assure the protection, security, and timely availability of trusted corporate data resources for supporting business operations ranging from highly responsive on-line transaction processing, to data warehousing and ‘big data’ decision support services. ​

They ensure the availability of a robust high-quality 'data supply chain' for servicing transaction processing and decision support business applications. Effectively employed data management techniques and technologies ensure the delivery of robust data management environments, capable of efficiently evolving and scaling in accordance with the needs of changing business environments.

Data management solutions are planned and designed in accordance with corporate data architecture policies, standards, and blueprints. Assuring responsive agility in adapting to business data needs, with minimal financial and operational impact.

One of the primary references in the development of robust integral data management solutions is the enterprise data architecture conceptual data model. This maps the essential business data entities, together with and identifying the ‘systems of record’ for the respective data assets. The practice assures consistency in the delivery of individual data management solutions across the enterprise. Eliminating reconciliation and data exchange inefficiencies by assuring consistency in each occurrence of the business data set.

Data architects refine conceptual data models into logical data models covering the scope of the data requirements for the business application delivery. Conceptual data entities are refined into logical data entities and elaborated with the business data elements required to support the application’s data processing needs. All data relationships between the logical data entities are implemented to ensure the integrity and quality of the data management solution in completing the logical data model.

​Logical data models are subsequently converted into physical data structures and tuned to assure optimal execution time performance within the target data management environment, which can include: relational databases, file stores, No-SQL databases, and big data repositories.

​Relational solutions are designed for implementation as either normalized Third Normal Form (TNF) or Star-Schema deployments within either SMP or MPP data management technologies. SMP technologies include SQL-Server, Azure SQL Database, and AWS Aurora  platforms, while MPP environments such as Azure SQL Data Warehouse or AWS Redshift ensure for high-performance of exceptionally large relational data volumes. Relational implementations are referred to as a ‘schema on write’ approach. In this case information technology experts design and implement the data structures by prior to the data being captured.

​Rapidly evolving on-line digital transaction processing applications need the ability to quickly flex the application’s data structures in order to accommodate rapidly evolving products and business lines. Relational solutions unable to respond  within such time constraints. Document based JSON (No-SQL) data management solutions provide the ability to swiftly enhance and extend the application data structures.

​Big data and data lake implementations are exceptional solutions for capturing, managing, and disseminating extremely large volumes of semi and unstructured data arriving at high-velocity in various forms. They are designed in the forms of hierarchical folders and files, logically partitioned into 'landing', 'curation', and 'publication' zones to manage the flow of data within the data management environment. User accessibility to the data is authorized at either folder or file levels. The approach is referred to as either ‘Extract-Load-Transform’ (ELT) or ‘schema on read’, as the data structures for provisioning the data are specified by the consumer at the time the of retrieval.

In-memory data management solutions are implemented with technologies such as Apache Spark when time critical applications such as investment securities and real-time analytics require micro-second throughput and response-time rates.

Rapidly arriving data sourced from data stores such as Azure Event Hubs can be cached and processed in-memory. The data activity transactional logs are persisted on disk for recovery and audit purposes in case of process or server failure. The resultant real-time results are able to be published to decision makers either through custom applications, or with data visualization tools that have the capabilities for presenting streaming data such as Microsoft Power BI.

    © 2019 Data2Intelligence Inc.