Deep Data example

DevOps sometimes requires data that is stored deeply within a data source to be stored and retained as part of either a Code Governance requirement or as a part of a DevOps automation build or release sequence.   

Deep data is data stored within other some other data within a Data Source such as a database or file store.  A simple example would be content stored within a file that is stored within a Zip file, and potentially where the zip file is itself stored in a database row.

 The example below extracts a part of JSON content that’s stored within a text row within a database, and requires the following sequence of extraction configurations:
  • Read the Database
  • For each Row parse the JSON from the text row
  •    Select a part of the JSON content
  •    Save (export) this piece of JSON into a specific filename that relates to the ID column of the database table
 

Easy Access

Data stored deeply within a data store can be difficult to obtain through normal means.  ExaOps provides easy to use tools that integrate into a DevOps pipeline.  These tools make it easy to define how to obtain deeply held data and then output it in a variety of forms – for example to the file system, a database or to an API endpoint.

ExaOp’s provides powerful tools that makes accessing deeply held data straightforward.  It requires approximately 30 lines of ExaOps configuration to deeply access content through steps such as:

  • Query a database
  • Open a Zip file within the database
  • Select a specific JSON file within the Zip file
  • Select an element of the JSON
  • Finally store that element in a Git style source repository, database or on the file system. 

This is purely a configuration driven style and is hence dramatically less effort to manage and maintain than an equivalent coding approach.

Summary 

  • Both Source Code and key Configuration Data is potentially stored deeply within data sources
  • The information is required for both Code Governance and DevOps purposes
  • ExaOps provides a range of tools that simplify the process of accessing and automating deep data requirements

Recommended Posts