System Reference Tables
Ganymede metadata tables enable users to query and reference information about flow runs, tables, and files stored in Ganymede. Some possible uses for these tables are to debug prior runs, find specific files / table records associated with prior runs runs, and systematically displaying tags used to contextualize and ensure traceability of captured data.
The tables described below are found in the Ganymede database, and can be referenced from Data Explorer, within Flow code, or in Dashboards.
Flow Runs
The flow_run_metadata table is written to once per flow execution.
| Field Name | Description |
|---|---|
| flow_run_id | Unique identifier for the flow run, stored as epoch time in milliseconds |
| flow_id | Flow name |
| inputs_dict | Run context for the flow; corresponds to the run context described in the Ganymede Class overview |
| initiator | Flow run initiator; agent name + MAC address for agent-initiated flows, user email for user-initiated flows, event name for event-triggered flows, or flow name if triggered from another flow |
| initiator_type | Type of the user who initiated the flow run (AGENT, USER, EVENT, FLOW) |
| flow_version | Commit hash associated with flow execution |
| state | Final disposition of flow run (Success, Failed) |
Table Writes
The public_flow_run_output table contains 1 record per output table written in Ganymede.
| Field Name | Description |
|---|---|
| id | ID for table save |
| node | Name of node that wrote to table |
| name | Name of table written to Ganymede table storage |
| timestamp | Time of table save; epoch time in fractional seconds |
| datastream_metadata | JSON containing UUID associated with table write transaction and flow run ID |
File Writes
The file_metadata table contains 1 record per file written in Ganymede.
| Field Name | Description |
|---|---|
| bucketType | Input or Output; indicates whether the file was input into Ganymede or an output from processing performed in Ganymede |
| uri | full URI to the file |
| path | Path within storage bucket to file |
| size | Size of file in bytes |
| createdAt | Time of file save; epoch time in milliseconds |
| creator | Name of the file creator (e.g. - email address for users, agent name + MAC address for agents) |
| fileTags | List of JSON objects containing tags associated with the file |
Python Packages
The ganymede_env_python_package_list shows a list of all Python packages installed in the workflow environment.
| Field Name | Description |
|---|---|
| package_name | Name of the Python package |
| version_number | Version of the Python package |
Pandas <-> Ganymede DB Field Reference
SQL columns have character restrictions for compability with SQL which are not present in Pandas. The __TABLE_METADATA contains the mapping between table field names referenced in NodeReturn and the corresponding SQL column names.
The retrieve_tables method associated with Ganymede class provides a convenient method to retrieve tables in notebooks with their original Pandas Dataframe column names.
This table contains 1 record per table write transaction.
| Field Name | Description |
|---|---|
| flow_id | Flow name |
| flow_run_id | Unique identifier for the flow run, stored as epoch time in milliseconds |
| table_id | Table name |
| timestamp | Time of table save; epoch time in fractional seconds |
| type | Whether write transaction is to create, replace, or append to database table |
| table_columns | List of column names in the table |
| table_column_types | List of column types in the table |
| table_row_count | Number of records in Pandas DataFrame being written to DB |