Ganymede Metadata Tables
Ganymede metadata tables allow users to query and reference details about flow runs, tables, and files stored within the Ganymede platform. These tables can be utilized for various purposes, such as debugging previous runs, locating specific files or table records associated with past runs, and systematically displaying tags used to contextualize and ensure traceability of captured data.
The tables described below are found in the Ganymede database, and can be referenced from Data Explorer, within Flow code, or in Dashboards.
Flow Runs
The flow_run_metadata table captures a record for each flow execution.
| Field Name | Description |
|---|---|
| start_time | Start time of the Flow run |
| end_time | End time of the Flow run |
| export_time | Time of the Flow run export |
| flow_run_id | Unique identifier for the Flow run, stored as epoch time in milliseconds |
| flow_id | Flow name |
| inputs_dict | Run context for the Flow; corresponds to the run context described in the Ganymede Class overview |
| initiator | Flow run initiator; agent name + MAC address for agent-initiated Flows, user email for user-initiated flows, event name for event-triggered Flows, or Flow name if triggered from another Flow |
| initiator_type | Type of the user who initiated the flow run (AGENT, USER, EVENT, FLOW) |
| flow_version | Commit hash associated with flow execution |
| state | Final disposition of Flow run (Success, Failed) |
| table_output | JSON containing database table write info associated with Flow run; table, type of write operation (e.g. - replace, append), table columns, and row count are available in data |
File Writes
The file_metadata table contains 1 record per file written in Ganymede.
| Field Name | Description |
|---|---|
| bucketType | Either "Input" or "Results"; indicates whether the file was input into Ganymede or an output from processing performed in Ganymede |
| uri | Full URI to the file |
| path | Path within storage bucket to file |
| size | Size of the file in bytes |
| createdAt | Time of the file save, stored as epoch time in milliseconds |
| creator | Name of the file creator (e.g. - email address for users, agent name + MAC address for Agents) |
| creatorType | Type of the file creator (e.g. - User, Agent) |
| fileTags | List of JSON objects containing Tags associated with the file; tag ID, display value, URL, tag type, background color, text color, tag applied date, and tag updated date are present for each Tag |
Python Packages
The ganymede_env_python_package_list shows a list of all Python packages installed in the workflow environment.
| Field Name | Description |
|---|---|
| package_name | Name of the Python package |
| version_number | Version of the Python package |
Pandas <-> Ganymede DB Field Reference
SQL columns have character restrictions for compatibility with SQL which are not present in Pandas. __TABLE_SCHEMA contains the mapping between table field names referenced in NodeReturn and the corresponding SQL column names.
The retrieve_tables method associated with Ganymede class provides a convenient method to retrieve tables in notebooks with their original Pandas Dataframe column names.
This table contains 1 record per table write transaction.
| Field Name | Description |
|---|---|
| pd_field_name | Pandas field name |
| pd_field_type | Pandas field column type |
| bq_table_name | Table name in Ganymede data lake |
| bq_field_name | Column name in Ganymede data lake |
| flow_run_id | Flow run that the table write is associated with |