Thepage is the starting point for creating and modifying flows.
The header bar contains buttons for creating flows and monitoring flow runs.
Choose a Flow: Selection box for specifying the flow to load.
Node: Opens right sidebar for selecting new node to add.
View: Opens right sidebar for specifying whether flow should be laid out horizontally or vertically
Analysis: Enables access to Jupyter notebooks which can be used for ad-hoc observation and data analysis. These notebooks are hosted on a virtual machine (VM) separate from the workflow management software running the flow.
Save: Button to save flow layout to git repo and deploy code to workflow management system.
Manage: Opens right sidebar to create new flows or delete existing ones. When a flow is selected, we can apply new configurations to the flow.
- Owner - Owner of the flow. Does not affect how the flow runs.
- Retries - How many times the flow would retry to execute a node, in case of an error or a timeout.
Run Queue Bar
The Run Queue Bar located at the bottom of the screen contains buttons for executing flow runs. The full menu can be accessed by clicking on theicon in the bottom right corner of the bar.
- Add run: Adds a row to the run queue.
- Run all: Runs all jobs loaded into the run queue.
- Re-run: This requeues the last row that was ran for this specific flow. The button is greyed out if the current session had no prior runs, or if the page has been refreshed. If inputs have changed, this button will fill out all parameters that exist using the existing run.
- Processing: Shows the number of files in the queue for your next run. Clicking this button brings up the page.
To create a new flow, click thebutton in the upper right hand corner of the screen. Then click the button on the sidebar. This action exposes a modal for naming and describing the flow.
Scheduling Flow Execution
Flows can be configured to automatically run on a schedule. The schedule is set by selecting from the dropdowns shown below and can be configured in intervals of months, weeks, days, hours, and minutes.
Scheduled jobs are specified in UTC time.
Configuring Email Notifications
Email notifications are sent to specified users upon flow failure; an example email for this is shown below.
Retries and email notifications associated with the flow can be specified in this modal. Retries is the number of times that a node run within the flow is retried before processing is considered to have failed.
Loading and Saving Flows
To load a flow, specify the flow to load in theselection box. Upon doing so, the primary pane displays the graphical structure of the flow.
After saving a flow, the entire environment (which contains all flows) is saved and deployed. When this save and deploy is taking place, users are temporarily restricted from running flows. A tooltip will be displayed to the user if they try to run a flow during this time. Users can observe the status of environment updates on the page.
To select input files for flows, users can clickto choose a file or drag-and-drop a file into the box labeled "Upload file here". If is clicked, users have the option to either select an input file to process from the local computer or from Ganymede Cloud Storage (for files previously executed on flows in Ganymede).
If the input file is uploaded from local computer and has previously been uploaded to the corresponding flow and node (as determined by file name), users are prompted with an option to use the file that was previously uploaded or to save a new version with a unique id appended to the end of the file name.
To run the same flow with different sets of input files, add additional rows to the input box by clicking on thebutton.
Clicking thebutton on the Flow Inputs node kicks off the flow run. Run progress can be tracked by clicking on the Runs button in the header, or by navigating to the page.
All inputs MUST be filled out otherwise the run will not be started. A message box will appear saying which inputs have not been filled.
Autorun can be configured by clicking on the Input Settings button, which can be found on the top left corner of the Run Queue form:
Clicking here will bring up this modal:
Enabling this feature will allow for runs to start whenever all inputs for a row in the run queue are filled. This button is disabled if any nodes take more than 1 file, or if there are no input parameters for the Flow. This feature melds nicely with file watcher, and will work even after navigating away from thepage.
Modifying and Deleting Flows
Edges can be removed by selecting them and pressing "delete" or "backspace" on the keyboard. A selected edge is slightly darker and wider than unselected edges.
To make two nodes dependent on each other within a run, click and drag between two orbs two different nodes. Flow dependencies run top-to-bottom (if vertical layout is specified) or left-to-right (if horizontal layout is specified).
In order to delete flows, click theButton on the Flow Editor page, which opens up the right sidebar offering a button to
Deleting flows is not reversible from within the Ganymede UI, so undoing a flow deletion will require restoring code from a prior commit in the git repo corresponding to the Ganymede cloud tenant.