Build a Flow

This guide walks you through building a Flow for ingesting plate reader absorbance data captured at two timepoints and computing the difference between the two values.

Flows and Nodes

Flows are programs represented as directed graphs comprised of user-editable nodes. For precise definitions of Flows and nodes, check out the Ganymede Concepts page.

Step 1: Download sample plate reader file

Click here to download the plate reader file used for this tutorial.

Step 2: Sign into Ganymede

Using Google Chrome, sign into Ganymede. The

Sign In

button will be available once authentication is configured for your tenant. This is generally a web address accessible via any web browser with the URL https://<your_tenant>.ganymede.bio.

You may need to enable pop-ups for Ganymede or disable any ad-blocking software for the ganymede.bio domain.

Step 3: Create a new Flow

Click the

New Flow

button on the Home page.

 

This exposes a modal for adding a new Flow. Name the Flow "Quickstart_Absorbance_Change" as shown below, add a description, and then hit the "Create Flow" button.

After a few seconds, your Flow will be created and you'll be redirected to a blank Flow editor page where you can edit your new Flow. Select your Flow from the Choose A Flow dropdown.

Step 4: Add nodes to ingest and process data

Click the

Node

button in the header bar and search for the term "input" to filter for the relevant

Input_File

node. Click the

Add

button to add it to your Flow.

input file vs input file multi

The Input File Multi node can be used to ingest multiple files on a single node. For this quickstart, we will use the Input File node to ingest a single file.

 

Click the downward-pointing carat in the corner of the

Input_File

node to expand it and edit its attributes. Change the file_pattern node attribute from "file_pattern" to "*.xlsx" so that only files with .xlsx as an extension will be processed by this node (other file types, such as .csv will be blocked).

 

file extension validation

You can remove extension validation by leaving the file pattern attribute blank.

Click

Save

in the header to save this change.

tip

A green check box will pop up when the save and deploy are complete. You can also click the button in the upper right corner to view the Notifications page, where you can track the progress of a save.

Now you have a one-node Flow that is capable of ingesting Excel files. Let's add a second node to process this incoming raw data. Click the

Node

button in the header bar again, and search for the term "Python". This time, add a

Python

node to your Flow by clicking the

Add

button.

After adding the Node, the

Flow Editor

window should look like this:

 

We want data processing to flow from the

Input_File

Node to the

Python

Node. To do this, click on the

Input_File

node's bottom attachment point and drag the connection to the top of the

Python

node. Then click

Save

.

Step 5: Modify the Input File node to store results in a table

Expand the

Input_File

node by clicking on the downward-pointing caret in the corner of the node. Click the command prompt icon in the upper-right-hand corner of the node to edit the Python code associated with it.

We will output the contents of the plate reader spreadsheet into a cloud database table called Quickstart_Absorbance_Change_data by modifying the contents of the User-Defined Function associated with the

Input_File

node. Copy and paste the code below into the notebook cell.

from typing import Dict
from ganymede_sdk.io import NodeReturn
import pandas as pd


def execute(file_data: Dict[str, bytes], ganymede_context=None) -> NodeReturn:
    """
    Processes file data for saving in cloud storage

    Parameters
    ----------
    file_data : Dict[str, bytes]
        Bytes object to process, indexed by filename
    ganymede_context : GanymedeContext
        Ganymede context variable, which stores Flow run metadata

    Returns
    -------
    NodeReturn
        Table(s) and File(s) to store in Ganymede.  To write to the table referenced on the node,
        return a DataFrame in the "results" key of the tables_to_upload dictionary.  For more info,
        type '?NodeReturn' into a cell in the editor notebook.
    """

    excel_file = list(file_data.values()).pop()
    df_excel_file = pd.read_excel(excel_file)

    return NodeReturn(
        tables_to_upload={'Quickstart_Absorbance_Change_data': df_excel_file},
    )

non-tabular data

Non-tabular data can be stored by returning a dictionary of files to upload in the files_to_upload key of the NodeReturn object. Stored files can be retrieved from the Files page or the Flow Runs panel.

Run the User-Defined Function cell by selecting the cell and then clicking on the

▶️ Run

button in the header, which runs the selected cell. In the "Testing" section below, you can test uploading the original plate reader excel file and passing it into the User-Defined Function. This is a great way to debug your code before deploying the Flow.

Save and deploy changes made to the Python node by clicking the button in the toolbar, or by pressing Cmd+S (Mac) or Ctrl+S (Windows/Linux).

tip

Below the User-Defined Function cell in the notebook, you'll find the Testing Section. Here, you can upload a test input file and test the processing performed by the User-Defined Function interactively. This is a great way to debug your code before deploying the Flow.

Step 6: Modify the Python node to calculate absorbance difference

Currently the

Python

node is in its default state. Let's edit it to perform some processing specific to our Plate Reader data. To do so, open up the backing notebook for the

Python

node by clicking the command prompt icon in the upper-right-hand corner of the node.

 

Edit the query_sql string to query for the output of the

Input_File

node.

query_sql = """
  SELECT * FROM Quickstart_Absorbance_Change_data
"""

Since the plate reader file has been ingested in the

Input_File

node, we now can query the results of that node using a SQL query on the table Quickstart_Absorbance_Change_data. The query above simply selects all the data from this table.

Run this notebook cell by clicking on the

▶️ Run

button in the header.

Next, modify the User-Defined Function cell to calculate the absorbance difference as shown below, and run the cell:

import pandas as pd
from ganymede_sdk.io import NodeReturn
from typing import Union, Dict, List
from io import BytesIO


def execute(
    df_sql_result: Union[pd.DataFrame, List[pd.DataFrame]], ganymede_context=None
) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:

    # remove fields that do not contain well measurements
    df_out = df_sql_result[
        ~df_sql_result["field"].isin(["Cycle Nr.", "Time [s]", "Temp. [°C]"])
    ].copy()

    # calculate absorbance difference
    df_out["run_diff"] = df_out["run2"] - df_out["run1"]

    # output to CSV file
    bio = BytesIO()
    df_out.to_csv(bio, index=False)
    bio.seek(0)
    csv_file_output = bio.read()

    return NodeReturn(
        tables_to_upload={"quickstart_Python_results": df_out},
        files_to_upload={"absorbance_diff.csv": csv_file_output},
    )

This code calculates the difference between the two columns run1 and run2, and stores the result in a new column called run_diff. It also outputs a file called absorbance_diff.csv, which contains the results in CSV format.

tip

Near the bottom of the notebook you'll find the Testing Section. As with the Input_File node, you can interactively test the SQL query and the User-Defined Function. Note that tables are not generated until the Flow is run (from the Flow Editor, not the notebook), and so the Testing section for this Node will not run successfully unless the Input File Node has been run in a Flow.

After saving the Flow, close the editor notebook associated with the Python node to return to the

Flow Editor

view.

Step 7: Run the Flow

On the

Flow Editor

page, click

Expand

in the Run Queue bar at the bottom of the screen to expand the input form for running nodes. Click the "Drag and drop files here or Browse" button and upload the "PlateReader.xlsx" file downloaded at the beginning of this quickstart.

note

The associated file ingested into the Flow can be now found in the

Files

tab of Ganymede.

Click the

Run

button to kick off the Flow. The running Flow run will now be visible below.

tip

You can also click on

Flow Runs

in the left sidebar to monitor the current run or browse past runs.

When the job is complete, the Flow status should indicate success:

Step 8: Observe results in Data Explorer

Click on

Data Explorer

in the left sidebar and look for Quickstart_Absorbance_Change_data in the table browser.

This allows you to observe and query the Quickstart_Absorbance_Change_data table.

For example, clicking on the Query tab and running the query below:

select 
    left(field, 1) as well_row, 
    cast(right(field, length(field)-1) as int) as well_column,
    *
from Quickstart_Absorbance_Change_data where field not in ('Cycle Nr.', 'Time [s]', 'Temp. [°C]')
order by left(field, 1), cast(right(field, length(field)-1) as int)

will yield a view of the plate reader results submitted:

 

If you encounter any challenges with this Quickstart, please visit the Troubleshooting Flows page or contact Ganymede Support.