Node Overview

Nodes provide template structure for performing data extraction, processing, and API communication. The diagram below shows how a node performs a base function, but exposes an interface for user-defined code to enable tailoring.

In Jupyter notebooks associated backing editable nodes, user-defined SQL and user-defined Python cells share a light blue background and are committed to a repo + deployed to a workflow orchestrator upon running the Save Pipeline Code cell at the bottom of the notebook, or by pressing Cmd+Shift+S or Ctrl+Shift+S (Windows/Linux).

Key Node Characteristics

A useful way to interact with nodes is to consider how nodes interact with other systems, their input and output types, and whether or not they contain a user-editable component. The table below shows the full list of available nodes, along with whether there is a user-editable component associated with the node.

Available Nodes with Key Characteristics

The table below lists available nodes.

• Is Editable indicates that the node has a user-editable components
• Is Multi indicates that the node can accept multiple inputs.

Category	Name	Input Types	Output Types	Is Editable	Is Multi
Analysis	Branch_Python	Table(s) and/or File(s)	NodeReturn	True	False
Analysis	Python	Table(s) and/or File(s)	NodeReturn	True	False
Analysis	RunContainer	FileAny	FileAny	True	False
Analysis	Transform_SQL	Table	Table	True	False
Analysis	Transform_py	Table or List[Table]	NodeReturn	True	False
Analysis	Trigger_Python	Table(s) and/or File(s)	FileAny	True	False
App	APINode	API	NodeReturn	True	False
App	AirtableExport	Table	API	True	False
App	AirtableImport	API	NodeReturn	False	False
App	Azure_Query	API	NodeReturn	True	False
App	Azure_Read	API	FileAny	False	False
App	Azure_Read_Multi	API	FileAny	False	True
App	Azure_Write	FileAny	API	False	False
App	Benchling_Api	API	NodeReturn	True	False
App	Benchling_Event	App	FileAny	True	False
App	Benchling_Read	API	NodeReturn	True	False
App	Benchling_Read_Object	API	NodeReturn	True	False
App	Benchling_Write	Table	NodeReturn	True	False
App	Benchling_Write_Object	Optional[FileAny or List[FileAny]] and Optional[Table or List[Table]]	NodeReturn	True	False
App	Coda_Write	Table	NodeReturn	True	False
App	ELabNext_Write	Table	NodeReturn	True	False
App	Load_Parquet_to_Table	API	FileAny	False	False
App	S3_Event	App	FileAny	True	False
App	S3_Read	API	FileAny	False	False
App	S3_Write	FileAny	API	False	False
App	SciNote_API	Table	NodeReturn	True	False
App	Smartsheet_Read	API	NodeReturn	True	False
App	Snowflake_Write	Table	NodeReturn	False	False
File	AVI_Read	FileAVI	NodeReturn	True	False
File	AVI_Read_Multi	Set[FileAVI]	NodeReturn	True	True
File	CSV_Read	FileCSV	NodeReturn	True	False
File	CSV_Read_Multi	Set[FileCSV]	NodeReturn	True	True
File	CSV_Write	Table or List[Table]	NodeReturn	True	False
File	Excel_Read	FileExcel	NodeReturn	True	False
File	Excel_Read_Multi	Set[FileExcel]	NodeReturn	True	True
File	Excel_Write	Table or List[Table]	NodeReturn	True	False
File	FCS_Extract_Load	FileFCS	NodeReturn	True	False
File	HDF5_Read	FileHDF5	NodeReturn	True	False
File	Image_Read	FileImage	NodeReturn	True	False
File	Image_Read_Multi	List[FileImage]	NodeReturn	True	True
File	Image_Write	Table or List[Table]	NodeReturn	True	False
File	Input_File	FileAny	NodeReturn	True	False
File	Input_File_Multi	Set[FileAny]	NodeReturn	True	True
File	PDF_Read	FilePDF	NodeReturn	True	False
File	PDF_Read_Multi	Set[FilePDF]	NodeReturn	True	True
File	Powerpoint_Write	Table or List[Table]	NodeReturn	True	False
File	XML_Read	FileXML	NodeReturn	True	False
File	Zip_Read	FileZip	NodeReturn	True	False
Instrument	Agilent_HPLC_Read	File	NodeReturn	True	False
Instrument	Instron_Tensile_Read	FileIsTens	NodeReturn	True	False
Instrument	Profilometer_Read	FileHDF5	NodeReturn	True	False
Instrument	Synergy_Read	FileTxt	NodeReturn	True	False
Instrument	Synergy_Read_Multi	List[FileTxt]	NodeReturn	True	True
Tag	Benchling_Tag	TagBenchling	string	False	False
Tag	Input_Param	string	string	False	False

Node Categories

• App: Accesses third-party APIs for processing; in many cases, key exchange between third-party and Ganymede are necessary for functionality
• Analysis: Performs Python / SQL manipulations
• Instrument: Lab instrument-specific functions
• File: For ETL operations on data of specified type into Ganymede cloud
• Tag: For specifying parameters at flow runtime

Node Input/Output Types

Input/output types are split into the following categories:

• NodeReturn: Object holding tables and files to return from Node. Initializing a NodeReturn object involves passing the following parameters:
  • tables_to_upload: Dictionary of Pandas DataFrames keyed by table name to store in Ganymede
  • files_to_upload: Dictionary of files (bytes objects) keyed by file name to store in Ganymede

example

Running the following in a cell in the editor notebook

import pandas as pd

def execute():
    message = "Message to store in file"
    byte_message = bytes(message, "utf-8")

    df = pd.DataFrame({"col1": [1, 2, 3], "col2": [4, 5, 6]})

    # upload a table named 'my_table' and a file named 'my_file'
    return NodeReturn(files_to_upload={"my_file": message}, tables_to_upload={"my_table": df})

execute()

returns the following summary of the NodeReturn object:

Docstrings and source code can be viewed by typing ?NodeReturn and ??NodeReturn respectively in a cell in the editor notebook.

• Table: Tabular data retrieved from or passed to tenant-specific Ganymede data lake. Tables are retrieved from Ganymede data lake via ANSI SQL queries, and are passed to Ganymede data lake as pandas DataFrames
• API: access via third-party API
• File-related inputs/outputs: File of specified type.
  • FileAVI: AVI file
  • FileCSV: CSV file
  • FileExcel: Excel file (xls, xlsx, ..)
  • FileImage: Image file (png, bmp, ..)
  • FileHDF5: HDF5 file
  • FileXML: XML file
  • FileZip: Zip file
  • FileAny: generic data file, which may be unstructured
• TagBenchling: Benchling run tag
• string: String parameter

Set, List, and Dict correspond to Python sets, lists, and dictionaries respectively.

Optional indicates that the input or output is optional.

User-editable Nodes

User-editable nodes present an interface for modifying and testing code that is executed by the workflow management system. These Jupyter notebooks are split into the following sections:

• Node Description: A short blurb about the node that the user-editable function corresponds to
• Node Input Data: For nodes that retrieve tabular data from the data lake as input, the query string in this cell specifies the query (-ies) that are executed and presented to the user-defined function for processing.
• User-Defined Function: The execute function within this cell processes data. The workflow management system calls the execute function within this cell during flow execution.

info

The execute function may call classes and functions found within the User-Defined Function cell.

• Save Pipeline Code: This cell stores any changes to the Node Input Data (if present) and User-Defined Function cells.

• Testing Section: The cells in this section can be used for testing modifications to the SQL query and user-defined python function. This enables rapid iteration on user-defined code; after necessary edits are made, changes can be saved in by running the Save Pipeline Code cell.

List of Available Nodes

Category	Name	Brief Description
Analysis	Branch_Python	Process data with Python and conditionally execute downstream nodes
Analysis	Python	Process data with python
Analysis	RunContainer	Run container node
Analysis	Transform_SQL	SQL analysis Function
Analysis	Transform_py	Manipulate data with python
Analysis	Trigger_Python	Process data with Python and trigger subsequent flow
App	APINode	Generic API Access Node
App	AirtableExport	Export data from Ganymede data lake to Airtable
App	AirtableImport	Import data from Airtable into Ganymede data lake
App	Azure_Query	Query data from Azure SQL Server
App	Azure_Read	Read data from Azure Blob Storage
App	Azure_Read_Multi	Read all data from Azure Blob Storage
App	Azure_Write	Write data to Azure Blob storage
App	Benchling_Api	Read Benchling data into data lake
App	Benchling_Event	Capture events from Benchling for triggering flows
App	Benchling_Read	Read Benchling data into data lake using run tag
App	Benchling_Read_Object	Read Benchling data into data lake using object ID
App	Benchling_Write	Write to Benchling
App	Benchling_Write_Object	Write object to Benchling
App	Coda_Write	Write Coda tables
App	ELabNext_Write	Create and write eLabNext entry
App	Load_Parquet_to_Table	Create datalake table from parquet files
App	S3_Event	Capture events from AWS S3 for triggering flows
App	S3_Read	Ingest data into Ganymede data storage from AWS S3 storage
App	S3_Write	Write data to an S3 bucket
App	SciNote_API	Create and write SciNote entry
App	Smartsheet_Read	Read sheet from Smartsheet
App	Snowflake_Write	Sync tables in Ganymede data lake to Snowflake
File	AVI_Read	Read in contents of an AVI file to a table
File	AVI_Read_Multi	Read in contents of multiple avi files to a table
File	CSV_Read	Read in contents of a CSV file
File	CSV_Read_Multi	Read in contents of multiple CSV files
File	CSV_Write	Write table to CSV file
File	Excel_Read	Read Excel spreadsheet
File	Excel_Read_Multi	Read Excel spreadsheets
File	Excel_Write	Write Excel spreadsheet
File	FCS_Extract_Load	Load FCS file to data lake
File	HDF5_Read	Read HDF5 data
File	Image_Read	Process image data; store processed images to data store
File	Image_Read_Multi	Process image data for multiple images; store processed images to data store
File	Image_Write	Process tabular data; write an image to data lake
File	Input_File	Read data file and process in Ganymede
File	Input_File_Multi	Read data files and process in Ganymede
File	PDF_Read	Read in contents of an PDF file to a table
File	PDF_Read_Multi	Read in contents of multiple pdf files to a table
File	Powerpoint_Write	Process tabular data; write a powerpoint presentation to data lake
File	XML_Read	Read XML file into data lake
File	Zip_Read	Extract Zip file
Instrument	Agilent_HPLC_Read	Read an Agilent HPLC file of type .uv, .ms, .ch, or .bin
Instrument	Instron_Tensile_Read	Load .is_tens file to data lake
Instrument	Profilometer_Read	Read Mx Profiler data file
Instrument	Synergy_Read	Load Synergy text file to data lake
Instrument	Synergy_Read_Multi	Load multiple Synergy texts file to data lake
Tag	Benchling_Tag	Read benchling tag
Tag	Input_Param	Input parameter into Flow