Flowchart Overview

The flowchart allows you to analyze samples in your project and create plots by arranging analysis nodes. Each node takes an input, performs an operation, and produces an output. For example the Derivative node takes use-specified numerical arrays, computes the derivative of these arrays, and then outputs the result.

The Flowchart is based on the pyqtgraph flowchart widgets

Flowchart Window


Add node: Right click -> Add node -> Choose from selection

Click on a node to highlight the Control Widget

Remove node: Right click -> Remove node

Connecting nodes: Click on a node terminal and drag to another terminal

Save the flowchart layout: Click “Save as…” to save the layout to a new file. You must specify the file extension as “.fc”. If you save this file within the “flowcharts” directory of your project it will show up in the Welcome Window when you open your project.


This does not save the data, use the Save node to save data.


Due to a weird Qt or pyqtgraph bug certain parameter values (such as those in drop-down menus) can’t be saved. Similarly, parameters values are lost when you save to an existing .fc file. If you’re interested take a look at pyqtgraphCore.WidgetGroup. Anyways you shouldn’t be using the flowchart layout to save this information, that’s what the History Trace in Transmission objects is for.

Load an .fc file: Click the “Load” button.

Reset View button: Reset the view, for example if you zoom out or pan too far.

Video Tutorial

Part 5 - 9 of the Main Tutorial series also provide various examples for how the flowchart can be used: https://www.youtube.com/playlist?list=PLgofWiw2s4REPxH8bx8wZo_6ca435OKqg


API Reference

Almost every node uses a Transmission object for input and output. A Transmission is basically a DataFrame and a History Trace (analysis log) of the data within the DataFrame.

Transmission DataFrame

The Transmission DataFrame is created from your Project DataFrame (or sub-DataFrame) by the Load_Proj_DF node. This initial DataFrame will contain the same columns as your Project DataFrame, and a new column named _RAW_CURVE. Each element (row) in the _RAW_CURVE column is a 1-D numerical array representing a single raw curve extracted from an ROI.

A new column named _BLOCK_ is also added which contains the UUID for logging the analysis history of this newly created block of DataFrame rows, known as a data block. This allows you to merge Transmissions (see Merge node) and maintain their independent analysis logs prior to the merge.

Naming conventions for DataFrame columns according to the data types

  • numerical data: single leading underscore ( _ ). All caps if produced by a flowchart node.

  • categorial data: no leading underscore. All caps if produced by flowhchart node.

  • special cases: Peak detection data are placed in a column named peaks_bases where each element is a DataFrame.

  • uuid data: has uuid or UUID in the name


_BLOCK_ is an exception, it contains UUIDs not numerical data.

History Trace

The History Trace of a Transmission is a log containing the discrete analysis steps, known as operations, along with their parameters and any other useful information. When a flowchart node performs an operation it stores the output(s) data in the Transmission DataFrame and appends the operation parameters to this log. A seperate log is kept for each data block present in the Transmission DataFrame.


You have direct access to the data within the nodes through the console in the flowchart. To show the console go to View -> Console.

See also

If you are unfamiliar with the console see the overview on Consoles

Call get_nodes() to view a dict of all nodes in the flowchart. You can access the output Transmission in most nodes through the attribute t. You can access the transmission dataframe through t.df.

See also

See the Transmission API for more information. Sources for the nodes at mesmerize/pyqtgraphCore/flowchart/library.

Example, directly accessing DataFrame elements through the flowchart console


Transmission Files

You can save a Transmission files using the Save node and work with the data directly in scripts, jupyter notebooks etc. You can also save them through the flowchart console (and plot consoles) through Transmission.to_hdf5.

Load Transmission

Load a saved Transmission instance using Transmission.from_hdf5

 1>>> from mesmerize import Transmission
 2>>> from uuid import UUID
 4>>> t = Transmission.from_hdf5('/share/data/temp/kushal/data.trn')
 5<mesmerize.analysis.data_types.Transmission at 0x7f4d42f386a0>
 7# The DataFrame is always the 'df' attribute
 8>>> t.df.head()
10                                           CurvePath  ... FCLUSTER_LABELS
110  curves/a2-_-1-_-843c2d43-75f3-421a-9fef-483d1e...  ...               8
121  curves/brn3b_a6-_-2-_-21557a64-6868-4ff4-8db1-...  ...               4
132  curves/brn3b_a6-_-2-_-21557a64-6868-4ff4-8db1-...  ...               5
143  curves/brn3b_day1_3-_-2-_-ff3e95df-0e15-495c-9...  ...               8
154  curves/brn3b_day1_3-_-2-_-ff3e95df-0e15-495c-9...  ...               6
17[5 rows x 27 columns]

View History Log

Transmissions have a history_trace attribute which is an instance of HistoryTrace.

Use the get_data_block_history and get_operations_list methods to view the history log of a data block.

 1# To view the history log, first get the block UUID of the dataframe row of which you want the history log
 3# Block UUIDs are stored in the _BLOCK_ column
 4>>> bid = t.df.iloc[10]._BLOCK_
 5>>> bid
 9# Get the history log of this data block
10# HistoryTrace.get_operations_list() returns a list of operations, without parameters
11# HistoryTrace.get_data_block_history() returns the operations list with the parameters
12>>> t.history_trace.get_operations_list(bid)
15 'splice_arrays',
16 'normalize',
17 'rfft',
18 'absolute_value',
19 'log_transform',
20 'splice_arrays',
21 'fcluster']
23# View the entire history log with all params
24>>> t.history_trace.get_data_block_history(bid)
26[{'spawn_transmission': {'sub_dataframe_name': 'neuronal',
27'dataframe_filter_history': {'dataframe_filter_history': ['df[~df["promoter"].isin([\'cesa\', \'hnk1\'])]',
28    'df[~df["promoter"].isin([\'cesa\', \'hnk1\'])]',
29    'df[~df["cell_name"].isin([\'not_a_neuron\', \'non_neuronal\', \'untagged\', \'ependymal\'])]']}}},
30{'splice_arrays': {'data_column': '_RAW_CURVE',
31'start_ix': 0,
32'end_ix': 2990,
33'units': 'time'}},
34{'normalize': {'data_column': '_SPLICE_ARRAYS', 'units': 'time'}},
35{'rfft': {'data_column': '_NORMALIZE',
36'frequencies': [0.0,
37    0.0033444816053511705,
38    0.0033444816053511705,
39    0.006688963210702341,
40    ...
42# Get the parameters for the 'fcluster' operation
43>>> fp = t.history_trace.get_operation_params(bid, 'fcluster')
45# remove the linkage matrix first so we can view the other params
46>>> fp.pop('linkage_matrix');fp
48{'threshold': 8.0,
49 'criterion': 'maxclust',
50 'depth': 1,
51 'linkage_params': {'method': 'complete',
52 'metric': 'wasserstein',
53 'optimal_ordering': True}}