Flowchart Overview¶

The flowchart allows you to analyze samples in your project and create plots by arranging analysis nodes. Each node takes an input, performs an operation, and produces an output. For example the Derivative node takes use-specified numerical arrays, computes the derivative of these arrays, and then outputs the result.

The Flowchart is based on the pyqtgraph flowchart widgets

Flowchart Window

Add node: Right click -> Add node -> Choose from selection

Click on a node to highlight the Control Widget

Remove node: Right click -> Remove node

Connecting nodes: Click on a node terminal and drag to another terminal

Save the flowchart layout: Click “Save as…” to save the layout to a new file. You must specify the file extension as “.fc”. If you save this file within the “flowcharts” directory of your project it will show up in the Welcome Window when you open your project.

Note

This does not save the data, use the Save node to save data.

Warning

Due to a weird Qt or pyqtgraph bug certain parameter values (such as those in drop-down menus) can’t be saved. Similarly, parameters values are lost when you save to an existing .fc file. If you’re interested take a look at pyqtgraphCore.WidgetGroup. Anyways you shouldn’t be using the flowchart layout to save this information, that’s what the History Trace in Transmission objects is for.

Load an .fc file: Click the “Load” button.

Reset View button: Reset the view, for example if you zoom out or pan too far.

Video Tutorial¶

Part 5 - 9 of the Main Tutorial series also provide various examples for how the flowchart can be used: https://www.youtube.com/playlist?list=PLgofWiw2s4REPxH8bx8wZo_6ca435OKqg

Transmission¶

API Reference

Almost every node uses a Transmission object for input and output. A Transmission is basically a DataFrame and a History Trace (analysis log) of the data within the DataFrame.

Transmission DataFrame

The Transmission DataFrame is created from your Project DataFrame (or sub-DataFrame) by the Load_Proj_DF node. This initial DataFrame will contain the same columns as your Project DataFrame, and a new column named _RAW_CURVE. Each element (row) in the _RAW_CURVE column is a 1-D numerical array representing a single raw curve extracted from an ROI.

A new column named _BLOCK_ is also added which contains the UUID for logging the analysis history of this newly created block of DataFrame rows, known as a data block. This allows you to merge Transmissions (see Merge node) and maintain their independent analysis logs prior to the merge.

Naming conventions for DataFrame columns according to the data types

numerical data: single leading underscore ( _ ). All caps if produced by a flowchart node.

categorial data: no leading underscore. All caps if produced by flowhchart node.

special cases: Peak detection data are placed in a column named peaks_bases where each element is a DataFrame.

uuid data: has uuid or UUID in the name

Note

_BLOCK_ is an exception, it contains UUIDs not numerical data.

History Trace

The History Trace of a Transmission is a log containing the discrete analysis steps, known as operations, along with their parameters and any other useful information. When a flowchart node performs an operation it stores the output(s) data in the Transmission DataFrame and appends the operation parameters to this log. A seperate log is kept for each data block present in the Transmission DataFrame.

Console¶

You have direct access to the data within the nodes through the console in the flowchart. To show the console go to View -> Console.

Transmission Files¶

You can save a Transmission files using the Save node and work with the data directly in scripts, jupyter notebooks etc. You can also save them through the flowchart console (and plot consoles) through Transmission.to_hdf5.

Load Transmission¶

Load a saved Transmission instance using Transmission.from_hdf5

>>> from mesmerize import Transmission
>>> from uuid import UUID

>>> t = Transmission.from_hdf5('/share/data/temp/kushal/data.trn')
<mesmerize.analysis.data_types.Transmission at 0x7f4d42f386a0>

# The DataFrame is always the 'df' attribute
>>> t.df.head()

                                           CurvePath  ... FCLUSTER_LABELS
0  curves/a2-_-1-_-843c2d43-75f3-421a-9fef-483d1e...  ...               8
1  curves/brn3b_a6-_-2-_-21557a64-6868-4ff4-8db1-...  ...               4
2  curves/brn3b_a6-_-2-_-21557a64-6868-4ff4-8db1-...  ...               5
3  curves/brn3b_day1_3-_-2-_-ff3e95df-0e15-495c-9...  ...               8
4  curves/brn3b_day1_3-_-2-_-ff3e95df-0e15-495c-9...  ...               6

[5 rows x 27 columns]

View History Log¶

Transmissions have a history_trace attribute which is an instance of HistoryTrace.

Use the get_data_block_history and get_operations_list methods to view the history log of a data block.

# To view the history log, first get the block UUID of the dataframe row of which you want the history log

# Block UUIDs are stored in the _BLOCK_ column
>>> bid = t.df.iloc[10]._BLOCK_
>>> bid

'248a6ece-e60e-4a09-845e-188a5199d262'

# Get the history log of this data block
# HistoryTrace.get_operations_list() returns a list of operations, without parameters
# HistoryTrace.get_data_block_history() returns the operations list with the parameters
>>> t.history_trace.get_operations_list(bid)

['spawn_transmission',
 'splice_arrays',
 'normalize',
 'rfft',
 'absolute_value',
 'log_transform',
 'splice_arrays',
 'fcluster']

# View the entire history log with all params
>>> t.history_trace.get_data_block_history(bid)

[{'spawn_transmission': {'sub_dataframe_name': 'neuronal',
'dataframe_filter_history': {'dataframe_filter_history': ['df[~df["promoter"].isin([\'cesa\', \'hnk1\'])]',
    'df[~df["promoter"].isin([\'cesa\', \'hnk1\'])]',
    'df[~df["cell_name"].isin([\'not_a_neuron\', \'non_neuronal\', \'untagged\', \'ependymal\'])]']}}},
{'splice_arrays': {'data_column': '_RAW_CURVE',
'start_ix': 0,
'end_ix': 2990,
'units': 'time'}},
{'normalize': {'data_column': '_SPLICE_ARRAYS', 'units': 'time'}},
{'rfft': {'data_column': '_NORMALIZE',
'frequencies': [0.0,
    0.0033444816053511705,
    0.0033444816053511705,
    0.006688963210702341,
    ...

# Get the parameters for the 'fcluster' operation
>>> fp = t.history_trace.get_operation_params(bid, 'fcluster')

# remove the linkage matrix first so we can view the other params
>>> fp.pop('linkage_matrix');fp

{'threshold': 8.0,
 'criterion': 'maxclust',
 'depth': 1,
 'linkage_params': {'method': 'complete',
 'metric': 'wasserstein',
 'optimal_ordering': True}}