Flowchart Overview

The flowchart allows you to analyze samples in your project and create plots by arranging analysis nodes. Each node takes an input, performs an operation, and produces an output. For example the Derivative node takes use-specified numerical arrays, computes the derivative of these arrays, and then outputs the result.

The Flowchart is based on the pyqtgraph flowchart widgets

Flowchart Window

../../_images/flowchart_overview.png

Add node: Right click -> Add node -> Choose from selection

Click on a node to highlight the Control Widget

Remove node: Right click -> Remove node

Connecting nodes: Click on a node terminal and drag to another terminal

Save the flowchart layout: Click “Save as…” to save the layout to a new file. You must specify the file extension as “.fc”. If you save this file within the “flowcharts” directory of your project it will show up in the Welcome Window when you open your project.

Note

This does not save the data, use the Save node to save data.

Warning

Due to a weird Qt or pyqtgraph bug certain parameter values (such as those in drop-down menus) can’t be saved. Similarly, parameters values are lost when you save to an existing .fc file. If you’re interested take a look at pyqtgraphCore.WidgetGroup. Anyways you shouldn’t be using the flowchart layout to save this information, that’s what the History Trace in Transmission objects is for.

Load an .fc file: Click the “Load” button.

Reset View button: Reset the view, for example if you zoom out or pan too far.

Video Tutorial

Part 5 - 9 of the Main Tutorial series also provide various examples for how the flowchart can be used: https://www.youtube.com/playlist?list=PLgofWiw2s4REPxH8bx8wZo_6ca435OKqg

Transmission

API Reference

Almost every node uses a Transmission object for input and output. A Transmission is basically a DataFrame and a History Trace (analysis log) of the data within the DataFrame.

Transmission DataFrame

The Transmission DataFrame is created from your Project DataFrame (or sub-DataFrame) by the Load_Proj_DF node. This initial DataFrame will contain the same columns as your Project DataFrame, and a new column named _RAW_CURVE. Each element (row) in the _RAW_CURVE column is a 1-D numerical array representing a single raw curve extracted from an ROI.

A new column named _BLOCK_ is also added which contains the UUID for logging the analysis history of this newly created block of DataFrame rows, known as a data block. This allows you to merge Transmissions (see Merge node) and maintain their independent analysis logs prior to the merge.

Naming conventions for DataFrame columns according to the data types

  • numerical data: single leading underscore ( _ ). All caps if produced by a flowchart node.

  • categorial data: no leading underscore. All caps if produced by flowhchart node.

  • special cases: Peak detection data are placed in a column named peaks_bases where each element is a DataFrame.

  • uuid data: has uuid or UUID in the name

Note

_BLOCK_ is an exception, it contains UUIDs not numerical data.

History Trace

The History Trace of a Transmission is a log containing the discrete analysis steps, known as operations, along with their parameters and any other useful information. When a flowchart node performs an operation it stores the output(s) data in the Transmission DataFrame and appends the operation parameters to this log. A seperate log is kept for each data block present in the Transmission DataFrame.

Console

You have direct access to the data within the nodes through the console in the flowchart. To show the console go to View -> Console.

See also

If you are unfamiliar with the console see the overview on Consoles

Call get_nodes() to view a dict of all nodes in the flowchart. You can access the output Transmission in most nodes through the attribute t. You can access the transmission dataframe through t.df.

See also

See the Transmission API for more information. Sources for the nodes at mesmerize/pyqtgraphCore/flowchart/library.

Example, directly accessing DataFrame elements through the flowchart console

../../_images/flowchart_console.png

Transmission Files

You can save a Transmission files using the Save node and work with the data directly in scripts, jupyter notebooks etc. You can also save them through the flowchart console (and plot consoles) through Transmission.to_hdf5.

Working with Transmission files

Load a saved Transmission instance using Transmission.from_hdf5

  1>>> from mesmerize import Transmission
  2>>> from uuid import UUID
  3
  4# load transmission file
  5>>> t = Transmission.from_hdf5('/share/data/temp/kushal/data.trn')
  6<mesmerize.analysis.data_types.Transmission at 0x7f4d42f386a0>
  7
  8# The DataFrame is always the 'df' attribute
  9>>> t.df.head()
 10
 11                                           CurvePath  ... FCLUSTER_LABELS
 120  curves/a2-_-1-_-843c2d43-75f3-421a-9fef-483d1e...  ...               8
 131  curves/brn3b_a6-_-2-_-21557a64-6868-4ff4-8db1-...  ...               4
 142  curves/brn3b_a6-_-2-_-21557a64-6868-4ff4-8db1-...  ...               5
 153  curves/brn3b_day1_3-_-2-_-ff3e95df-0e15-495c-9...  ...               8
 164  curves/brn3b_day1_3-_-2-_-ff3e95df-0e15-495c-9...  ...               6
 17
 18[5 rows x 27 columns]
 19
 20# the `df` is just a pandas dataframe
 21# View a list of samples in the current file
 22>>> t.df.SampleID.unique()
 23
 24array(['a2-_-1', 'a5-_-1', 'brn3b_a6-_-2', 'brn3b_day1_3-_-2',
 25    'brn3b_day1_a1-_-2', 'brn3b_day1_a2-_-2', 'brn3b_day1_a4-_-2',
 26    'brn3b_day2_a1-_-2', 'brn3b_day2_a1-_-t', 'brn3b_day2_a10-_-2',
 27    'brn3b_day2_a2-_-1', 'brn3b_day2_a2-_-3', 'brn3b_day2_a8-_-1',
 28    'cesa_a1-_-1', 'cesa_a1-_-2', 'cesa_a1_jan_2019-_-1',
 29    'cesa_a1_jan_2019-_-2', 'cesa_a2-_-2', 'cesa_a6-_-1',
 30    'cesa_a7-_-1', 'cesa_a7-_-2', 'cesa_a8-_-1', 'cesa_a9-_-1',
 31    'cng_ch4_day1_a2-_-t1', 'cng_ch4_day1_a2-_-t2',
 32    'cng_ch4_day2_a4-_-t1', 'dmrt1_day1_a2-_-2', 'dmrt1_day1_a4-_-t2',
 33    'dmrt1_day1_a5-_-', 'dmrt1_day1_a6-_-t', 'dmrt1_day1_a6-_-t2',
 34    'dmrt1_day2_a1-_-t1', 'dmrt1_day2_a1-_-t2', 'dmrt1_day2_a2-_-t1',
 35    'dmrt1_day2_a3-_-t1', 'dmrt1_day2_a3-_-t2', 'dmrt1_day2_a4-_-t1',
 36    'dmrt1_day2_a4-_-t2', 'hnk1_a5-_-2', 'hnk1_a6-_-1', 'hnk1_a7-_-1',
 37    'hnk1_a7-_-2', 'hnk1_a8-_-1', 'pc2_a10-_-1', 'pc2_a11-_-1',
 38    'pc2_a13-_-1', 'pc2_a14-_-1', 'pc2_a15-_-1', 'pc2_a16-_-1',
 39    'pc2_a9-_-1', 'pde9_day1_a2-_-2', 'pde9_day1_a3-_-1',
 40    'pde9_day1_a4-_-1', 'pde9_day1_a4-_-2', 'pde9_day2_a2-_-t2',
 41    'pde9_day2_a2-_-t4', 'pde9_day2_a4-_-t1', 'pde9_day2_a4-_-t2',
 42    'pde9_day2_a4-_-t3', 'pde9_day2_a5-_-t1', 'pde9_day2_a5-_-t2',
 43    'pde9_day2_a6-_-t1', 'pde9_day2_a7-_-t1', 'pde9_day2_a7-_-t2'],
 44    dtype=object)
 45
 46# Show data associated with a single sample
 47>>> t.df[t.df['SampleID'] == 'brn3b_day1_a1-_-2']
 48
 49                                            CurvePath  ... FCLUSTER_LABELS
 506   curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...  ...               6
 517   curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...  ...               6
 528   curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...  ...               5
 539   curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...  ...               7
 5410  curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...  ...               5
 55
 56# View the data associated with one ROI
 57# the `uuid_curve` is a unique identifier for each curve/ROI
 58>> t.df[t.df['SampleID'] == 'brn3b_day1_a1-_-2'].iloc[0]
 59
 60CurvePath              curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...
 61ImgInfoPath            images/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...
 62ImgPath                images/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...
 63ImgUUID                             d3c5f225-7039-4abd-a7a1-5e9ef2150013
 64ROI_State              {'roi_xs': [554, 553, 553, 552, 552, 551, 551,...
 65SampleID                                               brn3b_day1_a1-_-2
 66anatomical_location                                                 palp
 67cell_name                                                           palp
 68comments                                                        untagged
 69date                                                     20190425_110103
 70dorso_ventral_axis                                              untagged
 71misc                                                                  {}
 72morphology                                                      untagged
 73promoter                                                           brn3b
 74rostro_caudal_axis                                              untagged
 75stimulus_name                                                 [untagged]
 76uuid_curve                          f44fbd3d-6eaa-4e19-a677-496908565fde
 77_RAW_CURVE             [81.41972198848178, 75.61356993008134, 70.0493...
 78meta                   {'origin': 'AwesomeImager', 'version': '4107ff...
 79stim_maps                                                       [[None]]
 80_BLOCK_                             3e069e2d-d012-47ee-830c-93d85197e2f4
 81_SPLICE_ARRAYS         [2.646593459501195, 1.8252819116136887, 1.7422...
 82_NORMALIZE             [0.0681729940259753, 0.06533186950232853, 0.06...
 83_RFFT                  [443.19357880089615, -66.8777897472859, 55.244...
 84_ABSOLUTE_VALUE        [443.19357880089615, 66.8777897472859, 55.2443...
 85_LOG_TRANSFORM         [2.646593459501195, 1.8252819116136887, 1.7422...
 86FCLUSTER_LABELS                                                        6
 87Name: 6, dtype: object
 88
 89# Show the ROI object data
 90>>> t.df[t.df['SampleID'] == 'brn3b_day1_a1-_-2'].iloc[0]['ROI_State']
 91
 92{'roi_xs': array([554, 553, 553, 552, 552, 551, 551, 551, 551, 550, 550, 550, 549,
 93    548, 547, 547, 546, 546, 545, 545, 544, 543, 543, 542, 541, 541,
 94    540, 540, 539, 539, 538, 537, 536, 535, 534, 533, 532, 531, 531,
 95    530, 529, 528, 527, 527, 526, 526, 525, 525, 525, 524, 524, 523,
 96    522, 522, 521, 521, 520, 521, 521, 521, 521, 521, 522, 522, 522,
 97    522, 522, 522, 522, 522, 521, 521, 521, 521, 521, 521, 522, 523,
 98    524, 524, 525, 525, 525, 526, 526, 527, 528, 528, 529, 529, 529,
 99    530, 530, 531, 532, 532, 533, 534, 535, 535, 536, 536, 537, 538,
100    539, 540, 540, 541, 541, 542, 542, 543, 544, 545, 546, 546, 547,
101    548, 548, 549, 549, 549, 549, 550, 550, 550, 550, 551, 551, 551,
102    552, 552, 552, 553, 553, 553, 554, 554, 554, 553, 554, 554, 554,
103    554, 554]),
104'roi_ys': array([155, 156, 156, 157, 157, 158, 159, 160, 160, 161, 162, 162, 162,
105        162, 163, 163, 164, 164, 165, 165, 165, 166, 166, 166, 167, 167,
106        167, 166, 167, 167, 167, 167, 167, 167, 167, 167, 167, 168, 168,
107        168, 168, 168, 168, 167, 167, 166, 166, 165, 164, 164, 163, 163,
108        163, 162, 162, 161, 161, 160, 160, 159, 158, 157, 156, 156, 155,
109        154, 153, 152, 151, 150, 150, 149, 148, 147, 146, 145, 144, 144,
110        144, 144, 143, 143, 142, 141, 141, 140, 140, 140, 139, 139, 138,
111        137, 137, 136, 136, 136, 135, 135, 135, 136, 136, 137, 137, 137,
112        137, 137, 138, 138, 138, 137, 137, 136, 136, 136, 136, 137, 137,
113        137, 138, 138, 139, 140, 141, 141, 142, 143, 144, 144, 145, 146,
114        146, 147, 148, 148, 149, 150, 150, 151, 151, 152, 152, 153, 154,
115        155, 155]),
116'curve_data': (array([   0,    1,    2, ..., 2996, 2997, 2998]),
117array([ 81.41972199,  75.61356993,  70.04934883, ..., 195.4416283 ,
118        184.8844155 , 174.76708104])),
119'tags': {'anatomical_location': 'palp',
120'cell_name': 'palp',
121'morphology': 'untagged'},
122'roi_type': 'CNMFROI',
123'cnmf_idx': 2}

View History Log

Transmissions have a history_trace attribute which is an instance of HistoryTrace.

Use the get_data_block_history and get_operations_list methods to view the history log of a data block.

 1# To view the history log, first get the block UUID of the dataframe row of which you want the history log
 2
 3# Block UUIDs are stored in the _BLOCK_ column
 4>>> bid = t.df.iloc[10]._BLOCK_
 5>>> bid
 6
 7'248a6ece-e60e-4a09-845e-188a5199d262'
 8
 9# Get the history log of this data block
10# HistoryTrace.get_operations_list() returns a list of operations, without parameters
11# HistoryTrace.get_data_block_history() returns the operations list with the parameters
12>>> t.history_trace.get_operations_list(bid)
13
14['spawn_transmission',
15 'splice_arrays',
16 'normalize',
17 'rfft',
18 'absolute_value',
19 'log_transform',
20 'splice_arrays',
21 'fcluster']
22
23# View the entire history log with all params
24>>> t.history_trace.get_data_block_history(bid)
25
26[{'spawn_transmission': {'sub_dataframe_name': 'neuronal',
27'dataframe_filter_history': {'dataframe_filter_history': ['df[~df["promoter"].isin([\'cesa\', \'hnk1\'])]',
28    'df[~df["promoter"].isin([\'cesa\', \'hnk1\'])]',
29    'df[~df["cell_name"].isin([\'not_a_neuron\', \'non_neuronal\', \'untagged\', \'ependymal\'])]']}}},
30{'splice_arrays': {'data_column': '_RAW_CURVE',
31'start_ix': 0,
32'end_ix': 2990,
33'units': 'time'}},
34{'normalize': {'data_column': '_SPLICE_ARRAYS', 'units': 'time'}},
35{'rfft': {'data_column': '_NORMALIZE',
36'frequencies': [0.0,
37    0.0033444816053511705,
38    0.0033444816053511705,
39    0.006688963210702341,
40    ...
41
42# Get the parameters for the 'fcluster' operation
43>>> fp = t.history_trace.get_operation_params(bid, 'fcluster')
44
45# remove the linkage matrix first so we can view the other params
46>>> fp.pop('linkage_matrix');fp
47
48{'threshold': 8.0,
49 'criterion': 'maxclust',
50 'depth': 1,
51 'linkage_params': {'method': 'complete',
52 'metric': 'wasserstein',
53 'optimal_ordering': True}}
54
55# Draw the analysis history as a graph
56# This will open your defeault pdf viewer with the graph
57>>> t.history_trace.draw_graph(bid, view=True)
58
59# If you are using the API to perform analysis on
60# transmission files, you can use the `HistoryTrace`
61# to log the analysis history
62# For example, add a number `3.14` to all datapoints in a curve
63>>> t.df['_RAW_CURVE'] = t.df['_RAW_CURVE'].apply(lambda x: x + 3.14)
64
65# Append the analysis log
66>>> t.history_trace.add_operation(data_block_id='all', operation='addition', parameters={'value': 3.14}