Flowchart Overview¶
The flowchart allows you to analyze samples in your project and create plots by arranging analysis nodes. Each node takes an input, performs an operation, and produces an output. For example the Derivative node takes use-specified numerical arrays, computes the derivative of these arrays, and then outputs the result.
The Flowchart is based on the pyqtgraph flowchart widgets
Flowchart Window
Add node: Right click -> Add node -> Choose from selection
Click on a node to highlight the Control Widget
Remove node: Right click -> Remove node
Connecting nodes: Click on a node terminal and drag to another terminal
Save the flowchart layout: Click “Save as…” to save the layout to a new file. You must specify the file extension as “.fc”. If you save this file within the “flowcharts” directory of your project it will show up in the Welcome Window when you open your project.
Note
This does not save the data, use the Save node to save data.
Warning
Due to a weird Qt or pyqtgraph bug certain parameter values (such as those in drop-down menus) can’t be saved. Similarly, parameters values are lost when you save to an existing .fc file. If you’re interested take a look at
pyqtgraphCore.WidgetGroup. Anyways you shouldn’t be using the flowchart layout to save this information, that’s what the History Trace in Transmission objects is for.
Load an .fc file: Click the “Load” button.
Reset View button: Reset the view, for example if you zoom out or pan too far.
Video Tutorial¶
Part 5 - 9 of the Main Tutorial series also provide various examples for how the flowchart can be used: https://www.youtube.com/playlist?list=PLgofWiw2s4REPxH8bx8wZo_6ca435OKqg
Transmission¶
Almost every node uses a Transmission object for input and output. A Transmission is basically a DataFrame and a History Trace (analysis log) of the data within the DataFrame.
Transmission DataFrame
The Transmission DataFrame is created from your Project DataFrame (or sub-DataFrame) by the Load_Proj_DF node. This initial DataFrame will contain the same columns as your Project DataFrame, and a new column named _RAW_CURVE. Each element (row) in the _RAW_CURVE column is a 1-D numerical array representing a single raw curve extracted from an ROI.
A new column named _BLOCK_ is also added which contains the UUID for logging the analysis history of this newly created block of DataFrame rows, known as a data block. This allows you to merge Transmissions (see Merge node) and maintain their independent analysis logs prior to the merge.
Naming conventions for DataFrame columns according to the data types
numerical data: single leading underscore ( _ ). All caps if produced by a flowchart node.
categorial data: no leading underscore. All caps if produced by flowhchart node.
special cases: Peak detection data are placed in a column named peaks_bases where each element is a DataFrame.
uuid data: has uuid or UUID in the name
Note
_BLOCK_ is an exception, it contains UUIDs not numerical data.
History Trace
The History Trace of a Transmission is a log containing the discrete analysis steps, known as operations, along with their parameters and any other useful information. When a flowchart node performs an operation it stores the output(s) data in the Transmission DataFrame and appends the operation parameters to this log. A seperate log is kept for each data block present in the Transmission DataFrame.
Console¶
You have direct access to the data within the nodes through the console in the flowchart. To show the console go to View -> Console.
See also
If you are unfamiliar with the console see the overview on Consoles
Call get_nodes() to view a dict of all nodes in the flowchart. You can access the output Transmission in most nodes through the attribute t. You can access the transmission dataframe through t.df.
See also
See the Transmission API for more information. Sources for the nodes at mesmerize/pyqtgraphCore/flowchart/library.
Example, directly accessing DataFrame elements through the flowchart console
Transmission Files¶
You can save a Transmission files using the Save node and work with the data directly in scripts, jupyter notebooks etc. You can also save them through the flowchart console (and plot consoles) through Transmission.to_hdf5.
Working with Transmission files¶
Load a saved Transmission instance using Transmission.from_hdf5
1>>> from mesmerize import Transmission
2>>> from uuid import UUID
3
4# load transmission file
5>>> t = Transmission.from_hdf5('/share/data/temp/kushal/data.trn')
6<mesmerize.analysis.data_types.Transmission at 0x7f4d42f386a0>
7
8# The DataFrame is always the 'df' attribute
9>>> t.df.head()
10
11 CurvePath ... FCLUSTER_LABELS
120 curves/a2-_-1-_-843c2d43-75f3-421a-9fef-483d1e... ... 8
131 curves/brn3b_a6-_-2-_-21557a64-6868-4ff4-8db1-... ... 4
142 curves/brn3b_a6-_-2-_-21557a64-6868-4ff4-8db1-... ... 5
153 curves/brn3b_day1_3-_-2-_-ff3e95df-0e15-495c-9... ... 8
164 curves/brn3b_day1_3-_-2-_-ff3e95df-0e15-495c-9... ... 6
17
18[5 rows x 27 columns]
19
20# the `df` is just a pandas dataframe
21# View a list of samples in the current file
22>>> t.df.SampleID.unique()
23
24array(['a2-_-1', 'a5-_-1', 'brn3b_a6-_-2', 'brn3b_day1_3-_-2',
25 'brn3b_day1_a1-_-2', 'brn3b_day1_a2-_-2', 'brn3b_day1_a4-_-2',
26 'brn3b_day2_a1-_-2', 'brn3b_day2_a1-_-t', 'brn3b_day2_a10-_-2',
27 'brn3b_day2_a2-_-1', 'brn3b_day2_a2-_-3', 'brn3b_day2_a8-_-1',
28 'cesa_a1-_-1', 'cesa_a1-_-2', 'cesa_a1_jan_2019-_-1',
29 'cesa_a1_jan_2019-_-2', 'cesa_a2-_-2', 'cesa_a6-_-1',
30 'cesa_a7-_-1', 'cesa_a7-_-2', 'cesa_a8-_-1', 'cesa_a9-_-1',
31 'cng_ch4_day1_a2-_-t1', 'cng_ch4_day1_a2-_-t2',
32 'cng_ch4_day2_a4-_-t1', 'dmrt1_day1_a2-_-2', 'dmrt1_day1_a4-_-t2',
33 'dmrt1_day1_a5-_-', 'dmrt1_day1_a6-_-t', 'dmrt1_day1_a6-_-t2',
34 'dmrt1_day2_a1-_-t1', 'dmrt1_day2_a1-_-t2', 'dmrt1_day2_a2-_-t1',
35 'dmrt1_day2_a3-_-t1', 'dmrt1_day2_a3-_-t2', 'dmrt1_day2_a4-_-t1',
36 'dmrt1_day2_a4-_-t2', 'hnk1_a5-_-2', 'hnk1_a6-_-1', 'hnk1_a7-_-1',
37 'hnk1_a7-_-2', 'hnk1_a8-_-1', 'pc2_a10-_-1', 'pc2_a11-_-1',
38 'pc2_a13-_-1', 'pc2_a14-_-1', 'pc2_a15-_-1', 'pc2_a16-_-1',
39 'pc2_a9-_-1', 'pde9_day1_a2-_-2', 'pde9_day1_a3-_-1',
40 'pde9_day1_a4-_-1', 'pde9_day1_a4-_-2', 'pde9_day2_a2-_-t2',
41 'pde9_day2_a2-_-t4', 'pde9_day2_a4-_-t1', 'pde9_day2_a4-_-t2',
42 'pde9_day2_a4-_-t3', 'pde9_day2_a5-_-t1', 'pde9_day2_a5-_-t2',
43 'pde9_day2_a6-_-t1', 'pde9_day2_a7-_-t1', 'pde9_day2_a7-_-t2'],
44 dtype=object)
45
46# Show data associated with a single sample
47>>> t.df[t.df['SampleID'] == 'brn3b_day1_a1-_-2']
48
49 CurvePath ... FCLUSTER_LABELS
506 curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-... ... 6
517 curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-... ... 6
528 curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-... ... 5
539 curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-... ... 7
5410 curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-... ... 5
55
56# View the data associated with one ROI
57# the `uuid_curve` is a unique identifier for each curve/ROI
58>> t.df[t.df['SampleID'] == 'brn3b_day1_a1-_-2'].iloc[0]
59
60CurvePath curves/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...
61ImgInfoPath images/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...
62ImgPath images/brn3b_day1_a1-_-2-_-d3c5f225-7039-4abd-...
63ImgUUID d3c5f225-7039-4abd-a7a1-5e9ef2150013
64ROI_State {'roi_xs': [554, 553, 553, 552, 552, 551, 551,...
65SampleID brn3b_day1_a1-_-2
66anatomical_location palp
67cell_name palp
68comments untagged
69date 20190425_110103
70dorso_ventral_axis untagged
71misc {}
72morphology untagged
73promoter brn3b
74rostro_caudal_axis untagged
75stimulus_name [untagged]
76uuid_curve f44fbd3d-6eaa-4e19-a677-496908565fde
77_RAW_CURVE [81.41972198848178, 75.61356993008134, 70.0493...
78meta {'origin': 'AwesomeImager', 'version': '4107ff...
79stim_maps [[None]]
80_BLOCK_ 3e069e2d-d012-47ee-830c-93d85197e2f4
81_SPLICE_ARRAYS [2.646593459501195, 1.8252819116136887, 1.7422...
82_NORMALIZE [0.0681729940259753, 0.06533186950232853, 0.06...
83_RFFT [443.19357880089615, -66.8777897472859, 55.244...
84_ABSOLUTE_VALUE [443.19357880089615, 66.8777897472859, 55.2443...
85_LOG_TRANSFORM [2.646593459501195, 1.8252819116136887, 1.7422...
86FCLUSTER_LABELS 6
87Name: 6, dtype: object
88
89# Show the ROI object data
90>>> t.df[t.df['SampleID'] == 'brn3b_day1_a1-_-2'].iloc[0]['ROI_State']
91
92{'roi_xs': array([554, 553, 553, 552, 552, 551, 551, 551, 551, 550, 550, 550, 549,
93 548, 547, 547, 546, 546, 545, 545, 544, 543, 543, 542, 541, 541,
94 540, 540, 539, 539, 538, 537, 536, 535, 534, 533, 532, 531, 531,
95 530, 529, 528, 527, 527, 526, 526, 525, 525, 525, 524, 524, 523,
96 522, 522, 521, 521, 520, 521, 521, 521, 521, 521, 522, 522, 522,
97 522, 522, 522, 522, 522, 521, 521, 521, 521, 521, 521, 522, 523,
98 524, 524, 525, 525, 525, 526, 526, 527, 528, 528, 529, 529, 529,
99 530, 530, 531, 532, 532, 533, 534, 535, 535, 536, 536, 537, 538,
100 539, 540, 540, 541, 541, 542, 542, 543, 544, 545, 546, 546, 547,
101 548, 548, 549, 549, 549, 549, 550, 550, 550, 550, 551, 551, 551,
102 552, 552, 552, 553, 553, 553, 554, 554, 554, 553, 554, 554, 554,
103 554, 554]),
104'roi_ys': array([155, 156, 156, 157, 157, 158, 159, 160, 160, 161, 162, 162, 162,
105 162, 163, 163, 164, 164, 165, 165, 165, 166, 166, 166, 167, 167,
106 167, 166, 167, 167, 167, 167, 167, 167, 167, 167, 167, 168, 168,
107 168, 168, 168, 168, 167, 167, 166, 166, 165, 164, 164, 163, 163,
108 163, 162, 162, 161, 161, 160, 160, 159, 158, 157, 156, 156, 155,
109 154, 153, 152, 151, 150, 150, 149, 148, 147, 146, 145, 144, 144,
110 144, 144, 143, 143, 142, 141, 141, 140, 140, 140, 139, 139, 138,
111 137, 137, 136, 136, 136, 135, 135, 135, 136, 136, 137, 137, 137,
112 137, 137, 138, 138, 138, 137, 137, 136, 136, 136, 136, 137, 137,
113 137, 138, 138, 139, 140, 141, 141, 142, 143, 144, 144, 145, 146,
114 146, 147, 148, 148, 149, 150, 150, 151, 151, 152, 152, 153, 154,
115 155, 155]),
116'curve_data': (array([ 0, 1, 2, ..., 2996, 2997, 2998]),
117array([ 81.41972199, 75.61356993, 70.04934883, ..., 195.4416283 ,
118 184.8844155 , 174.76708104])),
119'tags': {'anatomical_location': 'palp',
120'cell_name': 'palp',
121'morphology': 'untagged'},
122'roi_type': 'CNMFROI',
123'cnmf_idx': 2}
View History Log¶
Transmissions have a history_trace attribute which is an instance of HistoryTrace.
Use the get_data_block_history and get_operations_list methods to view the history log of a data block.
1# To view the history log, first get the block UUID of the dataframe row of which you want the history log
2
3# Block UUIDs are stored in the _BLOCK_ column
4>>> bid = t.df.iloc[10]._BLOCK_
5>>> bid
6
7'248a6ece-e60e-4a09-845e-188a5199d262'
8
9# Get the history log of this data block
10# HistoryTrace.get_operations_list() returns a list of operations, without parameters
11# HistoryTrace.get_data_block_history() returns the operations list with the parameters
12>>> t.history_trace.get_operations_list(bid)
13
14['spawn_transmission',
15 'splice_arrays',
16 'normalize',
17 'rfft',
18 'absolute_value',
19 'log_transform',
20 'splice_arrays',
21 'fcluster']
22
23# View the entire history log with all params
24>>> t.history_trace.get_data_block_history(bid)
25
26[{'spawn_transmission': {'sub_dataframe_name': 'neuronal',
27'dataframe_filter_history': {'dataframe_filter_history': ['df[~df["promoter"].isin([\'cesa\', \'hnk1\'])]',
28 'df[~df["promoter"].isin([\'cesa\', \'hnk1\'])]',
29 'df[~df["cell_name"].isin([\'not_a_neuron\', \'non_neuronal\', \'untagged\', \'ependymal\'])]']}}},
30{'splice_arrays': {'data_column': '_RAW_CURVE',
31'start_ix': 0,
32'end_ix': 2990,
33'units': 'time'}},
34{'normalize': {'data_column': '_SPLICE_ARRAYS', 'units': 'time'}},
35{'rfft': {'data_column': '_NORMALIZE',
36'frequencies': [0.0,
37 0.0033444816053511705,
38 0.0033444816053511705,
39 0.006688963210702341,
40 ...
41
42# Get the parameters for the 'fcluster' operation
43>>> fp = t.history_trace.get_operation_params(bid, 'fcluster')
44
45# remove the linkage matrix first so we can view the other params
46>>> fp.pop('linkage_matrix');fp
47
48{'threshold': 8.0,
49 'criterion': 'maxclust',
50 'depth': 1,
51 'linkage_params': {'method': 'complete',
52 'metric': 'wasserstein',
53 'optimal_ordering': True}}
54
55# Draw the analysis history as a graph
56# This will open your defeault pdf viewer with the graph
57>>> t.history_trace.draw_graph(bid, view=True)
58
59# If you are using the API to perform analysis on
60# transmission files, you can use the `HistoryTrace`
61# to log the analysis history
62# For example, add a number `3.14` to all datapoints in a curve
63>>> t.df['_RAW_CURVE'] = t.df['_RAW_CURVE'].apply(lambda x: x + 3.14)
64
65# Append the analysis log
66>>> t.history_trace.add_operation(data_block_id='all', operation='addition', parameters={'value': 3.14}