Nodes

The easiest way to create a new node is create a class that inherits from CtrlNode. You can place this class in one of the existing modules in mesmerize/pyqtgraphCore/flowchart/library

Become familiar with the Transmission object before creating a node. Almost all nodes work with a Transmission object for storing data. Make sure to conform to the conventions for naming of data columns and categorical columns.

Simple node

The simplest type of node performs an operation on a user-specified data column and doesn’t take any parameters.

Basic structure

class MyNode(CtrlNode):
"""Doc String that is shown when node is clicked on"""
nodeName = 'MyNode'
uiTemplate = <list of tuples, see below>

def processData(self, transmission: Transmission):
    self.t = transmission.copy()  #: input to this node

    # .. do stuff to the Transmission DataFrame

    params = <dict of analysis params>

    # log the analysis that was done
    self.t.history_trace.add_operation('all', 'mynode', params)

    # Send the output
    return self.t

Required class attributes:

If the node only has one input and one output terminal it is sufficient to create a processData method that performs the node’s analysis operation(s).

Example

 1class Derivative(CtrlNode):
 2"""Return the Derivative of a curve."""
 3nodeName = 'Derivative'
 4uiTemplate = [('data_column', 'combo', {}),
 5              ('Apply', 'check', {'checked': False, 'applyBox': True})
 6              ]
 7
 8# If there is only one input and one output temrinal, processData will
 9# always have a single argument which is just the input transmission,
10# i.e. the output from the previous node.
11def processData(self, transmission: Transmission):
12    # the input transmission
13    self.t = transmission
14
15    # If a comboBox widget named 'data_column' is specified in the
16    # uiTemplate, you can update its contents using the following method.
17    # This will populate the comboBox with all the data columns from the
18    # input transmission and select the input data column as the
19    # output data column from the previous node.
20    self.set_data_column_combo_box()
21
22    # Check if the Apply checkbox is checked
23    if self.ctrls['Apply'].isChecked() is False:
24        return
25
26    # Make a copy of the input transmission so we can modify it to create an output
27    self.t = transmission.copy()
28
29    # By convention output columns are named after the node's name and in all caps
30    # Columns containing numerical data have a leading underscore
31    output_column = '_DERIVATIVE'
32
33    # Perform this nodees operation
34    self.t.df[output_column] = self.t.df[self.data_column].apply(np.gradient)
35
36    # Set tranmission's `last_output` attribute as the name of the output column
37    # This is used by the next node to know what thet last output data was
38    self.t.last_output = output_column
39
40    # Create a dict of parameters that this node used
41    # Usually a dict that captures the state of the uiTemplate
42    # the transmission `last_unit` attribute is the data units of the data
43    # in the output column (i.e. `t.last_output`). Change it only if the data units change
44    params = {'data_column': self.data_column,
45              'units': self.t.last_unit
46              }
47
48    # Add a log of this node's operation to the transmission's `HistoryTrace` instance
49    # Nodes usually perform an operation on all datablocks pass 'all' to the data_block_id argument
50    # By convention the operation name is the name of the node in lowercase letters
51    self.t.history_trace.add_operation(data_block_id='all', operation='derivative', parameters=params)
52
53    # return the modified transmission instance, which is then the output of this node
54    return self.t

Complex node

For a more complex node with multiple inputs and/or outputs you will need to explicitly specify the terminals when instantiating the parent CtrlNode and create a simple override of the process() method.

Format of the dict specifying the node’s terminals:

{
    <terminal name (str)>:             {'io': <'in' or 'out'>},
    <another terminal name (str)>:     {'io', <'in' or 'out'>},
    <another terminal name (str)>:     {'io', <'in' or 'out'>}
    ...
}

Override the process() method simply pass all kwargs to a processData() method and return the output The processData() method must return a dict. This dict must have keys that correspond to the specified output terminals. The values of these keys are the outputs from the respective terminals.

Here is a trimmed down example from the LDA node:

 1class LDA(CtrlNode):
 2"""Linear Discriminant Analysis, uses sklearn"""
 3nodeName = "LDA"
 4uiTemplate = [('train_data', 'list_widget', {'selection_mode': QtWidgets.QAbstractItemView.ExtendedSelection,
 5                                                'toolTip': 'Column containing the training data'}),
 6              ('train_labels', 'combo', {'toolTip': 'Column containing training labels'}),
 7              ('solver', 'combo', {'items': ['svd', 'lsqr', 'eigen']}),
 8              ('shrinkage', 'combo', {'items': ['None', 'auto', 'value']}),
 9              ('shrinkage_val', 'doubleSpin', {'min': 0.0, 'max': 1.0, 'step': 0.1, 'value': 0.5}),
10              ('n_components', 'intSpin', {'min': 2, 'max': 1000, 'step': 1, 'value': 2}),
11              ('tol', 'intSpin', {'min': -50, 'max': 0, 'step': 1, 'value': -4}),
12              ('score', 'lineEdit', {}),
13              ('predict_on', 'list_widget', {'selection_mode': QtWidgets.QAbstractItemView.ExtendedSelection,
14                                             'toolTip': 'Data column of the input "predict" Transmission\n'
15                                                        'that is used for predicting from the model'}),
16              ('Apply', 'check', {'applyBox': True, 'checked': False})
17              ]
18
19def __init__(self, name, **kwargs):
20    # Specify the terminals with a dict
21    CtrlNode.__init__(self, name, terminals={'train': {'io': 'in'},
22                                             'predict': {'io': 'in'},
23
24                                             'T': {'io': 'out'},
25                                             'coef': {'io': 'out'},
26                                             'means': {'io': 'out'},
27                                             'predicted': {'io': 'out'}
28                                             },
29                      **kwargs)
30    self.ctrls['score'].setReadOnly(True)
31
32# Very simple override
33def process(self, **kwargs):
34    return self.processData(**kwargs)
35
36def processData(self, train: Transmission, predict: Transmission):
37    self.t = train.copy()  #: Transmisison instance containing the training data with the labels
38    self.to_predict = predict.copy()  #: Transmission instance containing the data to predict after fitting on the the training data
39
40    # function from mesmerize.analysis.utils
41    dcols, ccols, ucols = organize_dataframe_columns(self.t.df.columns)
42
43    # Set available options for training data & labels
44    self.ctrls['train_data'].setItems(dcols)
45    self.ctrls['train_labels'].setItems(ccols)
46
47    dcols = organize_dataframe_columns(self.to_predct.df.columns)
48    # Set available data column options for predicting on
49    self.ctrls['predict_on'].setItems(dcols)
50
51    # Process further only if Apply is checked
52    if not self.ctrls['Apply'].isChecked():
53        return
54
55    # Get the user-set parameters
56    train_column = self.ctrls['train_data'].currentText()
57
58    # ... get other params
59    n_components = self.ctrls['n_components'].value()
60
61    # ... do stuff
62
63    # This node ouputs separate transmissions that are all logged
64    self.t.history_trace.add_operation('all', 'lda', params)
65    self.t_coef.history_trace.add_operation('all', 'lda', params)
66    self.t_means.history_trace.add_operation('all', 'lda', params)
67
68    # the `to_predict` transmission is logged differently
69    self.to_predict.history_trace.add_operations('all', 'lda-predict', params_predict)
70
71    # dict for organizing this node's outputs
72    # The keys MUST be the same those specified for this node's output terminals
73    out = {'T': self.t,
74           'coef': self.t_coef,
75           'means': self.t_means,
76           'predicated': self.to_predct
77           }
78
79    return out

uiTemplate

Specify the uiTemplate attribute as a list of tuples.

One tuple per UI element with the following structure:

(<name (str)>, <type (str)>, <dict of attributes to set>)

Examples:

('dist_metric', 'combo', {'items': ['euclidean', 'wasserstein', 'bah'], 'toolTip': 'distance metric to use'})
('n_components', 'intSpin', {'min': 2, 'max': 10, 'value': 2, 'step': 1, 'toolTip': 'number of components'})
('data_columns', 'list_widget', {'selection_mode': QtWidgets.QAbstractItemView.ExtendedSelection})

The name can be anything. Accepted types and accepted attributes are outlined below

widget type

attributes that can be set

intSpin

min (int): minimum value allowed in the spinbox
max (int): maximum value allowed
step (int): step size
value (int): default value

doubleSpin

min (float): minimum value allowed in the spinbox
max (float): maximum value allowed
step (float): step size
value (float): default value

check

checked (bool): default state of the checkBox
applyBox (bool): Whether this is an “Apply checkbox”

radioBtn

checked (bool): default state of this radioButton

combo

items (list): default list of items that will be set in the comboBox

list_widget

items (list): default list of items that will be set in the list_widget
selection_mode: One of the accepted QAbstractItemView selection modes

lineEdit

text (str): default text in the line edit
placeHolder (str): placeholder text
readOnly (bool): set as read only

plainTextEdit

text (str): default text in the text edit
placeHolder (str): placeholder text

label

text (str): default text

button

text (str): default text on the button
checkable (bool): whether this button is checkable

color

Does not take any attributes

All UI widget types outlined above take ‘toolTip’ as an attribute which can be used to display tooltips