Python Transformation
Contents
The Python Transformation pipeline step (Figure 9.170) allows FCS Express and Python to communicate with each other through a text script that can be created directly within FCS Express.
Please note that both Python (version 3.8 or higher) and all the Python libraries required to run the calculation/algorithm of interest have to be installed on the user computer to allow the Python Transformation pipeline step to be run properly. Python can be downloaded and installed via the official Python website.
Figure 9.169 The Python Transformation pipeline step dialog.
The Python Script dialog allows to:
1.Define the options (and their default values) that will be used to tune the calculation. For example, if a clustering algorithm requires the user to define the number of clusters that need to be created, the number of clusters will be defined as a User Defined Option. A name and a default value can be specified for each option. This action is accomplished via the RegisterOptions() function (see below).
2.Define the output parameter(s) that needs to be re-imported from Python into FCS Express at the end of the calculation. For example, if the calculation runs a clustering algorithm, the output parameter that needs to be imported into FCS Express will be the clustering assignment. A name can be assigned to each output parameter. The name will be used by FCS Express to name that parameter everywhere within the layout. This action is accomplished via the RegisterParameters() function (see below).
3.Run the calculation (e.g. the clustering) on the Input Data (i.e. the events and the parameters defined in the pipeline), using the User Defined Options, and import the Output Parameter(s). This action is accomplished via the Execute() function (see below).
The three actions above are executed by the three functions presented below.
RegisterOptions(params)
The RegisterOptions(params) function is used to define the options that will be used by the user to tune the calculation. This function allows to specify a name and a default value for each option. Registered options will appear under the User Defined Options panel of the Python Transformation dialog (see screenshot above) when the script is run the first time.
Once options appear in the User Defined Options panel, they can be adjusted directly from the panel, without further action on the script. It is suggested to not further change the default values defined in the RegisterOptions(params) function once the script is run the first time.
The argument of this function is the {list} of the names of the parameters currently selected in the Parameter Option panel of Python Transformation dialog (see screenshot above). This is an object of type list.
The RegisterOptions(params) function does not generate any output, it sends the name and the default value of each registered option to FCS Express so that FCS Express can populate the interactive User Defined Options panel of the Python Transformation dialog.
Several types of options can be defined through the RegisterOptions(params) function:
Type |
Function to use within the RegisterOptions(params) function |
Description |
Boolean |
RegisterBooleanOption(OptionName: str, DefaultValue: bool) |
Allows users to create a User Defined Options that allows True/False values as input. |
Floating |
RegisterFloatOption(OptionName: str, DefaultValue: float, OptionMaximum: float, OptionMinimum: float) |
Allows users to create a User Defined Options that allows a floating number as input. Optionally, a maximum and a minimum values can be provided. Default for OptionMaximum and OptionMinimum is None. |
Integer |
RegisterIntegerOption(OptionName: str, DefaultValue: int, OptionMaximum: int, OptionMinimum: int) |
Allows users to create a User Defined Options that allows an integer number as input. Optionally, maximum and minimum values may be provided. Default for OptionMaximum and OptionMinimum is None. |
String |
RegisterStringOption(OptionName: str, DefaultValue: str) |
Allows users to create a User Defined Options that allows a string as input. |
/ |
ClearOptions() |
This function will clear all registered options. |
/ |
DeleteOption(OptionName: str) |
This function will delete a registered option based on OptionName. |
An example of RegisterOptions(params) function is depicted below. In this example, six different options are defined and initialized:
•A user option of type string named Distance Metric and initializes it to euclidean.
•A user option of type integer named Number Initialization and initializes it with 10.
•A user option of type integer named Number Iterations and initializes it to 100, and sets a min/max values to 0 and 1000 respectively.
•A user option of type integer named Float without Range and initializes it with 10.67.
•A user option of type floating named Tolerance and initializes it to 0.04, and sets a min/max values to 0 and 1000 respectively.
•A user option of type boolean named Precompute Distances and initializes it with True.
def RegisterOptions(params):
RegisterStringOption("Distance Metric", "euclidean")
RegisterIntegerOption("Number Initialization", 10)
RegisterIntegerOption("Number Iterations", 100, 0, 1000)
RegisterFloatOption("Float without Range", 10.67)
RegisterFloatOption("Tolerance", 0.004, 0.0, 1.0)
RegisterBooleanOption("Precompute Distances", True)
Running the lines above will populate the User Defined Options panel as follow:
Once options appear in the User Defined Options panel, the user will then be able to tune each option directly through the User Defined Options panel. It is suggested to not further change the default values defined in the RegisterOptions(params) function once the script is run the first time.
User-defined options are imported into Python in an object of type dictionary (see the opts argument in the RegisterParameters() and in the Execute() functions below), with keys matching the labels defined in the RegisterOptions() function, and values matching the value defined in the User Defined Options panel.
This dictionary containing user-defined options is an argument of both the RegisterParameters(opts, params) and the Execute(opts, data, res) functions (see below).
RegisterParameters(opts, params)
Results generated by the Python script are saved into an object of type dictionary and then imported into FCS Express as new parameters. While the result are imported into FCS Express, dictionary keys are used as parameter names, and dictionary values are used as parameter values by FCS Express.
The RegisterParameters() function allows to define the keys of the dictionary, thus the name of the new parameters.
Results generated by the calculation (e.g. a cluster assignment resulting from a clustering algorithm, or a the parameters resulting from a dimensionality reduction algorithm) will be then saved as values of each key (this is done via the Execute() function).
Arguments:
•opts {dict} This is the dictionary containing names and values for each user-defined option. Names are the keys defined in the RegisterOptions(params) function, while values are pulled down from the User Defined Options panel;
•params {list} is the list of the input parameters selected in the Parameter Option panel of Python Transformation step (see screenshot above).
Two types of parameters can be defined through the RegisterParameters(opts, params) function:
Type |
Function to use within the RegisterParameters(opts, params) function |
Description |
Classification assignment |
RegisterClassification(ParamName: str, Optional Argument 1 , Optional Argument 2) |
Register the name for a classification parameter. The parameter should be returned with integer values only. Optionally, the following two optional arguments can also be passed: •Optional Argument 1. This should be a string in the form of desc="". It allows to define a description for the classification •Optional Argument 2. This should be either a list of labels in the form of labels=[] or a single integer number defining the number of classifications (classification values will start from 0) in the form of num_class=integer_value. If the former, the number of labels provided should be equal to the number of classifications. |
Floating values |
RegisterParameter(ParamName: str) |
Register the name for a floating number parameter. |
An example of RegisterParameters(opts, params) function is depicted below. In this example, three types of results are initialized:
•A classification assignment named Cluster assignments with a description (i.e. Kmeans Cluster assignments)
•A classification assignment named Cell Status with no description (see the second argument which is "") and 2 classification levels (i.e. 0 and 1).
•A classification assignment named Cell type with no description (see the second argument which is "") and 2 classification levels with specific names (i.e. "B Cells" and "T Cells").
•A floating point parameter named Custom Parameter.
def RegisterParameters(opts,params):
RegisterClassification("Cluster assignments", "Kmeans Cluster assignments")
RegisterClassification("Cell Status", "", 2, ["live", "dead"])
RegisterClassification("Cell type", "", 2, ["B Cells", "T Cells"])
RegisterParameter("Custom Parameter")
Execute(opts, data, res)
The Execute() function is used to run the calculation of interest and save the results as dictionary values into the dictionary object defined by the RegisterParameters() function.
Arguments:
•opts is an object of type {dict} containing name and value for each user-defined option. Names are the keys defined in the RegisterOptions() function, while values are pulled down from the User Defined Options panel;
•data is an object of type {dict} containing the input data. Keys are derived from the parameters selected in the Parameter Option panel of Python Transformation step (see screenshot at the beginning of this chapter). The length of the dictionary matches the number of selected input parameters. The value for each key is a a list containing the values for that parameter. The length of each list of the dictionary matches the number of input data points for the Python Transformation step.
Spectral parameters and KNN parameters are also allowed (their value in the dictionary object, is a list of list).
•res is an object of type {dict} with keys matching the registered parameter names (see the RegisterParameters() function) and vales preallocated to nan. This dictionary is setup to be used as the return value of this method.
The function:
1.Runs the calculation of interest
2.Populates the values of the res object with the output values generated by the calculation. Length of each value list should be equal to number of input data points. A key should exist for each registered parameter.
3.Return the populated res object to FCS Express.
Variables
While the running the calculation of interest within the Execute(), the user may access the following variables:
Variable |
Description |
NumberDataPoints {int} [read-only] |
Returns the number of cells being transformed (i.e. the number of input events). |
NumberClassifications {dict} [read-write] |
Returns a dictionary where each key is a classification parameter registered with the RegisterParameters function, and each value is the number of classifications for that parameter allowing modification to the number of classifications after performing a given transformation. |
InputClassificationLabels {dict} [read-only] |
Returns a dictionary where each key is a selected classification input parameter (i.e. selected in the Parameter Options panel), and each value is a list of labels for that classification parameter. The list may be accessed using the classification value provided in the data dictionary. If no selected input parameter are classification parameters, this dictionary will be empty.
For example, let's consider the following scenario:
•CD8 APC-Cy7-A. This parameter containing fluorescence intensity values for each data point. •File Identifier. This parameter contain the numerical identifier created when files are merged together. •SAMPLE ID. This parameter contain and additional classification identifier defined in the Instrument Specific Settings. •Classification Identifier. This parameter contain the classification identifier created when files are merged together. •Kmeans 10 Cluster Assignments: This parameter contain the classification generated by a Kmeans pipeline step executed upstream the Python Transformation pipeline step.
InputClassificationLabels would return an object of type dictionary with 3 keys: Classification Identifier, Kmeans 10 Cluster Assignments and SAMPLE ID. The values for each key is the list of classification labels for that classification parameter.
|
OutputClassificationLabels {dict} [read-write] |
Returns a dictionary where each key is a classification parameter registered with the RegisterParameters function, and each value is a list of labels (if provided when registering, empty otherwise) to use for the classification, allowing modification to the the labels after performing a given transformation.
In the example below a registered parameter named "Kmeans" with 5 integer cluster_labels (from 0 to 4) is modified with the OutputClassificationLabels function so that the new cluster labels will be ["Cluster 1", "Cluster 2", "Cluster 3", "Cluster 4", "Cluster 5"].
OutputClassificationLabels["KMeans"] = ["Cluster {}".format(i+1) for i in range(max(cluster_labels)+1)]
|
ClassificationCentroids {dict} [read-write] |
Returns a dictionary where each key is a classification parameter registered with the RegisterParameters function, and each value is a list of centroids (if provided when registering, empty otherwise) to use for each classification, if not automatically calculating centroids. This allows registered classification to be used by downstream pipeline steps that require high dimensional information. |
Importing FCS Express functions
All the functions presented above can be loaded by running the following command at the beginning of the script:
from FcsExpress import *
Specific functions may also be loaded. As an example, the line below loads the RegisterStringParameter function:
from FcsExpress import RegisterStringParameter
The function name can also be customized at the time of loading. As an example, the line below loads the RegisterStringParameter function and will assign a new name to it (i.e. regstr):
from FcsExpress import RegisterStringParameter as regstr
Example scripts
In the following chapters some training and example scripts are provide:
•Example scripts with popular algorithms
oPARC
oIvis