flowrunner.runner package

Submodules

flowrunner.runner.flow module

This module contains two main exposed classes to uses BaseFlow: A base class to build flows off of FlowRunner: A class to run any subclass of BaseFlow

class flowrunner.runner.flow.BaseFlow(data_store: dict = <factory>, param_store: dict = <factory>)

Bases: object

BaseFlow is the base class on which all flows are derived from

These are not meant to be used directly, but subclass/parent class for any Flow

Attrs:

data_store: A dict that is meant to be used to store data by method name and output eg. {‘dataframe1’: DataFrame}. param_store: A dict that stores parameter values eg. {‘snapshot_date’: ‘2023-01-01}.

dag(save_file: bool = False, path: str = None, description: bool = True)

Method to generate html flowchart for Flow

We first run a validation check without raising an error and do not show the output. Then we use the FlowRunner class to run it

Parameters:

  • save_file – Optional Bool value to save file or not

  • path – Optional path to provide to save file, if path is provided, save_file is True implicitly

Returns:

HTMl data in the form of string

Return type:

content

data_store: dict
display(description: bool = True)

Method to show html output of the flowchart

Parameters:

description – An optional bool argument which can turn off/on description. Defaults to True

Returns:

displays an html flowchart of the Flow

Return type:

None

param_store: dict
run()

Method to run a flow

We first run a validation check with raise error and do not show the output. Then we use the FlowRunner class to run it

Parameters:

None –

Returns:

None

Raises:

InvalidFlowException – If an invalid flow is detected

show()

Method to show the levels and order of iteration of the Flow

We first run a validation check without raising an error and do not show the output. Then we use the FlowRunner class to run it

Parameters:

None –

Returns:

None

validate(terminal_output: bool = True)

Method to validate a flow

We use the FlowRunner class to run the validation checks.

Parameters:

  • flow_instance – An instance of the Flow class

  • terminal_ouput – An optional bool value of whether to display the terminal output for validate we choose to show the output, defaults to True

Returns:

None

validate_with_error(terminal_output: bool = True)

Method to validate a flow with an error

We use the FlowRunner class to run the validation checks. :param flow_instance: An instance of the Flow class :param terminal_ouput: An optional bool value of whether to display the terminal output

for validate we choose to show the output, defaults to True

Returns:

None

Raises:

InvalidFlowException – If an invalid flow is detected

class flowrunner.runner.flow.FlowRunner

Bases: object

FlowRunner class is used to run all subclasses of BaseFlow class.

We use class methods to run each of the flows

classmethod run(flow_instance)

Class method to run a Flow

This method actually runs the flow and each method in order of iteration. We also do a validation check before we run it(without terminal output, just notification we are running it). During the run we iterate of Graph using the Graph.levels attribute, which is a list

[

[node1, node2], # first level [node3, node4] # second level

]

We also store the output of a function in that instance of the Flow, inside the attribute BaseFlow.data_store.

eg. {

‘method_1’: 7, ‘method_2’: 8

}

Parameters:

flow_instance – An instance of the Flow class

Returns:

None

Raises:

InvalidFlow – Raised if ANY of the validation checks are failed

classmethod show(flow_instance)

Class method to show a Flow

This method we show the order of iteration by going over the levels in the Graph.levels attribute. This DOES NOT run the actual method but just uses the ‘docstring’ and ‘__name__’ to show the Flow.

Parameters:

flow_instance – An instance of the Flow class

Returns:

None

classmethod validate(flow_instance, terminal_output: bool = False)

Class method to validate a Flow

We use the GraphValidator to do the heavy lifting for this part

Parameters:

  • flow_instance – An instance of the Flow class

  • terminal_output – An optional bool value to show output in terminal, defaults to False

Returns:

None

classmethod validate_with_error(flow_instance, terminal_output: bool = False)

Class method to validate a Flow with an error

We use the GraphValidator to do the heavy lifting for this part. We raise an error if validation check fails

Parameters:

  • flow_instance – An instance of the Flow class

  • terminal_output – An optional bool value to show output in terminal, defaults to False

Returns:

None

Raises:

InvalidFlow – Raised if ANY of the validation checks are failed

Module contents