pyf.manager.network | PyF, flow-based python programming pyf.manager.network

pyf.manager.network

A network-style manager. The Network object sets up nodes and relation between them from a dictionnary representation. The nodes object do prelink and link from the bottom up (consumers to producers) The Network can then set up a status handler and furnish a final iterator over statuses.

class pyf.manager.network.DataJunction(joiner=None, strip_bypass=True)

A DataJunction is an object specialised in taking multiple sources, setting up a joiner runner and permitting to get a generator output.

It handles BYPASS_VALS constants to pass them to the output (only one if there are many).

add_input_source(source, name)

Appends a named input source

continuation_stripper(values)

Removes the continuation values (values defined in BYPASS_VALS) and puts them in the queue. Returns an iterator over non bypassed values

This is done so the joiner runner doesn’t have to handle the bypassing values.

get_generator()

Sets up all the sources in the runner and returns an iterator over the output port of the junction runner.

Warning, this function shouldn’t be called twice !

output_checker(values)

Outputs the values passed in, and if there is something in the queue, pass it, but discard all other objects in the queue until next input data. (If there is a continuation, just pass one, and not the others... It’s usefull if there is a split then a merge with continuation vals, to avoid duplication of sync packets)

class pyf.manager.network.Network(nodes=None)

Initialises a network of nodes from a dictionary structure. Once the nodes have been set up, you can either get the consumer iterators, or get a status handler checking that all the outputs resolve to True.

add_node(node)

Adds a node to the main node dictionnary

consumer_nodes

Returns the consumers Node objects

consumers

Return the outputs from the consumer Node objects

get_status_handler(advanced=True)

Returns a status handler that returns True each time it receives a True from parents.

Experimental: if “advanced” is set to True, uses an intelligent status handler that looks for parent nodes until it finds a splitting node and consumes all the buffer on each branch to avoid high memory usage

initialize()

Creates the Node objects from the dictionnary data structure, set up the links between them and prelink_up then link_up the consumers.

producer_nodes

Return the producers Node objects

class pyf.manager.network.Node(node_id, orunner, in_port=None, out_port=None, joiner=None, splitter=None, joiner_strip_bypass=True, deffer_to_process=False)

Object that is here to be linked to other nodes. Requires a node_id and a runner object (orunner).

First step is to prelink, then to link. Order is from consumers up to producers :

  • Consumers take output from up nodes than themselves take sources up...
  • If there is multiple source, the joiner argument is used (or a default) and encapsulated in a DataJunction
  • If there is multiple output, the splitter argument is used (or a default)
add_input(node)

Adds a node as an input

add_output(node)

Adds a node as an output

get_default_splitter(size)

Returns a default splitter (basically the same than an itertools tee: pass the same data in all outputs)

get_output(node=None)

Returns an output generator. If there is one output, return the main runner out port, else, return a free port from the splitter

init_component()

Connect data inputs in the main runner or in a junction connected to the main runner. If needed, connect output in the splitter.

is_consumer

Returns true if there are no output ports

is_producer

Returns true if there are no input ports

Link to the input nodes. (ask up to do the same) If not inited yet, init the current node

name

Returns the name (id) of the current node

If there is a splitter needed, initialize it. If there is a junction needed, initialize it.

Prelinks the input nodes... (ask up to do the same)