flowctl

flowctl is a lightweight process orchestration tool for NetBSD.
It allows starting, connecting, and monitoring processes ("workers") as well as linking their input and output streams via so-called "pumps".

License and Author

• Author: Matthias Petermann
• License: MIT

Directory Layout

flowctl uses /var/flowctl as its base directory.
Worker and pump directories with control files and FIFO pipes are created there.

/var/flowctl/
    workers/<alias>/
        stdin
        stdout
        stderr.log
        pid
    pumps/<from->to1,to2,...>/
        pid
        targets

Installation

Build and install using the provided Makefile:

make
make install

Usage

General syntax:

flowctl <command> [args]

Available Commands

Command	Description
worker <alias> <cmd...>	Start a new worker process with alias and command.
pump <from> <to...>	Connect stdout of one worker to stdin of one or more others.
status [alias]	Show status of all workers and pumps or a specific alias.
inject <alias> <text>	Send a line of text into the worker's stdin.
list	List all workers and pumps.
destroy <alias>	Stop and remove a worker or pump.
destroy-all	Stop and remove all workers and pumps.
graph	Print the current network in DOT format for Graphviz.
help	Show help message.

Example Setup

In this example:

• extractor tails the system log file /var/log/messages
• transformer transforms lowercase letters to uppercase using tr
• loader1 writes the transformed output to /tmp/out1.txt
• loader2 writes the transformed output to /tmp/out2.txt

We then set up pumps to connect the workers.

Step 1: Start the workers

flowctl worker extractor /usr/bin/tail -f /var/log/messages
flowctl worker transformer /usr/bin/tr a-z A-Z
flowctl worker loader1 /bin/sh -c "cat > /tmp/out1.txt"
flowctl worker loader2 /bin/sh -c "cat > /tmp/out2.txt"

Step 2: Connect the pumps

• Pump from extractor to transformer
• Pump from transformer to loader1 and loader2

flowctl pump extractor transformer
flowctl pump transformer loader1 loader2

Step 3: Verify the setup

flowctl status
[WORKERS]
extractor: RUNNING PID=2486
transformer: RUNNING PID=26087
loader1: RUNNING PID=26088
loader2: RUNNING PID=1016
[PUMPS]
extractor->transformer: RUNNING PID=27218
transformer->loader1,loader2: RUNNING PID=5059

Step 4: Visualize the graph

flowctl graph | dot -Tpng > graph.png

This will show a data flow:

extractor -> transformer -> loader1
                            loader2

Architecture

• Worker: A process with defined stdin, stdout, and stderr.
• Pump: A daemon that forwards the stdout of one worker into the stdin of other workers.
• Supervisor: Ensures that workers are started with defined pipes and monitored.

Logging

• Worker error output is written to stderr.log inside the worker’s directory.
• Events are logged via syslog.

Notes

• Designed for Unix-like systems with FIFO pipes (e.g. NetBSD).
• Processes are terminated with SIGTERM and, if needed, SIGKILL on destroy.
• Use destroy-all during shutdown to ensure all processes are stopped.