Skip to content

Tasks, Trackers, and Actions: Your guide to Aeromancy's Terminological Soup#

The terminology can get confusing here, as there are several similarly named but functionally different pieces. The following diagram summarizes the key relationships:

graph LR
  ActionBuilder -->|Convertible to| ActionRunner
  ActionBuilder -->|Builds| Action
  Action -->|Instantiates| Tracker
  WandbTracker -->|Subclasses| Tracker
  FakeTracker -->|Subclasses| Tracker
  Action -->|Convertible to| Task[pydoit.Task]
  ActionRunner -->|Executes| Task

Tracker#

Creates an environment for to track code for reproducible ML/data science/data pipelines/etc. For example, WandbTracker (a subclass) integrates with Weights and Biases (W&B) for ML experiment tracking and S3 for external artifact storage. This class doesn't know anything task-specific, dependencies between tasks, or how to actually execute the code.

Action#

A single task-specific node in an execution graph. Each node knows its inputs (other Actions it depends on and the results of their computation) and outputs as well as the actual code to run the action. We wrap the execution of an Action in an Aeromancy Tracker when run (potentially a FakeTracker for development work). Actions know a little about W&B artifact names since they need to declare and look these up. Actions are agnostic about how/where they actually gets executed -- that's up to a task runner which we can swap out as needed (see ActionRunner).

ActionBuilder#

A factory class that constructs Actions with specified dependencies for a given configuration.

ActionRunner#

Uses pydoit to execute Actions in an ActionBuilder (according to dependencies it laid out). This includes logic to convert Actions to pydoit Tasks.

Task (pydoit)#

A single executable task with dependencies and metadata in pydoit's framework. All logic for working with these lives in the aeromancy.action_runner module.