Install the job scripting DSL

Storm is a Python DSL to generate a graph of task dependencies. It consists of an API and a command-line interface. The API can be used by other tools to build a graphs of tasks. The command-line interface is convenient for validation, serialization, and more.

pip install cwstorm

Now run storm --version to test that it works.

Install for development

If you would like to contribute to the project, it's best to install in editable mode in a virtual environment.

> git clone git@github.com/ConductorTechnologies/cwstorm.git
> cd cwstorm

# Create a virtual environment
> python3 -m venv cwstorm.venv
> . cwstorm.venv/bin/activate

# Install in editable mode
> pip install -e .

> storm --version

Quick Start API

Create nodes link them together to create a graph of tasks.

You can add tasks to the job or to other tasks. The job is the root node of the graph.

from cwstorm.dsl.job import Job
from cwstorm.dsl.cmd import Cmd
from cwstorm.dsl.task import Task
from cwstorm.serializers import default

# Create a job node to hold data about the job
j = Job("Pitch Black")
j.metadata({"shot": "0130-28", "code": "PB", "producer": "Matthew Plumber"})
j.project("Pitch Black")
j.location("4A:1C:3F:7B:2E:9D")
j.comment('This is a comment about the job.')

# Add a task as a dependency of the job.
t = Task("Make Quicktime")
t.commands(Cmd( "ffmpeg", "-i", "foo.*.exr", "-c:v", "libx264", "-pix_fmt", "yuv420p", "./media/robots.mp4"))
t.hardware("cw-instance-1")
t.env({ "PATH": "/usr/local/sbin/ffmpeg" })
t.output_path("/media/")

serialized = default.serialize(self.job)
print(serialized)

Quick Start CLI

Serialize

Use the storm CLI command to serialize one of the example jobs. The pretty option generates human readable JSON output. The filename will be the same as the example but with a .json extension, and will be placed in the folder you specify.

> storm serialize -x simple_qt -f pretty ~/Desktop/graphs/
> cat ~/Desktop//graphs/simple_qt.json

Validate

Several properties of nodes have validators. The validate subcommand Validates a JSON file.

> storm validate /path/to/my_graph.json

You'll see a report in a browser window.

You can output the report to markdown if you prefer.

In order to validate a graph, the storm command uses the DSL to reconstruct the graph from JSON. See deserializer.py.

from cwstorm.deserializer import deserialize

with open("my_job.json", "r", encoding="utf-8") as fh:
    data = json.load(fh)
  
job = deserialize(data)

print("name", job.name())
print("comment", job.comment())
print("num_tasks", job.count_descendents())

Command-line interface

To see the full list of options, run:

storm serialize --help

Examples

Look through the examples folder to familiarize yourself with the API. All classes derive from Node.