Tangle Templates

Looking recently at hsfiles template made me fiddle around with a relatively old script for project template management using org-babel. If you have used any scaffolding tool like cookiecutter, you already know what I mean by template management for projects.

I tried using cookiecutter long ago but then stopped. Not sure of the exact reason but here are few points that cover parts of the dissatisfaction (in context of the new scheme from this post):

UPDATION: Keeping template-ish content spread around in a directory makes them cumbersome to update.
UNIFICATION: Different languages have different scaffolding tools, like quickproject for Lisp, poetry's for python etc. There should be a simple way to wrap them around.
REUSABILITY: Once we have the above wrapping around, there should be a way for extending templates with reusable components. For example, I might want to add my common testing structure to poetry's init process and a conda based project.
FLEXIBILITY: Pre-execution editing of the templates makes them very practical since I might want to change something which was not planned as a template variable. Almost all other scaffolding tools are horrifically bad in this situation because of their terminal centered approach.

1. obtt

obtt (Org Babel Tangle Templates) is a little tool which extends the idea of hsfiles templates and solves a lot of the issues I have with tools like cookiecutter. Here is how its supposed to be used:

Describe project templates in org files using yasnippet like templates and org babel tangle blocks.
Fire obtt-new to select the root project directory and select the template to use.
The next buffer (seed buffer) will drop you in yasnippet expansion state. Edit all you want and then fire obtt-tangle to finish.

Using org mode lets you do other useful things like evaluating a code block. This can be useful in cases where you want to generate content of a file from an external tool. For example if you use generate-license for licenses, you can add the following block with :obtt eval in the header.

#+BEGIN_SRC shell :obtt eval
gen license:gpl-3.0
#+END_SRC

There are other obvious benefits like including another template using the #+INCLUDE: sub-template.obtt. You might also template the parameter out to switch easily between, say, CI config files like below:

#+INCLUDE: ${1:gitlab-ci.obtt}

Having an org file as your template also lets you add useful documentation which makes the whole workflow more shareable and easier to tweak without bringing in inconsistencies.

2. A sample template

Here is a sample template from my config. As described in the preamble, its not the one I use right now and things might change. Good for conveying a general idea though.

# -*- mode: org -*-
#+TITLE: obtt-experiment

This is a snakemake + poetry based template for experimental projects. Heavily
inspired by other data science templates. Most of the experiments use a conda
based setup but this template just uses poetry for now. I will tune this when
needed.

* Variables

- Name :: $1
- Description :: $2

* README

#+BEGIN_SRC org :tangle ./README.org
#+TITLE: $1

$2
#+END_SRC

* toml file

#+BEGIN_SRC toml :tangle ./pyproject.toml
[tool.poetry]
name = "$1"
version = "0.1.0"
description = "$2"
authors = []

[tool.poetry.dependencies]
python = "^3.6"
snakemake = "^5.2"

[tool.poetry.dev-dependencies]
pytest = "^3.0"
mypy = "^0.620.0"
#+END_SRC

* General structure

We have a data directory with external and processed sub dirs. A src directory
keeps the modules that we build and use. scripts keep snakemake scripts.

#+BEGIN_SRC shell :obtt eval
mkdir -p data/external data/processed
mkdir scripts
mkdir src
touch ./src/__init__.py
#+END_SRC

The snakefile rules now bind everything together.

#+BEGIN_SRC python :tangle ./Snakefile
rule all:
    shell: "echo 'hi'"
#+END_SRC

* Testing

Tests are important even in experimental code, blah blah...

#+BEGIN_SRC python :tangle ./tests/__init__.py
#+END_SRC

#+BEGIN_SRC python :tangle ./mypy.ini
[mypy]
ignore_missing_imports=True
#+END_SRC

#+BEGIN_SRC text :tangle ./tox.ini
[tox]
skipsdist = True
envlist = py36

[testenv]
whitelist_externals = poetry
skip_install = true
commands =
    poetry install -v
    poetry run pytest tests/
#+END_SRC

* License

#+BEGIN_SRC shell :obtt eval
gen license:gpl-3.0
#+END_SRC

3. The package

The obtt package is in my config as of now. I will be moving it out after a while. Other obtt templates are here right now. One issue with this scheme is that conflicts can popup among yasnippet variables when you include other templates. But it should be easy to resolve by allowing users to name variables and then recreating the numbered variables, $1, $2 etc that yasnippet expects.