Looking recently at hsfiles template made me fiddle around with a relatively old
script for project template management using
org-babel. If you have used any
scaffolding tool like cookiecutter, you already know what I mean by template
management for projects.
I tried using cookiecutter long ago but then stopped. Not sure of the exact reason but here are few points that cover parts of the dissatisfaction (in context of the new scheme from this post):
- UPDATION: Keeping template-ish content spread around in a directory makes them cumbersome to update.
- UNIFICATION: Different languages have different scaffolding tools, like quickproject for Lisp, poetry's for python etc. There should be a simple way to wrap them around.
- REUSABILITY: Once we have the above wrapping around, there should be a way
for extending templates with reusable components. For example, I might want
to add my common testing structure to poetry's
initprocess and a conda based project.
- FLEXIBILITY: Pre-execution editing of the templates makes them very practical since I might want to change something which was not planned as a template variable. Almost all other scaffolding tools are horrifically bad in this situation because of their terminal centered approach.
obtt (Org Babel Tangle Templates) is a little tool which extends the idea of
hsfiles templates and solves a lot of the issues I have with tools like
cookiecutter. Here is how its supposed to be used:
- Describe project templates in org files using yasnippet like templates and org babel tangle blocks.
obtt-newto select the root project directory and select the template to use.
- The next buffer (seed buffer) will drop you in yasnippet expansion state.
Edit all you want and then fire
Using org mode lets you do other useful things like evaluating a code block.
This can be useful in cases where you want to generate content of a file from an
external tool. For example if you use generate-license for licenses, you can add
the following block with
:obtt eval in the header.
#+BEGIN_SRC shell :obtt eval gen license:gpl-3.0 #+END_SRC
There are other obvious benefits like including another template using the
#+INCLUDE: sub-template.obtt. You might also template the parameter out to
switch easily between, say, CI config files like below:
Having an org file as your template also lets you add useful documentation which makes the whole workflow more shareable and easier to tweak without bringing in inconsistencies.
2. A sample template
Here is a sample template from my config. As described in the preamble, its not the one I use right now and things might change. Good for conveying a general idea though.
# -*- mode: org -*- #+TITLE: obtt-experiment This is a snakemake + poetry based template for experimental projects. Heavily inspired by other data science templates. Most of the experiments use a conda based setup but this template just uses poetry for now. I will tune this when needed. * Variables - Name :: $1 - Description :: $2 * README #+BEGIN_SRC org :tangle ./README.org #+TITLE: $1 $2 #+END_SRC * toml file #+BEGIN_SRC toml :tangle ./pyproject.toml [tool.poetry] name = "$1" version = "0.1.0" description = "$2" authors =  [tool.poetry.dependencies] python = "^3.6" snakemake = "^5.2" [tool.poetry.dev-dependencies] pytest = "^3.0" mypy = "^0.620.0" #+END_SRC * General structure We have a data directory with external and processed sub dirs. A src directory keeps the modules that we build and use. scripts keep snakemake scripts. #+BEGIN_SRC shell :obtt eval mkdir -p data/external data/processed mkdir scripts mkdir src touch ./src/__init__.py #+END_SRC The snakefile rules now bind everything together. #+BEGIN_SRC python :tangle ./Snakefile rule all: shell: "echo 'hi'" #+END_SRC * Testing Tests are important even in experimental code, blah blah... #+BEGIN_SRC python :tangle ./tests/__init__.py #+END_SRC #+BEGIN_SRC python :tangle ./mypy.ini [mypy] ignore_missing_imports=True #+END_SRC #+BEGIN_SRC text :tangle ./tox.ini [tox] skipsdist = True envlist = py36 [testenv] whitelist_externals = poetry skip_install = true commands = poetry install -v poetry run pytest tests/ #+END_SRC * License #+BEGIN_SRC shell :obtt eval gen license:gpl-3.0 #+END_SRC
3. The package
The obtt package is in my config as of now. I will be moving it out after a
while. Other obtt templates are here right now. One issue with this scheme is
that conflicts can popup among yasnippet variables when you include other
templates. But it should be easy to resolve by allowing users to name variables
and then recreating the numbered variables,
$2 etc that yasnippet expects.