Autodoc - an automated documentation system for Clojure

Introduction

Autodoc is a system for generating HTML documentation for Clojure projects. It is used to create the up-to-the minute published API documentation for Clojure itself, the various contributed libraries, and the statistics package, Incanter.

Autodoc has been created by Tom Faulhaber.

Autodoc leverages two aspects of Clojure in particular:

When run over a project, Autodoc will generate:

  1. An overview page (called index.html) with a description of the project, and an overview of each namespace (from the namespace’s metadata) with a list of documented, public variables and functions in that namespace.

    (A sample of the overview page)

  2. A page for each namespace in the project which has the detailed documentation for each variable, function and macro. This includes name, type, usage and documentation.

    (A sample of the namespace page)

  3. A global index of all the documented variables, their namespace, and a snippet of their documentation.

    (A sample of the index page)

  4. A Clojure index of all the namespaces and vars in those namespaces for use by other tools. See the file index.clj in the output directory:

{:namespaces
       ({:source-url nil,
         :wiki-url "noir.cookies-api.html",
         :name "noir.cookies",
         :doc "Stateful access to cookie values"}
        {:source-url nil,
         :wiki-url "noir.core-api.html",
        ...
       :vars
       ({:arglists ([k] [k default]),
         :name "get",
         :namespace "noir.cookies",
         :source-url nil,
         :raw-source-url nil,
         :wiki-url "/noir.cookies-api.html#noir.cookies/get",
         :doc
         "Get the value of a cookie from the request. k can either be a string or keyword.",
         :var-type "function",
         :line 20,
         :file "src/noir/cookies.clj"}
        {:arglists ([sign-key k] [sign-key k default]),
         :name "get-signed",
        ...

By default, Autodoc produces documentation that looks a lot like Clojure’s own API documentation. However, it is possible to customize both the style and layout of the pages generated by Autodoc to your tastes.

Autodoc can be run directly from the command line or can be used from build tools like ant or leiningen.

Things not yet implemented

Most of the features of the Autodoc service have been implemented in the Autodoc command. A couple of features are not set up yet, however:

Stay tuned. These features are coming.

Autodoc continues to evolve. If there are features you’d like to see, let me know.

Adding Documentation to your Project

Autodoc uses Clojure’s metadata mechanism to find documentation that you’ve built into your Clojure programs. Autodoc uses the same doc strings that the regular Clojure documentation system uses and adds some extra functionality as well.

Autodoc reads metadata attached to namespaces and vars. Since defined functions are a type of var, their doc strings and other metadata are included automatically.

Every Clojure namespace and var has a Clojure map for its metadata. Metadata is generally added when you use the ns macro or the def macro (or one of its relatives such as defn).

Autodoc uses the following keys in these maps:

Metadata key Use Namespace or Def
:arglists A list of the allowed argument lists for a function. The defn and defmacro macros will add this automatically. You may want to add this manually to defmulti to get the same effect. Def
:author A string with the author of this namespace. Used to attribute the namespace when multiple authors have worked on a project. Namespace
:file Used to create the source link when autodoc has a defined source repository to link to. This key is generated automatically by def and its related macros. Def
:line Used to create the source link. See :file. Def
:macro Used to determine the type of the symbol. If the var has a :macro key, it is assumed to be a macro. This key is added automatically by defmacro. Def
:see-also A list of titles and URLs for related documents. These can either be documents built as part of the Autodoc process or links to external references on the web. Namespace
:skip-wiki If non-nil, skip this namespace or var. Both
:tag Determines whether the var is a multimethod. defmulti sets this automatically (though there are plans to change this to a different key). Def

Running Autodoc

Getting Autodoc

Unless you are using Autodoc from leiningen, you will need to get a copy of the standalone jar. Download the latest version from github.

If you’re using leiningen, the autodoc code will be pulled automatically from its home on clojars.org.

From the command line

The simplest way to use autodoc is to launch it from the command line in the root directory of your project:

java -jar autodoc-<version>-standalone.jar

This will scan the source in the src/ directory and produce an HTML tree in autodoc/ directory.

To modify the behavior of autodoc, use parameters. For example:

java -jar autodoc-<version>-standalone.jar --source-directory=master

will use the sub-directory master/ as the location of the source files.

As shown above, to set parameters from the command line, simply use options like --<parameter-name>=<value> or --<parameter-name> <value>. The full list of available parameters is below.

If you need extra classpath entries (that is, if your project uses a library other than Clojure or Clojure-Contrib), you won’t be able to run the executable jar directly. Instead, specify your full classpath and the class autodoc.autodoc as follows:

java -cp autodoc-<version>-standalone.jar:<other jars> autodoc.autodoc <autodoc params>

Autodoc parameters

Parameter Description Default Value
name The name of this project Taken from the directory name where autodoc is running
description A description of this project nil
param-dir A directory from which to load custom project data autodoc-params
root The directory in which to find the project .
source-path The relative path within the project directory where we find the source src
web-src-dir The web address for source files (e.g., http://github.com/clojure/clojure/blob/) nil
web-home Where these autodoc pages will be stored on the web (for gh-pages, http://<user>.github.com/<project>/) nil
output-path Where to create the output html tree. autodoc
external-doc-tmpdir The place to store temporary doc files during conversion (i.e., when converting markdown). /tmp/autodoc/doc
namespaces-to-document The list of namespaces to include in the documentation, separated by commas nil
trim-prefix The prefix to trim off namespaces in page names and references (e.g. "clojure.contrib") nil
load-except-list A list of regexps that describe files that shouldn't be loaded []
build-json-index Set to true if you want to create an index file in JSON (currently slow) false
page-title A title to put on each page nil
copyright Copyright (or other page footer data) to put at the bottom of each page No copyright info

Integrating Autodoc with your build

Building with Leiningen

lein-autodoc provides an Autodoc plug-in for Leiningen. This is the very easiest way to integrate Autodoc into a build process.

There are two ways you can add the plugin to your leiningen environment: add it to the project or add it to a user profile. This is described in the Leiningen plugin documentation. If generating autodoc is a regular artifact of your project, put autodoc in the project.clj. If you want to be able to generate autodoc for arbitrary projects that you may not own, it makes sense to add the plugin to your user profile.

Adding to a project.clj file

The autodoc plugin should always be added to the :dev profile within your project.clj file, as follows:

:profiles {:dev {:plugins [[lein-autodoc "1.0.0"]]}}

Adding to your user profile

To add the plugin to your user profile so that all projects that you work on get autodoc, simply add it to your ~/.lein/profiles.clj file:

{:user
 {:plugins [[lein-autodoc "1.0.0"]
            ...]}
  ...}}

Running autodoc from Leiningen

To build the documentation, simply type:

lein autodoc

Autodoc will set the name, description, and source-path parameters from the Leiningen configuration. Other parameters can be set via a map attached to the :autodoc key:

(defproject bugs "0.1.0-SNAPSHOT"
      :description "Statistical routines for thinking about a bugzilla DB"
      :dependencies [[incanter "1.0-master-SNAPSHOT"]
                     [org.clojure/clojure "1.1.0-alpha-SNAPSHOT"]
                     [org.clojure/clojure-contrib "1.0-SNAPSHOT"]]
      :profiles {:dev {:plugins [[lein-autodoc "1.0.0"]]}}
      :autodoc { :name "Bugs", :page-title "Bugs API Documentation"})

Notes

There are two things to consider with Leiningen integration:

  1. Options that take lists (like :load-except-list) are not yet supported.
  2. In order to provide more independence from shared dependencies with Leiningen, Autodoc and its dependents are loaded into their own directory and executed in a separate process from Leiningen. This won’t work if you run it for the first time when you’re untethered from the Internet.

Building with Ant

Building with ant is straightforward. Just add an autodoc target to your build.xml file that looks like this:

<!-- Adjust the pathname of the jar to wherever you've stored it and set the version appropriately -->
<property name="autodoc-standalone-jar" location="${user.home}/.clojure/autodoc-<version>-standalone.jar"/>

<!-- Adjust the depends as appropriate -->
<target name="autodoc" depends="build"
        description="Build the HTML documentation">
  		<java classname="autodoc.autodoc" fork="true" failonerror="true">
    <classpath>
      <pathelement location="${autodoc-standalone-jar}"/>
      <!-- other pathelements as required by the project -->
    </classpath>
    <arg value="--name=Bugs"/>
    <arg value="--description=Statistical routines for thinking about a bugzilla DB"/>
    <arg value="--page-title=Bugs API Documentation"/>
  </java>
</target>

The comments refer to the parts that will have to change depending on the configuration of your build system.

Maven

Autodoc currently does not support Maven. It is planned, though. Patches welcome.

Making Autodoc work with source code management

Many users will want the files generated by Autodoc to be excluded in their code repositories. This is easy to do in most SCMs.

For example, in git, create a .gitignore file in the root directory of the project with the following lines:

#ignore generated autodoc files
autodoc/**

Other SCMs support similar mechanisms.

Publishing your documentation on GitHub using GitHub Pages

When the project is using GitHub for sharing source, GitHub pages provide a great way to share documentation. This is how the Clojure core API documentation is hosted at http://clojure.github.com/clojure, for instance.

See http://pages.github.com/ for a complete description of GitHub pages and how they work.

Setting up your Autodoc output to link to the GitHub pages for your project is easy:

  1. Add autodoc/** to your .gitignore file as described above.

  2. remove any autodoc directory that you’ve already generated.

  3. While in your project root directory, create a clone of your private github repository into the autodoc directory:

     git clone git@github.com:<user name>/<project name>.git autodoc
    
  4. Create a new root branch in autodoc with the name gh-pages:

     $ cd autodoc
     $ git symbolic-ref HEAD refs/heads/gh-pages
     $ rm .git/index
     $ git clean -fdx
     $ cd ..
    
  5. Now build the autodoc using the command line, Leiningen, ant or your preferred build tool.

  6. Commit the documentation to GitHub with this set of commands:

     $ cd autodoc
     $ git add -A
     $ git commit -m"Documentation update"
     $ git push origin gh-pages
    

Now, after a few minutes, you should be able to go to http://<user name>.github.com/<project name> and see your documentation rendered for the whole world to see. (Once in a while I’ve had to create an extra checkin to kick GitHub when it didn’t notice my first checkin.

Now that you have GitHub pages set up, updating them is straightforward:

  1. Build the autodoc with your normal method.
  2. Commit your changes:

     $ cd autodoc
     $ git add -A
     $ git commit -m"Documentation update"
     $ git push origin gh-pages
    

Customizing Autodoc

Autodoc supports customization of page layout, styles and graphical elements.

You can add customized layout templates and styles in the autodoc-params directory of your project.

More documentation about customizing will be coming soon.