"; */ ?>

clojurescript

boot / clojure / clojurescript / lein / mount — Comments Off on The Story of Booting Mount
22
Dec 15

The Story of Booting Mount

Feeling The Code

I don’t agree with the opinion that “cool kids now use boot“. People who say that are just missing out on the power of “feeling the code” rather than being abstracted from the code by a “better XML”. Same deal with people 10 years ago who said “cool kids are using functional languages”.

Don’t get me wrong I like lein a lot. It is simple to start with, it is well documented, it is very googlable, it is sharing platform (i.e. templates), mature, etc.. But boot is very different, it does not aim to do what lein does, it aims to do “what you want”. There is a difference.

Mounting a Bootable Partition

Since the late 90s when I got in to Linux, I found bootable partitions most exciting, they actually bootstrap everything, they were these wizards waving their magic wands and systems appeared. Granted the wave could take minutes, but we are humans, we always wait for the magic, even if it takes the whole life.

First thing that needs to be done for the magic to happen, this bootable partition needs to be mounted.

I wanted to do it for some time now, when, I could not figure out why ClojureScript brought in as a dependency with :classifier “aot” caused compilation problems with lein/cljsbuild, David Nolen suggested that this is rather due to the lein environment issues. So 2 and 2 together: it was the right time to “boot” myself up.

And since the partition was already mounted it was ready to boot.

Grokking the New Simple

Rather than tell you how great boot is, I’ll share non obvious (to me) things that I stumbled upon converting mount from lein to boot. Let’s rock & roll:

REPL is just REPL

Since I needed a support for both Clojure and ClojureScript, I looked at many examples and noticed a pattern: usually in a dev mode one task groups several, where most of the examples have a (watch) task in that group.

I just wanted to start out, so I decided that at a minimum I need a REPL and (I guess) this watcher to be able to mimic the lein repl behavior, so I did:

(deftask dev [] 
  (comp
    (watch)
    (repl)))

And it worked! I ran boot dev and I got a REPL which would see all the updates from vim (via the updated vim-fireplace).

But then I decided to stop the REPL, and it just froze.. I ran jstack on the PID and saw lots of watcher threads locking and derefing futures. Ok, so that’s not a good combination.

The answer is simpler than I expected: it’s just boot repl. Nothing else is needed to get to the lein repl functionality.

“Bring on Your Own Data Readers” Party

The Clojure mount example app uses in memory Datomic, so when I tried to start the app, boot told me:

no reader to handle the #db/id tag

This was easily googlable, and revealed that boot has a (load-data-readers!) function that “refreshes *data-readers* with readers from newly acquired dependencies”.

An interesting bit here is that (load-data-readers!) can’t be a part of a “top level” task that is executed with boot since:

java.lang.IllegalStateException: Can't set!: *data-readers* from non-binding thread

So calling boot dev, in case “load-data-readers!” is there, is not an option. But getting into a REPL “boot repl“, and then calling (dev) works beautifully.

REPL Logging

At this point I could get into the boot REPL and start the mount example app. A slight problem was that I did not see any logging from the app within the REPL.

That’s when I found boot-logservice that brought the logging back to the REPL:

(def log4b
  [:configuration
   [:appender {:name "STDOUT" :class "ch.qos.logback.core.ConsoleAppender"}
    [:encoder [:pattern "%-5level %logger{36} - %msg%n"]]]
   [:root {:level "TRACE"}
    [:appender-ref {:ref "STDOUT"}]]])
;; ...
 
(deftask dev []
 
  ;; ...
 
  (alter-var-root #'log/*logger-factory* 
                  (constantly (log-service/make-factory log4b)))
  ;; ... 
)
Shaking up tools.namespace

While it is not a requirement, and most of the time unnecessary, the example app uses tools.namespace to make it easier for people who rely on it heavily to get into mount.

By default “tools.namespace” won’t find anything to refresh, since boot uses its own “secret” temp directories for sources, and “tools.namespace” simply does not know about them.

This was an easy one, since it is well documented by boot. Hence having (apply set-refresh-dirs (get-env :directories)) in the “dev” task pointed “tools.namespace” to the right directories to refresh.

The Joy of Deploy: Build and Publish

At this point having the Clojure part figured out, before moving to the ClojureScript support, I decided to deploy mount to Clojars, to understand how it’s done with boot.

I found bootlaces, and just plugged it in, it was very straightforward:

(def +version+ "0.1.7-SNAPSHOT")
 
(bootlaces! +version+)
 
;; other things.. and
 
(task-options!
  pom {:project     'mount
       :version     +version+
       :description "managing Clojure and ClojureScript app state since (reset)"
       :url         "https://github.com/tolitius/mount"
       :scm         {:url "https://github.com/tolitius/mount"}
       :license     {"Eclipse Public License"
                     "http://www.eclipse.org/legal/epl-v10.html"}})

Then I did:

boot build-jar push-snapshot

and everything was going smoothly, it asked for my Clojars username, then password.. but then:

clojure.lang.ExceptionInfo: java.lang.AssertionError: 
Assert failed: current git branch is 0.1.7 but must be master
               (or (not ensure-branch) (= b ensure-branch))

Boot told me that it prefers publishing snapshots from the “master”. I don’t disagree, but for some projects I like snapshots from version branches. I don’t really like “git flow”, I like “git freedom”.

Looking at the bootlaces code it seems that “master” is hardcoded. By this time I already started to feel the concept of a “boot task” and noticed that it is hardcoded under the “push” internal task, which means that this task’s options can potentially be overridden:

;; ...
 
(task-options!
 
  push {:ensure-branch nil}       ;; <<<<<<<<<<
 
  pom {:project     'mount
       :version     +version+
       ;; ... 
       })

And what d’you know, it worked! This was most likely the first “aha moment” which wired some of my neurons in boot ways.

Shall Not Pass!

Mount’s “test” root has both cljc tests and clj/cljs test apps that these tests use. The structure looks similar to:

|~test/
| |~clj/...
| | `+tapp/
| |~cljs/...
| | `+tapp/
| |~mount/
| | |+test/...
| | `-test.cljc

In lein, I can give “test” + “test/clj” for Clojure tests, and “test” + “test/cljs” for ClojureScript tests as the sources paths.

In boot I can’t do that, boot says:

java.lang.AssertionError: Assert failed: 
The :source-paths, :resource-paths, and :asset-paths must not overlap.
    (empty? (set/intersection paths parents))

Since boot already read everything under “test”, it does not want to merge things from “test/clj”. Fair enough, so I had to change the structure a bit to make it work:

|~test/
| |~clj/
| | `+tapp/
| |~cljs/
| | `+tapp/
| |~core/
| | `~mount/
| |   |+test/
| |   `-test.cljc

Now I can give “test/core” + “test/clj” and “test/core” + “test/cljs” respectively.

ClojureScript is Clojure, but.. not Always

ClojureScript took some time to get right. Many examples helped a lot especially these three: boot-cljs-example, tenzing and boot-cljs-multiple-builds.

The concept of dividing “cljs” options between “xyz.cljs.edn” and “task options” did not sink in immediately, and required some code digging to figure out where to put what and how to make sure it is being used.

It ends up to be quite simple. Options that are provided via “xyz.cljs.edn” can be referenced from task options via ids option:

(cljs :optimizations :advanced :ids #{"mount"})

would mean that it would look for mount.cljs.edn file within the classpath. That file should point to the entry point of the ClojureScript app. In case of the mount example app it would just be:

{:require  [app.example]}

where init-fns and compiler-options can be also added.

Testing ClojureScript

“mount does doo” for ClojureScript testing, and boot-cljs-test does it as well.

I would expect it to pick up “xyz.cljs.edn” files in the same way as “boot-cljs”, but it does it a bit differently. It is not all that obvious at first, but looking at the code I saw that it has a different name for ids, it calls it out-id. It also does not just take an “id”, it takes an “id” + “.js”, as I saw from the code.

So to get it to work is quite simple:

(tcs/test-cljs :out-file "mount.js"))

which would look for the same mount.cljs.edn file within the classpath.

Power it Up

There were other discoveries, like

* tasks are functions, but not really, they take arguments in the particular format and they better return a fileset

* tasks: “comp us please”. They like to be (comp ..)ed. Otherwise no go.

* there were others, but I liked Pods the most.

At this point I got all up and pumping, deployed to CircleCi using boot to build and run tests, published to Clojars as snapshot and release, etc.

One of the greatest things that I loved while debugging dependencies is boot show -p, it’s amazing!

Get up! Boot yourself up! Enjoy the runtime!

build / clojure / clojurescript / java / lein / maven / protobufs — 3 Comments
2
Jul 14

Pom Pom Clojure

Fun with lein, Money with maven

While doing Clojure projects, it is the second time I faced a problem with a customer’s “build team” that knows what Java is, loves Maven, but does not believe in Mr. Leiningen, hence all of the lein niceties (plugins, once liners, tasks, etc..) need to now be converted to “pom.xml”s.

A good start is “lein pom”. While it only scratches the surface, it does generate a “pom.xml” with most of the dependencies. But in most cases it needs to be “massaged” well in order to fit а real maven build process.

Usual Suspects

Besides the most common “lein repl”, here is what I usually need lein to do:

* Compile Clojure code
* Some files need to be AOT compiled
* Run Clojure tests
* Compile ClojureScript

(not Clojure specific, but I’ll include it anyway)

* Compile protobuf
* Create a JAR for most projects
* Create a self executing “uberjar” for others

When Clojure is “Ahead Of Time”

Compiling, AOTing and running tests can be done with Clojure Maven Plugin:

<plugin>
    <groupId>com.theoryinpractise</groupId>
    <artifactId>clojure-maven-plugin</artifactId>
    <version>1.3.20</version>
    <extensions>true</extensions>
    <executions>
        <execution>
            <id>compile</id>
            <phase>compile</phase>
            <goals>
                <goal>compile</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <namespaces>
            <namespace>whatsapp.core</namespace>
        </namespaces>
        <compileDeclaredNamespaceOnly>true</compileDeclaredNamespaceOnly>
        <sourceDirectories>
            <sourceDirectory>src</sourceDirectory>
        </sourceDirectories>
        <testSourceDirectories>
            <testSourceDirectory>test</testSourceDirectory>
        </testSourceDirectories>
    </configuration>
</plugin>

notice “namespaces” and “compileDeclaredNamespaceOnly”, this is how AOT is done for selected namespaces.

For AOT it’s good to remember that “a side effect of compiling Clojure code is loading the namespaces in order to make macros and functions they use available”, here are AOT compilation gotchas to keep in mind.

Compiling ClojureScript

This one is a bit trickier. If it is possible to convince a build team to install lein as a library that is used for the build process (e.g. similar to “protoc” to compile protobufs), then to compile ClojureScript, a lein cljsbuild can be added to the profile:

vi ~/.lein/profiles.clj
{:user {:plugins [[lein-cljsbuild "1.0.0"]]}}

and an exec maven plugin can be used to relay the execution to “lein”:

<plugin>
    <artifactId>exec-maven-plugin</artifactId>
    <groupId>org.codehaus.mojo</groupId>
    <executions>
        <execution>
            <id>compiling ClojureScript</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>exec</goal>
            </goals>
            <configuration>
                <executable>lein</executable>
                <arguments>
                    <argument>cljsbuild</argument>
                    <argument>once</argument>
                </arguments>
            </configuration>
        </execution>
    </executions>
</plugin>

In fact, if “lein” is installed, it can be used via “exec-maven-plugin” to do everything else as well, but it all depends on build teams’ restrictions. For example, financial customers may have extremely strict “policies”/”rules”/”opinions”.

A couple more options to explore for building ClojureScript would be lein maven plugin and zi-cljs. Here is a related discussion on a ClojureScript google group.

Making Shippables

“lein uberjar” with some config in “project.clj” is all that is needed in “lein” world. In maven universe maven shade plugin will do just that:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.3</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>org.gitpod.WhatsApp</mainClass>
                    </transformer>
                </transformers>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
        </execution>
    </executions>
</plugin>

above will create a self executing JAR with all dependencies included and with an entry point (-main) in “org.gitpod.WhatsApp”.

Google Protocol Buffers

With lein it is as simple as pluging in lein protobuf. In maven, it is not as simple, but also not terribly difficult and solved via maven-protoc-plugin:

<plugin>
    <groupId>com.google.protobuf.tools</groupId>
    <artifactId>maven-protoc-plugin</artifactId>
    <version>0.3.2</version>
    <extensions>true</extensions>
    <executions>
        <execution>
            <goals>
                <goal>compile</goal>
            </goals>
            <phase>generate-sources</phase>
        </execution>
    </executions>
    <configuration>
        <protocExecutable>${PROTOBUF_HOME}/src/protoc</protocExecutable>
        <protoSourceRoot>resources/proto</protoSourceRoot>
        <outputDirectory>target/classes</outputDirectory>
        <!--<additionalProtopathElements>-->
        <!--    <param>${PROTOBUF_HOME}/src/google/protobuf</param>-->
        <!--</additionalProtopathElements>-->
    </configuration>
</plugin>

here is a repository it currently lives at:

<pluginRepositories>
    <pluginRepository>
        <id>protoc-plugin</id>
        <url>http://sergei-ivanov.github.com/maven-protoc-plugin/repo/releases/</url>
    </pluginRepository>
</pluginRepositories>

notice “additionalProtopathElements”. In case clojure-protobuf is used with extensions, a path to “descriptor.proto” can be specified in “additionalProtopathElements”.

clojure / clojurescript / lein — Comments Off on Leiningen Templates with Arguments
8
Mar 14

Leiningen Templates with Arguments

This template is so wrong!

Project templates can be as excellent as they can be awful since they are very opinionated beings:

  • “a web project MUST be Compojure based!”
  • “a network project MUST be Netty based!”
  • “there is no way I am building a web project based on Compojure!”
  • “a network project? of course ZeroMQ!”

But they can be really useful for quick prototypes, for learning new things, even for “real deal”, if of course you agree with their opinion.

Whatsapp WWW

I’ve recently built a template WWW to bootstrap web apps based on Clojure, ClojureScript, Compojure and Ring. Reason? I needed a faster way of getting apps up and running, especially when prototyping ideas.

Building a Leiningen template is quite a simple task. Once the template is build and installed/deployed, I could just do lein new www whatsapp and make my next $19 billion. But I wanted more!

WWW by default would create a project template with a very simple structure that can be immediately brought up (e.g. “lein ring server”) removing any obstacles on the way of making billions, everything is setup: just open vi and start hacking.

However, what if I want a ClojureScript REPL connected to my browser? I would need to go through some docs, and then depending on my experience with lein, Clojure and ClojureScript, I could quickly (or not) set it up myself.

Well, I wanted WWW template to have that setup for me right away. But here is a dilemma: sometimes I do want ClojureScript browser connected REPL, other times I don’t, and the way this REPL setup goes, it requires a code change to enable/disable it.

It’s Good to Have Options

The documented way of creating a lein template does not really talk about “options” that I want with my REPL. What do I do? Well.. It’s a Clojure universe, and, if anything, two things hold true most of the time:

  • It’s Simple
  • It’s a Function

A brand new lein template is done with lein new template www. It creates.. ready? “a template for a template”. That’s right, a template project to create a lein template.

An entry point into this template (of a template) will be located in src/leiningen/new/www.clj. Here www is the name of this template. Let’s peek inside:

$ lein new template www
Generating fresh 'lein new' template project.
$ cd www
$ cat src/leiningen/new/www.clj
(ns leiningen.new.www
  (:use [leiningen.new.templates :only [renderer name-to-path ->files]]))
 
(def render (renderer "www"))
 
(defn www
  "FIXME: write documentation"
  [name]
  (let [data {:name name
              :sanitized (name-to-path name)}]
    (->files data
             ["src/{{sanitized}}/foo.clj" (render "foo.clj" data)])))

See? www here is just a function, that in this case, takes a name parameter. And after this template is installed lein new www whatsapp will pass whatsapp to it.

Just a Function, Just Beatiful

Since www is just a function, why not make it take optional parameters which could potentially change the resulting template?

(defn www
  "Create a new Clojure + ClojureScript + Compojure + Ring project"
  ([name] 
   (www name :noop))
  ([name opts]
  ...))

Great! Now www can either take just name as before, or name and some magic opts. How about one of the options will determine whether or not a template with ClojureScript browser connected REPL is added? I think yes.

And now we can do:

  • either lein new www [app name]
  • or lein new www [app name] :with-brepl

which will create two different flavors of the same template.

clojure / clojurescript — Comments Off on ClojureScript: Use Any JavaScript Library
15
Jul 13

ClojureScript: Use Any JavaScript Library

Since ClojureScript relies on Google Closure compiler to “get down” to JavaScript, in order to take advantage of an “advanced” compilation, external JavaScript libraries have to follow certain Google Closure standards.

A Google Closure compiler introduces the concept of the “extern”: a symbol that is defined in code external to the code processed by the compiler. This is needed to exclude certain JS vars and functions that do not follow the standard that the g-closure advanced compilation relies on.

Many JS libraries do not follow g-closure standards, and there is a closure-compiler repository with some pre-built externs for JQuery, Jasmine, AngularJS and others. However there are about X thousand more JS libraries that could be useful while writing apps with ClojureScript.

While there is a way to manually go through all the ClojureScript code, find “external” JS vars/functions and write externs for them, there is a much nicer alternative written by Chris Houser in a gist: “Externs for ClojureScript” that creates a “DummyExternClass” and attaches all the vars and functions that are not part of (not recognized by) “core.cljs”.

Here is an example of creating externs for an arbitrary ClojureScript file that uses a nice chap timeline JS library:

user=> (print (externs-for-cljs ".../cljs/timeline.cljs"))
var document={};
var links.Timeline={};
var links.events={};
var DummyExternClass={};
DummyExternClass.toString=function(){};
DummyExternClass.substring=function(){};
DummyExternClass.getVisibleChartRange=function(){};
DummyExternClass.getTime=function(){};
DummyExternClass.start=function(){};
DummyExternClass.end=function(){};
DummyExternClass.getSelection=function(){};
DummyExternClass.row=function(){};
DummyExternClass.setSelection=function(){};
DummyExternClass.setCustomTime=function(){};
DummyExternClass.setVisibleChartToDate=function(){};
DummyExternClass.getElementById=function(){};
DummyExternClass.addListener=function(){};
DummyExternClass.draw=function(){};

The original file itself is not important, the output is. “externs-for-cljs” treated a couple of namespaced functions as vars, but it is a easy fix:

 
var links={};
links.Timeline=function(){};
links.events=function(){};

At this point the whole output can be saved as “timeline-externs.js”, and pointed to by “lein-cljsbuild”:

span style="color: #0086b3;"> :cljsbuild
   {:builds
    [{:source-paths [...],
      :compiler
      {:source-map ...,
       :output-to ...,
       :externs ["resources/public/js/externs/timeline-externs.js"]
       :optimizations :advanced}}]}

ClojureScript files based on other JS libraries that are not in a closure compiler repo: e.g. Twitter Bootstrap, Raphael and others can be “extern”ed the same way in order to take advantage of g-closure advanced compilation.

Interesting bit here that is not related to externs, but is to an advanced compilation is a “:source-map” attribute which is a way to map a combined/minified file back to an unbuilt state. It generates a source map which holds information about original JS files. When you query a certain line and column number in the (advanced) generated JavaScript you can do a lookup in the source map which returns the original location. Very handy to debug “:advanced” compiled ClojureScript.

For more info: