"; */ ?>

Posts Tagged: mount


20
Apr 16

The Way Nature Creates Things

Dogma From The Top


The best thing about the Clojure community is a large number of independent thinkers. The second best thing that allows this community to exist is the Clojure language itself, that have the hammock philosophy and great tools to enable thinkers to think and create. Often I notice myself following a “90 / 10” rule: where I spend 90% of the time thinking (brain, REPL, google, papers, people, sleep, etc..) and 10% of the time creating once thoughts are solidified.

As any vibrant community Clojure has “the powers that be” that provide guidance to people who need it. This guidance could be either: very helpful or lacking context + real world examples => let’s call it “dogma from the top”.

While I am not a biologist, I like to draw a parallel between software programs created by developers and things “naturally” created by nature. When I am in the beautiful garden or diving in the ocean, I find a lot less dogma, and a lot more fruits of mutation and centuries of trial and error development. It’s amazing. So much bigger and more honest than… “singletons are bad”.

Yes, this short story is about Mount, or rather it is about perception and understanding.

Singletons are Bad


If the context is not provided, “singletons”, or “global singletons” when referring to Clojure vars, have no quality of being bad or good. It’s just the word. Is “paper” bad? How about “inheritance”, is it bad? Well, what if it’s in millions of dollars?

The claim from the top is that keeping state in Clojure vars is bad, because global singletons are bad. Ok more context, but still no context to measure the goodness. When I ask for a concrete example, or two, that would “teach me” why Clojure var is such a horrible place to keep state, the only concrete example I get is:

“Because you can develop and run tests within the same REPL / runtime”

It turns out that some people develop and run tests within the same REPL.

I like my tests run in a separate REPL, moreover I like to run a watcher in that REPL which will rerun tests on any change I make and will notify me with a pass/fail sound / color in that terminal. This can be easily done with Boot: boot watch speak test. Can also be done with Leiningen via lein-test-refresh: lein test-refresh.

Another benefit of running tests in a separate REPL / runtime: it ensures a clean, isolated environment where tests are run. And since most people still use Leiningen this saves a lot of time.

But I do respect that people could have different workflows and they like running tests within the same REPL they develop. And there is Yurt that allows to create as many different yurts as needed in the same runtime.

False Dmitry I


… “the real Dmitriy died in Uglich” [source]

All other “examples” of why keeping state in vars is bad are simply.. not cutting it. Here are some.

“Because there are instances I will have more than one database”

– “Why would I ever have more than one database?”
– “Why would I not!?”

And the example usually goes like this:

(component/system-map
 ;;...
 :user-db (database "sql://user-db")
 :product-db (database "sql://product-db")
 ;;...)

“I can’t do that with namespaces and vars”. Yes you can. Namespaces and vars are beautiful, very capable things. Here is a simple example:

(defn database [uri]
  (connect-to uri))
 
(defstate user-db :start (database "sql://user-db"))
(defstate product-db :start (database "sql://product-db"))
 
;; + :stop functions

notice how “database” is just a function, can be reused as many times as needed. user and product DBs can live within different namespaces if needed.

Yes, if “defstate” was a simple “def”, “def user-db” would be limiting, since how would you restart it for example. But “defstate” plugs “user-db” into a lifecycle aware Mount where “user-db” becomes a managed component (in a good sense) of an application.

Context is King


And here we arrived at the context: (defstate state :start (f1) :stop (f2)) is not just some global singleton which is claimed to be bad, but it is a living component that can be started, restarted, referenced and healed when it’s abused. Is it bad? Not from my experience, it’s actually quite excellent :)

“Because if you recompile a namespace you end up with a stale state”

Now with the context provided, it is simply not true with Mount.

When a namespace is recompiled Mount will stop/restart/clean all the states in this namespace. It will also clean the deleted states. Here is a lot more details about it: Recompiling Namespaces with Running States and Cleaning up Deleted States

Just Ask


Again, the Clojure community is full of people who think and reason independently. The next time you hear something like:

” Whenever you’re about to make a singleton, ask yourself “When will I need 2 of these?” ”

or

” More than one instance of a resource in an app? Inconceivable! ”

Just ask for more context, you’ll see that most of the time this is just “dogma from the top” and has nothing to do with the way nature creates things.


31
Jan 16

Yurt: Mount’s Local Real (e)State

Components or states in mount are kept in Clojure vars, which means that they are accessible from any other namespaces that :require them. They can be made private of course but nothing stops a developer from accessing a private var in Clojure by its full name: i.e. #’any.namespace/any-var.

The flip side is that they “lie near”: they are really easy to use. Can usage of these vars be abused? Of course. Can any “other state management solutions” be abused? Of course. I like my fine balance between “easy” things for development and “simple” things for the architecture.

Several, Local and Simultaneous


In several reddit discussions, people pointed out that since mount keeps components in Clojure vars, those are singletons, hence you can’t have more than one.

While I honestly don’t know how often I would want to have more than one database connection to

* the same database
* with the same host / port / sid
* the same credentials and
* the same schema

(because if any of the above is not the same it would be a different resource / component all together) The example people gave would always come down to this one use case where you want to run a development “system” and a test “system” (potentially more than one) in the same REPL.

The way I do it today is simply running:

boot watch speak test

in a different REPL.

The need to run multiple systems in the same REPL might have something to do with the limitation of the framework (i.e. component), since you can’t just start and stop parts of a system, which means you can’t iterate quickly with sub systems in the REPL. So instead you end up predefining and running several sub systems simultaneously. But this is just an assumption.

I personally never needed to run multiple “systems” within the same REPL. But I truly believe there are multiple great ways to do things, an I simply can’t just dismiss the fact that people have different development flows.

So I sat down and started to think.

Yurts are Comfy


Mount relies on the Clojure compiler to tell it what states are defined and what order they should be started / stopped in. So, I thought, I can just use this intel, create a “system” and simply detach it from vars. In other words, using the intel mount has and only using the vars for the bootstrap I can spawn as many local systems as needed.

I don’t think this should be mount’s core functionality, plus I really like the way things work with vars and namespaces, but it would be really cool to have these local systems that people can use for testing, whether it is from REPL or for running tests in parallel.

That’s how Yurt was born. The docs go over Yurt’s API (i.e. blueprint, build, destroy) and have a couple of examples of running multiple Yurts simultaneously in the same REPL.

This of course comes down to a choice of either using vars directly or encapsulating components in Yurts, or maybe even both (?) in the same application. But what’s cool about either choice, there is still “nothing to buy”: no full app buy in.


17
Jan 16

Swapping Alternate Implementations with Mount

Getting ready for a talk at Clojure Remote gave me an excuse to work on several mount example apps that, I feel, should help others to start with mount, as well as just present certain ways to structure applications.

Sending SMS over Web


Using a great twilio library I wrote a small web app that sends texts over web (i.e. via HTTP POST). It serves as a good example of how to test mount states by swapping them with stubs/mocks. This application has 3 states:

  • config which is loaded from an external file

  • web-server a Jetty web server

  • send-sms which, once started, becomes a function with Twilio creds that sends texts

The app receives an HTTP POST request and sends an SMS message:

(POST "/sms/:from/:to/:msg" [from to msg]
  (generate-string
    @(send-sms {:from from
                :to to
                :body msg}))))

The

send-sms is a mount state that is setup with Twilio credentials that come from the config:

(defn create-sms-sender [{:keys [sid auth-token]}]
  (fn [{:keys [from to body]}]
    (twilio/with-auth sid auth-token
      (twilio/send-sms 
        (twilio/sms from to body)))))
 
(defstate send-sms :start (create-sms-sender 
                            (:sms config)))

When an HTTP request is sent:

$ curl -X POST "http://localhost:4242/sms/+15104266868/+17180000000/mount%20is%20fun%20:)"

it gets wrapped to an SMS payload and is passed to Twilio which successfully delivers it:

I am sure you noticed, but the Twilio phone number this SMS is sent from is:

+1 (510) 42 MOUNT :)

Testing the App


Notice that while send-sms is a state, when started, it becomes just a function that takes args and passes them to Twilio as an SMS payload. Which means that if it is needed to be replaced, or swapped, during testing, it can be replaced with a test function that, for example, receives an SMS and puts in on a core.async channel:

(fn [sms] 
  (go (>! sms-ch sms)))

One thing to note, the real twilio/send-sms returns a future, which means it might be dereferenced somewhere in the codebase, and it is just safer to stay true to the “real thing”, so we’ll return a future as well:

(fn [sms] 
  (go (>! sms-ch sms))
  (future))

Now all that needs to be done is to create a test function and let mount know to use it instead if the real one.

;; ...
(let [sms-ch (chan)
      send-sms (fn [sms]
                 (go (>! sms-ch sms))
                 (future))]                        ;; twilio API returns a future
  (mount/start-with {#'app.sms/send-sms send-sms})
;; ...

This way mount will start an application with swapping the real app.sms/send-sms for a send-sms stub function. Which means that any reference to app.sms/send-sms at runtime will be using this locally scoped send-sms stub.

Check out the working test to get a visual on how all the above pieces come together.
And here is more details on swapping alternate implementations from mount docs.


7
Jan 16

Clojure Mindsets

After Dan’s post, we had a great discussion on reddit that was not a religious discussion, but a “civil and thought-provoking” discussion. And that, in my mind, is the greatest power of Clojure.

And The Winner is…


Yes, we have not established which is better mount or component, but that was never the point. The great summary of the discussion was a realization that @yogthos and @weavejester made at the end:

Overall, it sounds like both approaches are actually fairly similar: Mount uses namespaces and vars where Component uses records and protocols, but the purpose is broadly the same.

This is interesting for two major reasons:

* It advocates the proper usage of Component with using protocols over records/components

* It draws a clear parallel between Mount and Component, so it is a lot easier to choose which one fits your coding style / (formal vs. simple) mindset.

Libraries vs. Frameworks


Since I am obviously biased towards Mount :)..

Component manages protocols and records, and in order to do that it requires a whole app buyin, which makes it a framework. Mount does not need to manage namespaces and vars, since it is very well managed by the Clojure Compiler. Which makes it a library, which I prefer. You may not, and I respect that.

Math vs. Music


Another thing I keep noticing is that “formally” inclined people prefer records and protocols for many solutions, and for them it is easier to reason about applications since it fits better into the formal mindset.

What I disagree with is equating “more formal” with simpler. It is simpler for a formal mindset, yes, but it is not simpler for me. Explicit formalism complects things for my mindset. I do not believe there is an objective truth here, since we are just different. But I believe Clojure has both: which allows for both mindsets to coexist.

What I like protocols for is an optimized dynamic dispatch, creating libraries (which use protocols internally), and a way to tame the expression problem if and when it arises. Protocols make it simple for me.

“Frogs All Look The Same”


And to make it more interesting, I am currently on the train going back to Philly from the great NYC Lisp meetup, where Gerald Sussman presented a way to write systems that evolve and expand to handle any future functionality that is thrown at them (“extensible generics”).

He drew an interesting parallel between software systems and biology saying that “many biological mutations are fatal, but as a result we end up with extremely robust systems“, doesn’t sound very formal to me ;)


22
Dec 15

The Story of Booting Mount

Feeling The Code


I don’t agree with the opinion that “cool kids now use boot“. People who say that are just missing out on the power of “feeling the code” rather than being abstracted from the code by a “better XML”. Same deal with people 10 years ago who said “cool kids are using functional languages”.

Don’t get me wrong I like lein a lot. It is simple to start with, it is well documented, it is very googlable, it is sharing platform (i.e. templates), mature, etc.. But boot is very different, it does not aim to do what lein does, it aims to do “what you want”. There is a difference.

Mounting a Bootable Partition


Since the late 90s when I got in to Linux, I found bootable partitions most exciting, they actually bootstrap everything, they were these wizards waving their magic wands and systems appeared. Granted the wave could take minutes, but we are humans, we always wait for the magic, even if it takes the whole life.

First thing that needs to be done for the magic to happen, this bootable partition needs to be mounted.

I wanted to do it for some time now, when, I could not figure out why ClojureScript brought in as a dependency with :classifier “aot” caused compilation problems with lein/cljsbuild, David Nolen suggested that this is rather due to the lein environment issues. So 2 and 2 together: it was the right time to “boot” myself up.

And since the partition was already mounted it was ready to boot.

Grokking the New Simple


Rather than tell you how great boot is, I’ll share non obvious (to me) things that I stumbled upon converting mount from lein to boot. Let’s rock & roll:

REPL is just REPL

Since I needed a support for both Clojure and ClojureScript, I looked at many examples and noticed a pattern: usually in a dev mode one task groups several, where most of the examples have a (watch) task in that group.

I just wanted to start out, so I decided that at a minimum I need a REPL and (I guess) this watcher to be able to mimic the lein repl behavior, so I did:

(deftask dev [] 
  (comp
    (watch)
    (repl)))

And it worked! I ran boot dev and I got a REPL which would see all the updates from vim (via the updated vim-fireplace).

But then I decided to stop the REPL, and it just froze.. I ran jstack on the PID and saw lots of watcher threads locking and derefing futures. Ok, so that’s not a good combination.

The answer is simpler than I expected: it’s just boot repl. Nothing else is needed to get to the lein repl functionality.

“Bring on Your Own Data Readers” Party

The Clojure mount example app uses in memory Datomic, so when I tried to start the app, boot told me:

no reader to handle the #db/id tag

This was easily googlable, and revealed that boot has a (load-data-readers!) function that “refreshes *data-readers* with readers from newly acquired dependencies”.

An interesting bit here is that (load-data-readers!) can’t be a part of a “top level” task that is executed with boot since:

java.lang.IllegalStateException: Can't set!: *data-readers* from non-binding thread

So calling boot dev, in case “load-data-readers!” is there, is not an option. But getting into a REPL “boot repl“, and then calling (dev) works beautifully.

REPL Logging

At this point I could get into the boot REPL and start the mount example app. A slight problem was that I did not see any logging from the app within the REPL.

That’s when I found boot-logservice that brought the logging back to the REPL:

(def log4b
  [:configuration
   [:appender {:name "STDOUT" :class "ch.qos.logback.core.ConsoleAppender"}
    [:encoder [:pattern "%-5level %logger{36} - %msg%n"]]]
   [:root {:level "TRACE"}
    [:appender-ref {:ref "STDOUT"}]]])
;; ...
 
(deftask dev []
 
  ;; ...
 
  (alter-var-root #'log/*logger-factory* 
                  (constantly (log-service/make-factory log4b)))
  ;; ... 
)
Shaking up tools.namespace

While it is not a requirement, and most of the time unnecessary, the example app uses tools.namespace to make it easier for people who rely on it heavily to get into mount.

By default “tools.namespace” won’t find anything to refresh, since boot uses its own “secret” temp directories for sources, and “tools.namespace” simply does not know about them.

This was an easy one, since it is well documented by boot. Hence having (apply set-refresh-dirs (get-env :directories)) in the “dev” task pointed “tools.namespace” to the right directories to refresh.

The Joy of Deploy: Build and Publish

At this point having the Clojure part figured out, before moving to the ClojureScript support, I decided to deploy mount to Clojars, to understand how it’s done with boot.

I found bootlaces, and just plugged it in, it was very straightforward:

(def +version+ "0.1.7-SNAPSHOT")
 
(bootlaces! +version+)
 
;; other things.. and
 
(task-options!
  pom {:project     'mount
       :version     +version+
       :description "managing Clojure and ClojureScript app state since (reset)"
       :url         "https://github.com/tolitius/mount"
       :scm         {:url "https://github.com/tolitius/mount"}
       :license     {"Eclipse Public License"
                     "http://www.eclipse.org/legal/epl-v10.html"}})

Then I did:

boot build-jar push-snapshot

and everything was going smoothly, it asked for my Clojars username, then password.. but then:

clojure.lang.ExceptionInfo: java.lang.AssertionError: 
Assert failed: current git branch is 0.1.7 but must be master
               (or (not ensure-branch) (= b ensure-branch))

Boot told me that it prefers publishing snapshots from the “master”. I don’t disagree, but for some projects I like snapshots from version branches. I don’t really like “git flow”, I like “git freedom”.

Looking at the bootlaces code it seems that “master” is hardcoded. By this time I already started to feel the concept of a “boot task” and noticed that it is hardcoded under the “push” internal task, which means that this task’s options can potentially be overridden:

;; ...
 
(task-options!
 
  push {:ensure-branch nil}       ;; <<<<<<<<<<
 
  pom {:project     'mount
       :version     +version+
       ;; ... 
       })

And what d’you know, it worked! This was most likely the first “aha moment” which wired some of my neurons in boot ways.

Shall Not Pass!

Mount’s “test” root has both cljc tests and clj/cljs test apps that these tests use. The structure looks similar to:

|~test/
| |~clj/...
| | `+tapp/
| |~cljs/...
| | `+tapp/
| |~mount/
| | |+test/...
| | `-test.cljc

In lein, I can give “test” + “test/clj” for Clojure tests, and “test” + “test/cljs” for ClojureScript tests as the sources paths.

In boot I can’t do that, boot says:

java.lang.AssertionError: Assert failed: 
The :source-paths, :resource-paths, and :asset-paths must not overlap.
    (empty? (set/intersection paths parents))

Since boot already read everything under “test”, it does not want to merge things from “test/clj”. Fair enough, so I had to change the structure a bit to make it work:

|~test/
| |~clj/
| | `+tapp/
| |~cljs/
| | `+tapp/
| |~core/
| | `~mount/
| |   |+test/
| |   `-test.cljc

Now I can give “test/core” + “test/clj” and “test/core” + “test/cljs” respectively.

ClojureScript is Clojure, but.. not Always

ClojureScript took some time to get right. Many examples helped a lot especially these three: boot-cljs-example, tenzing and boot-cljs-multiple-builds.

The concept of dividing “cljs” options between “xyz.cljs.edn” and “task options” did not sink in immediately, and required some code digging to figure out where to put what and how to make sure it is being used.

It ends up to be quite simple. Options that are provided via “xyz.cljs.edn” can be referenced from task options via ids option:

(cljs :optimizations :advanced :ids #{"mount"})

would mean that it would look for mount.cljs.edn file within the classpath. That file should point to the entry point of the ClojureScript app. In case of the mount example app it would just be:

{:require  [app.example]}

where init-fns and compiler-options can be also added.

Testing ClojureScript

“mount does doo” for ClojureScript testing, and boot-cljs-test does it as well.

I would expect it to pick up “xyz.cljs.edn” files in the same way as “boot-cljs”, but it does it a bit differently. It is not all that obvious at first, but looking at the code I saw that it has a different name for ids, it calls it out-id. It also does not just take an “id”, it takes an “id” + “.js”, as I saw from the code.

So to get it to work is quite simple:

(tcs/test-cljs :out-file "mount.js"))

which would look for the same mount.cljs.edn file within the classpath.

Power it Up


There were other discoveries, like

* tasks are functions, but not really, they take arguments in the particular format and they better return a fileset

* tasks: “comp us please”. They like to be (comp ..)ed. Otherwise no go.

* there were others, but I liked Pods the most.

At this point I got all up and pumping, deployed to CircleCi using boot to build and run tests, published to Clojars as snapshot and release, etc.

One of the greatest things that I loved while debugging dependencies is boot show -p, it’s amazing!

Get up! Boot yourself up! Enjoy the runtime!