"; */ ?>
clojure / component / mount — Comments Off on The Way Nature Creates Things
20
Apr 16

The Way Nature Creates Things

Dogma From The Top

The best thing about the Clojure community is a large number of independent thinkers. The second best thing that allows this community to exist is the Clojure language itself, that have the hammock philosophy and great tools to enable thinkers to think and create. Often I notice myself following a “90 / 10” rule: where I spend 90% of the time thinking (brain, REPL, google, papers, people, sleep, etc..) and 10% of the time creating once thoughts are solidified.

As any vibrant community Clojure has “the powers that be” that provide guidance to people who need it. This guidance could be either: very helpful or lacking context + real world examples => let’s call it “dogma from the top”.

While I am not a biologist, I like to draw a parallel between software programs created by developers and things “naturally” created by nature. When I am in the beautiful garden or diving in the ocean, I find a lot less dogma, and a lot more fruits of mutation and centuries of trial and error development. It’s amazing. So much bigger and more honest than… “singletons are bad”.

Yes, this short story is about Mount, or rather it is about perception and understanding.

Singletons are Bad

If the context is not provided, “singletons”, or “global singletons” when referring to Clojure vars, have no quality of being bad or good. It’s just the word. Is “paper” bad? How about “inheritance”, is it bad? Well, what if it’s in millions of dollars?

The claim from the top is that keeping state in Clojure vars is bad, because global singletons are bad. Ok more context, but still no context to measure the goodness. When I ask for a concrete example, or two, that would “teach me” why Clojure var is such a horrible place to keep state, the only concrete example I get is:

“Because you can develop and run tests within the same REPL / runtime”

It turns out that some people develop and run tests within the same REPL.

I like my tests run in a separate REPL, moreover I like to run a watcher in that REPL which will rerun tests on any change I make and will notify me with a pass/fail sound / color in that terminal. This can be easily done with Boot: boot watch speak test. Can also be done with Leiningen via lein-test-refresh: lein test-refresh.

Another benefit of running tests in a separate REPL / runtime: it ensures a clean, isolated environment where tests are run. And since most people still use Leiningen this saves a lot of time.

But I do respect that people could have different workflows and they like running tests within the same REPL they develop. And there is Yurt that allows to create as many different yurts as needed in the same runtime.

False Dmitry I

… “the real Dmitriy died in Uglich” [source]

All other “examples” of why keeping state in vars is bad are simply.. not cutting it. Here are some.

“Because there are instances I will have more than one database”

– “Why would I ever have more than one database?”
– “Why would I not!?”

And the example usually goes like this:

(component/system-map
 ;;...
 :user-db (database "sql://user-db")
 :product-db (database "sql://product-db")
 ;;...)

“I can’t do that with namespaces and vars”. Yes you can. Namespaces and vars are beautiful, very capable things. Here is a simple example:

(defn database [uri]
  (connect-to uri))
 
(defstate user-db :start (database "sql://user-db"))
(defstate product-db :start (database "sql://product-db"))
 
;; + :stop functions

notice how “database” is just a function, can be reused as many times as needed. user and product DBs can live within different namespaces if needed.

Yes, if “defstate” was a simple “def”, “def user-db” would be limiting, since how would you restart it for example. But “defstate” plugs “user-db” into a lifecycle aware Mount where “user-db” becomes a managed component (in a good sense) of an application.

Context is King

And here we arrived at the context: (defstate state :start (f1) :stop (f2)) is not just some global singleton which is claimed to be bad, but it is a living component that can be started, restarted, referenced and healed when it’s abused. Is it bad? Not from my experience, it’s actually quite excellent :)

“Because if you recompile a namespace you end up with a stale state”

Now with the context provided, it is simply not true with Mount.

When a namespace is recompiled Mount will stop/restart/clean all the states in this namespace. It will also clean the deleted states. Here is a lot more details about it: Recompiling Namespaces with Running States and Cleaning up Deleted States

Just Ask

Again, the Clojure community is full of people who think and reason independently. The next time you hear something like:

” Whenever you’re about to make a singleton, ask yourself “When will I need 2 of these?” ”

or

” More than one instance of a resource in an app? Inconceivable! ”

Just ask for more context, you’ll see that most of the time this is just “dogma from the top” and has nothing to do with the way nature creates things.

clojure / cprop — Comments Off on Config You Know Plus Some Love
03
Mar 16

Config You Know Plus Some Love

Development and Structure Factors

12 Factor clearly states that configuration should live in environment variables:

“The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.”

Makes sense. But for me:

* it makes development harder
* it is simply not needed for certain apps I work on
* environment variables lack structure

But it does not mean nothing can be done to have both: 12 factor and the solution for these three points above. We are programmers, we create things.

The Golden Goose

I’ve done quite a few apps in Java over the last decade, and I know a thing or two on how to bridge the gap between applications and configurations.

In Clojure universe, the most popular library to manage configs is environ. Normally apps would have their properties live in lein profiles / .lein-env / .boot-env and could be overridden by environment variables and system properties. Currently this is the way to manage configs in Clojure.

Things I find lacking are:

1. Ultimately configuration is solely based on ENV variables (+ some system props) exported as individual properties: 100 properties? 100 env variables exported..

2. It allows only string values: no data structures, no numbers, etc.

3. It allows no structure / hierarchy, just one (top) level pile of properties

4. Ironically :) it keeps a single global internal config state, which makes it hard to have app (sub) modules with separate configs

This is not to say environ got it wrong, or it is no good, it is really useful and works great for a lot of people, but I feel I need an alternative, because of the 4 things above, and simply because I like to have alternatives. So I wrote one.

12 Factor + Some Love

My take on configuration and properties is called cprop. It is young and brave, and currently is used by me in several applications, a few others, and it is a default configuration approach in Luminus which is a micro-framework that is based on a set of lightweight libraries.

Instead of repeating cprop documentation, it would be important to notice that the factor number III out of 12 factors, which is called “Config“, is fully supported by cprop, while still allowing for more flexibility.

For example let’s take a concept of “structure” or “hierarchy”. Say you have a config in development (i.e. somewhere in dev-resources):

{:datomic {:url "CHANGE ME"},
 :aws
 {:access-key "AND ME",
  :secret-key "ME TOO",
  :region "FILL ME IN AS WELL",
  :visiblity-timeout-sec 30,
  :max-conn 50,
  :queue "cprop-dev"},
 :io
 {:http
  {:pool
   {:socket-timeout 600000,
    :conn-timeout :I-SHOULD-BE-A-NUMBER,
    :conn-req-timeout 600000,
    :max-total 200,
    :max-per-route :ME-ALSO}}},
 :other-things
 ["I am a vector and also like to play the substitute game"]}

Looks like it passes the 12 factor litmus test which says that “the codebase could be made open source at any moment, without compromising any credentials” while preserving the structure.

There are several ways to provide real values for this config with cprop, such as other maps merged at runtime, system properties, runtime arguments, etc.. but the most interesting way, from the 12 factor point of view, is “environment variables”. Which can be simply done with cprop as:

export AWS__ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
export AWS__SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS__REGION=us-east-1
export IO__HTTP__POOL__CONN_TIMEOUT=60000
export IO__HTTP__POOL__MAX_PER_ROUTE=10
export OTHER__THINGS='[1 2 3 "42"]'

Check out the documentation, there are several more useful concepts that cprop brings to life.

I love bringing things to life.

clojure / component / mount / yurt — Comments Off on Yurt: Mount’s Local Real (e)State
31
Jan 16

Yurt: Mount’s Local Real (e)State

Components or states in mount are kept in Clojure vars, which means that they are accessible from any other namespaces that :require them. They can be made private of course but nothing stops a developer from accessing a private var in Clojure by its full name: i.e. #’any.namespace/any-var.

The flip side is that they “lie near”: they are really easy to use. Can usage of these vars be abused? Of course. Can any “other state management solutions” be abused? Of course. I like my fine balance between “easy” things for development and “simple” things for the architecture.

Several, Local and Simultaneous

In several reddit discussions, people pointed out that since mount keeps components in Clojure vars, those are singletons, hence you can’t have more than one.

While I honestly don’t know how often I would want to have more than one database connection to

* the same database
* with the same host / port / sid
* the same credentials and
* the same schema

(because if any of the above is not the same it would be a different resource / component all together) The example people gave would always come down to this one use case where you want to run a development “system” and a test “system” (potentially more than one) in the same REPL.

The way I do it today is simply running:

boot watch speak test

in a different REPL.

The need to run multiple systems in the same REPL might have something to do with the limitation of the framework (i.e. component), since you can’t just start and stop parts of a system, which means you can’t iterate quickly with sub systems in the REPL. So instead you end up predefining and running several sub systems simultaneously. But this is just an assumption.

I personally never needed to run multiple “systems” within the same REPL. But I truly believe there are multiple great ways to do things, an I simply can’t just dismiss the fact that people have different development flows.

So I sat down and started to think.

Yurts are Comfy

Mount relies on the Clojure compiler to tell it what states are defined and what order they should be started / stopped in. So, I thought, I can just use this intel, create a “system” and simply detach it from vars. In other words, using the intel mount has and only using the vars for the bootstrap I can spawn as many local systems as needed.

I don’t think this should be mount’s core functionality, plus I really like the way things work with vars and namespaces, but it would be really cool to have these local systems that people can use for testing, whether it is from REPL or for running tests in parallel.

That’s how Yurt was born. The docs go over Yurt’s API (i.e. blueprint, build, destroy) and have a couple of examples of running multiple Yurts simultaneously in the same REPL.

This of course comes down to a choice of either using vars directly or encapsulating components in Yurts, or maybe even both (?) in the same application. But what’s cool about either choice, there is still “nothing to buy”: no full app buy in.

clojure / mount / testing — Comments Off on Swapping Alternate Implementations with Mount
17
Jan 16

Swapping Alternate Implementations with Mount

Getting ready for a talk at Clojure Remote gave me an excuse to work on several mount example apps that, I feel, should help others to start with mount, as well as just present certain ways to structure applications.

Sending SMS over Web

Using a great twilio library I wrote a small web app that sends texts over web (i.e. via HTTP POST). It serves as a good example of how to test mount states by swapping them with stubs/mocks. This application has 3 states:

  • config which is loaded from an external file

  • web-server a Jetty web server

  • send-sms which, once started, becomes a function with Twilio creds that sends texts

The app receives an HTTP POST request and sends an SMS message:

(POST "/sms/:from/:to/:msg" [from to msg]
  (generate-string
    @(send-sms {:from from
                :to to
                :body msg}))))

The

send-sms is a mount state that is setup with Twilio credentials that come from the config:

(defn create-sms-sender [{:keys [sid auth-token]}]
  (fn [{:keys [from to body]}]
    (twilio/with-auth sid auth-token
      (twilio/send-sms 
        (twilio/sms from to body)))))
 
(defstate send-sms :start (create-sms-sender 
                            (:sms config)))

When an HTTP request is sent:

$ curl -X POST "http://localhost:4242/sms/+15104266868/+17180000000/mount%20is%20fun%20:)"

it gets wrapped to an SMS payload and is passed to Twilio which successfully delivers it:

I am sure you noticed, but the Twilio phone number this SMS is sent from is:

+1 (510) 42 MOUNT :)

Testing the App

Notice that while send-sms is a state, when started, it becomes just a function that takes args and passes them to Twilio as an SMS payload. Which means that if it is needed to be replaced, or swapped, during testing, it can be replaced with a test function that, for example, receives an SMS and puts in on a core.async channel:

(fn [sms] 
  (go (>! sms-ch sms)))

One thing to note, the real twilio/send-sms returns a future, which means it might be dereferenced somewhere in the codebase, and it is just safer to stay true to the “real thing”, so we’ll return a future as well:

(fn [sms] 
  (go (>! sms-ch sms))
  (future))

Now all that needs to be done is to create a test function and let mount know to use it instead if the real one.

;; ...
(let [sms-ch (chan)
      send-sms (fn [sms]
                 (go (>! sms-ch sms))
                 (future))]                        ;; twilio API returns a future
  (mount/start-with {#'app.sms/send-sms send-sms})
;; ...

This way mount will start an application with swapping the real app.sms/send-sms for a send-sms stub function. Which means that any reference to app.sms/send-sms at runtime will be using this locally scoped send-sms stub.

Check out the working test to get a visual on how all the above pieces come together.
And here is more details on swapping alternate implementations from mount docs.

clojure — Comments Off on Disturbance in The Force
10
Jan 16

Disturbance in The Force

I’d like to lay out a couple of thoughts here to discuss further to get more insight from people with different mindsets / opinions / facts / views. Reasons are mostly selfish: I know what I don’t know and I’d like to understand it better.

Folds and Braids

As Clojure developers we are in this “Cult of Simple”. Simple “was defined to us” :) as one fold/braid/twist, which does not mean “one thing”, and is really about the interleaving, not the cardinality. We also know that simple is “objective”, we can look at it and see the “number of folds”.

I don’t think it’s such a bad cult to be in, but.. It seems that the above “simple definition of simple” is now taken to an extreme where, as definition, it is interleaved / complected with real problems and is losing its intended power.

The definition of simple is now used as a shield, rather than a tool.

It might have to do with the very subjective definition of “one fold”. It is easier to understand what “one fold” is when thinking about “primitives”: i.e. Rich’s example of “Sets vs. Lists”, where Sets are simpl(er) since Lists introduce order. But it is not as clear what “one fold” is in a more “complected problems” whether these are business problems or tool libraries.

Keep Your Functions Close, but “Just In Case” Closer

Clojure protocols are super powerful, and I would say “simple”.

I don’t think Clojure records are powerful, but good (in my experience) for type driven polymorphism, and they hold fields a bit more efficiently than maps. In reality “type driven polymorphism” would be the only reason I would use them for.

If “type driven polymorphism” is all that’s needed I would first reach out to deftype instead of records, since records complect data with types: two folds :)

However, since intuitively, building solutions with protocols feels a lot more extensible, robust and flexible, I think they get applied where a simple set of functions over multiple namespaces should have been used instead. This problem, I think, is caused by the invisible seduction of “easy”, which is defined to us as “lie near”:

“since my solution should be robust and extensible, I’ll use records and protocols, since I know they will make it so“.

In other words, for initial application design: “records and protocols lie near”.

That is not to say that protocols or types should be avoided, quite the contrary, dynamic type dispatch, libraries with (internal) abstractions, host interop: all great cases to use and love them.

But I don’t think it is wise, in the Clojure Universe, to make people create records / types and use protocols when they need to use your library to develop products. This is not the absolute truth, but for most cases, when developing a business application, I would rather use a Clojure function: one fold :)

Humor Driven Development

I spend a couple of years working in Scala. An implicit type in Scala was one of the most common causes of confusion. It is used everywhere internally in the language itself, as well as advocated to use in every day Scala programs. This and many other examples, teach us that “implicit” is “complex”. The flip side of that is where the problem lies. In Clojure “formal circles” the “inferential” belief is that “explicit” is “simple”.

I believe that neither implicit nor explicit can be applied to simple without a context. And no, a “well implied implicit” does not mean complex. And no, explicit does not mean simple.

For example, I need to create a local scope and bind some values: if I am given a choice to write an identity monad or to use a Clojure let binding, I would choose the let binding, because it is a great syntactic implicit. An identity monad would also work, but being explicit here does not buy me any simplicity.

An interesting quality of a good implicit, by the way, is “automatic” understanding of what’s implied: that’s how we laugh :)

Respect and Doubts

This brings me to the overuse of “explicit formalism” in Clojure. On one hand it cannot be subjectively complex, since we know that “simple” is objective. On the other hand it can, because a “single fold” can be defined very differently in the explicitly formal solution by me and by people who created it.

I say: let’s listen to each other, rather than teach and preach.

← Older Entries
Newer Entries →