A couple of interesting fact to keep in mind about iterator-seq:
- it calls hasNext right away
- it “caches” reads by chunks of 32 items
Its implementation is quite simple and returns a lazy seq, but the above two is good to keep in mind when working with iteratees:
private static final int CHUNK_SIZE = 32; public static ISeq chunkIteratorSeq(final Iterator iter){ if(iter.hasNext()) { return new LazySeq(new AFn() { public Object invoke() { Object[] arr = new Object[CHUNK_SIZE]; int n = 0; while(iter.hasNext() && n < CHUNK_SIZE) arr[n++] = iter.next(); return new ChunkedCons(new ArrayChunk(arr, 0, n), chunkIteratorSeq(iter)); } }); } return null; } |
Why 32? I like 42 better!
32 is a good choice for the CHUNK_SIZE since it matches the number of child nodes in Clojure (persistent) collections:
static public PersistentVector create(ISeq items){ Object[] arr = new Object[32]; int i = 0; for(;items != null && i < 32; items = items.next()) arr[i++] = items.first(); if(items != null) { // >32, construct with array directly PersistentVector start = new PersistentVector(32, 5, EMPTY_NODE, arr); TransientVector ret = start.asTransient(); for (; items != null; items = items.next()) ret = ret.conj(items.first()); return ret.persistent(); } else if(i == 32) { // exactly 32, skip copy return new PersistentVector(32, 5, EMPTY_NODE, arr); } else { // <32, copy to minimum array and construct Object[] arr2 = new Object[i]; System.arraycopy(arr, 0, arr2, 0, i); return new PersistentVector(i, 5, EMPTY_NODE, arr2); } } |
The Dark Side of “hasNext()”
But before creating a lazy seq, the first call “iterator-seq” does is iter.hasNext(). While this makes sense (why create a seq, if there is nothing to create it from), a thing to keep in mind is the implementation of the iteratee which is passed to “iterator-seq”. Here is an example from my recent HBase journey.
cbass wraps an HBase Scanner in “iterator-seq”:
(let [results (-> (.iterator (.getScanner h-table (scan-filter criteria)))
iterator-seq)
Once “iterator-seq” makes a call to iter.hasNext(), HBase scanner goes out and fetches the first result based on its filter. While this sounds ok, internally, depending on HBase client caching configuration, it may end up in fetching lots a lots of data to “cache” locally before returning a single item. Not exactly a “lazy seq behavior” the one can expect. More about it here.
To conclude: it is always good to keep a fresh copy of Clojure source code in the head :)