Clojure - memoize on disk
I would like to improve the performance of a function that returns resized images. The requested image size shouldn't be much different (device dependent), so it makes sense to cache the results somehow.
I could of course save it to disk and check if the modified image exists, and make sure that if the original image is deleted, the modified versions do too ...
Or, I could use a memoized function. But since the result is potentially quite large (the image is about 5 - 10 MB, I think), it makes no sense to store them in memory (several tens of GB of images and their modified versions will fill the memory pretty quickly).
So, is there a way to have a memoized function that acts like a normal Clojure defmemo
, but is it supported by a folder on local disk instead of memory? Then I could use a strategy ttl
to make sure the images don't stay out of sync for too long.
Something similar to crache but backed up by the filesystem?
source to share
Do not overdo it. Your filesystem as a cache is the right idea. If one file becomes popular and many are accessed, then your operating system will keep an eye on it in RAM. This is the same strategy as many databases. For example, Elasticsearch requires that you leave enough RAM in order to have Lucene index files in RAM.
Never change your files! Do it in a functional way: treat it as immutable data. Your input file should not be modified. If so, then this is a new file. Hard disk space is not cheap cheap. Don't be afraid to have a lot of files lying around. If you need to, you can do garbage collection, which will remove the old / flagged files after a while.
To check if a file is in the cache, you simply check if the file exists. If it is not: you write it once.
So, let's summarize:
- Let your O / S run with caching
- Don't edit your files. Treat them as immutable data. Write once
- Your O / S will free up RAM for unused files. Hard disk space is super cheap.
source to share
Why not implement TTL-Cache from clojure.core.cache, wrapping it in the functionality you want? Your key can be anything that identifies your modified image, and the value will be its location on disk. Then you could implement some kind of set or set! a function by passing it a function that will be called to generate the image when it doesn't exist. eg.
(def Cache (atom (cache/ttl-cache-factory {} :ttl 20000)))
(defn get-or-update!
"wraps the recommended has-hit-get pattern
https://github.com/clojure/core.cache/wiki/Using"
[key fn]
(if (cache/has? @Cache key)
(get (swap! Cache #(cache/hit % key)) key)
(get (swap! Cache #(cache/miss % key (fn))) key)))
source to share
What you need, it sounds like the perfect Datomic app. It is easily used from Clojure, is reasonably efficient, and like any good database, it has a least recently used (LRU) cache in memory. It can also use a wide variety of database databases as a substrate, from strictly on-board memory (best for testing and experimentation) to Postgres, Redis, DynamoDB, Riak, and more. There is also a "dev" mode that uses local files for the entire repository.
See here for more details:
There is a standard version with a free, perpetual license that works for most purposes. A paid version is available for advanced features.
source to share