Clever Geek Handbook

How do I execute synchronous parallel functions in Clojure?

I have an application that has an initial thread that can be executed in parallel:

  • Get two JSON documents (I am using clj-http for this )
  • Parse these documents (extract only required data)
  • Join results
  • Dump them into one file

So something like this:

some-entry-point
      /\
     /  \
    /    \
   /      \
fetchA   fetchB
  |         |
  |         |
parseA   parseB
  \        /
   \      /
    \    /
     \  /
      \/
     join
      |
      |
     dump

What is the correct and most recent way to achieve this?

What I have found so far:

+3


source to share


1 answer


Since you have exactly two branches, it's best to submit parallel jobs to split threads using a future . future will return you a future object (a special promise that will be automatically resolved when the job is completed).

This is how it will look:

(defn some-entry-point
  [obja objb]
  (let [pa (-> (fetchA obja)
               parseA
               future)
        pb (-> (fetchB objb)
               parseB
               future)]
    (-> (join @pa @pb)
        dump)))

@

here is the shortcut (reader macro) for the deref function .



Or, of course, you could execute one of the branches on the main thread, creating only one future object:

(defn some-entry-point
  [obja objb]
  (let [p (-> (fetchB objb)
              parseB
              future)]
    (-> (fetchA obja)
        parseA
        (join @p)
        dump)))

You can also generalize this approach to extract and analyze multiple objects using some common functionality fetch

and parse

using pmap to parallelize:

(defn some-entry-point
  "Fetch and parse multiple objects in parallel"
  [objs]
  (->>  objs
        (pmap (comp parse fetch))
        (apply join)
        dump))
+4


source






More articles:


All Articles