How to lazy the result stream from MySQL db using clojure.java.jdbc?

I am having an issue with streaming a large result set from a MySQL database using clojure.java.jdbc. Here's what I'm doing now:

(defn etl! [query result-set-fn]
  (jdbc/with-db-transaction [t-conn db-spec]
    (let [conn (jdbc/get-connection t-conn)
          statement (jdbc/prepare-statement conn query
                                            :fetch-size Integer/MIN_VALUE
                                            :concurrency :read-only
                                            :result-type :forward-only)]
      (jdbc/query conn [statement] :result-set-fn first))))

      

I've also tried it without transaction and with (.setAutoCommit conn False)

(which is needed for Postgres). According to the MySQL docs, setting the fetch size, concurrency and result type should tell MySQL to execute the results one at a time, but that doesn't seem to be happening; the request hangs and heap consumption is steadily increasing by at least hundreds of megabytes.

A similar question was asked a few years ago, but the answer is now obsolete clojure.java.jdbc: Streaming from MySQL with clojure.java.jdbc

Any ideas?

+3


source to share





All Articles