How to implement data streaming using Snap framework?
I would like to implement big data streams (in both directions) using a Snap server. To explore the possibilities, I created a sample program that has two endpoints, read and write. There is a very simple internal buffer that contains one ByteString
, and whatever is written to the write endpoint appears in read. (There is currently no way to terminate the thread, but this is fine for this purpose.)
{-# LANGUAGE OverloadedStrings #-}
import Control.Applicative
import Control.Concurrent.MVar.Lifted
import Control.Monad
import Data.ByteString (ByteString)
import Blaze.ByteString.Builder (Builder, fromByteString)
import Data.Enumerator
import qualified Data.Enumerator.List as E
import Data.Enumerator.Binary (enumFile, iterHandle)
import Snap.Core
import Snap.Http.Server
main :: IO ()
main = do
buf <- newEmptyMVar
quickHttpServe (site buf)
site :: MVar ByteString -> Snap ()
site buf =
route [ ("read", modifyResponse (setBufferingMode False
. setResponseBody (fromBuf buf)))
, ("write", runRequestBody (toBuf buf))
]
fromBuf :: MVar ByteString -> Enumerator Builder IO a
fromBuf buf = E.repeatM (liftM fromByteString $ takeMVar buf)
toBuf :: MVar ByteString -> Iteratee ByteString IO ()
toBuf buf = E.mapM_ (putMVar buf)
Then I start different terminals
curl http://localhost:8000/read >/dev/nul
and
dd if=/dev/zero bs=1M count=100 | \
curl --data-binary @- http://localhost:8000/write
But the failing part is not executed if the exception has moved to toplevel: too many bytes read. It's obviously an instance TooManyBytesReadException
, but I couldn't find where it was dropped. Writing less data like 1MB works as expected.
My questions:
- Where / how to fix the read limit?
- Would this data stream be without loading the entire POST request into memory? If not, how do I fix it?
source to share
It will work if you add to your /write
content type, not "application/x-www-form-urlencoded"
like:
dd if=/dev/zero bs=1M count=100 | \
curl -H "Content-Type: application/json" --data-binary @- http://localhost:8000/write
This bit in Snap does something like
if contentType == Just "application/x-www-form-urlencoded" then readData maximumPOSTBodySize
where
maximumPOSTBodySize = 10*1024*1024
and x-www-form-urlencoded
hangs by default.
source to share
To follow up on the previous answer: because type forms are application/x-www-form-urlencoded
so common that the convenience of Snap will automatically decode them for you and put them in a parameter map in the query. The idea is similar in spirit, for example $_POST
from PHP.
However, since these cards are read into RAM, naively decoding unlimited amounts of this data would allow an attacker to trivially perform a DoS server by sending arbitrary amounts of this input before dumping the heap. For this reason, it snap-server
limits the amount of data that he wants to read in this way.
source to share