Using DeflateStream and / or CryptoStream on TPL TransformBlock data stream
I am using TPL dataflow blocks to implement a packet based network protocol. This protocol is fixed and cannot be changed by me. Mostly there will be small but many packages.
I have a client component that connects to the server and reads raw packages. These original packets are then sent as MemoryStreams to BufferBlock and then to TransformBlock, decoded into packet structures based on packet type / id.
To send packets, this process calls on another data flow blockchain. This all works well as far as I can tell.
The problem is that these packets may or may not be compressed and may or may not be encrypted based on server responses. I would solve this with new TransformBlocks placed inbetween (using unboxing as an example):
static TransformBlock<Stream, Stream> CreateDecompressorBlock(ProtocolContext context)
{
return new TransformBlock<Stream, Stream>(stream =>
{
if (!context.CompressionEnabled) return stream;
return new DeflateStream(stream, CompressionMode.Decompress);
}
}
It seems to me that this is the wrong way. As I understand it, DeflateStream (and CryptoStream) decodes data as it reads it. This means the data is decoded when I decode the contents of the packet, not inside the TransformBlock itself, where only the skins stream is generated. It looks like this will bypass the strengths of Dataflow blocks.
So another solution came to my mind: instead of returning a DeflateStream / CryptoStream, I read it into another MemoryStream:
static TransformBlock<Stream, Stream> CreateDecompressorBlock(ProtocolContext context)
{
return new TransformBlock<Stream, Stream>(stream =>
{
if (!context.CompressionEnabled) return stream;
using (var deflate = new DeflateStream(stream, CompressionMode.Decompress))
{
var ms = new MemoryStream();
deflate.CopyTo(ms);
return ms;
}
}
}
Now it seems like a waste of memory.
So my question is, is it enough to just wrap the stream and let the TransformBlock, which subsequently decodes the packet content, does the job, or should I use a little more memory, and then perhaps better share and perhaps parallelism? (Although I don't think decoding will be the bottleneck, it will be the network).
Or is there a pattern I can use with TPL dataflow that solves my problem even better?
source to share
As always, this is a compromise and the decision can only be made by you.
I would use the simplest solution, which is only to connect both blocks, and let the write block absorb the added complexity of compression, since TDF blocks can increase their parallelism when they need it. But , I would only do it if this block does not have limited parallelism ( MaxDegreeOfParallelism
).
If there are limitations, I would treat the actual compression in the compression unit in the same way as you described. This behavior can be accomplished to a very high degree of parallelism, and this memory buffer doesn't seem to be such a big problem for everyone.
If so, you can add a buffer pool and thus your buffers will be limited to a block MaxDegreeOfParallelism
, and you only need to allocate those buffers once during the life of your application.
source to share