Optimal buffer writing process
We have our own streaming algorithm which includes some metadata + records + field values.
We are currently using TStream and writing to add values to the stream. Now I am wondering if it is possible to speed up the operation faster this time using some technique.
Edit: We're only adding data to the end, not moving around and looking.
Some of the things I was thinking about are the following:
- Do not use buf streams for some large allocated amount of memory to copy data, the problem is that if we go out of the buffer size, then we need to move to some new memory space.
- Use a Stream filled with C # 0 up to some size and then start adding values. The rationale is that Tstream has to allocate its own buffer every time I write (I don't know how this works, just wondering ...)
We add strings to TStream and binary data in the form # 0 # 0 # 0 # 1, for example.
The data is then transferred over TCP, so it's not about writing to a file.
So what's the best way to do this?
source to share
Overwrite TMemoryStream and remove size and capacity limitation. And don't call TMemoryStream.Clear, but call TMemoryStream.SetSize (0)
type
TMemoryStreamEx = class(TMemoryStream)
public
procedure SetSize(NewSize: Longint); override;
property Capacity;
end;
implementation
{ TMemoryStreamEx }
procedure TMemoryStreamEx.SetSize(NewSize: Integer);
var
OldPosition: Longint;
begin
if NewSize > Capacity then
inherited SetSize(NewSize)
else
begin
OldPosition := Position;
SetPointer(Memory, NewSize);
if OldPosition > NewSize then
Seek(0, soFromEnd);
end;
end;
source to share
- Use a profiler to see where it is really slow.
- If it is indeed due to multiple reallocations to increase the stream size, you can avoid this by setting the property
Size
to a large enough amount. - The only case where using a memory buffer might make an obvious difference is if you are using FileStreams, all other streams in use already do it for you.
source to share
First, you consider what TStream
the bottleneck is. You need to profane your code, for example with AQTime, to determine where the bottleneck is. Don't make assumptions.
Second, what type TStream
are you using? TMemoryStream
? TFileStream
? Something other? Different types of threads handle memory differently. TMemoryStream allocates a memory buffer and increments it by the specified number of bytes whenever the buffer is full. TFileStream
on the other hand it doesn't use any memory at all, it just writes directly to the file and allows the OS to handle any buffering.
No matter what type of stream you are using, one thing you can try is to implement your own custom class TStream
that has a fixed size internal buffer and a pointer to your real target TStream
. Then you can pass an instance of your custom class to your algorithm. Have your class override the method TStream::Write()
to copy the input into its buffer until it fills up, then you can Write()
save the buffer to the destination TStream
and flush the buffer. Your algorithm will never know the difference. How TMemoryStream
, and TFileStream
would benefit from an additional buffer - a smaller number of records will mean a more efficient allocation of memory and input-output files. For example:
type
TMyBufferedStreamWriter = class(TStream)
private
fDest: TStream;
fBuffer: array[0..4095] of Byte;
fOffset: Cardinal;
public
constructor Create(ADest: TStream);
function Read(var Buffer; Count: Longint): Longint; override;
function Write(const Buffer; Count: Longint): Longint; override;
procedure FlushBuffer;
end;
...
uses
RTLConsts;
constructor TMyBufferedStreamWriter.Create(ADest: TStream);
begin
fDest := ADest;
end;
function TMyBufferedStreamWriter.Read(var Buffer; Count: Longint): Longint;
begin
Result := 0;
end;
function TMyBufferedStreamWriter.Write(const Buffer; Count: Longint): Longint;
var
pBuffer: PByte;
Num: Cardinal;
begin
Result := 0;
pBuffer := PByte(@Buffer);
while Count > 0 do
begin
Num := Min(SizeOf(fBuffer) - fOffset, Cardinal(Count));
if Num = 0 then FlushBuffer;
Move(pBuffer^, fBuffer[fOffset], Num);
Inc(fOffset, Num);
Inc(pBuffer, Num);
Dec(Count, Num);
Inc(Result, Num);
end;
end;
procedure TMyBufferedStreamWriter.FlushBuffer;
var
Idx: Cardinal;
Written: Longint;
begin
if fOffset = 0 then Exit;
Idx := 0;
repeat
Written := fDest.Write(fBuffer[Idx], fOffset - Idx);
if Written < 1 then raise EWriteError.CreateRes(@SWriteError);
Inc(Idx, Written);
until Idx = fOffset;
fOffset := 0;
end;
...
Writer := TMyBufferedStreamWriter.Create(RealStreamHere);
try
... write data to Writer normally as needed...
Writer.FlushBuffer;
finally
Writer.Free;
end;
source to share