Finding a Binary Encoding Engine

We have implemented an optimized binary format for serializing records and arrays. You can also easily serialize any memory structure object. It's optimized for speed and usable space.

This is part of our open source mORMot project, running from Delphi 5 to XE2. You don't need to use all the functionality of the ORM / Client-Server service of the project, just a block SynCommons.pas

.

Then you can use our SynLZ real-time compression format to make the resulting content even smaller.

See this blog post and related source code.

It has more options than serialization (i.e. sorting, searching, hashing, slicing, reverse ...).

It can be used with classes TFileBufferWriter/TFileBufferReader

to create any custom variable length integer encoding format and some other optimizations.

For example, we use this serialization to store a .map file of all characters in a binary .mab format: it uses multiple TDynArray

+ SynLZ instances. For a text file with a 4.44MB .map extension, it creates a .mab of 378KB. See TSynMapFile.SaveToStream

others.

We use this very format to store a list of objects in memory (see TSQLRestServerStaticInMemory

class c SQLite3Commons.pas

). For example, 502KB content is People.json

saved in a 92KB binary People.data

.

Just a piece of code:

function TSQLRestServerStaticInMemory.SaveToBinary(Stream: TStream): integer;
var W: TFileBufferWriter;
    MS: THeapMemoryStream;
    IDs: TIntegerDynArray;
    i, n, f: integer;
begin
  result := 0;
  if (self=nil) or (Stream=nil) then
    exit;
  MS := THeapMemoryStream.Create;
  W := TFileBufferWriter.Create(MS);
  try
    // primitive magic and fields signature for file type identification
    W.Write(RawUTF8(ClassName));
    W.Write(StoredClassProps.SQLTableName);
    n := Length(StoredClassProps.FieldsName);
    W.WriteRawUTF8DynArray(StoredClassProps.FieldsName,n);
    W.Write(pointer(StoredClassProps.FieldType),sizeof(TSQLFieldType)*n);
    // write IDs
    SetLength(IDs,Count);
    with fValue do
      for i := 0 to Count-1 do
        IDs[i] := TSQLRecord(List[i]).fID;
    W.WriteVarUInt32Array(IDs,Count,wkSorted); // efficient ID storage
    // write content, grouped by field (for better compression)
    for f := 0 to High(fStoredClassProps.Fields) do
      with fStoredClassProps.Fields[f]^, fValue do
        for i := 0 to Count-1 do
          GetBinary(TSQLRecord(List[i]),W);
    W.Flush;
    result := StreamSynLZ(MS,Stream,TSQLRESTSERVERSTATICINMEMORY_MAGIC);
  finally
    W.Free;
    MS.Free;
  end;
end;