Reading a File in Mathematica - How?

I have a large file (written in Mathematica) containing n "records", and each of those records is a fixed length list m, where n> 10000 and 500 <m <600 (bytes). Please note that my system does not have the ability to keep all records in memory - the reason for writing them to a file. I have an application (in Mathematica) that needs to process these records in reverse order; those. the last recorded entry is the first processed entry. How can I read these records from a file in reverse order?

In the meantime (after some trial and error using Mathematica I / O) I found one solution. Note that this is a stripped down example of a possible solution.

    fname = "testfile";

    strm = OpenWrite[fname];
    n = 10; (* In general, n could be very large *)
    For[k = 1, k <= n, k++,
      (* Create list on each pass through this loop ... *)

      POt = {{k, k + 1}, {k + 2, k + 3}};
      Print[POt];

      (* Save to a file *)
      Write[strm, POt];
    ];
    Close[strm];

    (* 2nd pass to get byte offsets of each record written to file *)
    strm = OpenRead[fname];
    ByteIndx = {0};
    For[i = 1, i <= n, i++,
      PIn = Read[strm];
      AppendTo[ByteIndx, StreamPosition[strm]];
    ];
    Drop[ByteIndx, -1]

    (* Read records in reverse order *)
    For[i = n, i >= 1, i--,
      SetStreamPosition[strm, ByteIndx[[i]]];
      PIn = Read[strm];
      Print[PIn];

      (* Process PIn ... *)

    ];
    Close[strm];

      

It would be nice if the 2nd pass (to get the byte offsets) could be eliminated, but I haven't found how to do it yet ... Also, these byte offsets can be written to a file (similar to how writes are handled ) and then read back one at a time if the memory problem is still not resolved.

+3


source to share


1 answer


for the sake of an answer, your second pass can be written concisely:



strm = OpenRead[fname];
ByteIndx=Reap[While[Sow[StreamPosition[strm]]; !TrueQ[Read[strm ] == EndOfFile]]][[2,1,;;-2]]
n=Length[ByteIndx]

      

0


source







All Articles