Reading a File in Mathematica - How?
I have a large file (written in Mathematica) containing n "records", and each of those records is a fixed length list m, where n> 10000 and 500 <m <600 (bytes). Please note that my system does not have the ability to keep all records in memory - the reason for writing them to a file. I have an application (in Mathematica) that needs to process these records in reverse order; those. the last recorded entry is the first processed entry. How can I read these records from a file in reverse order?
In the meantime (after some trial and error using Mathematica I / O) I found one solution. Note that this is a stripped down example of a possible solution.
fname = "testfile";
strm = OpenWrite[fname];
n = 10; (* In general, n could be very large *)
For[k = 1, k <= n, k++,
(* Create list on each pass through this loop ... *)
POt = {{k, k + 1}, {k + 2, k + 3}};
Print[POt];
(* Save to a file *)
Write[strm, POt];
];
Close[strm];
(* 2nd pass to get byte offsets of each record written to file *)
strm = OpenRead[fname];
ByteIndx = {0};
For[i = 1, i <= n, i++,
PIn = Read[strm];
AppendTo[ByteIndx, StreamPosition[strm]];
];
Drop[ByteIndx, -1]
(* Read records in reverse order *)
For[i = n, i >= 1, i--,
SetStreamPosition[strm, ByteIndx[[i]]];
PIn = Read[strm];
Print[PIn];
(* Process PIn ... *)
];
Close[strm];
It would be nice if the 2nd pass (to get the byte offsets) could be eliminated, but I haven't found how to do it yet ... Also, these byte offsets can be written to a file (similar to how writes are handled ) and then read back one at a time if the memory problem is still not resolved.
source to share