Long term hiccups for registering stdout to file
I have a C program that writes 3 lines every 10ms to stdout. If I redirect the output to a file (using>), there will be long delays (60ms) in starting the program. Delays are periodic (say, every 5 seconds).
If I just let it write to the console or redirect to / dev / null, no problem.
I suspected it was a stdout buffer issue, but using fflush (stdout) did not solve the problem.
How can I solve the problem?
source to share
You need to use fsync
. Following:
fsync(fileno(stdout))
Should help. Note that the Linux kernel will still buffer the I / O limit according to its internal scheduler limits. Running as root and setting a very low value nice
can make a difference if you are not getting the frequency you want.
If it's still too slow, try using instead fdatasync
. Each fflush
and fsync
makes the file system update the node metadata (file size, access times, etc.) as well as the actual data. If you know in blocks how much data you will be writing, you can try the following trick:
#define _XOPEN_SOURCE 500
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
int main(int argc, char **argv){
FILE *fp = fopen("test.txt", "w");
char *line = "Test\n";
char *fill = "\0";
fwrite(fill, 1, 100*strlen(line), fp);
fflush(fp);
fsync(fileno(fp));
rewind(fp);
for (int i = 0; i < 100; i++){
fwrite(line, strlen(line), 1, fp);
fflush(fp);
fdatasync(fileno(fp));
}
}
The first call to fwrite writes 5 * 100 zeros to the file in one chunk, and fsyncs so that it gets written to disk and the node information is updated. We can now write up to 500 bytes to a file without filesystem metadata. rewind(3)
returns the file pointer position to the beginning of the file, so we can write data without resizing the file node.
The timing of this program:
$ time ./fdatasync
./fdatasync 0.00s user 0.01s system 1% cpu 0.913 total
So it would run fdatasync and sync to disk 100 times in 0.913 seconds, averaging ~ 9ms per write and fdatasync call.
source to share
If I redirect the output to a file (using>) there will be a long delay (60ms) when the program starts.
This is because when stdout
is a terminal device, it is usually (although not necessary) line buffered, that is, the output buffer is flushed when a newline is written, whereas in the case of regular files, the output is fully buffered, which means that the buffers are flushed either when full , either when you close the file (or you explicitly call fflush()
, of course).
fflush(stdout)
might not be enough for you, as this only flushes the standard I / O library buffers, but the kernel also buffers and delays writes to disk. You can call fsync()
in a file descriptor to flush modified buffer cache pages to disk after being called fflush()
, as in fsync(STDOUT_FILENO)
.
Be careful not to call fsync()
without calling fflush()
earlier.
UPDATE . You can also try sync()
, which, unlike fsync()
, does not block waiting for the original records to return. Or, as suggested in another answer, fdatasync()
might be a good choice as it avoids the overhead of updating file times.
source to share