How to fix memory leaks D "
So, I have been looking for a solution to this problem for some time now. I wrote a program to get data from two separate text files, parse and output to another text file and an ARFF file for Weka analysis. The problem I am running into is that the function I wrote to handle read and parse operations does not deselect proper memory allocation. Each subsequent call uses an additional 100MB or so, and I need to call this function more than 60 times during this function. Is there a way to force D to allocate memory in relation to arrays, dynamic arrays, and associative arrays in particular?
An example of my problem:
struct Datum {
string Foo;
int Bar;
}
Datum[] Collate() {
Datum[] data;
int[] userDataSet;
int[string] secondarySet;
string[] raw = splitLines(readText(readFile)).dup;
foreach (r; raw) {
userDataSet ~= parse(r);
secondarySet[r.split(",").dup] = parseSomeOtherWay(r);
}
data = doSomeOtherCalculation(userDataSet, secondarySet);
return data;
}
source to share
Are the strings stored in the returned data inside the original text file?
Split operations on an array in D do not make a copy of the data - instead, they simply store the pointer and length. This also applies to splitLines
, split
and possibly to doSomeOtherCalculation
. This means that as long as a substring of the original text file exists anywhere in the program, the contents of the entire file cannot be freed.
If the data you are returning is only a small fraction of the size of the text file you are reading, you can use it .dup
to create a copy of the string. This will prevent small strings from being bound from the entire contents of the file in memory.
source to share
If the content of the result is Collate()
duplicated after the call, it is likely that it is not collected by the GC and thus remains in memory until it is no longer used. If so, you can use the global container, which you reset for each Collate()
:
void Collate(out Datum[] data) {
// data content is cleared because of 'out' param storage class
// your processing to fill data
}
source to share