OpenCSV Reader-Java limitation
I used OpenCSV Reader - java to download my CSV file (file size: -1.47 GB (1,585,965,952 bytes)).
However, inside my coding, whenever it manages to insert 10950 records into PostgreSQL database.
CSVReader csvReader = new CSVReader(new FileReader(csvFilename));
String[] row = null;
String sqlInsertCSV = "insert into ip2location_tmp_test
(ip_from, ip_to, xxxxx, "
+ "xxxxx, xxxxx,xxxxx, "
+ "xxxxx,xxxxx, xxxxx, xxxxx, xxxxx, xxxxx,xxxxx,xxxxx)"
+ " VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
while((row = csvReader.readNext()) != null) {
PreparedStatement insertCSV = conn.prepareStatement(sqlInsertCSV);
insertCSV.setLong(1, Long.parseLong(row[0]));
....
....
insertCSV.setString(14, row[13]); // usage_type
insertCSV.executeUpdate();
}
csvReader.close();
PreparedStatement insertCSV = conn.prepareStatement(sqlInsertCSV);
insertCSV.executeUpdate();
}
Is there an OpenCSV limitation?
I need to use setString function to handle single quote in PostgreSQL.
source to share
First question: are you telling us how many bytes are in the file, but how many records (lines) are in the file? If the file has 10,950 lines everything works fine.
Second question: why do you have prepareStatement / executeUpdate method outside of the while loop? It seems to me that I will try to either insert the last record twice, or insert an empty record, because when you are out of bounds, while you have no data, because the csvReader returned null.
By using the notation in the time method that you use, there is no limit to the number of entries you can read. Check the data file to see if there are any random carriage returns in the file around line 10950.
source to share