DataAdapter.Update () performance

I have a relatively simple procedure that looks at the database records for media files, calculates the width, height, and size of the file, and writes them back to the database.

SQLite database, using System.Data.SQLite library, processes ~ 4000 rows. I load all rows into an ADO table, update the rows / columns with the new values, and then run the adapter .Update (table); On him.

Loading dataset from db tables for half a second or so, updating all rows with image width / height and getting file length from FileInfo took maybe 30 seconds. Good.

.Update adapter (table); the team took somewhere around 5-7 minutes to run.

It seems terribly over the top. The id is PK INTEGER, and hence - according to the SQLite docs, is indexed per se, but I can't help but think that if I ran a separate update command for each individual update, it would be much faster.

I considered ADO / adapters to be relatively low (as opposed to ORM) and this awful performance surprised me. Can anyone shed some light on why it would take 5-7 minutes to update a batch of ~ 4000 records across a local SQLite database?

As possible aside, is there a way to "peek" into how ADO handles this? Internal libraries step by step or ... ??

thank

public static int FillMediaSizes() {
        // returns the count of records updated

        int recordsAffected = 0;

        DataTable table = new DataTable();
        SQLiteDataAdapter adapter = new SQLiteDataAdapter();

        using (SQLiteConnection conn = new SQLiteConnection(Globals.Config.dbAppNameConnectionString))
        using (SQLiteCommand cmdSelect = new SQLiteCommand())
        using (SQLiteCommand cmdUpdate = new SQLiteCommand()) {

            cmdSelect.Connection = conn;
            cmdSelect.CommandText =
                "SELECT ID, MediaPathCurrent, MediaWidth, MediaHeight, MediaFilesizeBytes " +
                "FROM Media " +
                "WHERE MediaType = 1 AND (MediaWidth IS NULL OR MediaHeight IS NULL OR MediaFilesizeBytes IS NULL);";

            cmdUpdate.Connection = conn;
            cmdUpdate.CommandText =
                "UPDATE Media SET MediaWidth = @w, MediaHeight = @h, MediaFilesizeBytes = @b WHERE ID = @id;";

            cmdUpdate.Parameters.Add("@w", DbType.Int32, 4, "MediaWidth");
            cmdUpdate.Parameters.Add("@h", DbType.Int32, 4, "MediaHeight");
            cmdUpdate.Parameters.Add("@b", DbType.Int32, 4, "MediaFilesizeBytes");
            SQLiteParameter param = cmdUpdate.Parameters.Add("@id", DbType.Int32);
            param.SourceColumn = "ID";
            param.SourceVersion = DataRowVersion.Original;

            adapter.SelectCommand = cmdSelect;
            adapter.UpdateCommand = cmdUpdate;

            try {
                conn.Open();
                adapter.Fill(table);
                conn.Close();
            }
            catch (Exception e) {
                Core.ExceptionHandler.HandleException(e, true);
                throw new DatabaseOperationException("", e);
            }

            foreach (DataRow row in table.Rows) {

                try {

                    using (System.Drawing.Image img = System.Drawing.Image.FromFile(row["MediaPathCurrent"].ToString())) {

                        System.IO.FileInfo fi;

                        fi = new System.IO.FileInfo(row["MediaPathCurrent"].ToString());

                        if (img != null) {

                            int width = img.Width;
                            int height = img.Height;
                            long length = fi.Length;

                            row["MediaWidth"] = width;
                            row["MediaHeight"] = height;
                            row["MediaFilesizeBytes"] = (int)length;
                        }
                    }
                }
                catch (Exception e) {
                    Core.ExceptionHandler.HandleException(e);
                    DevUtil.Print(e);
                    continue;
                }
            }                


            try {
                recordsAffected = adapter.Update(table);
            }
            catch (Exception e) {
                Core.ExceptionHandler.HandleException(e);
                throw new DatabaseOperationException("", e);
            }


        }

        return recordsAffected;
    }

      

+3


source to share


1 answer


Loading dataset from db tables in half a second or so

This is one SQL statement (so it is fast). Exclude SQL SELECT, populate dataset, do.

updating all lines with image width / height and getting file length from FileInfo took maybe 30 seconds. Good.

It's updating data in memory (just as fast), changing row x in the dataset, not talking to SQL at all.

The team adapter.Update(table);

took somewhere around 5 to 7 minutes to run.

This will update the SQL for every updated row . This is why it is slow.

however, I can't help but think that if I ran a separate update command for each individual update, it would complete much faster.



This is basically what he does anyway!


From MSDN

The update is done line by line . For each row inserted, modified, and deleted, the Update method determines the type of change that was made to it (insert, update, or delete). Depending on the type of change, the template executes the Insert, Update, or Delete command to propagate the changed row to the data source. When the application calls the Update method, the DataAdapter examines the RowState property and executes the required INSERT, UPDATE, or DELETE statements iteratively for each row , based on the order-based indexes configured in the DataSet.


is there a way to "peek" into how ADO handles this?

Yes: Debug .NET Framework source code in Visual Studio 2012?

+2


source







All Articles