Speed up this search / filtering operation - (VB6, TextFile, ADO, VFP 6.0 database)
I am trying to figure out how to speed up this operation. Before importing a record from a text file, I first need to see if it exists in the database. If it exists, I'm going to perform an update operation. If it doesn't exist, I'm going to create a new entry.
Running the code that you see below this operation takes somewhere around 3 hours.
I've tried using ADO's search method and it actually looks slower than the filter method.
The database is a Visual Foxpro 6 database. The table has an index on the item_cd field, but the table does not have a primary key. This is beyond my power since I did not write software and I try to stay away from any structural changes to the database.
There are 46652 lines in a text file and about 650,000 records / lines in the ADO Recordset. I think slimming down the recordset will be the biggest step in doing this, but I didn't come up with it. I am trying to prevent duplicate records from being created since there is no primary key, so I really need to have the entire table in my recordset.
Since I am running this on my local machine, it looks like the operation is CPU-bound. This can actually be used all over the web, especially if I can get it to go faster.
Dim sFileToImport As String
sFileToImport = Me.lstFiles.Text
If sFileToImport = "" Then
MsgBox "You must select a file from the listbox to import."
Exit Sub
End If
If fConnectToDatabase = False Then Exit Sub
With gXRst
.CursorLocation = adUseClient
.CursorType = adOpenKeyset
.LockType = adLockReadOnly
.Open "SELECT item_cd FROM xmsalinv ORDER BY item_cd ASC", gXCon
End With
Call fStartProgress("Running speed test.")
Dim rstTxtFile As ADODB.Recordset
Set rstTxtFile = New ADODB.Recordset
Dim con As ADODB.Connection
Set con = New ADODB.Connection
Dim sConString As String, sSQL As String
Dim lRecCount As Long, l As Long
Dim s As String
sConString = "DRIVER={Microsoft Text Driver (*.txt; *.csv)};Dbq=" & gsImportFolderPath & ";Extensions=asc,csv,tab,txt;Persist Security Info=False;"
con.Open sConString
sSQL = "SELECT * FROM [" & sFileToImport & "]"
rstTxtFile.Open sSQL, con, adOpenKeyset, adLockPessimistic
If Not (rstTxtFile.EOF And rstTxtFile.BOF) = True Then
rstTxtFile.MoveFirst
lRecCount = rstTxtFile.RecordCount
Do Until rstTxtFile.EOF = True
'This code appears to actually be slower than the filter method I'm now using
'gXRst.MoveFirst
'gXRst.Find "item_cd = '" & fPQ(Trim(rstTxtFile(0))) & "'"
gXRst.Filter = "item_cd = '" & fPQ(Trim(rstTxtFile(0))) & "'"
If Not (gXRst.EOF And gXRst.BOF) = True Then
s = "Item Found - " & Trim(rstTxtFile(0)) 'item found
Else
s = "Item Not Found - " & Trim(rstTxtFile(0)) 'Item not found found
End If
l = l + 1
Call subProgress(l, lRecCount, s)
rstTxtFile.MoveNext
Loop
End If
Call fEndProgress("Finished running speed test.")
Cleanup:
rstTxtFile.Close
Set rstTxtFile = Nothing
gXRst.Close
source to share
In response to Bob Rimersma's post, the text file does not cause speed issues. I changed my code to open a Recordset with a single item search query. This code now works after 1 minute and 2 seconds, as opposed to three to four hours I was looking at the other way.
Dim sFileToImport As String
sFileToImport = Me.lstFiles.Text
If sFileToImport = "" Then
MsgBox "You must select a file from the listbox to import."
Exit Sub
End If
If fConnectToDatabase = False Then Exit Sub
Call fStartProgress("Running speed test.")
Dim rstTxtFile As ADODB.Recordset
Set rstTxtFile = New ADODB.Recordset
Dim con As ADODB.Connection
Set con = New ADODB.Connection
Dim sConString As String, sSQL As String
Dim lRecCount As Long, l As Long
Dim sngQty As Single, sItemCat As String
sConString = "DRIVER={Microsoft Text Driver (*.txt; *.csv)};Dbq=" & gsImportFolderPath & ";Extensions=asc,csv,tab,txt;Persist Security Info=False;"
con.Open sConString
sSQL = "SELECT * FROM [" & sFileToImport & "]"
rstTxtFile.Open sSQL, con, adOpenKeyset, adLockPessimistic
If Not (rstTxtFile.EOF And rstTxtFile.BOF) = True Then
rstTxtFile.MoveFirst
lRecCount = rstTxtFile.RecordCount
Do Until rstTxtFile.EOF = True
l = l + 1
sItemCat = fItemCat(Trim(rstTxtFile(0)))
If sItemCat <> "[item not found]" Then
sngQty = fItemQty(Trim(rstTxtFile(0)))
End If
Call subProgress(l, lRecCount, sngQty & " - " & sItemCat & " - " & rstTxtFile(0))
sngQty = 0
rstTxtFile.MoveNext
Loop
End If
Call fEndProgress("Finished running speed test.")
Cleanup:
rstTxtFile.Close
Set rstTxtFile = Nothing
My functions:
Private Function fItemCat(sItem_cd As String) As String
'Returns blank if nothing found
If sItem_cd <> "" Then
With gXRstFind
.CursorLocation = adUseClient
.CursorType = adOpenKeyset
.LockType = adLockReadOnly
.Open "SELECT item_cd, ccategory FROM xmsalinv WHERE item_cd = '" & fPQ(sItem_cd) & "'", gXCon
End With
If Not (gXRstFind.EOF And gXRstFind.BOF) = True Then
'An item can technically have a blank category although it never should have
If gXRstFind!ccategory = "" Then
fItemCat = "[blank]"
Else
fItemCat = gXRstFind!ccategory
End If
Else
fItemCat = "[item not found]"
End If
gXRstFind.Close
End If
End Function
Private Function fIsStockItem(sItem_cd As String, Optional bConsiderItemsInStockAsStockItems As Boolean = False) As Boolean
If sItem_cd <> "" Then
With gXRstFind
.CursorLocation = adUseClient
.CursorType = adOpenKeyset
.LockType = adLockReadOnly
.Open "SELECT item_cd, bal_qty, sug_qty FROM xmsalinv WHERE item_cd = '" & fPQ(sItem_cd) & "'", gXCon
End With
If Not (gXRstFind.EOF And gXRstFind.BOF) = True Then
If gXRstFind!sug_qty > 0 Then
fIsStockItem = True
Else
If bConsiderItemsInStockAsStockItems = True Then
If gXRstFind!bal_qty > 0 Then
fIsStockItem = True
End If
End If
End If
End If
gXRstFind.Close
End If
End Function
Private Function fItemQty(sItem_cd As String) As Single
'Returns 0 if nothing found
If sItem_cd <> "" Then
With gXRstFind
.CursorLocation = adUseClient
.CursorType = adOpenKeyset
.LockType = adLockReadOnly
.Open "SELECT item_cd, bal_qty FROM xmsalinv WHERE item_cd = '" & fPQ(sItem_cd) & "'", gXCon
End With
If Not (gXRstFind.EOF And gXRstFind.BOF) = True Then
fItemQty = CSng(gXRstFind!bal_qty)
End If
gXRstFind.Close
End If
End Function
source to share
A simple solution to speed up Yours_Rs.find's response is to use Yours_Rs.move operator first, if possible for you. What I did was use the MyRs.move expression prior to using MyRs.find to get closer to my actual entry. This actually worked for me as the move operator's response is quite lively.
I used it to find a patient record. Thus, moving the pointer to the record near the actual record made the MyRs.find expression operate at the speed of light.
Respectfully,
MAS.
source to share
doesn't answer your question and this is a pretty old thread, but why don't you import a text file into a temp table on your db and then join? something like SELECT tt. * FROM texttemp tt left external join xmsalinv xal to tt.field1 = xal.item_cd where xal.item_cd is null
this should return the content of the imported text file, which does not have any item_cd matches in the database, since you are dealing with a text file that complicates the query, so I wonder not to import the content into a temporary table.
now, assuming you know the field mapping, you can probably also use that to insert, assuming your db accepts an insert to select the insert it will insert to xmsalinv (fields) select (matching fields) from (like stated above ...) this moves your shutter points into the import process, and I hope it's fast.
ado collections look like they're pretty dumb, so they don't get any data knowledge and are a bit slow.
ah next item on "vb6 filter" google http://www.techrepublic.com/article/why-ados-find-method-is-the-devil/1045830
this answer is based on basic knowledge of sql and is not intended for foxpro
source to share
Use a cursor to get the results of a VFP query if you don't, and have a look at your other post here for suggestions regarding text file recordset.
Perhaps even better, but you can try to get rid of your slow "loop and search" aproach.
I would probably create a temporary Jet 4.0 MDB from scratch for every text file you want to search for. Import text data by specifying your key field. Use ADOX to define a related table in a VFP database. Use a query to perform a search.
Then close and delete MDB.
source to share
First, you can try to create an index in memory item_cd
with gXRst!item_cd.Properties("OPTIMIZE").Value = True
which accelerate both Find
, and so Filter
.
For maximum speed in searches, initialize the helper index Collection
like this
Set cIndex = New Collection
On Error Resume Next
Do While Not gXRst.EOF
cIndex.Add gXRst.Bookmark, "#" & gXRst!item_cd.Value
gXRst.MoveNext
Loop
On Error GoTo ErrorHandler
And insert Find
uses some function like this
Public Function SearchCollection(Col As Object, Index As Variant) As Boolean
On Error Resume Next
IsObject Col(Index)
SearchCollection = (Err.Number = 0)
On Error GoTo 0
End Function
source to share
3 hours in just a few hundred thousand records !!! You are doing it wrong. Just: -append the text file to the VFP table - then insert the ones that don't exist in the existing table with one SQL - and update those that exist with another updated sql.
All this will take less than a minute (a minute is very slow). You can do all of this with the VFPOLEDB driver and it doesn't matter if you have a VFP6 database, VFPOLEDB has a built-in VFP9 module.
source to share