Speed ​​up this search / filtering operation - (VB6, TextFile, ADO, VFP 6.0 database)

I am trying to figure out how to speed up this operation. Before importing a record from a text file, I first need to see if it exists in the database. If it exists, I'm going to perform an update operation. If it doesn't exist, I'm going to create a new entry.

Running the code that you see below this operation takes somewhere around 3 hours.

I've tried using ADO's search method and it actually looks slower than the filter method.

The database is a Visual Foxpro 6 database. The table has an index on the item_cd field, but the table does not have a primary key. This is beyond my power since I did not write software and I try to stay away from any structural changes to the database.

There are 46652 lines in a text file and about 650,000 records / lines in the ADO Recordset. I think slimming down the recordset will be the biggest step in doing this, but I didn't come up with it. I am trying to prevent duplicate records from being created since there is no primary key, so I really need to have the entire table in my recordset.

Since I am running this on my local machine, it looks like the operation is CPU-bound. This can actually be used all over the web, especially if I can get it to go faster.

Dim sFileToImport As String
sFileToImport = Me.lstFiles.Text
If sFileToImport = "" Then
    MsgBox "You must select a file from the listbox to import."
    Exit Sub
End If

If fConnectToDatabase = False Then Exit Sub

With gXRst
    .CursorLocation = adUseClient
    .CursorType = adOpenKeyset
    .LockType = adLockReadOnly
    .Open "SELECT item_cd FROM xmsalinv ORDER BY item_cd ASC", gXCon
End With



Call fStartProgress("Running speed test.")

Dim rstTxtFile As ADODB.Recordset
Set rstTxtFile = New ADODB.Recordset
Dim con As ADODB.Connection
Set con = New ADODB.Connection

Dim sConString As String, sSQL As String
Dim lRecCount As Long, l As Long
Dim s As String

sConString = "DRIVER={Microsoft Text Driver (*.txt; *.csv)};Dbq=" & gsImportFolderPath & ";Extensions=asc,csv,tab,txt;Persist Security Info=False;"
con.Open sConString

sSQL = "SELECT * FROM [" & sFileToImport & "]"

rstTxtFile.Open sSQL, con, adOpenKeyset, adLockPessimistic
If Not (rstTxtFile.EOF And rstTxtFile.BOF) = True Then
    rstTxtFile.MoveFirst
    lRecCount = rstTxtFile.RecordCount
    Do Until rstTxtFile.EOF = True

        'This code appears to actually be slower than the filter method I'm now using
        'gXRst.MoveFirst
        'gXRst.Find "item_cd = '" & fPQ(Trim(rstTxtFile(0))) & "'"

        gXRst.Filter = "item_cd = '" & fPQ(Trim(rstTxtFile(0))) & "'"
        If Not (gXRst.EOF And gXRst.BOF) = True Then
            s = "Item Found  -  " & Trim(rstTxtFile(0)) 'item found
        Else
           s = "Item Not Found  -  " & Trim(rstTxtFile(0)) 'Item not found found
        End If
        l = l + 1
        Call subProgress(l, lRecCount, s)
        rstTxtFile.MoveNext
    Loop
End If

Call fEndProgress("Finished running speed test.")

Cleanup:
    rstTxtFile.Close
    Set rstTxtFile = Nothing
    gXRst.Close

      

0


source to share


6 answers


In response to Bob Rimersma's post, the text file does not cause speed issues. I changed my code to open a Recordset with a single item search query. This code now works after 1 minute and 2 seconds, as opposed to three to four hours I was looking at the other way.

Dim sFileToImport As String
sFileToImport = Me.lstFiles.Text
If sFileToImport = "" Then
    MsgBox "You must select a file from the listbox to import."
    Exit Sub
End If

If fConnectToDatabase = False Then Exit Sub


Call fStartProgress("Running speed test.")

Dim rstTxtFile As ADODB.Recordset
Set rstTxtFile = New ADODB.Recordset
Dim con As ADODB.Connection
Set con = New ADODB.Connection

Dim sConString As String, sSQL As String
Dim lRecCount As Long, l As Long
Dim sngQty As Single, sItemCat As String

sConString = "DRIVER={Microsoft Text Driver (*.txt; *.csv)};Dbq=" & gsImportFolderPath & ";Extensions=asc,csv,tab,txt;Persist Security Info=False;"
con.Open sConString

sSQL = "SELECT * FROM [" & sFileToImport & "]"

rstTxtFile.Open sSQL, con, adOpenKeyset, adLockPessimistic

If Not (rstTxtFile.EOF And rstTxtFile.BOF) = True Then
    rstTxtFile.MoveFirst
    lRecCount = rstTxtFile.RecordCount
    Do Until rstTxtFile.EOF = True
        l = l + 1
        sItemCat = fItemCat(Trim(rstTxtFile(0)))
        If sItemCat <> "[item not found]" Then
           sngQty = fItemQty(Trim(rstTxtFile(0)))
        End If
        Call subProgress(l, lRecCount, sngQty & " - " & sItemCat & " - " & rstTxtFile(0))
        sngQty = 0
        rstTxtFile.MoveNext
    Loop
End If

Call fEndProgress("Finished running speed test.")

Cleanup:
    rstTxtFile.Close
    Set rstTxtFile = Nothing

      



My functions:

Private Function fItemCat(sItem_cd As String) As String

    'Returns blank if nothing found

    If sItem_cd <> "" Then

        With gXRstFind
            .CursorLocation = adUseClient
            .CursorType = adOpenKeyset
            .LockType = adLockReadOnly
            .Open "SELECT item_cd, ccategory FROM xmsalinv WHERE item_cd = '" & fPQ(sItem_cd) & "'", gXCon
        End With
        If Not (gXRstFind.EOF And gXRstFind.BOF) = True Then
            'An item can technically have a blank category although it never should have
            If gXRstFind!ccategory = "" Then
                fItemCat = "[blank]"
            Else
                fItemCat = gXRstFind!ccategory
            End If
        Else
           fItemCat = "[item not found]"
        End If
        gXRstFind.Close
    End If

End Function

Private Function fIsStockItem(sItem_cd As String, Optional bConsiderItemsInStockAsStockItems As Boolean = False) As Boolean

    If sItem_cd <> "" Then

        With gXRstFind
            .CursorLocation = adUseClient
            .CursorType = adOpenKeyset
            .LockType = adLockReadOnly
            .Open "SELECT item_cd, bal_qty, sug_qty FROM xmsalinv WHERE item_cd = '" & fPQ(sItem_cd) & "'", gXCon
        End With
        If Not (gXRstFind.EOF And gXRstFind.BOF) = True Then
            If gXRstFind!sug_qty > 0 Then
                fIsStockItem = True
            Else
                If bConsiderItemsInStockAsStockItems = True Then
                    If gXRstFind!bal_qty > 0 Then
                        fIsStockItem = True
                    End If
                End If
            End If
        End If
        gXRstFind.Close
    End If

End Function


Private Function fItemQty(sItem_cd As String) As Single

    'Returns 0 if nothing found

    If sItem_cd <> "" Then

        With gXRstFind
            .CursorLocation = adUseClient
            .CursorType = adOpenKeyset
            .LockType = adLockReadOnly
            .Open "SELECT item_cd, bal_qty FROM xmsalinv WHERE item_cd = '" & fPQ(sItem_cd) & "'", gXCon
        End With
        If Not (gXRstFind.EOF And gXRstFind.BOF) = True Then
            fItemQty = CSng(gXRstFind!bal_qty)
        End If
        gXRstFind.Close
    End If

End Function

      

0


source


A simple solution to speed up Yours_Rs.find's response is to use Yours_Rs.move operator first, if possible for you. What I did was use the MyRs.move expression prior to using MyRs.find to get closer to my actual entry. This actually worked for me as the move operator's response is quite lively.

I used it to find a patient record. Thus, moving the pointer to the record near the actual record made the MyRs.find expression operate at the speed of light.



Respectfully,

MAS.

+1


source


doesn't answer your question and this is a pretty old thread, but why don't you import a text file into a temp table on your db and then join? something like SELECT tt. * FROM texttemp tt left external join xmsalinv xal to tt.field1 = xal.item_cd where xal.item_cd is null

this should return the content of the imported text file, which does not have any item_cd matches in the database, since you are dealing with a text file that complicates the query, so I wonder not to import the content into a temporary table.

now, assuming you know the field mapping, you can probably also use that to insert, assuming your db accepts an insert to select the insert it will insert to xmsalinv (fields) select (matching fields) from (like stated above ...) this moves your shutter points into the import process, and I hope it's fast.

ado collections look like they're pretty dumb, so they don't get any data knowledge and are a bit slow.

ah next item on "vb6 filter" google http://www.techrepublic.com/article/why-ados-find-method-is-the-devil/1045830

this answer is based on basic knowledge of sql and is not intended for foxpro

+1


source


Use a cursor to get the results of a VFP query if you don't, and have a look at your other post here for suggestions regarding text file recordset.

Perhaps even better, but you can try to get rid of your slow "loop and search" aproach.

I would probably create a temporary Jet 4.0 MDB from scratch for every text file you want to search for. Import text data by specifying your key field. Use ADOX to define a related table in a VFP database. Use a query to perform a search.

Then close and delete MDB.

0


source


First, you can try to create an index in memory item_cd

with gXRst!item_cd.Properties("OPTIMIZE").Value = True

which accelerate both Find

, and so Filter

.

For maximum speed in searches, initialize the helper index Collection

like this

Set cIndex = New Collection
On Error Resume Next
Do While Not gXRst.EOF
    cIndex.Add gXRst.Bookmark, "#" & gXRst!item_cd.Value
    gXRst.MoveNext
Loop
On Error GoTo ErrorHandler

      

And insert Find

uses some function like this

Public Function SearchCollection(Col As Object, Index As Variant) As Boolean
    On Error Resume Next
    IsObject Col(Index)
    SearchCollection = (Err.Number = 0)
    On Error GoTo 0
End Function

      

0


source


3 hours in just a few hundred thousand records !!! You are doing it wrong. Just: -append the text file to the VFP table - then insert the ones that don't exist in the existing table with one SQL - and update those that exist with another updated sql.

All this will take less than a minute (a minute is very slow). You can do all of this with the VFPOLEDB driver and it doesn't matter if you have a VFP6 database, VFPOLEDB has a built-in VFP9 module.

0


source







All Articles