Golan slow scan () for multiple lines

I am running a query in Golang where I am selecting multiple rows from my Postgresql database.

I am using the following imports for my request

"database/sql"
"github.com/lib/pq"

      

I narrowed it down to my loop for scanning the results in my structure.

// Returns about 400 rows
rows, err = db.Query('SELECT * FROM infrastructure')
if err != nil {
    return nil, err
}

var arrOfInfra []model.Infrastructure
for rows.Next() {
    obj, ptrs := model.InfrastructureInit()
    rows.Scan(ptrs...)
    arrOfInfra = append(arrOfInfra, *obj)
}
rows.Close()

      

The above code takes about 8 seconds to run, and when the request is fast the loop in rows.Next () takes a full 8 seconds.

Any ideas? Am I doing something wrong, or is there a better way?

My configuration for my database

// host, port, dbname, user, password masked for obvious reasons
db, err := sql.Open("postgres", "host=... port=... dbname=... user=... password=... sslmode=require")
if err != nil {
    panic(err)
}

// I have tried using the default, or setting to high number (100), but it doesn't seem to help with my situation
db.SetMaxIdleConns(1)
db.SetMaxOpenConns(1)

      


UPDATE 1:

I have placed the print statements in the for loop. Below is my updated snippet

for rows.Next() {
    obj, ptrs := model.InfrastructureInit()
    rows.Scan(ptrs...)
    arrOfInfra = append(arrOfInfra, *obj)
    fmt.Println("Len: " + fmt.Sprint(len(arrOfInfra)))
    fmt.Println(obj)
}

      

I noticed that in this loop it will pause halfway and continue after a short break. It looks like this:

Len: 221
Len: 222
Len: 223
Len: 224
<a short pause about 1 second, then prints Len: 225 and continues>
Len: 226
Len: 227
...
..
.

      

and this will happen again later in another row count and again after several hundred records.


UPDATE 2:

Below is a snippet of my InfrastructureInit () method

func InfrastructureInit() (*Infrastructure, []interface{}) {
    irf := new(Infrastructure)
    var ptrs []interface{}
    ptrs = append(ptrs,
        &irf.Base.ID,
        &irf.Base.CreatedAt,
        &irf.Base.UpdatedAt,
        &irf.ListingID,
        &irf.AddressID,
        &irf.Type,
        &irf.Name,
        &irf.Description,
        &irf.Details,
        &irf.TravellingFor,
    )
    return irf, ptrs
}

      

I'm not entirely sure what is causing this slowness, but I was currently hosting a quick patch on my server to use the redis database and fix my infrastructures by storing it as a string. Everything seems to be fine now, but now I have to support both redis and my postgres.

I'm still puzzled about this weird behavior, but I'm not quite sure how rows.Next () works - does it make a query to the database every time I call rows.Next ()?

+3


source to share





All Articles