Golan slow scan () for multiple lines
I am running a query in Golang where I am selecting multiple rows from my Postgresql database.
I am using the following imports for my request
"database/sql"
"github.com/lib/pq"
I narrowed it down to my loop for scanning the results in my structure.
// Returns about 400 rows
rows, err = db.Query('SELECT * FROM infrastructure')
if err != nil {
return nil, err
}
var arrOfInfra []model.Infrastructure
for rows.Next() {
obj, ptrs := model.InfrastructureInit()
rows.Scan(ptrs...)
arrOfInfra = append(arrOfInfra, *obj)
}
rows.Close()
The above code takes about 8 seconds to run, and when the request is fast the loop in rows.Next () takes a full 8 seconds.
Any ideas? Am I doing something wrong, or is there a better way?
My configuration for my database
// host, port, dbname, user, password masked for obvious reasons
db, err := sql.Open("postgres", "host=... port=... dbname=... user=... password=... sslmode=require")
if err != nil {
panic(err)
}
// I have tried using the default, or setting to high number (100), but it doesn't seem to help with my situation
db.SetMaxIdleConns(1)
db.SetMaxOpenConns(1)
UPDATE 1:
I have placed the print statements in the for loop. Below is my updated snippet
for rows.Next() {
obj, ptrs := model.InfrastructureInit()
rows.Scan(ptrs...)
arrOfInfra = append(arrOfInfra, *obj)
fmt.Println("Len: " + fmt.Sprint(len(arrOfInfra)))
fmt.Println(obj)
}
I noticed that in this loop it will pause halfway and continue after a short break. It looks like this:
Len: 221
Len: 222
Len: 223
Len: 224
<a short pause about 1 second, then prints Len: 225 and continues>
Len: 226
Len: 227
...
..
.
and this will happen again later in another row count and again after several hundred records.
UPDATE 2:
Below is a snippet of my InfrastructureInit () method
func InfrastructureInit() (*Infrastructure, []interface{}) {
irf := new(Infrastructure)
var ptrs []interface{}
ptrs = append(ptrs,
&irf.Base.ID,
&irf.Base.CreatedAt,
&irf.Base.UpdatedAt,
&irf.ListingID,
&irf.AddressID,
&irf.Type,
&irf.Name,
&irf.Description,
&irf.Details,
&irf.TravellingFor,
)
return irf, ptrs
}
I'm not entirely sure what is causing this slowness, but I was currently hosting a quick patch on my server to use the redis database and fix my infrastructures by storing it as a string. Everything seems to be fine now, but now I have to support both redis and my postgres.
I'm still puzzled about this weird behavior, but I'm not quite sure how rows.Next () works - does it make a query to the database every time I call rows.Next ()?
source to share
No one has answered this question yet
Check out similar questions: