Release question with psycopg2.ThreadConnectionPool, uWSGI and Flask
I am using Flask, psycopg2 and uWSGI. I am using psycopg2.ThreadConnectionPool
to join DB pools, and only cursor.callproc
used to query DB.
Problem: Sometimes during concurrent requests, the results of a procedure call are mixed, the code requests procedure_1
but gets the results for procedure_2
(and vice versa for another concurrent client). Even if in uwsgi.ini
there threads=1
, and only processes
used for concurrency. maxconn
for is psycopg2.ThreadConnectionPool
set to 1
. The problem goes away if the parameter is processes
set 1
touwsgi.ini
What could be causing the problem?
Here's a simplified code that reproduces the problem:
Flask app:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import contextlib
import flask
import psycopg2
import psycopg2.pool
app = flask.Flask(__name__)
class ThreadConnectionPool(psycopg2.pool.ThreadedConnectionPool):
@contextlib.contextmanager
def item(self):
close = True
connection = self.getconn()
connection.autocommit = True
try:
yield connection
close = False
finally:
self.putconn(connection, close=close or connection.closed)
pool = ThreadConnectionPool(maxconn=1, minconn=1, **kwargs) # kwargs are omitted here
# sp_1 always returns 1, sp_2 always returns 2
EXPECTED = {'sp_1': 1, 'sp_2': 2}
@app.route('/main/', methods=['GET'])
def main_handler():
procname = flask.request.args['procname']
with pool.item() as connection:
cursor = connection.cursor()
cursor.callproc(procname)
rows = cursor.fetchall()
if rows[0][0] == EXPECTED[procname]:
return 'OK\n'
else:
# procedure returned something that it shouldn't return
return 'ERROR\n'
if __name__ == '__main__':
app.run()
uwsgi.ini:
[uwsgi]
callable=app
chdir=/path/to/my/project/
module=mymodule
master=True
processes=4 ; if 1, there is no problem
socket=127.0.0.1:9000
threads=1
Procedures:
-- sp_1
BEGIN
Return 1;
END;
-- sp_2
BEGIN
Return 2;
END;
Playback (this does not happen on every request, but rather often):
curl 'http://mymodule.local/main/?procname=sp_1' & curl 'http://mymodule.local/main/?procname=sp_2'
[1] 10636
ERROR
ERROR
[1]+ Done curl 'http://mymodule.local/main/?procname=sp_1'
Thanks in advance!
source to share