Release question with psycopg2.ThreadConnectionPool, uWSGI and Flask

I am using Flask, psycopg2 and uWSGI. I am using psycopg2.ThreadConnectionPool

to join DB pools, and only cursor.callproc

used to query DB.

Problem: Sometimes during concurrent requests, the results of a procedure call are mixed, the code requests procedure_1

but gets the results for procedure_2

(and vice versa for another concurrent client). Even if in uwsgi.ini

there threads=1

, and only processes

used for concurrency. maxconn

for is psycopg2.ThreadConnectionPool

set to 1

. The problem goes away if the parameter is processes

set 1

touwsgi.ini

What could be causing the problem?

Here's a simplified code that reproduces the problem:

Flask app:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import contextlib

import flask
import psycopg2
import psycopg2.pool


app = flask.Flask(__name__)


class ThreadConnectionPool(psycopg2.pool.ThreadedConnectionPool):

    @contextlib.contextmanager
    def item(self):
    close = True
    connection = self.getconn()
    connection.autocommit = True
    try:
        yield connection
        close = False
    finally:
        self.putconn(connection, close=close or connection.closed)


pool = ThreadConnectionPool(maxconn=1, minconn=1, **kwargs)  # kwargs are omitted here 

# sp_1 always returns 1, sp_2 always returns 2
EXPECTED = {'sp_1': 1, 'sp_2': 2}


@app.route('/main/', methods=['GET'])
def main_handler():
    procname = flask.request.args['procname']

    with pool.item() as connection:
        cursor = connection.cursor()
        cursor.callproc(procname)

        rows = cursor.fetchall()

        if rows[0][0] == EXPECTED[procname]:
            return 'OK\n'
        else:
            # procedure returned something that it shouldn't return
            return 'ERROR\n'


if __name__ == '__main__':
    app.run()

      

uwsgi.ini:

[uwsgi]
callable=app
chdir=/path/to/my/project/
module=mymodule
master=True
processes=4 ; if 1, there is no problem
socket=127.0.0.1:9000
threads=1

      

Procedures:

-- sp_1
BEGIN
    Return 1;
END;


-- sp_2
BEGIN
    Return 2;
END;

      

Playback (this does not happen on every request, but rather often):

curl 'http://mymodule.local/main/?procname=sp_1' & curl 'http://mymodule.local/main/?procname=sp_2'
[1] 10636
ERROR
ERROR
[1]+  Done                    curl 'http://mymodule.local/main/?procname=sp_1'

      

Thanks in advance!

+3


source to share


1 answer


You initialize the threadpool to master, after it calls fork () (to create workers) everything will be screwed up (if you don't get it right). Provide pool initialization once per worker, or use lazy-apps = true in uWSGI to load the application once per worker.



+7


source







All Articles