Template with flags with Jinja

I have a flask app. In a specific view, I am showing a table that contains about 100k rows. It is clear that the page takes a long time to load and I am looking for ways to improve it. So far I figured I was querying the database and getting the result pretty quickly. I think the problem is with the rendering of the actual page. I found this page on streaming and am trying to work with it but keep running into problems all the time. I tried the stream_template solution provided there with this code:

@app.route('/thing/matches', methods = ['GET', 'POST'])
@roles_accepted('admin', 'team')
def r_matches():
    matches = Match.query.filter(
            Match.name == g.name).order_by(Match.name).all()

    return Response(stream_template('/retailer/matches.html',
        dashboard_title = g.name,
        match_show_option = True,
        match_form = form,
        matches = matches))

def stream_template(template_name, **context):
    app.update_template_context(context)
    t = app.jinja_env.get_template(template_name)
    rv = t.stream(context)
    rv.enable_buffering(5)
    return rv

      

A match query is one that returns 100k + items. However, whenever I run this page, the page just appears blank, there is nothing there. I also tried a solution with streaming data to json and loading it via ajax, but nothing seems to be found in the json file! This is what the solution looks like:

@app.route('/large.json')
def generate_large_json():
    def generate():
        app.logger.info("Generating JSON")
        matches = Product.query.join(Category).filter(
            Product.retailer == g.retailer,
            Product.match != None).order_by(Product.name)
        for match in matches:
            yield json.dumps(match)
    app.logger.info("Sending file response")
    return Response(stream_with_context(generate()))

      

Another solution I looked at was for pagination. This solution works well, except that I have to be able to sort the entire dataset using headers and couldn't find a way to do it without rendering the entire dataset in a table and then using JQuery to sort / paginate.

The file I get by going to /large.json is always empty. Please help or recommend another way to display such a large dataset!

Edit: I got the generate () part to work and update the code.

+3


source to share


1 answer


The problem in both cases is almost certainly that you are hooking on 100K + objects Match

and storing them in memory. You will also want to pipe the results from the DB using yield_per

. However, only PostGres + psycopg2 supports the required stream_result

argument ( here's a way to do it with MySQL ):

matches = Match.query.filter(
        Match.name == g.name).order_by(Match.name).yield_per(10)
        # Stream ten results at a time

      

Alternative



If you are using Flask-SQLAlchemy you can use a Pagination

class
to split the request to the server and not load all 100K + records in the browser. This has the added benefit of not requiring the browser to manage all of the DOM entries (assuming you are doing the HTML streaming option).

see also

+1


source







All Articles