How can I run a python module asynchronously and receive calls from other modules?
So, I'm currently working on adding a recommendation engine to a Django project and have to do some heavy processing (in an external module) for one of my view functions. This slows down the page load significantly because I have to load some data, transform it, do my calculations based on the parameters sent by the request, and then return the suggestions to the view. This must be done every time the view is loaded.
I was wondering if there is some way to load the recommender module and convert the data to memory, then wait for the parameters to be sent from the view, perform calculations on those parameters, and then send them back to the View.
Any help would be greatly appreciated.
source to share
Celery is a task queue that goes out of its way to trump things like this.
This will allow you to do something like:
- user makes a request to view
- view starts an async task that does the heavy lifting and then returns immediately to the user.
- you can poll javascript to see if your task is complete and load the results when it
Maybe not quite the flow you are looking for, but celery is definitely worth checking out Celery has a great django package , extremely easy to use
Rereading my question, I think it is also possible to create a local web service around the recommendation engine. On startup, it can load all data into memory, then can you just make requests to it from your django app?
source to share