Django Rest Framework bulk breakdown is extremely slow
I turned pagination into Django Rest frameworks and it seems incredibly slow. The graph looks like a criminal and takes hundreds of milliseconds to return every time due to the millions of rows in the tables.
I am using postgresql as my database. Is there a way to skip counting lines and still use pagination? The performance was fine until it was enabled if I manually filtered the request.
source to share
Override the method of get_paginated_response
your pagination class and don't include counting. You can refer to the base implementation of the class PageNumberPagination
to see what you should return.
from rest_framework.pagination import PageNumberPagination
from collections import OrderedDict # requires Python 2.7 or later
class PageNumberPaginationWithoutCount(PageNumberPagination):
# Set any other options you want here like page_size
def get_paginated_response(self, data):
return Response(OrderedDict([
('next', self.get_next_link()),
('previous', self.get_previous_link()),
('results', data)
]))
Then settings.py
set DEFAULT_PAGINATION_CLASS
pagination to your new class.
DEFAULT_PAGINATION_CLASS = 'path.to.PageNumberPaginationWithoutCount'
This approach is used in the example in pagination documents .
Edit: From the comments below, it sounds like it might not be enough to prevent a slow SQL query, so you might need to override paginate_queryset
.
source to share
The problem is that the query used to count is as potentially complex as it is to retrieve data. This is pretty wasteful. PageNumberPagination
internally uses Django's own Paginator
.
To make the request for the counter easier by overriding the paginator class, DRF uses:
from django.core.paginator import Paginator
from django.utils.functional import cached_property
from rest_framework.pagination import PageNumberPagination
class FasterDjangoPaginator(Paginator):
@cached_property
def count(self):
# only select 'id' for counting, much cheaper
return self.object_list.values('id').count()
class FasterPageNumberPagination(PageNumberPagination):
django_paginator_class = FasterDjangoPaginator
source to share
If you are ok with no counters, following and previous references, the following custom class can be used.
import sys
from collections import OrderedDict
from django.core.paginator import Paginator
from django.utils.functional import cached_property
from rest_framework.pagination import PageNumberPagination
from rest_framework.response import Response
class CustomPaginatorClass(Paginator):
@cached_property
def count(self):
return sys.maxsize
# To Avoid large table count query, We can use this paginator class
class LargeTablePagination(PageNumberPagination):
django_paginator_class = CustomPaginatorClass
def get_paginated_response(self, data):
return Response(OrderedDict([
('page', self.page.number),
('results', data)
]))
source to share