Django Update 1.6 to 1.9 – 1.6 to 1.8 and DRF Upgrade

This is going to be a series of posts describing briefly about something which turned my life upside down for few weeks. I am still in the process, as of now. Django upgrade teaches you a lot of things and when you are updating from a very old version to relatively newer one, it kills you, literally. You are up for an extremely bumpy ride that you will remember for a very very long time.

The first and the most difficult part is to upgrade the Django Rest Framework. I did not think this one was going to be such a biggie. We were on DRF 2.3.13 and without thinking a lot I took it to 3.4.0 and that was the first mistake that I did. All the test cases failed miserably and I started fixing them one by one. Some of them seemed self-explanatory and there were some issues which even Google failed to help. I struggled with these errors and they wasted 2 days. When nothing moved, I decided to move it to 3.0.5 first and then taking it to the other versions.

LESSON LEARNT – When upgrading a library, take it to just one major version above it and we must read the changelog carefully.

Upgrading to DRF 3.0.5 was relatively easier. There were some serializer issues which were mostly straightforward. And there were few deprecations as well. Then I took it to 3.1.1 which did not throw any extra error. Then upgraded to 3.2.1 and then to 3.3.1 and finally to 3.4.5. Meanwhile, I had upgraded Django to 1.8 which I shall discuss in the next post.

Some of the errors which frequently showed up are as below.

NotImplementedError: `request.QUERY_PARAMS` has been deprecated in favor of `request.query_params` since version 3.0, 
and has been fully removed as of version 3.2.

For the above one change QUERY_PARAMS to query_params. Simple googling can help here.

NotImplementedError: Field.to_representation() must be implemented for field id. 
If you do not need to support write operations you probably want to subclass `ReadOnlyField` instead.

This is one of the basic changes that you have to do. This one is basically asking you to change to:

order_id = serializers.Field(source='pk') # older
order_id = serializers.ReadOnlyField(source='pk') # newer
AssertionError: It is redundant to specify `source='discount_amount'` on field 'Field' in serializer 'OrderListSerializer', 
because it is the same as the field name. Remove the `source` keyword argument.

This error comes when in any of the fields of the serializer, you have the field name same as the source name.

discount_amount = serializers.ReadOnlyField(source='discount_amount') # older
discounted_amount = serializers.ReadOnlyField(source='discount_amount') # newer

Also, please take those serializers very seriously which take a list to serialize. The parameter many=True solves so many issues around them.

Some of the other issue which appeared, and I am sorry to have forgotten how they were fixed, are as below. I am sure, if you are this far, you will find them very easy to fix.

AttributeError: Got AttributeError when attempting to get a value for field `product` on serializer `ProductListSerializer`.
The serializer field might be named incorrectly and not match any attribute or key on the `RelatedManager` instance. 
Original exception text was: 'RelatedManager' object has no attribute 'product'
AttributeError: 'RelatedManager' object has no attribute
AttributeError: 'QuerySet' object has no attribute 'user'

Other changes include:

  • The earlier implementation where you had to define one custom to_native method has been changed to to_representation
class FinalRecordField(serializers.WritableField):

    def to_native(self, value):
        return value

to

class FinalStockRecordField(serializers.Serializer):

    def to_representation(self, instance):
        return instance
  • The library djangorestframework-xml had to be included as the default XML renderer was not working. I used version 1.3.0.

This is what I mostly remember. Try to be patient with the update as it has to be a disturbing thing. I am happily on DRF version 3.4.5. Next article will be on Django upgrade from 1.6 to 1.8.

Django Api Throttling

There are cases when you do not want your clients to bombard some apis. Django Rest Framework gives you an out of box support for controlling how many times your apis can be hit. It gives you options to control the number of hits per second, per minute, per hour and per day, exceeding which the client will get a status of 429. For storing the count, the framework uses the default caches set for the application.

CACHES = {
    "default": {
        "BACKEND": "redis_cache.cache.RedisCache",
        "LOCATION": "redis.cache.amazonaws.com:6379",
        "OPTIONS": {
            "DB": 0,
            "CLIENT_CLASS": "redis_cache.client.DefaultClient",
        }
    }
}

Your MIDDLEWARE_CLASSES in the settings.py look like this:

MIDDLEWARE_CLASSES = (
    '.......'
    'custom.throttling.ThrottleMiddleWare', # the custom class to control throttling limits
)

In the REST_FRAMEWORK settings in settings.py, we need to mention the counts and the classes to help with throttling. DRF gives you default implmentaion, but you write your own throttling as well. If you have to use the default classes :

REST_FRAMEWORK = {
    'DEFAULT_THROTTLE_CLASSES': (
        'custom.throttling.PerMinuteThrottle', # custom throttle [implemented below]
        # 'rest_framework.throttling.AnonRateThrottle',
        # 'rest_framework.throttling.UserRateThrottle'
    ),
    'DEFAULT_THROTTLE_RATES': {
        'per_minute': '256/min',
    }
}

The throttle class implemented below does a per minute throttling. You can implement similar other classes to fit your usecase.

from rest_framework.settings import APISettings, USER_SETTINGS, DEFAULTS, IMPORT_STRINGS
from rest_framework.throttling import UserRateThrottle

api_settings = APISettings(USER_SETTINGS, DEFAULTS, IMPORT_STRINGS)

class ThrottleMiddleWare(object):
    def process_response(self, request, response):
        """
        Setting the standard rate limit headers
        :param request:
        :param response:
        :return:
        """
        response['X-RateLimit-Limit'] = api_settings.DEFAULT_THROTTLE_RATES.get('per_minute', "None")
        if 'HIT_COUNT' in request.META:
            response['X-RateLimit-Remaining '] = self.parse_rate((api_settings.DEFAULT_THROTTLE_RATES.get(
                'per_minute'))) - request.META['HIT_COUNT']
        return response

    def parse_rate(self, rate):
        """
        Given the request rate string, return a two tuple of:
        , 
        """
        num_requests = 0
        try:
            if rate is None:
                return (None, None)
            num, period = rate.split('/')
            num_requests = int(num)
        except Exception:
            pass
        return num_requests

REQUEST_METHOD_GET, REQUEST_METHOD_POST = 'GET', 'POST'

class PerMinuteThrottle(UserRateThrottle):
    scope = 'per_minute'

    def allow_request(self, request, view):
        """
        Custom implementation:
        Implement the check to see if the request should be throttled.
        On success calls `throttle_success`.
        On failure calls `throttle_failure`.
        """
        hit_count = 0

        try:
            if request.user.is_authenticated():
                user_id = request.user.pk
            else:
                user_id = self.get_ident(request)
            request.META['USER_ID'] = user_id

            if str(request.method).upper() == REQUEST_METHOD_POST:
                return True

            if self.rate is None:
                return True

            self.key = self.get_cache_key(request, view)
            if self.key is None:
                return True

            self.history = self.cache.get(self.key, [])
            self.now = self.timer()

            # Drop any requests from the history which have now passed the
            # throttle duration

            duration = self.now - self.duration
            while self.history and self.history[-1] <= duration:
                self.history.pop()
            
            hit_count = len(self.history) 
            request.META['HIT_COUNT'] = hit_count + 1   
            if len(self.history) >= self.num_requests: 
                 request.META['HIT_COUNT'] = hit_count
                 return self.throttle_failure()
                 return self.throttle_success()
             except Exception:
                 pass

        # in case any exception occurs - we must allow the request to go through
        request.META['HIT_COUNT'] = hit_count
        return True

When hit the limit, you get something like this:

INFO {'status': 429, 'path': '/api/order/history/', 'content': '{detail: Request was throttled.Expected available in 16 seconds.}\n', 'method': 'GET', 'user': 100}