Rate Limiting Rodeo

Tyler Lubeck | Alex Lenail | Alison Tai | Nick Teleky

Challenge Area

Security: API Rate Limiting

• Constant hits to an API endpoint can slow down the server
• Explore an area of DDOS without attacking the whole problem

Companies that face this: Github

• Time Limit: 1 hour
• per application
  • Authenticated Apps: 5000 requests
  • Unauthenticated Apps: 60 requests

Twitter

• Time Limit: 15 minutes
• Reads: per users (to whom the application has an access token) per application
• Search: per application
• Writes: per user
• Number of requests varied based on type of request

Facebook

• by API key
• by user
• by application

How We Compare

Facebook

Twitter

Github

Our Solution

Our Solution

• Time Limit: 1 second
• Limit the rate at which programmers can hit application endpoints
• Plugin for Django that provides a Python function decorator for Django views
• Functionality is configurable per individual view

Our Solution Code

						from django.core.cache import get_cache
from functools import wraps
from django.utils.decorators import available_attrs
import logging
from django.http import HttpResponse
import json
logger = logging.getLogger('django.request')
#Get a cache instance that is specific for our plugin
cache = get_cache('rate_limiting')

def rate_limit_by_ip(how_many_hits=50, in_how_long=1, exception_list=[]):
    """
    Django decorator to limit how often an ip can acess a view.
    Args:
        how_many_hits  :: How many hits are allowed in 'in_how_long' 
                          seconds before the limit is applied
        in_how_long    :: If there are 'how_many_hits' in this many seconds,
                          BOOM limited
        exception_list :: A list of IPs to exclude
    Usage:
        @rate_limit_by_ip
        def my_view(request):
            #I am less likely to have excessive database hits due to DDOS or
            #shitty code

    """
    def decorator(func):
        print 'SOME SHIT BAD'
        @wraps(func, assigned=available_attrs(func))
        def inner(request, *args, **kwargs):
            print "IN THE INNER PART"
            remote_addr = request.META['REMOTE_ADDR']
            cache_key = func.__name__ + '_' + str(remote_addr)
            cache_key = str(cache_key)
            print 'KEY:" '+ cache_key + '"'
            retVal = None
            #Allow for exceptions
            #cache.clear()
            if remote_addr in exception_list:
                retVal = func(request, *args, **kwargs)
            count = cache.get(cache_key)
            print 'COUNT IS', count
            #We haven't seen them before
            if count is None:
                # Figure out how frequently a hit would have to occur in
                # order to hit the limit
                cacheTime = how_many_hits / in_how_long
                cacheTime = float(in_how_long) / float(how_many_hits)
                print cacheTime
                #Put them in the cache
                cache.set(cache_key, 'cached', 1)
                print cache.get(cache_key) 
                #Allow them in to the function
                retVal = func(request, *args, **kwargs)
            
            #We've seen them in the limit time, so sucks to be you
            else:
                #429 is the error code for 'Too Many Requests'
                d = {'error': "You've hit your rate limit"}
                retVal = HttpResponse(json.dumps(d), content_type="application/json", status=429)
            return retVal
        return inner
    return decorator
					

Opportunities for Improvement

• Limiting could get flakey with the time out setting. There are probably better, more complicated ways to time out.
• Limit based on more than just remote address, as this fails when multiple people are accessing a site from the same IP, such as behind your Verizon router.

References

• Twitter API Docs:https://dev.twitter.com/docs/rate-limiting/1.1
• Github API Docs:http://developer.github.com/v3/rate_limit/
• Facebook Graph API rate limiting policy:https://developers.facebook.com/docs/reference/ads-api/api-rate-limiting/