how-to tips   0   687
Server Access Logging in Django using middleware

Some application admins need to know which user performed what action in the application. We also felt the need of such tracking hence we started developing the access log system for our application.

In this article we will see how to develop the server access logging app for Django project.


We will be storing below information:

  • URL/link visited.
  • Request method, get or post.
  • Get or Post Data
  • Referrer
  • IP address of visitor
  • Session ID


What is access log:

An access log is a list of all the requests for individual files that website visitors have requested from the website.


Why access log:

We can analyse the access logs and figure out multiple aspects of website visitors and their behaviour:
  • Origin of request i.e. referrer
  • Location of visitor
  • What page or link is visited most
  • In case of audit, which visitor clicked on which page and searched what. etc.



Access Logs:

To start access logging, we will be using middleware. Create a Django project and create an app in project.

We strongly recommend to use virtual environment for python or Django project development.

To log the information, we need to create model.


Model:
Create a model in your app's models.py file.

from django.db import models


class AccessLogsModel(models.Model):
    sys_id = models.AutoField(primary_key=True, null=False, blank=True)
    session_key = models.CharField(max_length=1024, null=False, blank=True)
    path = models.CharField(max_length=1024, null=False, blank=True)
    method = models.CharField(max_length=8, null=False, blank=True)
    data = models.TextField(null=True, blank=True)
    ip_address = models.CharField(max_length=45, null=False, blank=True)
    referrer = models.CharField(max_length=512, null=True, blank=True)
    timestamp = models.DateTimeField(null=False, blank=True)

    class Meta:
        app_label = "django_server_access_logs"
        db_table = "access_logs"

Middleware:
Now create a file logging_middleware.py in your app.

from .models import AccessLogsModel
from django.conf import settings
from django.utils import timezone


class AccessLogsMiddleware(object):

    def __init__(self, get_response=None):
        self.get_response = get_response
        # One-time configuration and initialization.

    def __call__(self, request):
        # create session
        if not request.session.session_key:
            request.session.create()

        access_logs_data = dict()

        # get the request path
        access_logs_data["path"] = request.path

        # get the client's IP address
        x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
        access_logs_data["ip_address"] = x_forwarded_for.split(',')[0] if x_forwarded_for else request.META.get('REMOTE_ADDR')
        access_logs_data["method"] = request.method
        access_logs_data["referrer"] = request.META.get('HTTP_REFERER',None)
        access_logs_data["session_key"] = request.session.session_key

        data = dict()
        data["get"] = dict(request.GET.copy())
        data['post'] = dict(request.POST.copy())

        # remove password form post data for security reasons
        keys_to_remove = ["password", "csrfmiddlewaretoken"]
        for key in keys_to_remove:
            data["post"].pop(key, None)

        access_logs_data["data"] = data
        access_logs_data["timestamp"] = timezone.now()

        try:
            AccessLogsModel(**access_logs_data).save()
        except Exception as e:
            pass

        response = self.get_response(request)
        return response



In the file above, we are doing following things:

  1. Since we are using session key to uniquely identifying the request, we need to create session if session key doesn't exists.
  2. Get the path i.e. URL which user/visitor visited.
  3. Collect, IP address, request method, referrer URL and session key.
  4. Collect the post and get data and remove the sensitive information like password. You may edit the logic as per your requirement.
  5. Store the data with timestamp in table.


Settings:

For the above code to work, we need to complete below settings:

Add your app in installed app's list in settings.py file.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_server_access_logs'
]


Add your middleware class in middleware classes list.

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django_server_access_logs.logging_middleware.AccessLogsMiddleware',
]


Make migrations to create the Model table in database.

Now start hitting the application URLs and you can see the entry in your table.

server access logging in django using middleware  

Complete code is available on Github.



Future Work:

You may make changes in middleware code to
  • Ignore hits on static and media URL.
  • Log user_id  if user is logged in.


Data Cleanup:

Since access_log table will take a lot of space, it is good idea to delete the old entries.

You might want to create a management command and schedule it to delete the data periodically.

create a command and use below code for cleanup.

from django.core.management.base import BaseCommand, CommandError
import datetime
from django.utils import timezone
from .models import AccessLogsModel


class Command(BaseCommand):

    help = 'Clean the user access logs older than x days'

    def add_arguments(self, parser):
        pass

    def handle(self, *args, **options):
        days_to_keep_data = 7
        now = timezone.now()
        back_date = now - datetime.timedelta(days=days_to_keep_data)
        AccessLogsModel.objects.filter(timestamp__lt=back_date).delete()
        
   

Must read article for middleware :

How to develop a distributable Django app to block the crawling IP addresses

how-to tips   0   687

Related Articles:
Python Script 3: Validate, format and Beautify JSON string Using Python
Validating json using python code, format and beautify json file using python, validate json file using python, how to validate, format and beautify json...
Encryption-Decryption in Python Django
How to encrypt and decrypt the content in DJango. Encrypting the critical information in Django App. Encrypting username, email and password in Django...
How to upgrade to paid account on PythonAnyWhere
Which is the best server for hosting Django Apps. Best hosting provider for Django Apps. Cheapest Django Hosting. PythonAnyWhere Reviews. Django Hosting....
Scraping 10000 tweets in 60 seconds using celery, RabbitMQ and Docker cluster with rotating proxy
Scrapping large amount of tweets within minutes using celery, RabbitMQ and docker cluster. Scraping huge data quickly using docker cluster with TOR....

0 thoughts on 'Server Access Logging In Django Using Middleware'
Leave a comment:


*All Fields are mandatory. **Email Id will not be published publicly.


SUBSCRIBE
Please subscribe to get the latest articles in your mailbox.



Recent Posts:






© pythoncircle.com 2018-2019