Server Access Logging in Django using middleware

Some application admins need to know which user performed what action in the application. We also felt the need of such tracking hence we started developing the access log system for our application.

In this article we will see how to develop the server access logging app for Django project.


We will be storing below information:

  • URL/link visited.
  • Request method, get or post.
  • Get or Post Data
  • Referrer
  • IP address of visitor
  • Session ID


What is access log:

An access log is a list of all the requests for individual files that website visitors have requested from the website.


Why access log:

We can analyse the access logs and figure out multiple aspects of website visitors and their behaviour:
  • Origin of request i.e. referrer
  • Location of visitor
  • What page or link is visited most
  • In case of audit, which visitor clicked on which page and searched what. etc.



Access Logs:

To start access logging, we will be using middleware. Create a Django project and create an app in project.

We strongly recommend to use virtual environment for python or Django project development.

To log the information, we need to create model.


Model:
Create a model in your app's models.py file.

from django.db import models


class AccessLogsModel(models.Model):
    sys_id = models.AutoField(primary_key=True, null=False, blank=True)
    session_key = models.CharField(max_length=1024, null=False, blank=True)
    path = models.CharField(max_length=1024, null=False, blank=True)
    method = models.CharField(max_length=8, null=False, blank=True)
    data = models.TextField(null=True, blank=True)
    ip_address = models.CharField(max_length=45, null=False, blank=True)
    referrer = models.CharField(max_length=512, null=True, blank=True)
    timestamp = models.DateTimeField(null=False, blank=True)

    class Meta:
        app_label = "django_server_access_logs"
        db_table = "access_logs"

Middleware:
Now create a file logging_middleware.py in your app.

from .models import AccessLogsModel
from django.conf import settings
from django.utils import timezone


class AccessLogsMiddleware(object):

    def __init__(self, get_response=None):
        self.get_response = get_response
        # One-time configuration and initialization.

    def __call__(self, request):
        # create session
        if not request.session.session_key:
            request.session.create()

        access_logs_data = dict()

        # get the request path
        access_logs_data["path"] = request.path

        # get the client's IP address
        x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
        access_logs_data["ip_address"] = x_forwarded_for.split(',')[0] if x_forwarded_for else request.META.get('REMOTE_ADDR')
        access_logs_data["method"] = request.method
        access_logs_data["referrer"] = request.META.get('HTTP_REFERER',None)
        access_logs_data["session_key"] = request.session.session_key

        data = dict()
        data["get"] = dict(request.GET.copy())
        data['post'] = dict(request.POST.copy())

        # remove password form post data for security reasons
        keys_to_remove = ["password", "csrfmiddlewaretoken"]
        for key in keys_to_remove:
            data["post"].pop(key, None)

        access_logs_data["data"] = data
        access_logs_data["timestamp"] = timezone.now()

        try:
            AccessLogsModel(**access_logs_data).save()
        except Exception as e:
            pass

        response = self.get_response(request)
        return response



In the file above, we are doing following things:

  1. Since we are using session key to uniquely identifying the request, we need to create session if session key doesn't exists.
  2. Get the path i.e. URL which user/visitor visited.
  3. Collect, IP address, request method, referrer URL and session key.
  4. Collect the post and get data and remove the sensitive information like password. You may edit the logic as per your requirement.
  5. Store the data with timestamp in table.


Settings:

For the above code to work, we need to complete below settings:

Add your app in installed app's list in settings.py file.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_server_access_logs'
]


Add your middleware class in middleware classes list.

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django_server_access_logs.logging_middleware.AccessLogsMiddleware',
]


Make migrations to create the Model table in database.

Now start hitting the application URLs and you can see the entry in your table.

server access logging in django using middleware  

Complete code is available on Github.



Future Work:

You may make changes in middleware code to
  • Ignore hits on static and media URL.
  • Log user_id  if user is logged in.


Data Cleanup:

Since access_log table will take a lot of space, it is good idea to delete the old entries.

You might want to create a management command and schedule it to delete the data periodically.

create a command and use below code for cleanup.

from django.core.management.base import BaseCommand, CommandError
import datetime
from django.utils import timezone
from .models import AccessLogsModel


class Command(BaseCommand):

    help = 'Clean the user access logs older than x days'

    def add_arguments(self, parser):
        pass

    def handle(self, *args, **options):
        days_to_keep_data = 7
        now = timezone.now()
        back_date = now - datetime.timedelta(days=days_to_keep_data)
        AccessLogsModel.objects.filter(timestamp__lt=back_date).delete()
        
   

Must read article for middleware :

How to develop a distributable Django app to block the crawling IP addresses


Get latest Bitcoin and other crypto-currencies rates using python Django

Everybody is investing in bitcoins. 

James Howells is trying to dig a landfill site to get 7500 bitcoins that were dumped there in 2013.

To be a good investor, it is necessary that you keep track of ups and downs in the market. '

There are multiple platforms where you can track the price of bitcoin. But for a python programmer that is no fun. Being a python programmer we will develop our own project where we can get latest bitcoin and other crypto-currency prices.

Let's start.



Virtual environment setup:

It is always recommended to use virtual environment for all your python and Django projects.

Create a virtual environment using python3 using below command.

virtualenv -p /usr/bin/python3 crypto


Now activate the virtual environment.

source crypto/bin/activate


Install the latest Django version and other required libraries.

For now only requests package is required. We will add other packages later if required.

pip install django requests


This will install Django 2.0 and requests package along with some other package.

You can verify the same by running command pip freeze .

(crypto) rana@Brahma: crypto$ pip freeze
certifi==2017.11.5
chardet==3.0.4
Django==2.0
idna==2.6
pytz==2017.3
requests==2.18.4
urllib3==1.22


Creating new Django Project:

Once virtual environment has been setup and activated, create a new django project.

django-admin startproject crypto


Go to crypto project directory and list the files.

get latest bitcoin and other crypto currencies rates using python django

Since we are working on Django 2.0 , we need to take care of few things which we will highlight as we progress.

Now create a new app 'bitcoin'.

python manage.py startapp bitcoin


Add this app to the list of installed apps in settings.py  file.



Project Setup:

URLs:

Create urls.py  file in new app bitcoin.

from django.urls import path
from . import views

app_name = 'bitcoin'

urlpatterns = [
    path('', views.index, name="index"),
]


Django 2.0 Note: Adding app_name  in urls.py  is require now, otherwise you will get the below error.

'Specifying a namespace in include() without providing an app_name '
django.core.exceptions.ImproperlyConfigured: Specifying a namespace in include() without providing an app_name is not supported. Set the app_name attribute in the included module, or pass a 2-tuple containing the list of patterns and app_name instead.
 

Include urls of bitcoin app in project's urls.py  file.  


Teamplate:

Create a directory templates/bitcoin  in bitcoin app.

Inside this directory create a new html file, index.html . You can leave this file empty as of now or to make sure things are working fine, put some text there.  


Views:

Create a new function index in views.py  file.

For now this function will only render the index.html  created in previous step.

We will add more functionality in coming steps.

from django.shortcuts import render


def index(request):
    data = {}
    return render(request, "bitcoin/index.html", data)
 


Verifying basic setup:

Once above steps are completed, run the django server.

$python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have 14 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

December 28, 2017 - 09:00:43
Django version 2.0, using settings 'crypto.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.


Go to localhost:8000  and you can see the text which you putted in index.html  file.  



Getting bitcoin prices:

We will use the coinmarktecap api to fetch the latest data.

In views.py  file add a new function to get bitcoin data. This function will call the api url and get the currency data. Data returned is Json string. Convert it to Json Object.

# return the data received from api as json object
def get_crypto_data():
    api_url = "https://api.coinmarketcap.com/v1/ticker/?limit=10"

    try:
        data = requests.get(api_url).json()
    except Exception as e:
        print(e)
        data = dict()

    return data


Make a call to get_crypto_data  in index function and return the rendered response.

Complete views.py  file :

from django.shortcuts import render
import requests


def index(request):
    data = {}
    data["crypto_data"] = get_crypto_data()
    return render(request, "bitcoin/index.html", data)


# return the data received from api as json object
def get_crypto_data():
    api_url = "https://api.coinmarketcap.com/v1/ticker/?limit=10"

    try:
        data = requests.get(api_url).json()
    except Exception as e:
        print(e)
        data = dict()

    return data
 

Rendering Data in Template:

Render the coins data row wise using a for loop in template as shown in html code below.

<html>
    <head>
        <title>Latest Bitcoin Price - ThePythonDjango.Com</title>
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-beta.2/css/bootstrap.min.css" integrity="sha384-PsH8R72JQ3SOdhVi3uxftmaW6Vc51MKb0q5P2rRUpPvrszuE4W1povHYgTpBfshb" crossorigin="anonymous">
    </head>
    <body style="margin:20px;">
    
        <div class="alert alert-dark" role="alert">
            <span style="font-size:30px;">Bitcoin Latest Price</span> <span style="font-size:15px;">by <a href="http://ThePythonDjango.Com" target="_blank">ThePythonDjango.Com</a></span>
        </div>
        
        <div class="list-group">
            <div class="list-group-item list-group-item-primary">
                <div class="row">
                    <div class="col-md-3">
                        <label>Name</label>
                    </div>
                    <div class="col-md-3">
                        <label>USD Price</label>
                    </div>
                    <div class="col-md-3">
                        <label>BTC Price</label>
                    </div>
                    <div class="col-md-3">
                        <label>Change in Last Hour</label>
                    </div>
                </div>
            </div>
            {% for coin in crypto_data %}
            <div class="list-group-item list-group-item-{% if coin.percent_change_1h.0 == '-'%}danger{% else %}success{% endif %}">
                <div class="row">
                    <div class="col-md-3">
                        {{coin.name}}
                    </div>
                    <div class="col-md-3">
                        {{coin.price_usd}}
                    </div>
                    <div class="col-md-3">
                        {{coin.price_btc}}
                    </div>
                    <div class="col-md-3">
                        {{coin.percent_change_1h}}
                    </div>
                </div>
            </div>
            {% endfor %}
        </div>
    </body>
</html>


We are using bootstrap.css  for styling. Now if you go to localhost:8000  you can see the details as in below image.

get latest bitcoin and other crypto currencies rates using python django

Complete code is available on Github.

Download and setup the project on your machine. Experiment with it and let us know in case of any query.  


References:
[1] https://coinmarketcap.com/api/
[2] Django 2.0 official documentation

Difference between list, set and tuples in Python and more such comparisons

In this article we will see key differences between commonly used terms in python. For example difference between tuple and list, difference between range and xrange and so on.


List and Tuples:


List:
- are mutable i.e. we can add, extend or update a list.
- generally are homogeneous data structures.


>>> [x for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]


Tuples:
- are immutable i.e. we can not update a tuple.
- generally are heterogeneous data structure.
- is a sequence where position have semantic value.
- are more like records, collection of fixed number of fields.

>>> import time
>>> time.localtime()
time.struct_time(tm_year=2017, tm_mon=9, tm_mday=25, tm_hour=21, tm_min=52, tm_sec=24, tm_wday=0, tm_yday=268, tm_isdst=0)
 

Set:

- is just like mathematical set.
- is unordered.
- is mutable.
- doesn't contain duplicate values.

rana@Brahma: ~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> mylist = [1,2,5,2,3]
>>> a = set(mylist)
>>> a
{1, 2, 3, 5}



range() and xrange():


range():
- In python2 range() function returns a list of integers.

rana@Brahma: ~$ python2
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> range(10)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

 - In python3 range() do whatever xrange use to do in python2 i.e. returns the generator object that can be used to display numbers only by looping. Only particular range is displayed on demand and hence called "lazy evaluation".

rana@Brahma: ~$ python2
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = xrange(10)
>>> a
xrange(10)
>>> [i for i in xrange(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

xrange():
- There is no xrange() function in python3. We already know what xrange() use to do in python2.

>>> a = xrange(10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'xrange' is not defined

 - Since we are moving towards python3, we will not have any further discussion about it. If you are really interested visit this link.


raw_input() and input():


input():
- In python2, input() tries to run the user input as a valid python expression.

rana@Brahma: ~$ python2
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> input()
2^3
1
>>> input(2**3)
8

 - In python3, return the user input as string.


raw_input():
- In python2, return the user input as string.
- Doesn't exists in python3. To simulate the raw_input() in python3, use eval(input()).



Shallow and deep copy:


Before comparing shallow and deep copy, we must know that normal assignment is just pointing new variable to existing object.

rana@Brahma: ~$ python3
Python 3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:09:58) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [1,2]
>>> b = a
>>> print(id(a) == id(b))
True


Comparison of shallow copy and deep copy is relevant for compound objects i.e. where objects contains another objects.


Shallow copy: Shallow copy creates a new object and then use the reference to refer the inner objects.

>>> a = [[1,2],[3,4]]
>>> b = a.copy()
>>> print(id(a) == id(b))
False
>>> print(id(a[0]) == id(b[0]))
True


Deep copy: deep copy create a new object and recursively copy the inner objects too.

>>> from copy import deepcopy
>>> a = [[1,2],[3,4]]
>>> b = deepcopy(a)
>>> print(id(a) == id(b))
False
>>> print(id(a[0]) == id(b[0]))
False

id() function return the identity of the location of the object in memory.

>>> print(id(a))
139724773442120


Unicode and str:

Here is a nicely written article explaining the difference between str, unicode and byte.

unicode str byte
Python 2 unicode characters raw 8 bit values n/a
Python 3 n/a unicode characters raw 8 bit values


*args and **kwargs:

*args:
- is used to pass the variable length list of non-keyworded arguments to function.

*kwargs:
- is used to pass the variable length list of keyworded arguments to a function.

Always pass arguments in this order : formal arguments then non-keyworded argument list and then keyworded arguments list.    





SUBSCRIBE
Please subscribe to get the latest articles in your mailbox.



Recent Posts:






© pythoncircle.com 2018-2019