Elastic Search with Kibana and Django - part 2

In previous part, we installed Elasticsearch and Kibana. Now we will create a Django project and will link the elastic search with the project.


Django Project:

You can follow the official Django tutorials to create and start a Django project. Or you follow below steps. We will be following the official documentation, but only the necessary steps.

- Make sure python 3 is installed on your machine. Although not mandatory but it is recommended to use virtual environment.

- Install Django. Setup a database. We will be using MySQL.

- Create a project. $ django-admin startproject elasticsearchdjango

- Change directory to elasticsearch. Create a new app 'poll'. $ python manage.py startapp polls

- If you are using virtual environment than install the dependencies using pip install -r req.txt  file after saving below lines in req.txt  file.

certifi==2017.11.5
chardet==3.0.4
Django==1.10.6
elasticsearch==5.5.1
elasticsearch-dsl==5.3.0
idna==2.5
mysqlclient==1.3.10
python-dateutil==2.6.1
pytz==2017.2
requests==2.18.1
six==1.10.0
urllib3==1.21.1


- Create a model in polls/models.py file.

from django.db import models


class Question(models.Model):
    author = models.CharField(max_length=100)
    question_text = models.CharField(max_length=1000)
    pub_date = models.DateTimeField('date published')


- Register your model in admin.py

from django.contrib import admin
from .models import Question

# Register your models here.

# Need to register Question so it shows up in the admin
admin.site.register(Question)


- Add your app 'polls' in installed apps in settings.py file.

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'polls',
]


- Update your database configurations in settings.py file. We will be using MySQL. Create a database with name 'elasticsearchdb'.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'elasticsearchdb',
        'USER': 'root',
        'PASSWORD': 'root',
        'HOST': '127.0.0.1',
        'PORT': '3306',
    }
}


- Go to terminal, run below commands to create tables in DB.

python manage.py makemigrations
python manage.py migrate


- Create a super user to login in admin panel.

python manage.py createsuperuser


- Now run server with below command

python manage.py runserver


- Go to your browser http://localhost:8000/admin/  , login to admin panel and add question in Questions table.

elastic search with kibana and django part 2elastic search with kibana and django part 2


Now our Django project skeleton is ready. Lets connect Django Project to Elastic Search.



Connecting to Elastic Search:

- Install the python package elsaticsearch-dsl using pip. 


pip install elasticsearch-dsl

 - Create a new file elastic_search_connection.py  in your app 'polls'. Paste below content in it. Read comments to know what part of code is doing what?

from elasticsearch_dsl.connections import connections
from elasticsearch_dsl import DocType, Text, Date

# creates a global connection to elastic search
connections.create_connection()


# defines what needs to index in elastic search
class QuestionsIndex(DocType):
    author = Text()
    question_text = Text()
    pub_date = Date()

    class Meta:
        # name of index. Will be used in search
        index = 'questions-index'


So we are creating a global connection to elastic-search here. Then we are defining what we are indexing in elastic-search. Name of the index will be 'questions-index' .


- Now we will create a signal which will be triggered on every save event of question model instance. For this create a file signals.py in polls app directory.

from .models import Question
from django.db.models.signals import post_save
from django.dispatch import receiver


@receiver(post_save, sender=Question)
def index_question(sender, instance, **kwargs):
    instance.indexing()

This signal works as receiver on each post_save event where sender of signal is Question model. Everytime post_save event is triggered, it is received by this signal and it indexes the instance of Question model.

- We need to create indexing method in Question model for above code to work. Code in model class should look as below.

from django.db import models
from .elastic_search_connection import QuestionsIndex


class Question(models.Model):
    author = models.CharField(max_length=100)
    question_text = models.CharField(max_length=1000)
    pub_date = models.DateTimeField('date published')


    # indexing method of Question model
    def indexing(self):
       obj = QuestionsIndex(
            meta = {
                'id':self.id
            },
            author = self.author,
            pub_date = self.pub_date,
            question_text = self.question_text
       )
       obj.save()
       return obj.to_dict(include_meta=True)


- We also need to inform Django project that we are using signals. Add below config code to app.py .

from django.apps import AppConfig


class PollsConfig(AppConfig):
    name = 'polls'

    def ready(self):
        import polls.signals


- Now we need to tell Django that we are using new config defined in app.py . Add below line in __init__.py  file of polls app.

default_app_config = 'polls.apps.PollsConfig'


Now all the configurations are complete. On every save of model instance, data will be indexed in elastic-search.

Restart the Django project server. Now open a new terminal and start the elastic search as discussed in part 1 of this article.

Go to admin panel and save data in Question model.

Now navigate to this url http://127.0.0.1:9200/questions-index/questions_index/6?pretty  in your browser.

elastic search with kibana and django part 2

questions-index is the name of index. questions_index is the name of class QustionsIndex in snake notation.

This complete code is available on Github.


Using Kibana to search, browse and visualise data:

We already installed and started the kibana in previous article. Go to the homepage of kibana which will be running on localhost:5601. Initially there is no index pattern configured and it will ask us to configure one.


elastic search with kibana and django part 2


Type the name of your index in index pattern input box. You can use wild cards. Once index patter is configured, you will see the data in matching index pattern.

elastic search with kibana and django part 2


Now you can start exploring your data.

- Search and browse your data interactively from the Discover page.

- Chart and map your data from the Visualise page.

- Create and view custom dashboards from the Dashboard page. 

elastic search with kibana and django part 2 Pie chart showing distribution of Authors (column in questions model)


Happy learning.


Related Articles:
Python Script 3: Validate, format and Beautify JSON string Using Python
Validating json using python code, format and beautify json file using python, validate json file using python, how to validate, format and beautify json...
Encryption-Decryption in Python Django
How to encrypt and decrypt the content in DJango. Encrypting the critical information in Django App. Encrypting username, email and password in Django...
Adding Robots.txt file to Django Application
Adding robots.txt file in your Django application. Easiest way to add robots.txt file. Django application robots.txt file. Why should you add robots.txt file in your Django Application....
Scraping 10000 tweets in 60 seconds using celery, RabbitMQ and Docker cluster with rotating proxy
Scrapping large amount of tweets within minutes using celery, RabbitMQ and docker cluster. Scraping huge data quickly using docker cluster with TOR....

0 thoughts on 'Elastic Search With Kibana And Django - Part 2'
Leave a comment:


*All Fields are mandatory. **Email Id will not be published publicly.


SUBSCRIBE
Please subscribe to get the latest articles in your mailbox.



Recent Posts:






© pythoncircle.com 2018-2019