How to upload and process the CSV file in Django

In this article we will discuss how to upload a csv file and then process the content without storing file on server. 

One approach could be uploading the file, storing it in upload directory and then reading the file.

Another approach could be uploading file and reading it directly from post data without storing it in memory and displaying the data.

We will work with the later approach here.


Uploading CSV file:

First create HTML form to upload the csv file. Use below code for the same.

<form action="{% url "myapp:upload_csv" %}" method="POST" enctype="multipart/form-data" class="form-horizontal"> 
{% csrf_token %}
<div class="form-group">
    <label for="name" class="col-md-3 col-sm-3 col-xs-12 control-label">File: </label>
    <div class="col-md-8">
        <input type="file" name="csv_file" id="csv_file" required="True" class="form-control">
    </div>                    
</div>
<div class="form-group">                    
    <div class="col-md-3 col-sm-3 col-xs-12 col-md-offset-3" style="margin-bottom:10px;">
         <button class="btn btn-primary"> <span class="glyphicon glyphicon-upload" style="margin-right:5px;"></span>Upload </button>
    </div> 
</div>
</form>


Important: Do not forget to include enctype="multipart/form-data"  in form.

upload csv in django.png


Add a URL in URLpatterns.

url(r'^upload/csv/$', views.upload_csv, name='upload_csv'),
Create a function in views with the name upload_csv .


Process the CSV file:

In view function, get the file from post data and process it. I used below code in my project.

def upload_csv(request):
	data = {}
	if "GET" == request.method:
		return render(request, "myapp/upload_csv.html", data)
    # if not GET, then proceed
	try:
		csv_file = request.FILES["csv_file"]
		if not csv_file.name.endswith('.csv'):
			messages.error(request,'File is not CSV type')
			return HttpResponseRedirect(reverse("myapp:upload_csv"))
        #if file is too large, return
		if csv_file.multiple_chunks():
			messages.error(request,"Uploaded file is too big (%.2f MB)." % (csv_file.size/(1000*1000),))
			return HttpResponseRedirect(reverse("myapp:upload_csv"))

		file_data = csv_file.read().decode("utf-8")		

		lines = file_data.split("\n")
		#loop over the lines and save them in db. If error , store as string and then display
		for line in lines:						
			fields = line.split(",")
			data_dict = {}
			data_dict["name"] = fields[0]
			data_dict["start_date_time"] = fields[1]
			data_dict["end_date_time"] = fields[2]
			data_dict["notes"] = fields[3]
			try:
				form = EventsForm(data_dict)
				if form.is_valid():
					form.save()					
				else:
					logging.getLogger("error_logger").error(form.errors.as_json())												
			except Exception as e:
				logging.getLogger("error_logger").error(repr(e))					
				pass

	except Exception as e:
		logging.getLogger("error_logger").error("Unable to upload file. "+repr(e))
		messages.error(request,"Unable to upload file. "+repr(e))

	return HttpResponseRedirect(reverse("myapp:upload_csv"))
 

In the above code we are performing below actions:

      - If this is a GET request then render the upload csv html file.

      - If this is a POST request then proceed.

      - First check if file name is not ending with .csv then this is not the valid file. You may implement you own checks as well.

      - Then we check if file is too large. If these tests fail, we return to html form page with appropriate error message. For displaying error/success messages, we are using messages framework. Please import required modules.

      - Then we read the file and split the content by new line character.

      - Iterate over each line and split the line using comma.

      - We are assuming that our csv file have 4 columns of data. We stored the data in a dictionary and then pass the data dictionary to a form.

      - If form is valid then we proceed to save the form and hence creating entry in DB.

      - If form is not valid, or any other error is thrown then we log the error in log file.


      Read here about logging the errors in log files. This might be useful on live servers where debug is set to false.


        Please provide your inputs.



        try .. except .. else .. in python with example

        You will face at least two types of errors while coding with python. Syntax errors and exceptions.

        Syntax errors are also known as parsing errors. Compiler informs us about the parsing errors using an arrow.


        rana@brahma:~$ python3
        Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
        [GCC 5.4.0 20160609] on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> while True print("Hello")
          File "<stdin>", line 1
            while True print("Hello")
                           ^
        SyntaxError: invalid syntax
        >>> 
        


        Even when your code is perfect syntax wise, some lines may report an errors when you try to execute them. Errors reported at runtime are known as exceptions.

        For an uninterrupted execution, we must handle the exceptions properly. We can do so by using try except block. 

        In below example we are trying to add a string and an integer. When statement is executed it throws an exception. 

        >>> a = 'rana' + 10
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        TypeError: Can't convert 'int' object to str implicitly
        >>> 
        


        If we don't handle this exception, program will exit abruplty. Handle the exception using try except statements.


        >>> 
        >>> try:
        ...     a = 'rana' + 10
        ... except Exception as e:
        ...     print('Some useful message to debug the code')
        ... 
        Some useful message to debug the code
        >>> 
        

        When above piece of code is executed, exception is thrown but is then catched by except block which in turn print a useful message which helps in debugging the code.


        try ... except statement have an optional else clause. Else part is executed when there is no exception thrown in try clause. Else part is executed before finally clause.

        Lets say you are trying to open a file in try block (just an example) and it is possible there might occur some error in opening the file, we will handle exception in except block. If there is no exception thrown, file is opened successfully, we have to close the file descriptor. 


        >>> try:
        ...     f = open('filename.txt','r')
        ... except Exception as e:
        ...     print('Some exception in opening the file')
        ... else:
        ...     f.close()
        ...


        Its better to add code to else clause instead of adding code to try clause. This helps in avoiding catching the exception which was not raised by code being protected in try caluse. For example in addition to opening file in try clause, we were also trying to convert a variable to integer. Imaging file opened just fine but convertion to int threw an exception which will be reported by except block as exception in opening file, which will be misleading. See the example code below.


        >>> try:
        ...     f = open('filename.txt','r')
        ...     a = 'five'
        ...     b = int(a)
        ... except Exception as e:
        ...     print('can not open file')
        ... else:
        ...     f.close()
        ... 
        can not open file
        >>> 
        
         

        Above could should be rewritten as below.

        try:
            f = open('filename.txt','r')
            print('file was opened')
        except Exception as e:
            print('can not open file')
        else:
            print('going to close file')
            f.close()
        try:
            a = 'five'
            b = int(a)
        except Exception as e:
            print('exception in number conversion')

        Output:

        file was opened
        going to close file
        exception in number conversion

        This make sure that actual exception is reported by except clause.


        Now question arises, if we should use except or else block to control the execution flow?

        In python using exception to control the execution flow is normal. For example iterators uses stopIteration exception to singal the end of items.

        You can read more about exceptions by visitng the links in reference section below.

        References:

        1. https://stackoverflow.com/questions/16138232/is-it-a-good-practice-to-use-try-except-else-in-python
        2. https://docs.python.org/3/tutorial/errors.html

        Image source: quotemaster.org 




        Sending Emails Using Python and Gmail

        We already know how to send email using office 365 or how to send bulk emails using mailgun api.

        Here we will see how to send email from your python script or Django App using Gmail account.

        Gmail have some daily limits on the number of emails you can send in a day. If you need to send bulk emails, use mailgun api.


        Code:

        Import dependencies:

        import smtplib
        from email.mime.multipart import MIMEMultipart
        from email.mime.text import MIMEText
        from django.utils.html import strip_tags

        Create message:

        msg = MIMEMultipart('alternative')
        msg['Subject'] = "Test Mail : ThePythonDjango.Com"
        msg['From'] = "sender-email-id@gmail.com"
        msg['To'] = "receivers-email-id@gmail.com"


        Create message body and attach to message:

        html_text = '<div style="border:1px solid black">This is your message body in HTML format.</div>'
        plain_text = strip_tags(html_text)
        
        part1 = MIMEText(plain_text, 'plain')
        part2 = MIMEText(html_text, 'html')
        
        msg.attach(part1)
        msg.attach(part2)


        Send the message:

        host = "smtp.gmail.com"
        port = 587
        mail = smtplib.SMTP(host, port, timeout=60)
        mail.ehlo()
        mail.starttls()
        
        recepient = [msg["To"]]
        
        username = "your-user-name"
        password = "*****"
        mail.login(username, password)
        mail.sendmail(msg["From"], recepient, msg.as_string())
        mail.quit()
         

        Complete code with comments:


        # import dependencies
        
        import smtplib
        from email.mime.multipart import MIMEMultipart
        from email.mime.text import MIMEText
        import email
        import email.mime.application
        from django.utils.html import strip_tags
        
        # create message
        msg = MIMEMultipart('alternative')
        msg['Subject'] = "Test Mail : ThePythonDjango.Com"
        msg['From'] = "sender-email-id@gmail.com"
        msg['To'] = "receivers-email-id@gmail.com"
        
        # create body
        html_text = '<div style="border:1px solid black">This is your message body in HTML format.</div>'
        plain_text = strip_tags(html_text)
        
        # Create the body of the message (a plain-text and an HTML version).
        # Record the MIME types of both parts - text/plain and text/html.
        part1 = MIMEText(plain_text, 'plain')
        part2 = MIMEText(html_text, 'html')
        
        # Attach image if any
        
        # Attach parts into message container.
        # According to RFC 2046, the last part of a multipart message, in this case
        # the HTML message, is best and preferred.
        msg.attach(part1)
        msg.attach(part2)
        
        # Send the message via local SMTP server.
        host = "smtp.gmail.com"
        port = 587
        mail = smtplib.SMTP(host, port, timeout=60)
        mail.ehlo()
        mail.starttls()
        
        # Add recepiens, cc or bcc if any
        recepient = [msg["To"]]
        
        # username and password of gmail id which will be used to send email
        username = "your-user-name"
        password = "*****"
        
        # login using credentials
        mail.login(username, password)
        
        # send email
        mail.sendmail(msg["From"], recepient, msg.as_string())
        mail.quit()
        
        print("\nSent\n")
         

        Available on Gitlab as well. Please comment your views.

        Host your Django Application for free on PythonAnyWhere Servers.  





        SUBSCRIBE
        Please subscribe to get the latest articles in your mailbox.


        Recent Posts:






        © pythoncircle.com 2018-2019
        Contact Us: code108labs [at] gmail.com
        Address: 3747 Smithfield Avenue, Houston, Texas