how-to tips   0   1365
How to upload and process the CSV file in Django

In this article we will discuss how to upload a csv file and then process the content without storing file on server. 

One approach could be uploading the file, storing it in upload directory and then reading the file.

Another approach could be uploading file and reading it directly from post data without storing it in memory and displaying the data.

We will work with the later approach here.


Uploading CSV file:

First create HTML form to upload the csv file. Use below code for the same.

<form action="{% url "myapp:upload_csv" %}" method="POST" enctype="multipart/form-data" class="form-horizontal"> 
{% csrf_token %}
<div class="form-group">
    <label for="name" class="col-md-3 col-sm-3 col-xs-12 control-label">File: </label>
    <div class="col-md-8">
        <input type="file" name="csv_file" id="csv_file" required="True" class="form-control">
    </div>                    
</div>
<div class="form-group">                    
    <div class="col-md-3 col-sm-3 col-xs-12 col-md-offset-3" style="margin-bottom:10px;">
         <button class="btn btn-primary"> <span class="glyphicon glyphicon-upload" style="margin-right:5px;"></span>Upload </button>
    </div> 
</div>
</form>


Important: Do not forget to include enctype="multipart/form-data"  in form.

upload csv in django.png


Add a URL in URLpatterns.

url(r'^upload/csv/$', views.upload_csv, name='upload_csv'),
Create a function in views with the name upload_csv .


Process the CSV file:

In view function, get the file from post data and process it. I used below code in my project.

def upload_csv(request):
	data = {}
	if "GET" == request.method:
		return render(request, "myapp/upload_csv.html", data)
    # if not GET, then proceed
	try:
		csv_file = request.FILES["csv_file"]
		if not csv_file.name.endswith('.csv'):
			messages.error(request,'File is not CSV type')
			return HttpResponseRedirect(reverse("myapp:upload_csv"))
        #if file is too large, return
		if csv_file.multiple_chunks():
			messages.error(request,"Uploaded file is too big (%.2f MB)." % (csv_file.size/(1000*1000),))
			return HttpResponseRedirect(reverse("myapp:upload_csv"))

		file_data = csv_file.read().decode("utf-8")		

		lines = file_data.split("\n")
		#loop over the lines and save them in db. If error , store as string and then display
		for line in lines:						
			fields = line.split(",")
			data_dict = {}
			data_dict["name"] = fields[0]
			data_dict["start_date_time"] = fields[1]
			data_dict["end_date_time"] = fields[2]
			data_dict["notes"] = fields[3]
			try:
				form = EventsForm(data_dict)
				if form.is_valid():
					form.save()					
				else:
					logging.getLogger("error_logger").error(form.errors.as_json())												
			except Exception as e:
				logging.getLogger("error_logger").error(repr(e))					
				pass

	except Exception as e:
		logging.getLogger("error_logger").error("Unable to upload file. "+repr(e))
		messages.error(request,"Unable to upload file. "+repr(e))

	return HttpResponseRedirect(reverse("myapp:upload_csv"))
 

In the above code we are performing below actions:

      - If this is a GET request then render the upload csv html file.

      - If this is a POST request then proceed.

      - First check if file name is not ending with .csv then this is not the valid file. You may implement you own checks as well.

      - Then we check if file is too large. If these tests fail, we return to html form page with appropriate error message. For displaying error/success messages, we are using messages framework. Please import required modules.

      - Then we read the file and split the content by new line character.

      - Iterate over each line and split the line using comma.

      - We are assuming that our csv file have 4 columns of data. We stored the data in a dictionary and then pass the data dictionary to a form.

      - If form is valid then we proceed to save the form and hence creating entry in DB.

      - If form is not valid, or any other error is thrown then we log the error in log file.


      Read here about logging the errors in log files. This might be useful on live servers where debug is set to false.


        Please provide your inputs.


        how-to tips   0   1365

        Related Articles:
        Python Script 3: Validate, format and Beautify JSON string Using Python
        Validating json using python code, format and beautify json file using python, validate json file using python, how to validate, format and beautify json...
        Encryption-Decryption in Python Django
        How to encrypt and decrypt the content in DJango. Encrypting the critical information in Django App. Encrypting username, email and password in Django...
        Scraping 10000 tweets in 60 seconds using celery, RabbitMQ and Docker cluster with rotating proxy
        Scrapping large amount of tweets within minutes using celery, RabbitMQ and docker cluster. Scraping huge data quickly using docker cluster with TOR....
        How to use Google reCAPTCHA in Django
        How to use Google reCAPTCHA in Django. Preventing login attack in Django, Preventing multiple login attempts on login page in Django application. recaptca....

        0 thoughts on 'How To Upload And Process The Csv File In Django'
        Leave a comment:


        *All Fields are mandatory. **Email Id will not be published publicly.


        SUBSCRIBE
        Please subscribe to get the latest articles in your mailbox.



        Recent Posts:





        © pythoncircle.com 2018-2019