Django advanced form processing: Partial forms, AJAX and files

In most cases, Django form processing adheres to the steps and code sequences outlined in the first section of this chapter, shown in figure 6-1. However, Django form processing can require small adjustments for certain scenarios, like those involving partial form processing, AJAX forms and file uploads.

Partial forms

On occasions you may find yourself with the need to use a partial form based on a pre-existing Django form. For example, you get a request to add a similar form to the ContactForm used up to this point -- in listing 6-25 -- but without requiring the name and email fields. Since the ContactForm is tried and tested, including the necessary post-processing logic (e.g. adding the form data to a CRM[Customer Relationship Management] system) it's a worthwhile idea to reuse the ContactForm class. But how do you go about using a Django form partially ?

The first alternative to achieve partial forms is to hide the unneeded form fields from a user, allowing the original form class to remain intact while not requiring a new sub-class. This is the least invasive approach, but you need to be careful with two aspects. The first aspect is the form template layout, shown in listing 6-32.

Listing 6-32. Django form with fields marked as hidden

<form method="POST">
  {% csrf_token %}
    <div class="row">
      <div class="col-md-2">
        {{ form.comment.label_tag }}
          {% if form.comment.help_text %}
          <sup>{{ form.comment.help_text }}</sup>
          {% endif %}
          {% for error in form.comment.errors %}
           <div class="row">
             <div class="alert alert-danger">{{error}}</div>
           </div>
          {% endfor %}
       </div><div class="col-md-10 pull-left">
         {{ form.comment }}
       </div>
    </div>
    {{form.name.as_hiddden}}
    {{form.email.as_hidden}} 
<input type="submit" value="Submit form" class="btn btn-primary">
</form>

First notice in listing 6-32 the form fields are output individually and do not use a shortcut method (e.g. form.as_table) or loop like some of the previous form layout examples. The reason is, the last form fields are explicitly output as hidden through the as_hidden method -- described earlier in table 6-4. The as_hidden method generates a form field as a hidden HTML input (e.g. <input type="hidden"...>), irrespective of its form field data type. This mechanism effectively hides fields from end users, but keeps the fields as part of the form.

Why is it important to keep fields as part of the form and not just remove them ? Validation. Remember the form instance hasn't changed, only the requirements, you may not need the name and email data on this template page, but the form instance and validation logic doesn't know that. All of which takes us to the second aspect you need to take care of: required values.

If you publish the template in listing 6-32 without taking care of this second aspect, you might be surprised when you realize the form never passes validation! Why ? The underlying ContactForm class treats the email field as required, therefore it must be given a value, but since the field is now hidden from an end user, you must take care of giving the email field a value yourself -- note the name field doesn't present this problem because it's configured with required=False. Therefore, in order for the template in listing 6-32 to work, the form must be initialized with a value like the following snippet:

form = ContactForm(initial={'email':'anonymous@gmail.com'})

By initializing the form in this manner, the email form field is given this initial value as a hidden input. So once a user submits the form, the email value is transferred back to Django as part of the form, where validation passes since the post-processing logic gets all its expected values. Of course, if a form's fields are all optional (i.e. they're marked with required=False) this initialization process is unnecessary, since the validation won't be expecting any value anyways.

A second alternative to achieve partial forms is to create a sub-class from a form class and delete the unneeded fields. This is a little more straightforward approach, but it does require creating a form sub-class, which may be overkill for certain scenarios. Listing 6-33 shows how to sub-class a Django form class and remove some of its parent's fields.

Listing 6-33. Django form sub-class with removed parent fields

from coffeehouse.about.forms import ContactForm

class ContactCommentOnlyForm(ContactForm):
    def __init__(self, *args, **kwargs):
        super(ContactCommentOnlyForm, self).__init__(*args, **kwargs)
        del self.fields['name']
        del self.fields['email']

As you can see in listing 6-33, the ContactCommentOnlyForm form inherits its behavior from the ContactForm class. Next, inside the form class __init__ method, a call is made to the parent's __init__ class via super() and two calls are made on self.fields using Python's del to remove both the name and email fields from this form sub-class.

Once you have the form sub-class from listing 6-33, you can generate an unbound instance form in a view method and pass it to a template for rendering. Since this last sub-class removes the name and email fields, upon validation you won't face previously described problem of missing values for hidden fields, since the ContactCommentOnlyForm form now only has a single field.

AJAX form submission

AJAX is a technique where JavaScript on a web page communicates with a server-side application, re-renders the web page with the result of this communication, all without a web page transition. Django forms are often designed to submit their data via AJAX to create a smoother workflow (i.e. without changing the original web page with a web form)

The workflow for a form that uses AJAX as far as the initial form delivery is concerned -- steps 1 an 2 in figure 6-1 -- is identical to that of a regular Django non-AJAX form: you create an unbound form instance in a view method, which is passed to a template for rendering into a web page that's delivered to an end user.

The first difference in Django forms that submit their data via AJAX, is the web page on which the unbound form is delivered, must also contain the necessary JavaScript logic to send the form data to the server-side application and also process the server-side response. Listing 6-34 illustrates this JavaScript logic using the jQuery library to submit a Django form via AJAX.

Listing 6-34. JavaScript jQuery logic to submit Django form via AJAX

# NOTE: The following is only the Django (HTML) template with AJAX logic
# See listing 6-35 for AJAX processing views.py and urls.py 

<h4>Feedback</h4>
<div class="row">
  <div id="feedbackmessage"</div>
</div>
<form method="POST" id="feedbackform" action="{% url 'stores:feedback' %}">
  {% csrf_token %}
    <div class="row">
      <div class="col-md-12 pull-left">
         {{ form.comment }}
       </div>
    </div>
    {{form.name.as_hiddden}}
    {{form.email.as_hidden}} 
<input type="submit" value="Submit feedback" class="btn btn-primary">
</form>  
<script> $(document).ready(function() { $("#feedbackform").submit(function(event) { event.preventDefault(); $.ajax({ data: $(this).serialize(), type: $(this).attr('method'), url: $(this).attr('action'), success: function(response) { console.log(response); if(response['success']) { $("#feedbackmessage").html("<div class='alert alert-success'> Succesfully sent feedback, thank you!</div>"); $("#feedbackform").addClass("hidden"); } if(response['error']) { $("#feedbackmessage").html("<div class='alert alert-danger'>" + response['error']['comment'] +"</div>"); } }, error: function (request, status, error) { console.log(request.responseText); } }); }); }) </script>

The first part of listing 6-34 is a standard Django form layout. However, notice the <form> tag includes the action attribute. In previous Django forms, the action attribute was not used because the serving url -- GET request -- and processing url -- POST request -- was the same one. In listing 6-34, the action attribute explicitly tells the browser the url where it should send the POST request -- in this case {% url 'stores:feedback' %} is a Django template tag that gets translated into a url (e.g./stores/feedback/). Different urls are necessary in this case, because Django view methods that handle AJAX requests/responses are different from Django view methods that handle standard web requests/responses.

Now lets turn our attention to the bottom half of listing 6-34 and the JavaScript logic enclosed in <script> tags. In brief -- as it would go beyond the scope of our discussion to describe jQuery syntax -- the web page awaits a click on the #feedbackform form's submit button. Once this click is detected, an AJAX POST request using the form's data is sent to the url defined in the form's action attribute. Upon completion of the AJAX request (i.e. when the server-side sends a response back), the response -- in JSON format -- is analyzed. If the AJAX response contains a success message, the message is output at the top of the form and the form is hidden to avoid multiple submissions. If the AJAX response contains an error message, the message is output at the top of the form.

Now that you know how Django form data is submitted to the server-side via AJAX, listing 6-35 illustrates the Django view method necessary to handle this AJAX request payload.

Listing 6-35. Django view method to process Django form via AJAX

# urls.py (Main)
urlpatterns = [
    url(r'^stores/',include('coffeehouse.stores.urls',namespace="stores"))
]

# urls.py (App stores)
urlpatterns = [
    url(r'^$',views.index,name="index"),
    url(r'^feedback/$',views.feedback,name="feedback"),
]

# views.py (App stores)
from django.http import HttpResponse, JsonResponse
from coffeehouse.about.forms import ContactForm

def feedback(request):
    if request.POST:
        form = ContactForm(request.POST)
        if form.is_valid():
            return JsonResponse({'success':True})
        else:
            return JsonResponse({'error':form.errors})
    return HttpResponse("Hello from feedback!")

The first important point about the view method in listing 6-35 is that it only handles POST requests. Notice that in case the if request.POST: statement isn't true, the view method always responds with the Hello from feedback! string.

Next, inside the request.POST segment, the process starts out pretty similar to a standard Django form processing sequence (i.e. a bound form instance is created and a call is made to the form's is_valid() method to validate the user provided data). However, notice the responses for both valid and invalid form data use JsonResponse(). Since the AJAX form submission operates in a JavaScript context, the JSON format response is a common format to use to send a response back to the browser.

If you turn your attention back to listing 6-34, you can see the JavaScript logic is designed to handle and output either JSON response (success or error) returned by the view method in listing 6-35.

Files in forms

Django offers a couple of form fields to capture files through forms -- forms.ImageField() and forms.ImageField() described in table 6-2. However, although these form fields generate the necessary HTML and validation logic to enforce files are submitted via forms, there are a couple of subtleties related to processing files in forms.

The first issue is the HTML <form> tag for forms that transfer files must explicitly set the encoding type to enctype="multipart/form-data", in addition to using method=POST. The second issue is the contents of files are placed in the Django request object under the special FILES dictionary key. Listing 6-36 illustrates a form with file fields, its corresponding view method and its template layout.

Listing 6-36. Django form with file fields, corresponding view method and template layout.

# forms.py
from django import forms

class SharingForm(forms.Form):
    # NOTE: forms.PhotoField requires Python PIL & other operating system libraries,
    #       so generic FileField is used instead
    video = forms.FileField()
    photo = forms.FileField(widget=forms.ClearableFileInput(attrs={'multiple': True}))


# views.py
def index(request):
    if request.method == 'POST':
        # POST, generate form with data from the request
        form = SharingForm(request.POST,request.FILES)
        # check if it's valid:
        if form.is_valid():
            # Process file data in request.FILES
            # Process data, insert into DB, generate email,etc
            # redirect to a new URL:
            return HttpResponseRedirect('/about/contact/thankyou')
    else:
        # GET, generate blank form
        form = SharingForm()
    return render(request,'social/index.html',{'form':form})


# social/index.html
<form method="post" enctype="multipart/form-data">
  {% csrf_token %}
  <ul>
    {{form.as_ul}}
  </ul>
    <input type="submit" value="Submit photo" class="btn btn-primary">    
</form>

Notice how the HTML <form> tag declares the required attributes and in addition notice how the request.FILES reference is used to create a bound form -- along with the standard request.POST. The remaining form structures in listing 6-36 are identical to regular non-file forms (e.g. unbound form creation, call to is_valid()).

But similar to regular Django form processing, once a form is valid -- if is_valid() returns True -- you'll want do something with the contents of the form data, therefore in this case, you'll need to access request.FILES to do something with the files uploaded by a user.

But before describing how to process the contents of files in request.FILES it's important you understand the possible contents of the request.FILES dictionary:

Option 1) request.FILES = {'photo': [<InMemoryUploadedFile>,<TemporaryUploadedFile>],
                           'video': [<InMemoryUploadedFile>]}
Option 2) request.FILES = {'photo': [<TemporaryUploadedFile>],
                           'video': [<InMemoryUploadedFile>]}

The request.FILES dictionary contains keys corresponding to file field names. In this case, you can see the photo and video keys correspond to the form fields names declared in listing 6-36 that represent files. Next, notice the value for each key is a list, irrespective if the file form field can accept multiple files -- like photo due to the overridden widget with the 'multiple': True attribute -- or a single file -- in the case of video.

Now lets analyze the contents of the lists assigned to each file form field key. Each of the list elements represents an uploaded file. But notice how the files are represented by either the InMemoryUploadedFile or TemporaryUploadedFile class -- both of which are sub-classes of the django.core.files.uploadedfile.UploadedFile class. The reason uploaded files are represented by either an InMemoryUploadedFile or TemporaryUploadedFile instance class, is dependent on the size of an uploaded file.

As it turns out, before you even decide how to handle uploaded files, Django must decide what to do and where to place uploaded files. To do this, Django relies on upload handlers. By default and unless explicitly overridden in the FILE_UPLOAD_HANDLERS variable in settings.py,Django uses the following two file upload handlers:

FILE_UPLOAD_HANDLERS= [
    'django.core.files.uploadhandler.MemoryFileUploadHandler',
    'django.core.files.uploadhandler.TemporaryFileUploadHandler',
]

By default, if a file is less than 2.5 MB, Django uses MemoryFileUploadHandler to place the contents of an uploaded file into memory; if a file is larger than 2.5 MB, Django uses TemporaryFileUploadHandler to place the contents of an uploaded file in a temporary file (e.g. /tmp/Af534.upload).

You can define FILE_UPLOAD_HANDLERS in settings.py to only include the file upload handler you desire (e.g. if you wish to save memory resources remove MemoryFileUploadHandler). However, note a file upload handler must always be active in order for file uploads to work. If you don't like the behavior of Django's built-in file upload handlers, then you'll need to write your own file upload handler[5].

In addition to defining FILE_UPLOAD_HANDLERS in settings.py, there are two other parameters you can declare in settings.py to influence the behavior of file upload handlers. The FILE_UPLOAD_MAX_MEMORY_SIZE variable sets the threshold size at which files are held in memory vs. handled as temporary files, defaulting to 2621440 bytes (or 2.5 MB). And the FILE_UPLOAD_TEMP_DIR variable sets the directory to which temporary uploaded files must be saved, defaulting to None, meaning an operating system's temporary file directory is used (e.g. on Linux OS /tmp/).

Now that you know where and how Django stores uploaded files even before you decide how to handle them, let's take a look at how to process the contents of request.FILES. Listing 6-37 shows the continuation of listing 6-36 with the relevant snippets necessary to process uploaded files.

Listing 6-37. Django form file processing with save procedure to MEDIA_ROOT.

# views.py
from django.conf import settings

def save_uploaded_file_to_media_root(f):
    with open('%s%s' % (settings.MEDIA_ROOT,f.name), 'wb+') as destination:
        for chunk in f.chunks():
            destination.write(chunk)

def index(request):
    if request.method == 'POST':
        # POST, generate form with data from the request
        form = SharingForm(request.POST,request.FILES)
        # check if it's valid:
        if form.is_valid():
            for field in request.FILES.keys():
                for formfile in request.FILES.getlist(field):
                    save_uploaded_file_to_media_root(formfile)                    
            return HttpResponseRedirect('/about/contact/thankyou')
    else:
        # GET, generate blank form
        form = SharingForm()
    return render(request,'social/index.html',{'form':form})

After the form is_valid() method, you know each of the keys in the request.FILES dictionary are file form fields, so a loop is created to get each file field (i.e. video, photo). Next, using the getlist() method of the request.FILES dictionary, you obtain the list of file instances (i.e. InMemoryUploadedFile,TemporaryUploadedFile) for each file field and loop over each element to get file instances. Finally, you execute the save_uploaded_file_to_media_root() method on each file instance.

Each file instance -- represented as the formfile reference in listing 6-37 -- is either a InMemoryUploadedFile or TemporaryUploadedFile instance type. But as I mentioned earlier, these classes are derived from the parent class django.core.files.uploadedfile.UploadedFile, which means file instances are also UploadedFile instances.

By design, the Django django.core.files.uploadedfile.UploadedFile class has a series of methods and fields, specifically made to easily handle the contents of uploaded files, some of which include:

With the use of these UploadedFile fields and methods, you can see toward the top of listing 6-37 the save_uploaded_file_to_media_root() method logic includes: saving the uploaded file with its original name -- using Python's standard with...open syntax -- and then using the chunks() method to efficiently write all the pieces of the uploaded file to the file system.

Note The uploaded files in listing 6-37 are saved under the settings.MEDIA_ROOT folder. Although you can save files to any location you want, the standard practice for user uploaded files in Django is to save them under the MEDIA_ROOT folder defined in settings.py.
  1. https://docs.djangoproject.com/en/1.11/ref/files/uploads/#custom-upload-handlers