Thursday, 10 January 2013

Ajax and deferred tasks on App Engine

Quick post this week.  Development of the new website has been progressing well so I thought I'd share some of the details on developing nice client-side functionality using Ajax style calls and deferred tasks.

Just to recap, the new version of the socialsamplr site is being developed in Python using the Django framework.  My previous posts have outlined how it can be easily and securely it can be integrated with Google apps script.  This post will mainly be concerned around developing Ajax-style functionality on Python hosted on App Engine and using deferred tasks in the server-side processes to ensure parallel processes can be run for performance.  

Coming from a .NET development background I see this as similar to running multi-threaded server side processes.  I don't know want to give too much else away about what I'm specifically doing here as it will be touching on some of the IP I'm development - suffice to say running several server-side parallel processes from client side javascript calls to provide a result is what I'm doing here.

Note I won't be covering how to secure your ajax calls within your application in this post - I'm still finalising this in my app but I will make a mention of the approach I'm using.


So the basics of creating Ajax calls in your python web application are covered in the following excellent blog post  Basically this walks you through creating a basic adding and subtracting function to your web page and should be fairly easy to implement if you want.   Reading this article it'll take you through what you'll need to do, which is basically two parts.

  1. Create a server-side RPC process to take the client side call and return a result.
  2. Implement the relevant javascript code on the client to make the call to the RPC process and accept the result.

I'll cover the client side code first as it's fairly simple to implement, then I'll cover the server-side stuff where I do some funky functionality with the deferred tasks to enable parallel processing of the requests.

Client side code for enabling Ajax calls

To start with you'll need to include json2.js from for this implementation (json is the most common data format for processing Ajax requests).  Then you'll need to add the javascript code required and wire up the relevant events to HMTL controls on your page.  All pretty simple stuff if you've done any web development before, and not even that difficult if you're new to it.  

As my project uses the django implementation where I have a base.html file containing all core page code (for example header , footer and common javascript code) I've split the javascript into 2 parts.  Part one, as follows goes into the base.html file:

<script type="text/javascript" src="js/json2.js"></script>
    <script type="text/javascript">

    // As mentioned at
    if( !window.XMLHttpRequest ) XMLHttpRequest = function()
      try{ return new ActiveXObject("Msxml2.XMLHTTP.6.0") }catch(e){}
      try{ return new ActiveXObject("Msxml2.XMLHTTP.3.0") }catch(e){}
      try{ return new ActiveXObject("Msxml2.XMLHTTP") }catch(e){}
      try{ return new ActiveXObject("Microsoft.XMLHTTP") }catch(e){}
      throw new Error("Could not find an XMLHttpRequest alternative.")

    // Makes an AJAX request to a local server function w/ optional arguments
    // functionName: the name of the server's AJAX function to call
    // opt_argv: an Array of arguments for the AJAX function
    function Request(function_name, opt_argv) {

      if (!opt_argv)
        opt_argv = new Array();

      // Find if the last arg is a callback function; save it
      var callback = null;
      var len = opt_argv.length;
      if (len > 0 && typeof opt_argv[len-1] == 'function') {
        callback = opt_argv[len-1];
      var async = (callback != null);

      // Encode the arguments in to a URI
      var query = 'action=' + encodeURIComponent(function_name);
      for (var i = 0; i < opt_argv.length; i++) {
        var key = 'arg' + i;
        var val = JSON.stringify(opt_argv[i]);
        query += '&' + key + '=' + encodeURIComponent(val);
      query += '&time=' + new Date().getTime(); // IE cache workaround

      // Create an XMLHttpRequest 'GET' request w/ an optional callback handler
      var req = new XMLHttpRequest();'GET', 'http://localhost:8083/rpc?' + query, async);

      if (async) {
        req.onreadystatechange = function() {
          if(req.readyState == 4 && req.status == 200) {
            var response = null;
            try {
             response = JSON.parse(req.responseText);
            } catch (e) {
             response = req.responseText;

      // Make the actual request

    // Adds a stub function that will pass the arguments to the AJAX call
    function InstallFunction(obj, functionName) {
      obj[functionName] = function() { Request(functionName, arguments); }


Then add the code to call the actual RPC function to the actual page where it's  needed and wire it up to the controls.  In the case of my app this is the home.html file which is rendered within the body of base.html.  It contains a button (doscore) which makes the call to the RPC function and a label (result) which takes the result.  So you'll see this is pretty straightforward - the button calls doScore() which then calls server.LiveScore, passing in the subject being searched and the onScoreSuccess function as the callback.  The server.LiveScore corresponds to the server side RPC function implemented in the solution.

<script type="text/javascript">

    // Server object that will contain the callable methods
    var server = {};

    // Insert 'Add' as the name of a callable method
    InstallFunction(server, 'LiveScore');

    // Handy "macro"
    function $(id){
      return document.getElementById(id);

    // Client function that calls a server rpc and provides a callback
    function doScore() {
      $('progress').style.display= 'block';
      $('placeholder').style.display= 'none';
      $('resultgroup').style.display= 'none';
      $('doscore').disabled = true;
      server.LiveScore($('subject').value, onScoreSuccess);
    // Callback for after a successful doAdd
    function onScoreSuccess(response) {
      $('progress').style.display= 'none';
      $('result').innerHTML = response;
       $('doscore').disabled = false;
      $('resultgroup').style.display= 'block';

<div id="realtimeScore" align="center" class="form-search">    
<input id="subject" type="text" placeholder="e.g. Barack Obama" />
<input id="doscore" type="button" value="Get Real-time Score" class="btn" onclick="doScore()" />
<div style="height:150px">
<div style="height:30px"></div>
<div  align="center">
<div  class="progress progress-striped active" id="progress" style="width: 10%;height:20px;display:none">
  <div  class="bar" style="width: 100%">Calculating score...</div>
<div style="height:30px" id="placeholder"></div>
<div id="resultgroup"  align="center" style="display:none;height:30px"> 
<label id="result" class="lead"></label>
<button class="btn btn-info">How was this calculated?</button>

Important point to note, which should be obvious, is there's not really a security implementation here.  The one I will be implementing is to dynamically create a unique identifier key server side which will then be passed into the web page and subsequently passed back to the RPC function.  The RPC function will then verify the unique identifier and expire the key after a time of a few minutes or once it's been used.  In short, you should always ensure you have a robust, and well tested, solution for security.  On the server side you should also sanitise any requests made to ensure malicious requests that could possibly find their way in are filtered out.

Server side code for handling client-side calls and using deferred tasks for parallel processing

So now we've made the call from the javascript code we need to first handle the call via a generic RPC function on the server.  Hence implement the following code (again note no security implementation at this point) in the module.

def rpc(request):
   action= request.GET.get('action')
   if action=='LiveScore':
       return HttpResponse(json.dumps(result), content_type="text/plain")

So you can see here you could implement multiple functions from here.  For example if you wanted to add "CloseAccount" just call server.CloseAccount from the javascript and handle the "CloseAccount" action in the RPC function. 

So far so simple.  To get interesting we'll now looked at deferred tasks which are a very powerful capability that app engine does for you - so it's very easy to implement in your code.  The way I like to think of it is similar to multi-threading, where you can easily control the volume and speed of request processing.  First thing you'll need to do is create a queue.yaml file to define the speed and size of the queue processing (full explanation here  Mine looks like this at the moment:

# Set the total storage limit for all queues to 120MB
total_storage_limit: 120M
- name: default
  rate: 100/s
  bucket_size: 40

With this in place you can then use the deferred queuing functionality in your application.  This means you can spin up multiple calls and the appengine will do all the work of processing them in parallel, or queue them if the volume of queued processes is too high (based on your settings).  All your app has to do is then kick back and get the results.  Utilising this functionality means you can implement some pretty awesome functionality - your application can perform a lot of work server side by running in parallel and return the results very quickly to the user.  To implement the code, just start by implementing the libraries as follows:

from google.appengine.api import taskqueue
from google.appengine.ext import deferred

Then if you look at this code were performing an url get by calling the function by calling processResult as a deferred task, passing the url in as a parameter. We can make multiple deferred calls like this and they will then all be handled simultaneously by the app engine.

url=baseurl + '?sm=' + str(sm) + '&sj=' + search + '&sn=' + sessionid  +  '&bc=12&bs=' + str(bs) + '&appkey=' + key

Social Samplr Survey  - $100 Amazon gift voucher up for grabs

As a side note, the new site is coming along nicely.  In the meantime, I'm offering a US$100 Amazon gift voucher to one person who fills out the short survey on what I've done so far on Socialsamplr - click the link here.  Obviously I will need your email to send the voucher to if you're the lucky winner but otherwise I'd really appreciate anyone who fills it out - should take less than half a minute.

Thanks, until next time.


No comments:

Post a Comment