Once you have configured and working the django-storages package, you should fill the Date header or the x-amz-date header each time that you do a request to S3. So, the only way to do that is not to leave the module sending the date automatically, instead you should write the settings variable AWS_HEADERS.

import pytz
from datetime import datetime
from django.conf import settings

def get_aws_date(self):
    """
    Returns the server date formatted and using the server
    time zone localized to be used as Date header with
    Amazon S3.
    """
    stz = pytz.timezone(settings.TIME_ZONE)
    dtm = stz.localize(datetime.now()).strftime("%a, %d %b %Y %H:%M:%S %z")
    settings.AWS_HEADERS['Date'] = dtm
    settings.AWS_HEADERS['x-amz-date'] = dtm
    return dtm

Also, you must consider this requirement to work with S3 on all your requests. Since django-storages checks if the AWS_HEADERS has the Date or x-amz-date headers set, you must set that header each time that you make a request to S3, so you cannot use a batch read() from the storage or a batch write() to the storage, because it will use the previously sent Date header and it will fail, because it is considered inconsistent by the S3 authentication mechanism. If we use S3 as default storage, the example below will fail, because it will send file chunks with the same Date header, because the header was set previously.

from django.core.files.storage import default_storage

if default_storage.exist('test-large-file.mp3'):
    mp3file = open('test-large-file.mp3')
    s3file = default_storage.open('test-large-file.mp3')
    mp3file.write(s3file.read())
    mp3file.close()
    s3file.close()

So, you need to use small chunks to read from the storage, as the example below.

from django.core.files.storage import default_storage

if default_storage.exist('test-large-file.mp3'):
    get_aws_date()
    mp3file = open('test-large-file.mp3')
    s3file = default_storage.open('test-large-file.mp3')
    buff = s3file.read(settings.AWS_CHUNK_SIZE)
    while buff:
        mp3file.write(buff)
        try:
            buff = s3file.read(settings.AWS_CHUNK_SIZE)
        except Exception, exc:
            buff = None
    mp3file.close()
    s3file.close()

But also you must consider few issues about this. Since AWS_HEADERS is a global variable, writing to that variable will slow down your code because it lacks time on GIL usage, also it will lack your threaded application if it written twice, even if you are using locks and Django can handle parallel writes to that variable. So, be careful reading and writing large files from S3, and take a look on how the Date header is sent on each request.

def _add_aws_auth_header(self, headers, method,
                         bucket, key, query_args):
    if not headers.has_key('Date'):
        headers['Date'] = time.strftime("%a, %d %b %Y %X GMT",
                                        time.gmtime())

    c_string = canonical_string(method, bucket, key,
                                query_args, headers)
    headers['Authorization'] = \
        "AWS %s:%s" % (self.aws_access_key_id,
                       encode(self.aws_secret_access_key, c_string))

Where the %X specifier for localized machines with an environment variable LC_ALL different from C will throw the wrong date format disallowing your application to work with S3. This is a well known bug reported on this link. So, the right implementation, should be as follows.

def _add_aws_auth_header(self, headers, method,
                         bucket, key, query_args):
    if not 'Date' in headers:
        stz = pytz.timezone(settings.TIME_ZONE)
        dtm = stz.localize(datetime.now()).strftime("%a, %d %b %Y %H:%M:%S %z")
        headers['Date'] = dtm

    c_string = canonical_string(method, bucket, key,
                                query_args, headers)
    headers['Authorization'] = \
        "AWS %s:%s" % (self.aws_access_key_id,
                       encode(self.aws_secret_access_key, c_string))

Good luck using S3.

Copyright © 2012 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment
Post tags:

from django import template
from django.conf import settings

from owner.common.decorators import html_escape_output

register = template.Library()

def fmt_owner_name(_owner):
    """
    Formats the Owner Name
    """
    @html_escape_output()
    def _fmt_owner_name(owner):
        nms = u'%s %s' % (first_word_on_string(owner.first_name),
                          first_word_on_string(owner.last_name))
        if len(nms) > settings.MAX_NAME_ON_CARD:
            return u'%s...' % (nms[0:settings.MAX_NAME_ON_CARD])
        else:
            return nms
    return _format_owner_name(_owner)
fmt_owner_name = register.simple_tag(fmt_owner_name)

So, in this example, the template tag will escape the owner name, where all template tags in Django are considering HTML output as safe output, so they do not escape the HTML output automatically, even if you use is_safe and needs_autoescape properties, they require special treatment when you need to escape the HTML tags. In the example above the closure _fmt_owner_name is decorated with @html_escape_output which forces escaped HTML output to the Django template.

Also you can stack any number of decorators that you want. I think that this avoids using cgi.escape and similar stuff on each return statement. This helps you on Django template tag processing with common operations, like the example above. And it reduces the amount of code, making it more legible and simple, where we are using the fourth aphorism from the Zen of Python: «complex is better than complicated». Then we apply the template tag in the Django template as follows.

<div class="owner-name">
  {% fmt_owner_name owner_obj %}
</div>

I hope that this will help you on the task of reducing the amount of repeated calls.

Copyright © 2012 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment
Post tags:

Without falling in numerous details, I will try to explain how to mount selenium tests under Django. The very first step is to download and install the selenium plug-in, that allows you to create automated functional tests for the selenium API.

Selenium IDE

Once you have created the functional test using the record option, you can save it with the native selenium format, which is a simplified HTML file. Once you saved the test case, you can export the test case using the export option on the file menu, where you will see two formats provided for Python integration, called Web Driver and Remote Control. The first one called Web Driver is used with a local browser — on the local test server — and the second one called Remote Control should be used with grid servers, an option that supports multiple browsers by calling remote actions on a grid of parallel test servers. Once you export the file in the Web Driver format, you see that the python code is based on the standard unit testing suite, as the example that follows.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import unittest, time, re

class TestWebdriver(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.implicitly_wait(30)
        self.base_url = "https://myserver.com/"
        self.verificationErrors = []

    def test_login_view(self):
        driver = self.driver
        driver.get("/home/")
        driver.find_element_by_link_text("Login").click()
        driver.find_element_by_id("username").clear()
        driver.find_element_by_id("username").send_keys("test@myserver.com")
        driver.find_element_by_id("password").clear()
        driver.find_element_by_id("password").send_keys("testpass123123")
        driver.find_element_by_css_selector("input.submit.login-but").click()
        driver.find_element_by_link_text("MainPanel").click()

    def is_element_present(self, how, what):
        try: self.driver.find_element(by=how, value=what)
        except NoSuchElementException, e: return False
        return True

    def tearDown(self):
        self.driver.quit()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

To run that test, the file should be saved on the app/tests/ directory, and should run using a standard browser. In my case, I am using django-jenkins for continuous integration with Jenkins or Atlassian Bamboo. So, you should add the Python Selenium API, where you will find the webdriver package to use with the web driver API — remember that it is used with a local browser — or the selenium package that provides API calls to use grid servers for parallel tests.

So, the functional test integration runs as standard integration for any build. You can separate the test suites customising the build script for Jenkins or Bamboo and have separate reports. That is a preferred option if you want to keep detailed information about your tests. Enjoy integrating automated functional tests.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | No comment
Post tags:

tl;dr

I prefer multi-paradigm languages because you can do cool stuff with less code.

why do I prefer it?

If you reduce the amount of code that you are programming, you can decrease the amount of possible bugs by reducing its cyclomatic complexity. So, using functional approach in certain cases will reduce the amount of code that you are producing, increasing your productivity considerably. Take a look on the following code made on PHP to find the lowest value in an array of associative arrays.

<?php

/* PHP example on associative arrays */

$min = 9999;
$tst = array(array('a' => 30, 'b' => 20, 'c' => 10),
             array('a' => 29, 'b' => 19, 'c' => 9),
             array('a' => 28, 'b' => 18, 'c' => 8));

foreach ($tst as $t) {
    if ($t['a'] < $min) {
        $min = $t['a'];
    }
}

echo "{$min}\n";

 ?>

Now take a look on the same problem but implemented in Python using multi-paradigm approach, with functional features and using higher order functions.

### Python example using higher order functions and
### functional approach

tst = [{'a': 30, 'b': 20, 'c': 10},
       {'a': 29, 'c': 19, 'c': 9},
       {'a': 28, 'c': 18, 'c': 8}]

t = min(*tst, key=lambda x: x['a'])
print(t['a'])

You can clearly see that the code was reduced to only one line of code using a functional approach, using min as higher order function, so the η-reduction is evident. And you can handle a wide variety of problems using the functional approach, reducing the amount of lines of code considerably, increasing your productivity and doing better code. Do you remember how to calculate the factorial?, do you remember the factorial example in Python?. Take a look on the PHP implementation against the Python implementation.

<?php

/* PHP factorial example */

function fact($n) {
    if ($n == 0) return 1;
    $r = 1;
    for ($i = 1; $i <= $n; $i++) {
        $r = $r * $i;
    }
    return $r;
}

$f = fact(4);
echo "{$f}\n";

 ?>

Now take a look on the Python implementation. You will notice a multi-paradigm approach using procedural and functional paradigms, using higher order functions again.

### Python factorial example

def fact(n):
    if n == 0: return 1
    return reduce(int.__mul__, xrange(1, n + 1))

print(fact(4))

I really like to know how functional programming can allow you to reduce the amount of lines of code which are required to complete certain tasks, and how it is expressive as procedural code does, but it is pretty much well fluent and it has less cyclomatic complexity than the procedural approach. Functional code is very legible. The other advantage is the fact that some compilers supporting the functional paradigm are enabled with tail calls implementation, so you do not need to worry about creating stack frames and generating stack overflows with recursive functions. Languages that currently supports multi-paradigm approach are JavaScript, Python, Ruby and Scala, among other languages.

So, I recommend you that you should start learning some functional programming principles and techniques. On the past I got tired of those companies with silly restrictions about using reductions and making the code looks like literate programming code, instead of well qualified enterprise code, and since I am working as freelancer, I just use the best practices that each language requires.

Do not get stuck in object oriented and procedural paradigms only, learn other paradigms and approaches, so you can handle a wider number of problems reaching better solutions, more optimal in lines of code, cyclomatic complexity, algorithmic complexity and time complexity. In some manner, what I can say is the fact that more lines of code do not means better, it depends on the language that you are using and how do you implement your solutions, and having a multi-paradigm approach can lead you to deliver better code. Sometimes — certain languages — can allow you to implement the same solution with less code, you just need to know the proper technique.

Many languages now are including some functional features, and are going multi-paradigm. For example Java 8 has included Lambda Expressions, C# includes Lambda Expressions, they just need to implement Closures to start doing some cool stuff in certain cases. PHP 5.3 now includes Lambda Expressions or Anonymous Functions. So, you should learn about the functional paradigm, among other stuff that will be required in the future.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | No comment
Post tags:

tl;dr

You can setup a query API that throws XML through HTTP under Django using pyxser.

advice

All examples here are working, you must be really careful with the authentication and object permissions before using the routines in this post. So, try to wrap those routines correctly using the Django authentication components to filter query requests. Probably OAuth related modules may help. Also the examples are not using the Python and Django best practices, so you need to adjust them to fit the best practices requirements. Finally, do not take all examples very literal, they are just examples and this is just a proof of concept article.

serializing model objects

The pyxser extension — which is written in C and uses libxml2 as its basis for XML processing — has two main arguments for the serialization routines: obj and enc, where obj is the object to be serialized and enc is the XML encoding to be used, so you can serialize a valid object using pyxser.serialize(obj = my_object, enc = 'utf-8'). You can see the full pyxser documentation using the pydoc command and looking forward for the pyxser module.

To serialize Django models, you need to restrict some fields, so you need to filter them, you do not need to worry about processing each model field, you just need to worry to filter the model fields properly using the selector argument and the depth argument. Take a look on the following decorator.

import pyxser as px

def render_to_xml(**pyxser_args):
    def outer(f):
        @wraps(f)
        def inner_xml(request, *args, **kwargs):

            result = f(request, *args, **kwargs)
            r = HttpResponse(mimetype='text/xml')
            try:
                render = px.serialize(obj=result,
                                      enc='utf-8',
                                      **pyxser_args)
            except Exception, exc:
                render = "<pyxs:obj/>"
            if result:
                r.write(render)
            else:
                r.write("<pyxs:obj/>")
            return r
        return inner_xml
    return outer

If you apply the decorator above in a Django view, it will return the serialized object as text/xml to the HTTP client. So, your view must return a valid object to be serialized by pyxser. It applies the pyxser.serialize function to the given output from your view. Now take a look to a view which uses this decorator to throw XML.

def get_class(kls):
    try:
        parts = kls.split('.')
        module = ".".join(parts[:-1])
        m = __import__(module)
        for comp in parts[1:]:
            m = getattr(m, comp)
        return m
    except:
        return False

## use an URL as follows:
## (r'x/g/(?P<model>[\w\.]+)/(?P<oid>\d+)/',
##  u'views_xml.get_model_object'),
@require_http_methods(["GET", "OPTIONS", "HEAD"])
@render_to_xml(selector=do_value_attrs, depth=2)
def get_model_object(request, model=None, oid=None):
    obj = object()
    try:
        db_model = get_class(model)
        obj = db_model.objects.get(pk=oid)
        return obj
    except Exception, exc:
        log.error(exc)
    return obj

The view above returns an object from the given model name model and the given primary key oid, and passes the do_value_attrs selector function as attribute selector to pyxser, and restrict the serialization depth to two levels. Remember that pyxser allows to serialize circular references and cross references between objects, so we need to restrict the serialization depth, in case of Django models we can work with 2 levels in almost all models and the field selector do_value_attrs can be defined as follows.

DENIED_FIELDS = ['user', 'customer', 'users', 'customers']
DENIED_CLASSES = ['Related', 'Foreign', 'Many']

def is_allowed_class(fld):
    for nm in DENIED_CLASSES:
        if nm in fld.__class__.__name__:
            return False
    for nm in DENIED_FIELDS:
        if nm in fld.name:
            return False
    return True

def do_value_attrs(o):
    values = dict()
    if hasattr(o, '_meta') and hasattr(o._meta, 'fields'):
        for fldn in o._meta.fields:
            if is_allowed_class(fldn):
                values[fldn.name] = getattr(o, fldn.name)
    else:
        for kw in o.__dict__:
            values[kw] = getattr(o, kw)
    return values

Where we are filtering all fields in model objects that we do not want to serialize and all field classes that pyxser should not serialize for plain object transmision. Other objects which are not model related objects are serialized as plain Python objects using their dictionaries to get the object attributes, and also DENIED_FIELDS are skipped and DENIED_CLASSES are skipped too. The resulting XML for URLs like /p/x/g/offer.models.Marca/1/ is as follows.

<?xml version="1.0" encoding="utf-8"?>
<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
          version="1.0"
          type="Marca"
          module="prod.models"
          objid="id3128007116">
  <pyxs:prop type="unicode" name="nombre" size="4">Sony</pyxs:prop>
  <pyxs:prop type="long" name="id">1</pyxs:prop>
  <pyxs:prop type="unicode" name="slug" size="4">sony</pyxs:prop>
</pyxs:obj>

The pyxser serialization model holds type information, so any serialized object carries type information to be user in deserialization tasks, then you can handle the object back in any machine supporting pyxser and get the object deserialized to its original class using the unserialize function.

defining object containers

The pyxser extension cannot handle Django model containers directly, I mean those returned by the all method in query sets. So, you need to create a plain container to hold those objects that are retrieved from the database. Take a look on following view.

class Container(object):
    count = 0
    items = []
    def __init__(self):
        pass

def collect_filters(qd):
    data = qd.copy()
    filters = dict()
    for kw in data:
        if kw.startswith('filter__'):
            name = kw.replace('filter__', '')
            filters[name] = data[kw]
    return filters

### use an URL as follows:
### (r'x/l/(?P<model>[\w\.]+)/(?P<limit>\d+)/',
###  u'views_xml.get_model_list'),
@require_http_methods(["GET", "OPTIONS", "HEAD"])
@render_to_xml(selector=select_value_attrs, depth=4)
def get_model_list(request, model=None, limit=1):
    container = Container()
    container.count = 0
    container.items = []
    try:
        db_model = get_class(model)
        filters = collect_filters(request.GET)
        objs = db_model.objects.filter(**filters).all()[0:limit]
        container.count = len(objs)
        container.items = map(lambda x: x, objs)
        return container
    except Exception, exc:
        log.error(exc)
    return container

If you take a look carefully to this example, you will notice that we are using a very simple Container class to hold your objects. The resulting XML for the URL /p/x/l/prod.models.Marca/5/?filter__nombre__contains=son is as follows.

<?xml version="1.0" encoding="utf-8"?>
<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
          version="1.0"
          type="Container"
          module="prod.views_xml"
          objid="id3107379372">
  <pyxs:prop type="int" name="count">3</pyxs:prop>
  <pyxs:col type="list" name="items">
    <pyxs:obj type="Marca" module="prod.models" objid="id3107380204">
      <pyxs:prop type="unicode" name="nombre" size="5">Epson</pyxs:prop>
      <pyxs:prop type="long" name="id">10</pyxs:prop>
      <pyxs:prop type="unicode" name="slug" size="5">epson</pyxs:prop>
    </pyxs:obj>
    <pyxs:obj type="Marca" module="prod.models" objid="id3107380300">
      <pyxs:prop type="unicode" name="nombre" size="8">Ericsson</pyxs:prop>
      <pyxs:prop type="long" name="id">15</pyxs:prop>
      <pyxs:prop type="unicode" name="slug" size="8">ericsson</pyxs:prop>
    </pyxs:obj>
  </pyxs:col>
</pyxs:obj>

The resulting XML serialized object holds three Marca objects and all of them have their type to be deserialized once they are retrieved. If you want your objects to be deserialized back, you just need to use the pyxser.unserialize function properly, which is documented in the pyxser extension itself. I hope that you will like how pyxser works.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | One comment
Post tags: