A daemon, classically on most Unix™ first closes the three main I/O streams: stdin, stdout and stderr, then the fork(2) system call is used, creating an image of the current process, once the call is made, an exit(1) call is made on the parent process, and the child process keeps working in background. Due to the Python philosophy of being a productive language, this not so complex process is already implemented in the daemon module, so with this module you can create a daemon program.

import os
import sys
import daemon
import atexit

def main():
    """
    Main Program
    """

    install_signal_handler()
    atexit.register(at_exit_handler)

    opts = parse_opts()
    config = parse_config(opts)
    if not opts.cwd:
        print("No Working Directory")
        sys.exit(0)

    with daemon.DaemonContext():
        os.chdir(opts.cwd)
        install_signal_handler()
        start_schedule(config, opts)

""" Executes the main() Function """
if __name__ == '__main__':
    main()

If you observe the code, it changes the working directory using os.chdir(), because once a daemon is created, its working directory is automatically changed to the root directory or /. Some daemons or servers like the Apache HTTP server, have a precompiled working directory, but it also allows to use chroot directory to make it work outside the precompiled working directory. Also, it is installing signal handlers, to allow signal processing, like SIGHUP and SIGCHLD. The parse_opts() and parse_config() functions, are functions to parse daemon arguments and daemon configuration using the optparse and ConfigParser modules respectively.

The atexit module is used to ensure that most program resources — like files, connections and similar stuff — are released or closed, once the daemon process terminates. So, the following atexit example shows a routine releasing resources.

from django.db import connections
import atexit

def at_exit_handler():
    """
    At Exit Function (close all connections)
    """
    for con in connections.all():
        try:
            con.close()
        except Exception, exc:
            log().error(exc)
    log().close()

The signal handler have very similar behaviour, it register a processing callback, which is executed using each time a signal is received. For example on the SIGHUP signal, I am doing a daemon rehash, rather than doing a daemon termination.

import signal as sig
import traceback as tb

def rehash_daemon():
    """
    Rehash the daemon, reading configurations again.
    """
    global DAEMON_CONFIG
    DAEMON_CONFIG = parse_config()

def install_signal_handler():
    """
    Signal Handler Installer
    """
    try:
        sig.signal(sig.SIGHUP, rehash_daemon)
    except Exception, exc:
        log().error(repr(exc))
        log().error(tb.format_exc(exc))
        return False

This will allow you to create a robust daemon, capable to handle signals, resource releasing at the exit event and similar stuff. Remember that as any language with a garbage collector, Python has the disadvantage of leak memory if you do not cut the object references properly, due to the reference counting used by the garbage collector inside the GIL cycles. Your daemon design should be optimal.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Once you have configured and working the django-storages package, you should fill the Date header or the x-amz-date header each time that you do a request to S3. So, the only way to do that is not to leave the module sending the date automatically, instead you should write the settings variable AWS_HEADERS.

import pytz
from datetime import datetime
from django.conf import settings

def get_aws_date(self):
    """
    Returns the server date formatted and using the server
    time zone localized to be used as Date header with
    Amazon S3.
    """
    stz = pytz.timezone(settings.TIME_ZONE)
    dtm = stz.localize(datetime.now()).strftime("%a, %d %b %Y %H:%M:%S %z")
    settings.AWS_HEADERS['Date'] = dtm
    settings.AWS_HEADERS['x-amz-date'] = dtm
    return dtm

Also, you must consider this requirement to work with S3 on all your requests. Since django-storages checks if the AWS_HEADERS has the Date or x-amz-date headers set, you must set that header each time that you make a request to S3, so you cannot use a batch read() from the storage or a batch write() to the storage, because it will use the previously sent Date header and it will fail, because it is considered inconsistent by the S3 authentication mechanism. If we use S3 as default storage, the example below will fail, because it will send file chunks with the same Date header, because the header was set previously.

from django.core.files.storage import default_storage

if default_storage.exist('test-large-file.mp3'):
    mp3file = open('test-large-file.mp3')
    s3file = default_storage.open('test-large-file.mp3')
    mp3file.write(s3file.read())
    mp3file.close()
    s3file.close()

So, you need to use small chunks to read from the storage, as the example below.

from django.core.files.storage import default_storage

if default_storage.exist('test-large-file.mp3'):
    get_aws_date()
    mp3file = open('test-large-file.mp3')
    s3file = default_storage.open('test-large-file.mp3')
    buff = s3file.read(settings.AWS_CHUNK_SIZE)
    while buff:
        mp3file.write(buff)
        try:
            buff = s3file.read(settings.AWS_CHUNK_SIZE)
        except Exception, exc:
            buff = None
    mp3file.close()
    s3file.close()

But also you must consider few issues about this. Since AWS_HEADERS is a global variable, writing to that variable will slow down your code because it lacks time on GIL usage, also it will lack your threaded application if it written twice, even if you are using locks and Django can handle parallel writes to that variable. So, be careful reading and writing large files from S3, and take a look on how the Date header is sent on each request.

def _add_aws_auth_header(self, headers, method,
                         bucket, key, query_args):
    if not headers.has_key('Date'):
        headers['Date'] = time.strftime("%a, %d %b %Y %X GMT",
                                        time.gmtime())

    c_string = canonical_string(method, bucket, key,
                                query_args, headers)
    headers['Authorization'] =
        "AWS %s:%s" % (self.aws_access_key_id,
                       encode(self.aws_secret_access_key, c_string))

Where the %X specifier for localized machines with an environment variable LC_ALL different from C will throw the wrong date format disallowing your application to work with S3. This is a well known bug reported on this link. So, the right implementation, should be as follows.

def _add_aws_auth_header(self, headers, method,
                         bucket, key, query_args):
    if not 'Date' in headers:
        stz = pytz.timezone(settings.TIME_ZONE)
        dtm = stz.localize(datetime.now()).strftime("%a, %d %b %Y %H:%M:%S %z")
        headers['Date'] = dtm

    c_string = canonical_string(method, bucket, key,
                                query_args, headers)
    headers['Authorization'] =
        "AWS %s:%s" % (self.aws_access_key_id,
                       encode(self.aws_secret_access_key, c_string))

Good luck using S3.

© dmw for coder . cl, 2012. | Permalink | 6 comments | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

from django import template
from django.conf import settings

from owner.common.decorators import html_escape_output

register = template.Library()

def fmt_owner_name(_owner):
    """
    Formats the Owner Name
    """
    @html_escape_output()
    def _fmt_owner_name(owner):
        nms = u'%s %s' % (first_word_on_string(owner.first_name),
                          first_word_on_string(owner.last_name))
        if len(nms) > settings.MAX_NAME_ON_CARD:
            return u'%s...' % (nms[0:settings.MAX_NAME_ON_CARD])
        else:
            return nms
    return _format_owner_name(_owner)
fmt_owner_name = register.simple_tag(fmt_owner_name)

So, in this example, the template tag will escape the owner name, where all template tags in Django are considering HTML output as safe output, so they do not escape the HTML output automatically, even if you use is_safe and needs_autoescape properties, they require special treatment when you need to escape the HTML tags. In the example above the closure _fmt_owner_name is decorated with @html_escape_output which forces escaped HTML output to the Django template.

Also you can stack any number of decorators that you want. I think that this avoids using cgi.escape and similar stuff on each return statement. This helps you on Django template tag processing with common operations, like the example above. And it reduces the amount of code, making it more legible and simple, where we are using the fourth aphorism from the Zen of Python: «complex is better than complicated». Then we apply the template tag in the Django template as follows.

<div class="owner-name">
  {% fmt_owner_name owner_obj %}
</div>

I hope that this will help you on the task of reducing the amount of repeated calls.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Without falling in numerous details, I will try to explain how to mount selenium tests under Django. The very first step is to download and install the selenium plug-in, that allows you to create automated functional tests for the selenium API.

Selenium IDE

Once you have created the functional test using the record option, you can save it with the native selenium format, which is a simplified HTML file. Once you saved the test case, you can export the test case using the export option on the file menu, where you will see two formats provided for Python integration, called Web Driver and Remote Control. The first one called Web Driver is used with a local browser — on the local test server — and the second one called Remote Control should be used with grid servers, an option that supports multiple browsers by calling remote actions on a grid of parallel test servers. Once you export the file in the Web Driver format, you see that the python code is based on the standard unit testing suite, as the example that follows.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import unittest, time, re

class TestWebdriver(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.implicitly_wait(30)
        self.base_url = "https://myserver.com/"
        self.verificationErrors = []

    def test_login_view(self):
        driver = self.driver
        driver.get("/home/")
        driver.find_element_by_link_text("Login").click()
        driver.find_element_by_id("username").clear()
        driver.find_element_by_id("username").send_keys("test@myserver.com")
        driver.find_element_by_id("password").clear()
        driver.find_element_by_id("password").send_keys("testpass123123")
        driver.find_element_by_css_selector("input.submit.login-but").click()
        driver.find_element_by_link_text("MainPanel").click()

    def is_element_present(self, how, what):
        try: self.driver.find_element(by=how, value=what)
        except NoSuchElementException, e: return False
        return True

    def tearDown(self):
        self.driver.quit()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

To run that test, the file should be saved on the app/tests/ directory, and should run using a standard browser. In my case, I am using django-jenkins for continuous integration with Jenkins or Atlassian Bamboo. So, you should add the Python Selenium API, where you will find the webdriver package to use with the web driver API — remember that it is used with a local browser — or the selenium package that provides API calls to use grid servers for parallel tests.

So, the functional test integration runs as standard integration for any build. You can separate the test suites customising the build script for Jenkins or Bamboo and have separate reports. That is a preferred option if you want to keep detailed information about your tests. Enjoy integrating automated functional tests.

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

tl;dr

I prefer multi-paradigm languages because you can do cool stuff with less code.

why do I prefer it?

If you reduce the amount of code that you are programming, you can decrease the amount of possible bugs by reducing its cyclomatic complexity. So, using functional approach in certain cases will reduce the amount of code that you are producing, increasing your productivity considerably. Take a look on the following code made on PHP to find the lowest value in an array of associative arrays.

<?php

/* PHP example on associative arrays */

$min = 9999;
$tst = array(array('a' => 30, 'b' => 20, 'c' => 10),
             array('a' => 29, 'b' => 19, 'c' => 9),
             array('a' => 28, 'b' => 18, 'c' => 8));

foreach ($tst as $t) {
    if ($t['a'] < $min) {
        $min = $t['a'];
    }
}

echo "{$min}n";

 ?>

Now take a look on the same problem but implemented in Python using multi-paradigm approach, with functional features and using higher order functions.

### Python example using higher order functions and
### functional approach

tst = [{'a': 30, 'b': 20, 'c': 10},
       {'a': 29, 'c': 19, 'c': 9},
       {'a': 28, 'c': 18, 'c': 8}]

t = min(*tst, key=lambda x: x['a'])
print(t['a'])

You can clearly see that the code was reduced to only one line of code using a functional approach, using min as higher order function, so the η-reduction is evident. And you can handle a wide variety of problems using the functional approach, reducing the amount of lines of code considerably, increasing your productivity and doing better code. Do you remember how to calculate the factorial?, do you remember the factorial example in Python?. Take a look on the PHP implementation against the Python implementation.

<?php

/* PHP factorial example */

function fact($n) {
    if ($n == 0) return 1;
    $r = 1;
    for ($i = 1; $i <= $n; $i++) {
        $r = $r * $i;
    }
    return $r;
}

$f = fact(4);
echo "{$f}n";

 ?>

Now take a look on the Python implementation. You will notice a multi-paradigm approach using procedural and functional paradigms, using higher order functions again.

### Python factorial example

def fact(n):
    if n == 0: return 1
    return reduce(int.__mul__, xrange(1, n + 1))

print(fact(4))

I really like to know how functional programming can allow you to reduce the amount of lines of code which are required to complete certain tasks, and how it is expressive as procedural code does, but it is pretty much well fluent and it has less cyclomatic complexity than the procedural approach. Functional code is very legible. The other advantage is the fact that some compilers supporting the functional paradigm are enabled with tail calls implementation, so you do not need to worry about creating stack frames and generating stack overflows with recursive functions. Languages that currently supports multi-paradigm approach are JavaScript, Python, Ruby and Scala, among other languages.

So, I recommend you that you should start learning some functional programming principles and techniques. On the past I got tired of those companies with silly restrictions about using reductions and making the code looks like literate programming code, instead of well qualified enterprise code, and since I am working as freelancer, I just use the best practices that each language requires.

Do not get stuck in object oriented and procedural paradigms only, learn other paradigms and approaches, so you can handle a wider number of problems reaching better solutions, more optimal in lines of code, cyclomatic complexity, algorithmic complexity and time complexity. In some manner, what I can say is the fact that more lines of code do not means better, it depends on the language that you are using and how do you implement your solutions, and having a multi-paradigm approach can lead you to deliver better code. Sometimes — certain languages — can allow you to implement the same solution with less code, you just need to know the proper technique.

Many languages now are including some functional features, and are going multi-paradigm. For example Java 8 has included Lambda Expressions, C# includes Lambda Expressions, they just need to implement Closures to start doing some cool stuff in certain cases. PHP 5.3 now includes Lambda Expressions or Anonymous Functions. So, you should learn about the functional paradigm, among other stuff that will be required in the future.

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

tl;dr

You can setup a query API that throws XML through HTTP under Django using pyxser.

advice

All examples here are working, you must be really careful with the authentication and object permissions before using the routines in this post. So, try to wrap those routines correctly using the Django authentication components to filter query requests. Probably OAuth related modules may help. Also the examples are not using the Python and Django best practices, so you need to adjust them to fit the best practices requirements. Finally, do not take all examples very literal, they are just examples and this is just a proof of concept article.

serializing model objects

The pyxser extension — which is written in C and uses libxml2 as its basis for XML processing — has two main arguments for the serialization routines: obj and enc, where obj is the object to be serialized and enc is the XML encoding to be used, so you can serialize a valid object using pyxser.serialize(obj = my_object, enc = 'utf-8'). You can see the full pyxser documentation using the pydoc command and looking forward for the pyxser module.

To serialize Django models, you need to restrict some fields, so you need to filter them, you do not need to worry about processing each model field, you just need to worry to filter the model fields properly using the selector argument and the depth argument. Take a look on the following decorator.

import pyxser as px

def render_to_xml(**pyxser_args):
    def outer(f):
        @wraps(f)
        def inner_xml(request, *args, **kwargs):

            result = f(request, *args, **kwargs)
            r = HttpResponse(mimetype='text/xml')
            try:
                render = px.serialize(obj=result,
                                      enc='utf-8',
                                      **pyxser_args)
            except Exception, exc:
                render = "<pyxs:obj/>"
            if result:
                r.write(render)
            else:
                r.write("<pyxs:obj/>")
            return r
        return inner_xml
    return outer

If you apply the decorator above in a Django view, it will return the serialized object as text/xml to the HTTP client. So, your view must return a valid object to be serialized by pyxser. It applies the pyxser.serialize function to the given output from your view. Now take a look to a view which uses this decorator to throw XML.

def get_class(kls):
    try:
        parts = kls.split('.')
        module = ".".join(parts[:-1])
        m = __import__(module)
        for comp in parts[1:]:
            m = getattr(m, comp)
        return m
    except:
        return False

## use an URL as follows:
## (r'x/g/(?P<model>[w.]+)/(?P<oid>d+)/',
##  u'views_xml.get_model_object'),
@require_http_methods(["GET", "OPTIONS", "HEAD"])
@render_to_xml(selector=do_value_attrs, depth=2)
def get_model_object(request, model=None, oid=None):
    obj = object()
    try:
        db_model = get_class(model)
        obj = db_model.objects.get(pk=oid)
        return obj
    except Exception, exc:
        log.error(exc)
    return obj

The view above returns an object from the given model name model and the given primary key oid, and passes the do_value_attrs selector function as attribute selector to pyxser, and restrict the serialization depth to two levels. Remember that pyxser allows to serialize circular references and cross references between objects, so we need to restrict the serialization depth, in case of Django models we can work with 2 levels in almost all models and the field selector do_value_attrs can be defined as follows.

DENIED_FIELDS = ['user', 'customer', 'users', 'customers']
DENIED_CLASSES = ['Related', 'Foreign', 'Many']

def is_allowed_class(fld):
    for nm in DENIED_CLASSES:
        if nm in fld.__class__.__name__:
            return False
    for nm in DENIED_FIELDS:
        if nm in fld.name:
            return False
    return True

def do_value_attrs(o):
    values = dict()
    if hasattr(o, '_meta') and hasattr(o._meta, 'fields'):
        for fldn in o._meta.fields:
            if is_allowed_class(fldn):
                values[fldn.name] = getattr(o, fldn.name)
    else:
        for kw in o.__dict__:
            values[kw] = getattr(o, kw)
    return values

Where we are filtering all fields in model objects that we do not want to serialize and all field classes that pyxser should not serialize for plain object transmision. Other objects which are not model related objects are serialized as plain Python objects using their dictionaries to get the object attributes, and also DENIED_FIELDS are skipped and DENIED_CLASSES are skipped too. The resulting XML for URLs like /p/x/g/offer.models.Marca/1/ is as follows.

<?xml version="1.0" encoding="utf-8"?>
<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
          version="1.0"
          type="Marca"
          module="prod.models"
          objid="id3128007116">
  <pyxs:prop type="unicode" name="nombre" size="4">Sony</pyxs:prop>
  <pyxs:prop type="long" name="id">1</pyxs:prop>
  <pyxs:prop type="unicode" name="slug" size="4">sony</pyxs:prop>
</pyxs:obj>

The pyxser serialization model holds type information, so any serialized object carries type information to be user in deserialization tasks, then you can handle the object back in any machine supporting pyxser and get the object deserialized to its original class using the unserialize function.

defining object containers

The pyxser extension cannot handle Django model containers directly, I mean those returned by the all method in query sets. So, you need to create a plain container to hold those objects that are retrieved from the database. Take a look on following view.

class Container(object):
    count = 0
    items = []
    def __init__(self):
        pass

def collect_filters(qd):
    data = qd.copy()
    filters = dict()
    for kw in data:
        if kw.startswith('filter__'):
            name = kw.replace('filter__', '')
            filters[name] = data[kw]
    return filters

### use an URL as follows:
### (r'x/l/(?P<model>[w.]+)/(?P<limit>d+)/',
###  u'views_xml.get_model_list'),
@require_http_methods(["GET", "OPTIONS", "HEAD"])
@render_to_xml(selector=select_value_attrs, depth=4)
def get_model_list(request, model=None, limit=1):
    container = Container()
    container.count = 0
    container.items = []
    try:
        db_model = get_class(model)
        filters = collect_filters(request.GET)
        objs = db_model.objects.filter(**filters).all()[0:limit]
        container.count = len(objs)
        container.items = map(lambda x: x, objs)
        return container
    except Exception, exc:
        log.error(exc)
    return container

If you take a look carefully to this example, you will notice that we are using a very simple Container class to hold your objects. The resulting XML for the URL /p/x/l/prod.models.Marca/5/?filter__nombre__contains=son is as follows.

<?xml version="1.0" encoding="utf-8"?>
<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
          version="1.0"
          type="Container"
          module="prod.views_xml"
          objid="id3107379372">
  <pyxs:prop type="int" name="count">3</pyxs:prop>
  <pyxs:col type="list" name="items">
    <pyxs:obj type="Marca" module="prod.models" objid="id3107380204">
      <pyxs:prop type="unicode" name="nombre" size="5">Epson</pyxs:prop>
      <pyxs:prop type="long" name="id">10</pyxs:prop>
      <pyxs:prop type="unicode" name="slug" size="5">epson</pyxs:prop>
    </pyxs:obj>
    <pyxs:obj type="Marca" module="prod.models" objid="id3107380300">
      <pyxs:prop type="unicode" name="nombre" size="8">Ericsson</pyxs:prop>
      <pyxs:prop type="long" name="id">15</pyxs:prop>
      <pyxs:prop type="unicode" name="slug" size="8">ericsson</pyxs:prop>
    </pyxs:obj>
  </pyxs:col>
</pyxs:obj>

The resulting XML serialized object holds three Marca objects and all of them have their type to be deserialized once they are retrieved. If you want your objects to be deserialized back, you just need to use the pyxser.unserialize function properly, which is documented in the pyxser extension itself. I hope that you will like how pyxser works.

© dmw for coder . cl, 2011. | Permalink | One comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Some examples of Monoids, should be , , and so on, to describe some operations that should be mapped to natural numbers. Applied to the Python programming language, he describes sets as lists or tuples, and Python data types as Categories to treat certain operations as Monoids. Francisco has proposed a set of challenges regarding Monoids in Python. Here I will describe some solutions to those challenges.

1. Using only listm, how would you concatenate a list of lists. That is, how would you transform [[a, b, c], [d, e], [f], []] into [a, b, c, d, e, f]?
2. The reverse of a monoid is a monoid where the operator take its arguments in reverse order. Create a method, reverse, that creates a reverse monoid. What is listm.reverse()?
3. Are there monoids that are fixed-points of the star method? That is, is there a monoid m such that m and m.star() are indistinguishable? (Side challenge: formalize the notion of “indistinguishable”.)
4. Do monoids have a dual concept? Monoids, as defined here, are roughly an embodiment of “map/reduce”, which is used to process and analyze data, potentially taking a large body of data and reducing it. Is there a similar concept, a “comonoid”, that generates data, instead of reducing it?
5. Let lift = int, and let op return an int. Can we enumerate all possible monoids? If not, what can we enumerate?

the final monoid class

class Monoid(object):
    null = None
    lift = None
    op = None

    def __init__(self, null, lift, op):
        self.null = null
        self.lift = lift
        self.op = op

    def fold(self, xs):
        if hasattr(xs, '__fold__') and callable(xs.__fold__):
            return xs.__fold__(self)
        else:
            mxs = map(self.lift, xs)
            return reduce(self.op, mxs, self.null)

    def rfold(self, xs):
        def reverse_reduce(op, lst, null):
            acm = null
            if len(lst) <= 0:
                return acm
            cnt = 0
            lim = len(lst) - 1
            while cnt < lim:
                acm = op(lst[cnt], acm)
                cnt += 1
            return op(lst[lim], acm)
        mxs = map(self.lift, xs)
        return reverse_reduce(self.op, mxs, self.null)

    def __call__(self, *args):
        return self.fold(args)

    def star(self):
        return Monoid(self.null, self.fold, self.op)

    def reverse(self):
        class ReverseMonoid(Monoid):
            def __call__(self, *args):
                return self.rfold(*args)
            def star(self):
                return ReverseMonoid(self.null, self.rfold, self.op)
        return ReverseMonoid(self.null, self.lift, self.op)

1. listm as list concatenator

## challenge 1

listm = Monoid([], lambda x: [x], lambda a, b: a + b)
sample_list = [['a', 'b', 'c'], ['d', 'e'], ['f'], []]
a = listm.star().fold(sample_list)
print("a = %s" % (repr(a)))

## end challenge 1

2. reverse litsm monoid

A reverse Monoid, takes the operation arguments on reverse order, instead of passing a, b, it uses b, a as operation arguments on the reduce function. The reverse() method returns a ReverseMonoid instance.

## challenge 2

listmr = listm.reverse()
b = listmr(a)
print("---> Challenge 2.")
print("b = %s" % (repr(b)))

## end challenge 2

3. fixed-point monoids

A fixed-point monoid to its star() method requires a fold() or reduction method returning the same type as its arguments. The only type that allows such stuff in native Python is the string, so we will use the string type related Monoid to define a fixed-point Monoid to its star() method. The joinm monoid meets that requirement.

### challenge 3.1

sample_string1 = "hello 1!"
sample_string2 = "hello 2!"
sample_string3 = "hello 3!"

joinm = Monoid('', lambda x: str(x), lambda a, b: a + b)

sjoinm  = joinm.star()
rjoinm = joinm.reverse()

print("---> Challenge 3.2")

c0 = joinm(sample_string1,
           sample_string2,
           sample_string3)
print("c0 = %s" % (repr(c0)))

c1 = sjoinm(sample_string1,
            sample_string2,
            sample_string3)
print("c1 = %s" % (repr(c1)))

## end challenge 3

The example above, should be formalized as . Other example can be implemented using some type checking tricks.

## challenge 3.2

def sum_list(x):
    return ([lambda n: sum(n),
             lambda m: m][cmp(type(x),
                              list)])(x)

def reduce_list(a, b):
    return ([lambda n: sum(n),
             lambda m: m][cmp(type(a),
                              list)])(a)
           + ([lambda n: sum(n),
               lambda m: m][cmp(type(b),
                                list)])(b)

sample_list = [1, 2, 3, 4, 5]

summ = Monoid(0, sum_list, reduce_list)
ssumm  = summ.star()
rsumm = summ.reverse()

print("---> Challenge 3.1")

c0 = summ(sample_list)
print("c0 = %s" % (repr(c0)))

c1 = ssumm(sample_list)
print("c1 = %s" % (repr(c1)))

c2 = rsumm(sample_list)
print("c2 = %s" % (repr(c2)))

## end challenge 3.2

4. comonoid using monoid constructor

## challenge 4

def sum_list(x):
    return range(0, ([lambda n: sum(n),
                      lambda m: m][cmp(type(x),
                                       list)])(x))

def reduce_list(a, b):
    return range(0, ([lambda n: sum(n),
                      lambda m: m][cmp(type(a),
                                       list)])(a)
                 + ([lambda n: sum(n),
                     lambda m: m][cmp(type(b),
                                      list)])(b))

### challenge 4
sample_list = [1, 2, 3, 4, 5]
summ = Monoid(0, sum_list, reduce_list)
ssumm  = summ.star()
rsumm = summ.reverse()

print("---> Challenge 4")

c0 = summ(sample_list)
print("c0 = %s" % (repr(c0)))

c1 = ssumm(sample_list)
print("c1 = %s" % (repr(c1)))

c2 = rsumm(sample_list)
print("c2 = %s" % (repr(c2)))

## end challenge 4

5. enumerate int monoids

Since the number of operations that can be made using the int type in Python, we cannot enumerate such set of monoids. Such monoid, that should be declared as Monoid (0, lambda x: int(x), f), where f can be any function returning int. The only set that we can enumerate is the limited number of integer numbers that can operate with f. I hope that you will enjoy the examples above.

© dmw for coder . cl, 2011. | Permalink | One comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Django works using two layers to build queries. The first layer is model descriptive and the second one is the SQL compiler, which has its basis on the model description layer to construct each query. So you need to define both classes to build your own Aggregate — and skip the Django side processing that requires some complex queries. The first one is the SQL layer, which must define the proper SQL elements to be used to construct the query aggregate.

from django.db import models

class SumWithFunctionSQL(models.sql.aggregates.Aggregate):
    sql_function = 'SUM'
    sql_template = '%(function)s( %(proc_func)s( %(field)s ) )'

class SumWithFunction(models.Aggregate):
    name = 'Sum'

    def add_to_query(self, query, alias, col, source, is_summary):
        aggregate = SumWithFunctionSQL(col,
                                       source=source,
                                       is_summary=is_summary,
                                       **self.extra)
        query.aggregates[alias] = aggregate

On the example above, we will provide proc_func extra argument to the aggregate constructor, so we will be able to use the Sum aggregate wrapping a function call on each processed row, which will be provided by the proc_func argument. On the next example, we will see an If construct to be used with MySQL, which can be replaced by Case statements on PostgreSQL.

from django.db import models
from django.conf import settings

sum_if_sql_template = '%(%s)s(IF(%(%s)s, %(%s)s, %(%s)s))' %
                      ('function',
                       'condition',
                       'when_true',
                       'when_false')

class SumIfSQL(models.sql.aggregates.Aggregate):
    sql_function = 'SUM'
    sql_template = sum_if_sql_template

class SumIf(models.Aggregate):
    name = 'Sum'

    def add_to_query(self, query, alias, col, source, is_summary):
        aggregate = SumIfSQL(col,
                             source=source,
                             is_summary=is_summary,
                             **self.extra)
        query.aggregates[alias] = aggregate

The example above uses the condition, when_true and when_false extra arguments to be passed to the aggregate constructor, where the condition holds the SQL condition which must meet the SQL layer, when_true the value to be used in the Sum aggregate when the condition evaluates to true and when_false hold the value to be used when the condition evaluates to false.

query = AccountDetail.
        objects.
        values('account', 'currency', 'interest', 'dividend').
        annotate(quantity_sum=Sum('quantity'),
                 amount_sum=Sum('amount'),
                 applied_interest=SumIf('amount',
                                        condition='amount > 0.0',
                                        when_true='amount * interest',
                                        when_false='0.0'),
                 applied_dividend=SumIf('amount',
                                        condition='amount < 0.0',
                                        when_true='amount * dividend',
                                        when_false='0.0')).
        order_by('account', 'currency', 'interest', 'dividend').
        filter(**filter)
details = query.all()
return details

On the example above we have made a simple usage of the SumIf aggregate. I hope that you will like how it works and build a similar solution. The examples here do are not using the SQL layer compiler due to its complexity, but if you study enough you will be able to use the SQL compiler.

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

The first observation is the fact that all available algorithms have an exponential increment on CPU usage while they increase its complexity. Since hashlib do not uses locking, it is transparent to the module to be used in a threaded environment. There is no resource locking or thread blocking routines in the module, so it do not requires synchronisation, and even it doesn’t use synchronisation internally. Probably OpenSSL uses some kind of it, but it is transparent, without affecting the runtime.

HMAC Algorithm Performance

Is well known that the probability of hash collisions on lower algorithms, like MD5 and SHA1 is higher than SHA512 and similar ones. Probably we just need to use a safer algorithm that offers a better performance than others, but offers a safer digest like SHA256 as minimal requirement, so we store passwords for a long time without the risk of finding some password cracker that can crash our security system. We know that MD5 and SHA1 are not safe any more.

If you’re willing to spend about 2,000 USD and a week or two picking up CUDA, you can put together your own little supercomputer cluster which will let you try around 700,000,000 passwords a second. And that rate you’ll be cracking those passwords at the rate of more than one per second. [Source bcrypt].

On a threaded environment you can see that lower algorithms have a similar performance and behaviour. All of them seems to have a very similar behaviour, where thread creation is time consuming, but once all threads are created it begins increasing the time required to complete the task.

Non-Strong HMAC Algorithms Performance in Threaded Environment

The performance for strong algorithms seem to be similar, but more time consuming that the lower ones. Also on higher algorithms, such as SHA384 and SHA512 have a more stable behaviour, using more homogeneous times to complete the task. Seems that the best option by stability and being stronger enough is SHA384, which has better performance than SHA512 and it reliable enough to store strong passwords. The bcrypt advice should be taken seriously by some institutions that require strong password storage, but mere applications like blogs and similar ones, do not require that kind of strength.

Strong HMAC Algorithms Performance in Threaded Environment

Probably we will adopt SHA384 as our best option. We need that process to be finished early and we do not have a TRNG installed on the machine to obtain enough entropy. We are thinking to buy an Entropy Key device to guarantee a better performance on cryptographic routines, like randomise initial passwords and bring a better performance on the server. Also we are thinking to buy some devices from Soekris Engineering to enhance the performance of certain tasks. I well known that cryptographic tasks, like using TLS/SSL connections over TCP like HTTPS can lead to heavy CPU usage. I hope that all of those tools will serve us well.

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

The two well known variants of the inotify(7) API for Python are inotifyx and pyinotify. The most complete file system notification API is provided by the pyinotify Python package.

The inotifyx interface provides a simple function based API, with an object wrapper for event objects through the InotifyEvent class. It lacks a main loop for event notifications, so you need to create the event loop by your own. Also it is small and simple, do not requires too much knowledge of the notification API.

So, the best option for me is the pyinotify interface, which provides full integration with the file system event API, so you can track with those events with a full object oriented implementation where you can handle any kind of events over files and directories. One of my recent projects is a log monitor daemon that integrates file system monitoring tasks to log storage on a RDBMS engine. Each time that a log is written, it examines the remaining lines from the current offset to the end of the file that has been written, and extracts the log entries to be stored on the RDBMS. The event chosen to be handled in this case is the IN_CLOSE_WRITE event, which indicates that the directory has suffered a file writes and file close events.

class WatchLogProceesing(pyinotify.ProcessEvent):
    """
    This class is a monitor for those files that are placed
    on the log directory. Once the system writes those system
    and closes them, this class notifies to its handler to
    process the event.
    """
    def process_IN_CLOSE_WRITE(self, event):
        """
        Process the close/write event.
        """
        daemon_log("%s: processing %s"
                   % (s.argv[0], event.pathname))
        process_input(event.pathname)

def monitor_dir(directory):
    """
    Starts monitoring a directory for changes using the
    pyinotify API
    """
    try:
        wmn = pyinotify.WatchManager()
        notifier = pyinotify.Notifier(wmn,
                                      WatchLogProceesing())
        wmn.add_watch(directory,
                      pyinotify.IN_CLOSE_WRITE)
        notifier.loop()
    except Exception, exc:
        print(str(exc))
        print(format_exc(exc))

The notifier.loop() call, means that the notifier will enter in an event loop handler, catching all included events in the WatchManager.add_watch() call. This API is very similar to the BSD Kqueue interface, but less complete, since inotify(7) only can handle file system events, and kqueue(2) can handle file descriptor events, so you can apply kqueue system calls to sockets and related stuff, making of kqueue more universal system calls for file descriptor events. Try playing with both, kqueue and inotify API, you will find it interesting :)

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)