tl;dr

You can setup a query API that throws XML through HTTP under Django using pyxser.

advice

All examples here are working, you must be really careful with the authentication and object permissions before using the routines in this post. So, try to wrap those routines correctly using the Django authentication components to filter query requests. Probably OAuth related modules may help. Also the examples are not using the Python and Django best practices, so you need to adjust them to fit the best practices requirements. Finally, do not take all examples very literal, they are just examples and this is just a proof of concept article.

serializing model objects

The pyxser extension — which is written in C and uses libxml2 as its basis for XML processing — has two main arguments for the serialization routines: obj and enc, where obj is the object to be serialized and enc is the XML encoding to be used, so you can serialize a valid object using pyxser.serialize(obj = my_object, enc = 'utf-8'). You can see the full pyxser documentation using the pydoc command and looking forward for the pyxser module.

To serialize Django models, you need to restrict some fields, so you need to filter them, you do not need to worry about processing each model field, you just need to worry to filter the model fields properly using the selector argument and the depth argument. Take a look on the following decorator.

import pyxser as px

def render_to_xml(**pyxser_args):
    def outer(f):
        @wraps(f)
        def inner_xml(request, *args, **kwargs):

            result = f(request, *args, **kwargs)
            r = HttpResponse(mimetype='text/xml')
            try:
                render = px.serialize(obj=result,
                                      enc='utf-8',
                                      **pyxser_args)
            except Exception, exc:
                render = "<pyxs:obj/>"
            if result:
                r.write(render)
            else:
                r.write("<pyxs:obj/>")
            return r
        return inner_xml
    return outer

If you apply the decorator above in a Django view, it will return the serialized object as text/xml to the HTTP client. So, your view must return a valid object to be serialized by pyxser. It applies the pyxser.serialize function to the given output from your view. Now take a look to a view which uses this decorator to throw XML.

def get_class(kls):
    try:
        parts = kls.split('.')
        module = ".".join(parts[:-1])
        m = __import__(module)
        for comp in parts[1:]:
            m = getattr(m, comp)
        return m
    except:
        return False

## use an URL as follows:
## (r'x/g/(?P<model>[\w\.]+)/(?P<oid>\d+)/',
##  u'views_xml.get_model_object'),
@require_http_methods(["GET", "OPTIONS", "HEAD"])
@render_to_xml(selector=do_value_attrs, depth=2)
def get_model_object(request, model=None, oid=None):
    obj = object()
    try:
        db_model = get_class(model)
        obj = db_model.objects.get(pk=oid)
        return obj
    except Exception, exc:
        log.error(exc)
    return obj

The view above returns an object from the given model name model and the given primary key oid, and passes the do_value_attrs selector function as attribute selector to pyxser, and restrict the serialization depth to two levels. Remember that pyxser allows to serialize circular references and cross references between objects, so we need to restrict the serialization depth, in case of Django models we can work with 2 levels in almost all models and the field selector do_value_attrs can be defined as follows.

DENIED_FIELDS = ['user', 'customer', 'users', 'customers']
DENIED_CLASSES = ['Related', 'Foreign', 'Many']

def is_allowed_class(fld):
    for nm in DENIED_CLASSES:
        if nm in fld.__class__.__name__:
            return False
    for nm in DENIED_FIELDS:
        if nm in fld.name:
            return False
    return True

def do_value_attrs(o):
    values = dict()
    if hasattr(o, '_meta') and hasattr(o._meta, 'fields'):
        for fldn in o._meta.fields:
            if is_allowed_class(fldn):
                values[fldn.name] = getattr(o, fldn.name)
    else:
        for kw in o.__dict__:
            values[kw] = getattr(o, kw)
    return values

Where we are filtering all fields in model objects that we do not want to serialize and all field classes that pyxser should not serialize for plain object transmision. Other objects which are not model related objects are serialized as plain Python objects using their dictionaries to get the object attributes, and also DENIED_FIELDS are skipped and DENIED_CLASSES are skipped too. The resulting XML for URLs like /p/x/g/offer.models.Marca/1/ is as follows.

<?xml version="1.0" encoding="utf-8"?>
<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
          version="1.0"
          type="Marca"
          module="prod.models"
          objid="id3128007116">
  <pyxs:prop type="unicode" name="nombre" size="4">Sony</pyxs:prop>
  <pyxs:prop type="long" name="id">1</pyxs:prop>
  <pyxs:prop type="unicode" name="slug" size="4">sony</pyxs:prop>
</pyxs:obj>

The pyxser serialization model holds type information, so any serialized object carries type information to be user in deserialization tasks, then you can handle the object back in any machine supporting pyxser and get the object deserialized to its original class using the unserialize function.

defining object containers

The pyxser extension cannot handle Django model containers directly, I mean those returned by the all method in query sets. So, you need to create a plain container to hold those objects that are retrieved from the database. Take a look on following view.

class Container(object):
    count = 0
    items = []
    def __init__(self):
        pass

def collect_filters(qd):
    data = qd.copy()
    filters = dict()
    for kw in data:
        if kw.startswith('filter__'):
            name = kw.replace('filter__', '')
            filters[name] = data[kw]
    return filters

### use an URL as follows:
### (r'x/l/(?P<model>[\w\.]+)/(?P<limit>\d+)/',
###  u'views_xml.get_model_list'),
@require_http_methods(["GET", "OPTIONS", "HEAD"])
@render_to_xml(selector=select_value_attrs, depth=4)
def get_model_list(request, model=None, limit=1):
    container = Container()
    container.count = 0
    container.items = []
    try:
        db_model = get_class(model)
        filters = collect_filters(request.GET)
        objs = db_model.objects.filter(**filters).all()[0:limit]
        container.count = len(objs)
        container.items = map(lambda x: x, objs)
        return container
    except Exception, exc:
        log.error(exc)
    return container

If you take a look carefully to this example, you will notice that we are using a very simple Container class to hold your objects. The resulting XML for the URL /p/x/l/prod.models.Marca/5/?filter__nombre__contains=son is as follows.

<?xml version="1.0" encoding="utf-8"?>
<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
          version="1.0"
          type="Container"
          module="prod.views_xml"
          objid="id3107379372">
  <pyxs:prop type="int" name="count">3</pyxs:prop>
  <pyxs:col type="list" name="items">
    <pyxs:obj type="Marca" module="prod.models" objid="id3107380204">
      <pyxs:prop type="unicode" name="nombre" size="5">Epson</pyxs:prop>
      <pyxs:prop type="long" name="id">10</pyxs:prop>
      <pyxs:prop type="unicode" name="slug" size="5">epson</pyxs:prop>
    </pyxs:obj>
    <pyxs:obj type="Marca" module="prod.models" objid="id3107380300">
      <pyxs:prop type="unicode" name="nombre" size="8">Ericsson</pyxs:prop>
      <pyxs:prop type="long" name="id">15</pyxs:prop>
      <pyxs:prop type="unicode" name="slug" size="8">ericsson</pyxs:prop>
    </pyxs:obj>
  </pyxs:col>
</pyxs:obj>

The resulting XML serialized object holds three Marca objects and all of them have their type to be deserialized once they are retrieved. If you want your objects to be deserialized back, you just need to use the pyxser.unserialize function properly, which is documented in the pyxser extension itself. I hope that you will like how pyxser works.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | One comment
Post tags:

It has been downloaded by many people from different sources where it is distributed, reaching about 8.000 downloads from different locations. That makes me very happy, because it is being tested and used widely in real world applications.

Pyxser Downloads Stats

Is a real pleasure to know that it is helping in the development of many applications. Also some other projects like jquery-validators it’s being downloaded over 3.000 times. I think that the best features of pyxser, is the fact it can do circular and cross referenced serialization without crashing. Remember that .NET and Java does not support circular and cross referenced XML serialization, so I have reached the right serialization model with pyxser. I have plans to migrate the serialization model and algorithm to other platform, like Java.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | One comment
Post tags:

At other side, JSONx is an IBM standard that has as main purpose to build XML representation of JSON objects. It is using a model very similar to the pyxser model, but restricted to JSON supported data types, so it is not extensible like pyxser. It only supports objects, arrays, booleans, strings, numbers, and nulls. The pyxser side is quite simple, it has objects, collections and plain properties. Objects can hold other objects, collections and properties; collections can hold other objects and properties and properties can hold only plain text data types, but flexible enough to create a custom serialization and hold a Base 64 encoded binary string and similar stuff.

JSONx Structure

JSONx is more complex than pyxser. It is doing the same mistakes than other serializers, since it holds the data type on the XML element itself. Instead, pyxser is doing the task of placing the data type and namespace on separate attributes. The data type is stored on the type attribute and the namespace is stored on the module attribute. We can talk about JSONx as domain specific XML serialization, at the pyxser side, we can talk as domain abstraction serialization model.

JSONx Structure

The JSONx model is not available through public XML Schema, its XML Schema is held inside IBM products. You can search for the pyxser XML Schema here, on its public URL: pyxser XML Schema, and the serialization model implementation is available for the Python language as Python Extension (a module written in C) and you can find it here: pyxser. The JSONx reference can be found here: JSONx. Also you can find pyxser XML document definitions in the form of XML schemas and DTD, and it is distributed with the pyxser Python extension:

• pyxser XML Schema.
• pyxser XML Schema, C14N Compatible.
• pyxser XML DTD.
• pyxser XML DTD, C14N Compatible.

Understanding the pyxser XML Schema will drive you to understand its abstraction model. So, have fun implementing domain abstraction, instead of doing domain specific XML serializations.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | No comment
Post tags:

Some notes on performance. Windows has a very poor performance compared to Linux and FreeBSD platforms. Please check the following plot that compares the performance on Windows, Linux and FreeBSD:

Compare pyxser-1.5.2 on Windows, Linux and FreeBSD

The performance on Windows is very poor. The effort required to migrate the source from POSIX platforms to Windows platforms is minimum. The difference the major effort was made on porting the setup script. My code is almost standard and it requires a C99 compiler, and usually I use the flags -Wall -Wextra -Wshadow -pedantic -std=c99 to compile my sources, so I’m using very strict dialect. Seems that the Windows C compiler is C90 by default. No way to deal with the Windows compiler for some C keywords. I’m expecting to bring constant support for Win32 platform on future releases. I will be waiting your feedback :)

Some notes on the FreeBSD install, is the fact that it is a standard distribution, without kernel optimization and it is using distribution binaries — that means that everything is made for 80×386 machines — and also it does not have any compiler optimization. Also, is well known that FreeBSD default binaries are slow, mainly because it lacks kernel optimization and standard C library optimizations. The libxml2 port on FreeBSD seems to be the problem, since it is not using the jemalloc library, and it is using the internal libxml2 memory allocator.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | One comment
Post tags:

The ChangeLog for this release is as follows:

1.5.2r (2011.01.08):

        Daniel Molina Wegener 

        * Added support for Python 2.4
        * Replaced the use of the commands package by the
        subprocess package on the setup script.
        * On the next release will be added support
        for Python 3.X ;)

        Thanks to pyxser users for their feedback.

type backporting

I’ve received an email from Juha Tuomala reporting incomapibility on pyxser to install it under Python 2.4. Also he has sent me a patch to make it work with Python 2.4. I’ve reviewed the patch and it was including some system macros, so it would be a little bit hard to include in the pyxser distribution. I’ve decided to study the problem about type backporting to make pyxser compatible with Python 2.4. The main problem was the Py_ssize_t, that is not available on Python 2.4, and used from Python 2.5 on the _PyTuple_Resize Python internal. The patch that Juha tried to send me was creating some macros to replace that type definition. But researching a little a about the Py_ssize_t type, I’ve found the PEP353 that guides on how to backport the usage of the Py_ssize_t type.

#if PY_VERSION_HEX < 0x02050000 && !defined(PY_SSIZE_T_MIN)
typedef int Py_ssize_t;
#define PY_SSIZE_T_MAX INT_MAX
#define PY_SSIZE_T_MIN INT_MIN
#endif

The preprocessor conditional block above can be used to backport the Py_ssize_t type without overwriting system types, and since it is a PEP, it should be the official way to backport that specific type. The other problem was about the PyAnySet_CheckExact macro. In this case, I’ve used the that portion of the patch directly, allowing me to backport some pyxser internal functions to Python 2.4

#if PY_VERSION_HEX < 0x02050000 && !defined(PY_SSIZE_T_MIN)
#define Py_TYPE(ob)             (((PyObject*)(ob))->ob_type)
#define PyAnySet_CheckExact(ob) \
        (Py_TYPE(ob) == &PySet_Type || Py_TYPE(ob) == &PyFrozenSet_Type)
#endif

I’ve used the same technique to detect the Python version and use that specific macro just for the required Python version. This is not affecting the future versions of Pyxser. For the next pyxser release, I’m planning to port it to Python 3.X. I hope that you will enjoy trasmitting and sending Python objects in XML format using the pyxser serializer ;)

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | No comment
Post tags: