Hello Python Community.

I’m pleased to announce pyxser-1.4.4r, a python extension which contains functions to serialize and deserialize Python Objects into XML. It is a model based serializer. Here is the ChangeLog entry for this release:

1.4.4r (2010.02.10):

        Daniel Molina Wegener 

        * src/include/pyxser_collections.h - added set handling
        function prototypes. Added support for unicode
        key names, which are converted to the user settings
        encoding (ie utf-8) inside the XML output.
        * src/pyxser_serializer.c - removed memory leak. Addded
        support for unicode object names in dictionary/list types.
        * src/pyxser_collections.c - added set handling function
        prototypes. added name property handling algorithm, so
        non string name properties are not serialized. I shall
        extend it to other modules.
        * src/pyxser_tools.c - added set handling prototypes and
        set type checking function.
        * test-utf8-leak.py - added serialization of SQL Alchemy
        objects, so we can test more complex Python objects
        serialization.
        * test-utf8-sqlalchemy.py - added sql alchemy object
        serialization test.
        * src/pyxser_serializer.c - reduced serialization algorithms,
        replacing deep nested if statements by flatten ones.
        * src/pyxser_collections.c - reduced serialization algorithms
        replacing deep nested if statements by flatten ones.
        * src/pyxser_typem.c - reduced serialization algorithms,
        replacing deep nested if statements by flatten ones.

        Thanks to pyxser users for their feedback.

This release contains some bug fixes, mainly related to type checking and type handling. I hope this small extension will help you on your programming tasks.

• The project is hosted at:
  http://sourceforge.net/projects/pyxser/
• The web page for the project is located at:
  http://coder.cl/products/pyxser/
• PyPi entry is:
  http://pypi.python.org/pypi/pyxser/1.4.4r
• For a sample article on how to integrate pyxser with ZSI WebServices:
  http://coder.cl/2009/10/18/pyxser-and-zsi-webservices/

Thanks and best regards…

I hope that this module can help you in your development tasks and thanks for your feedback.

Copyright © 2010 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2010. | Permalink | No comment
Post tags:

pyxser

pyxser have about one year of development, some months doing research, and other months doing development or coding the Python Module.

And the SVN log show an average hour for commits between 22:00 and 23:00

r125 | damowe | 2010-02-10 16:15:49 -0300 (Wed, 10 Feb 2010) | 2 lines
r124 | damowe | 2010-02-10 15:52:31 -0300 (Wed, 10 Feb 2010) | 2 lines
r123 | damowe | 2010-02-10 13:30:15 -0300 (Wed, 10 Feb 2010) | 10 lines
r122 | damowe | 2010-02-02 08:47:07 -0300 (Tue, 02 Feb 2010) | 3 lines
r121 | damowe | 2009-12-26 21:52:49 -0300 (Sat, 26 Dec 2009) | 2 lines
r118 | damowe | 2009-12-26 21:40:33 -0300 (Sat, 26 Dec 2009) | 2 lines
r117 | damowe | 2009-12-25 15:20:30 -0300 (Fri, 25 Dec 2009) | 4 lines
r116 | damowe | 2009-12-22 20:09:25 -0300 (Tue, 22 Dec 2009) | 5 lines
r115 | damowe | 2009-12-21 19:47:50 -0300 (Mon, 21 Dec 2009) | 2 lines
r114 | damowe | 2009-12-20 01:38:45 -0300 (Sun, 20 Dec 2009) | 7 lines
r112 | damowe | 2009-12-03 09:48:41 -0300 (Thu, 03 Dec 2009) | 6 lines
r110 | damowe | 2009-11-30 14:52:10 -0300 (Mon, 30 Nov 2009) | 2 lines
r109 | damowe | 2009-11-30 14:46:43 -0300 (Mon, 30 Nov 2009) | 9 lines
r107 | damowe | 2009-08-23 11:43:52 -0400 (Sun, 23 Aug 2009) | 2 lines
r102 | damowe | 2009-08-23 11:29:07 -0400 (Sun, 23 Aug 2009) | 2 lines
r99 | damowe | 2009-08-23 11:12:20 -0400 (Sun, 23 Aug 2009) | 6 lines
r98 | damowe | 2009-08-08 15:15:07 -0400 (Sat, 08 Aug 2009) | 3 lines
r93 | damowe | 2009-07-28 21:59:10 -0400 (Tue, 28 Jul 2009) | 2 lines
r90 | damowe | 2009-07-27 22:22:55 -0400 (Mon, 27 Jul 2009) | 2 lines
r89 | damowe | 2009-07-27 22:21:50 -0400 (Mon, 27 Jul 2009) | 5 lines
r88 | damowe | 2009-07-26 22:27:23 -0400 (Sun, 26 Jul 2009) | 4 lines
r87 | damowe | 2009-07-20 16:02:27 -0400 (Mon, 20 Jul 2009) | 5 lines
r86 | damowe | 2009-07-17 22:24:53 -0400 (Fri, 17 Jul 2009) | 4 lines
r84 | damowe | 2009-05-09 23:19:49 -0400 (Sat, 09 May 2009) | 2 lines
r83 | damowe | 2009-05-09 22:31:00 -0400 (Sat, 09 May 2009) | 7 lines
r82 | damowe | 2009-05-09 11:00:44 -0400 (Sat, 09 May 2009) | 3 lines
r81 | damowe | 2009-05-08 21:31:56 -0400 (Fri, 08 May 2009) | 2 lines
r80 | damowe | 2009-04-28 00:32:53 -0400 (Tue, 28 Apr 2009) | 3 lines
r78 | damowe | 2009-04-26 19:21:16 -0400 (Sun, 26 Apr 2009) | 2 lines
r77 | damowe | 2009-04-26 19:13:26 -0400 (Sun, 26 Apr 2009) | 2 lines
r76 | damowe | 2009-04-26 17:34:09 -0400 (Sun, 26 Apr 2009) | 4 lines
r75 | damowe | 2009-04-25 19:31:45 -0400 (Sat, 25 Apr 2009) | 3 lines
r74 | damowe | 2009-04-24 20:49:40 -0400 (Fri, 24 Apr 2009) | 3 lines
r73 | damowe | 2009-04-23 23:06:46 -0400 (Thu, 23 Apr 2009) | 4 lines
r72 | damowe | 2009-04-22 08:28:31 -0400 (Wed, 22 Apr 2009) | 3 lines
r71 | damowe | 2009-04-22 00:52:12 -0400 (Wed, 22 Apr 2009) | 2 lines
r70 | damowe | 2009-04-22 00:28:48 -0400 (Wed, 22 Apr 2009) | 3 lines
r69 | damowe | 2009-04-21 23:09:09 -0400 (Tue, 21 Apr 2009) | 2 lines
r68 | damowe | 2009-04-21 16:41:11 -0400 (Tue, 21 Apr 2009) | 3 lines
r65 | damowe | 2009-04-19 01:24:34 -0400 (Sun, 19 Apr 2009) | 2 lines
r64 | damowe | 2009-04-19 01:22:15 -0400 (Sun, 19 Apr 2009) | 2 lines
r63 | damowe | 2009-04-19 01:17:29 -0400 (Sun, 19 Apr 2009) | 2 lines
r62 | damowe | 2009-04-19 01:10:33 -0400 (Sun, 19 Apr 2009) | 2 lines
r61 | damowe | 2009-04-19 00:01:47 -0400 (Sun, 19 Apr 2009) | 3 lines
r59 | damowe | 2009-04-18 22:57:01 -0400 (Sat, 18 Apr 2009) | 2 lines
r58 | damowe | 2009-04-18 22:30:30 -0400 (Sat, 18 Apr 2009) | 3 lines
r57 | damowe | 2009-04-18 04:26:58 -0400 (Sat, 18 Apr 2009) | 2 lines
r56 | damowe | 2009-04-18 03:28:56 -0400 (Sat, 18 Apr 2009) | 3 lines
r55 | damowe | 2009-04-15 23:38:34 -0400 (Wed, 15 Apr 2009) | 2 lines
r54 | damowe | 2009-04-15 23:10:32 -0400 (Wed, 15 Apr 2009) | 2 lines
r53 | damowe | 2009-04-15 22:46:15 -0400 (Wed, 15 Apr 2009) | 3 lines
r52 | damowe | 2009-04-10 18:30:22 -0400 (Fri, 10 Apr 2009) | 2 lines
r49 | damowe | 2009-04-10 00:17:25 -0400 (Fri, 10 Apr 2009) | 2 lines
r47 | damowe | 2009-04-08 23:43:50 -0400 (Wed, 08 Apr 2009) | 4 lines
r46 | damowe | 2009-04-08 14:29:42 -0400 (Wed, 08 Apr 2009) | 4 lines
r45 | damowe | 2009-04-08 01:31:22 -0400 (Wed, 08 Apr 2009) | 2 lines
r44 | damowe | 2009-04-08 01:30:18 -0400 (Wed, 08 Apr 2009) | 2 lines
r43 | damowe | 2009-04-08 01:28:36 -0400 (Wed, 08 Apr 2009) | 3 lines
r42 | damowe | 2009-04-06 21:58:57 -0400 (Mon, 06 Apr 2009) | 2 lines
r41 | damowe | 2009-04-06 21:46:19 -0400 (Mon, 06 Apr 2009) | 2 lines
r40 | damowe | 2009-04-06 21:36:55 -0400 (Mon, 06 Apr 2009) | 3 lines
r39 | damowe | 2009-04-05 22:25:50 -0400 (Sun, 05 Apr 2009) | 2 lines
r38 | damowe | 2009-04-05 22:19:10 -0400 (Sun, 05 Apr 2009) | 4 lines
r37 | damowe | 2009-04-04 20:18:15 -0400 (Sat, 04 Apr 2009) | 3 lines
r36 | damowe | 2009-03-31 23:49:23 -0400 (Tue, 31 Mar 2009) | 5 lines
r33 | damowe | 2009-03-28 12:13:41 -0400 (Sat, 28 Mar 2009) | 2 lines
r30 | damowe | 2009-03-26 22:17:10 -0400 (Thu, 26 Mar 2009) | 3 lines
r29 | damowe | 2009-03-26 21:59:57 -0400 (Thu, 26 Mar 2009) | 5 lines
r28 | damowe | 2009-03-26 21:17:10 -0400 (Thu, 26 Mar 2009) | 2 lines
r27 | damowe | 2009-03-25 23:26:30 -0400 (Wed, 25 Mar 2009) | 3 lines
r26 | damowe | 2009-03-25 22:58:16 -0400 (Wed, 25 Mar 2009) | 5 lines
r25 | damowe | 2009-03-25 22:03:16 -0400 (Wed, 25 Mar 2009) | 2 lines
r24 | damowe | 2009-03-25 21:00:44 -0400 (Wed, 25 Mar 2009) | 4 lines
r23 | damowe | 2009-03-25 20:48:26 -0400 (Wed, 25 Mar 2009) | 3 lines
r22 | damowe | 2009-03-25 20:34:30 -0400 (Wed, 25 Mar 2009) | 2 lines
r21 | damowe | 2009-03-25 20:15:58 -0400 (Wed, 25 Mar 2009) | 2 lines
r20 | damowe | 2009-03-24 20:47:42 -0400 (Tue, 24 Mar 2009) | 2 lines
r19 | damowe | 2009-03-23 23:41:47 -0400 (Mon, 23 Mar 2009) | 5 lines
r18 | damowe | 2009-03-23 21:03:13 -0400 (Mon, 23 Mar 2009) | 3 lines
r17 | damowe | 2009-03-23 20:16:58 -0400 (Mon, 23 Mar 2009) | 2 lines
r16 | damowe | 2009-03-19 22:57:46 -0400 (Thu, 19 Mar 2009) | 2 lines
r15 | damowe | 2009-03-19 22:30:56 -0400 (Thu, 19 Mar 2009) | 5 lines
r14 | damowe | 2009-03-19 20:37:13 -0400 (Thu, 19 Mar 2009) | 2 lines
r13 | damowe | 2009-03-19 00:37:39 -0400 (Thu, 19 Mar 2009) | 3 lines
r12 | damowe | 2009-03-19 00:11:23 -0400 (Thu, 19 Mar 2009) | 3 lines
r11 | damowe | 2009-03-18 23:47:08 -0400 (Wed, 18 Mar 2009) | 3 lines
r10 | damowe | 2009-03-17 22:25:47 -0400 (Tue, 17 Mar 2009) | 2 lines
r9 | damowe | 2009-03-17 22:23:21 -0400 (Tue, 17 Mar 2009) | 2 lines
r8 | damowe | 2009-03-17 22:16:37 -0400 (Tue, 17 Mar 2009) | 2 lines
r7 | damowe | 2009-03-17 00:55:43 -0400 (Tue, 17 Mar 2009) | 2 lines
r6 | damowe | 2009-03-16 23:33:31 -0400 (Mon, 16 Mar 2009) | 2 lines
r5 | damowe | 2009-03-15 17:21:57 -0400 (Sun, 15 Mar 2009) | 2 lines
r4 | damowe | 2009-03-15 17:07:39 -0400 (Sun, 15 Mar 2009) | 2 lines
r1 | damowe | 2009-03-15 15:35:31 -0400 (Sun, 15 Mar 2009) | 2 lines

The project has 6,198 lines of code… a good effort and a good product. During the day I must work on those projects which bring me money, at night and as hobbits I work on my FOSS projects. I have a little time to work on them.

caffeine

caffeine or C Application Framework has 14,177 lines of code, and it is evaluated as one project with three humans working on it, but it is really one, me.

And the SVN log show an average hour for commits between 22:00 and 23:00

r125 | damowe | 2009-09-30 14:45:06 -0400 (Wed, 30 Sep 2009) | 6 lines
r124 | damowe | 2009-09-26 18:49:31 -0400 (Sat, 26 Sep 2009) | 2 lines
r123 | damowe | 2009-09-26 17:46:26 -0400 (Sat, 26 Sep 2009) | 5 lines
r122 | damowe | 2009-09-26 13:14:50 -0400 (Sat, 26 Sep 2009) | 2 lines
r121 | damowe | 2009-09-26 12:25:19 -0400 (Sat, 26 Sep 2009) | 4 lines
r120 | damowe | 2009-09-22 05:18:11 -0400 (Tue, 22 Sep 2009) | 8 lines
r119 | damowe | 2009-09-19 23:27:06 -0400 (Sat, 19 Sep 2009) | 5 lines
r118 | damowe | 2009-09-19 19:06:50 -0400 (Sat, 19 Sep 2009) | 8 lines
r117 | damowe | 2009-09-01 22:18:54 -0400 (Tue, 01 Sep 2009) | 5 lines
r116 | damowe | 2008-10-27 07:40:14 -0300 (Mon, 27 Oct 2008) | 2 lines
r115 | damowe | 2008-10-27 07:38:30 -0300 (Mon, 27 Oct 2008) | 2 lines
r114 | damowe | 2008-10-27 07:35:09 -0300 (Mon, 27 Oct 2008) | 3 lines
r113 | damowe | 2008-10-25 13:01:20 -0300 (Sat, 25 Oct 2008) | 2 lines
r112 | damowe | 2008-06-10 21:10:30 -0400 (Tue, 10 Jun 2008) | 3 lines
r111 | damowe | 2008-06-06 20:05:39 -0400 (Fri, 06 Jun 2008) | 5 lines
r110 | damowe | 2008-05-08 21:12:50 -0400 (Thu, 08 May 2008) | 2 lines
r109 | damowe | 2008-05-06 23:53:10 -0400 (Tue, 06 May 2008) | 2 lines
r108 | damowe | 2008-05-05 23:37:42 -0400 (Mon, 05 May 2008) | 2 lines
r107 | damowe | 2008-04-27 02:08:49 -0400 (Sun, 27 Apr 2008) | 2 lines
r106 | damowe | 2008-04-26 11:59:49 -0400 (Sat, 26 Apr 2008) | 4 lines
r105 | damowe | 2008-04-22 23:06:33 -0400 (Tue, 22 Apr 2008) | 3 lines
r104 | damowe | 2008-04-21 21:13:06 -0400 (Mon, 21 Apr 2008) | 4 lines
r103 | damowe | 2008-02-10 13:16:32 -0300 (Sun, 10 Feb 2008) | 3 lines
r102 | damowe | 2008-02-10 10:22:34 -0300 (Sun, 10 Feb 2008) | 2 lines
r101 | damowe | 2008-02-10 10:08:35 -0300 (Sun, 10 Feb 2008) | 2 lines
r100 | damowe | 2008-01-06 12:07:57 -0300 (Sun, 06 Jan 2008) | 4 lines
r99 | damowe | 2008-01-05 20:03:27 -0300 (Sat, 05 Jan 2008) | 4 lines
r98 | damowe | 2008-01-01 17:29:14 -0300 (Tue, 01 Jan 2008) | 3 lines
r97 | damowe | 2007-12-31 15:47:20 -0300 (Mon, 31 Dec 2007) | 2 lines
r96 | damowe | 2007-12-30 12:15:46 -0300 (Sun, 30 Dec 2007) | 2 lines
r95 | damowe | 2007-12-30 12:06:41 -0300 (Sun, 30 Dec 2007) | 3 lines
r94 | damowe | 2007-12-30 07:59:58 -0300 (Sun, 30 Dec 2007) | 2 lines
r93 | damowe | 2007-12-25 10:37:15 -0300 (Tue, 25 Dec 2007) | 3 lines
r92 | damowe | 2007-12-22 09:57:27 -0300 (Sat, 22 Dec 2007) | 2 lines
r91 | damowe | 2007-12-22 09:56:15 -0300 (Sat, 22 Dec 2007) | 4 lines
r90 | damowe | 2007-12-22 07:26:39 -0300 (Sat, 22 Dec 2007) | 3 lines
r89 | damowe | 2007-12-16 22:21:51 -0300 (Sun, 16 Dec 2007) | 4 lines
r88 | damowe | 2007-12-03 21:35:27 -0300 (Mon, 03 Dec 2007) | 2 lines
r87 | damowe | 2007-12-01 23:56:53 -0300 (Sat, 01 Dec 2007) | 3 lines
r86 | damowe | 2007-12-01 23:44:02 -0300 (Sat, 01 Dec 2007) | 2 lines
r85 | damowe | 2007-11-11 09:06:24 -0300 (Sun, 11 Nov 2007) | 4 lines
r84 | damowe | 2007-11-10 21:43:36 -0300 (Sat, 10 Nov 2007) | 2 lines
r83 | damowe | 2007-11-10 21:25:58 -0300 (Sat, 10 Nov 2007) | 2 lines
r82 | damowe | 2007-10-07 11:39:55 -0400 (Sun, 07 Oct 2007) | 3 lines
r81 | damowe | 2007-10-07 08:32:09 -0400 (Sun, 07 Oct 2007) | 3 lines
r80 | damowe | 2007-09-30 21:53:52 -0400 (Sun, 30 Sep 2007) | 3 lines
r79 | damowe | 2007-09-19 18:27:20 -0400 (Wed, 19 Sep 2007) | 2 lines
r78 | damowe | 2007-09-18 00:34:21 -0400 (Tue, 18 Sep 2007) | 2 lines
r77 | damowe | 2007-09-17 17:10:57 -0400 (Mon, 17 Sep 2007) | 3 lines
r76 | damowe | 2007-09-15 17:27:45 -0400 (Sat, 15 Sep 2007) | 2 lines
r75 | damowe | 2007-09-08 11:19:19 -0400 (Sat, 08 Sep 2007) | 2 lines
r74 | damowe | 2007-09-02 12:11:38 -0400 (Sun, 02 Sep 2007) | 2 lines
r73 | damowe | 2007-09-02 10:57:54 -0400 (Sun, 02 Sep 2007) | 2 lines
r72 | damowe | 2007-09-02 10:55:09 -0400 (Sun, 02 Sep 2007) | 2 lines
r71 | damowe | 2007-09-02 09:38:43 -0400 (Sun, 02 Sep 2007) | 5 lines
r70 | damowe | 2007-09-02 02:13:20 -0400 (Sun, 02 Sep 2007) | 4 lines
r69 | damowe | 2007-08-31 20:05:06 -0400 (Fri, 31 Aug 2007) | 3 lines
r68 | damowe | 2007-08-26 13:34:53 -0400 (Sun, 26 Aug 2007) | 2 lines
r67 | damowe | 2007-08-26 13:15:04 -0400 (Sun, 26 Aug 2007) | 3 lines
r66 | damowe | 2007-08-26 10:10:19 -0400 (Sun, 26 Aug 2007) | 3 lines
r65 | damowe | 2007-08-25 19:00:10 -0400 (Sat, 25 Aug 2007) | 3 lines
r64 | damowe | 2007-08-19 00:26:00 -0400 (Sun, 19 Aug 2007) | 3 lines
r63 | damowe | 2007-08-18 10:36:16 -0400 (Sat, 18 Aug 2007) | 3 lines
r62 | damowe | 2007-08-15 17:23:28 -0400 (Wed, 15 Aug 2007) | 2 lines
r61 | damowe | 2007-08-13 07:48:31 -0400 (Mon, 13 Aug 2007) | 3 lines
r60 | damowe | 2007-08-12 17:06:08 -0400 (Sun, 12 Aug 2007) | 2 lines
r59 | damowe | 2007-08-12 16:33:09 -0400 (Sun, 12 Aug 2007) | 3 lines
r58 | damowe | 2007-08-12 13:39:59 -0400 (Sun, 12 Aug 2007) | 2 lines
r57 | damowe | 2007-08-12 01:26:32 -0400 (Sun, 12 Aug 2007) | 2 lines
r56 | damowe | 2007-08-11 13:26:10 -0400 (Sat, 11 Aug 2007) | 2 lines
r55 | damowe | 2007-08-11 13:21:17 -0400 (Sat, 11 Aug 2007) | 3 lines
r54 | damowe | 2007-08-06 21:58:22 -0400 (Mon, 06 Aug 2007) | 2 lines
r53 | damowe | 2007-08-04 23:03:06 -0400 (Sat, 04 Aug 2007) | 3 lines
r52 | damowe | 2007-08-04 22:39:10 -0400 (Sat, 04 Aug 2007) | 3 lines
r51 | damowe | 2007-08-04 17:28:45 -0400 (Sat, 04 Aug 2007) | 2 lines
r50 | damowe | 2007-08-04 16:54:48 -0400 (Sat, 04 Aug 2007) | 2 lines
r49 | damowe | 2007-07-29 09:15:12 -0400 (Sun, 29 Jul 2007) | 2 lines
r48 | damowe | 2007-07-28 17:38:23 -0400 (Sat, 28 Jul 2007) | 2 lines
r47 | damowe | 2007-07-16 23:45:16 -0400 (Mon, 16 Jul 2007) | 3 lines
r46 | damowe | 2007-07-16 16:47:06 -0400 (Mon, 16 Jul 2007) | 3 lines
r45 | damowe | 2007-07-15 20:03:51 -0400 (Sun, 15 Jul 2007) | 3 lines
r44 | damowe | 2007-07-15 12:10:35 -0400 (Sun, 15 Jul 2007) | 3 lines
r43 | damowe | 2007-07-02 17:30:56 -0400 (Mon, 02 Jul 2007) | 2 lines
r42 | damowe | 2007-07-02 10:46:27 -0400 (Mon, 02 Jul 2007) | 3 lines
r41 | damowe | 2007-06-30 16:23:19 -0400 (Sat, 30 Jun 2007) | 2 lines
r40 | damowe | 2007-06-23 21:08:20 -0400 (Sat, 23 Jun 2007) | 3 lines
r39 | damowe | 2007-06-23 10:37:22 -0400 (Sat, 23 Jun 2007) | 2 lines
r38 | damowe | 2007-06-23 10:33:24 -0400 (Sat, 23 Jun 2007) | 2 lines
r37 | damowe | 2007-06-22 23:10:58 -0400 (Fri, 22 Jun 2007) | 3 lines
r36 | damowe | 2007-06-20 21:57:34 -0400 (Wed, 20 Jun 2007) | 3 lines
r35 | damowe | 2007-06-18 21:22:26 -0400 (Mon, 18 Jun 2007) | 3 lines
r34 | damowe | 2007-06-17 21:33:13 -0400 (Sun, 17 Jun 2007) | 2 lines
r33 | damowe | 2007-06-17 21:31:58 -0400 (Sun, 17 Jun 2007) | 2 lines
r32 | damowe | 2007-06-17 18:48:40 -0400 (Sun, 17 Jun 2007) | 3 lines
r31 | damowe | 2007-06-17 12:03:32 -0400 (Sun, 17 Jun 2007) | 3 lines
r30 | damowe | 2007-06-17 11:50:15 -0400 (Sun, 17 Jun 2007) | 2 lines
r29 | damowe | 2007-06-16 21:48:05 -0400 (Sat, 16 Jun 2007) | 2 lines
r28 | damowe | 2007-06-16 21:43:38 -0400 (Sat, 16 Jun 2007) | 3 lines
r27 | damowe | 2007-06-09 18:25:07 -0400 (Sat, 09 Jun 2007) | 2 lines
r26 | damowe | 2007-06-09 18:05:07 -0400 (Sat, 09 Jun 2007) | 2 lines
r25 | damowe | 2007-06-09 18:03:56 -0400 (Sat, 09 Jun 2007) | 3 lines
r24 | damowe | 2007-06-03 20:45:25 -0400 (Sun, 03 Jun 2007) | 3 lines
r23 | damowe | 2007-06-03 19:08:34 -0400 (Sun, 03 Jun 2007) | 2 lines
r22 | damowe | 2007-06-03 19:07:35 -0400 (Sun, 03 Jun 2007) | 2 lines
r21 | damowe | 2007-05-21 10:48:01 -0400 (Mon, 21 May 2007) | 2 lines
r20 | damowe | 2007-05-20 22:00:12 -0400 (Sun, 20 May 2007) | 2 lines
r19 | damowe | 2007-05-20 21:45:38 -0400 (Sun, 20 May 2007) | 6 lines
r18 | damowe | 2007-05-20 10:57:12 -0400 (Sun, 20 May 2007) | 2 lines
r17 | damowe | 2007-05-19 22:22:24 -0400 (Sat, 19 May 2007) | 5 lines
r16 | damowe | 2007-05-18 21:41:36 -0400 (Fri, 18 May 2007) | 2 lines
r15 | damowe | 2007-05-14 23:02:09 -0400 (Mon, 14 May 2007) | 3 lines
r14 | damowe | 2007-05-13 12:03:47 -0400 (Sun, 13 May 2007) | 2 lines
r13 | damowe | 2007-05-13 01:01:13 -0400 (Sun, 13 May 2007) | 2 lines
r12 | damowe | 2007-05-12 22:48:18 -0400 (Sat, 12 May 2007) | 2 lines
r11 | damowe | 2007-05-12 22:41:14 -0400 (Sat, 12 May 2007) | 3 lines
r10 | damowe | 2007-05-07 08:57:33 -0400 (Mon, 07 May 2007) | 2 lines
r9 | damowe | 2007-05-03 08:37:04 -0400 (Thu, 03 May 2007) | 3 lines
r8 | damowe | 2007-05-02 20:50:25 -0400 (Wed, 02 May 2007) | 3 lines
r7 | damowe | 2007-05-01 17:43:00 -0400 (Tue, 01 May 2007) | 2 lines
r6 | damowe | 2007-05-01 09:31:08 -0400 (Tue, 01 May 2007) | 2 lines
r5 | damowe | 2007-05-01 09:24:07 -0400 (Tue, 01 May 2007) | 1 line
r4 | damowe | 2007-05-01 09:08:02 -0400 (Tue, 01 May 2007) | 2 lines
r3 | damowe | 2007-05-01 09:05:41 -0400 (Tue, 01 May 2007) | 2 lines
r2 | damowe | 2007-05-01 09:04:51 -0400 (Tue, 01 May 2007) | 2 lines
r1 | damowe | 2007-05-01 09:04:22 -0400 (Tue, 01 May 2007) | 2 lines

Both projects requires effort and research, and both project are not simple ones to handle, since they are written in C, a language difficult to learn and handle… I hope that I can continue the foss development that I’ve started, but seems that I have some health problems now

Copyright © 2010 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2010. | Permalink | No comment
Post tags:

"…premature optimization is the root of all evil".
— Donald Knuth

I agree with the fact that we must do our source code level optimizations when we have finshed the construction stage or it is almost complete. I was searching articles and papers about optimizing C source code to be applied on my programs and libraries. I’ve collected some of those optimizations. But you must not confuse algorithm optimization, source code optimization and compiler optimization, since the first one refers to algorithm design and the second one just refers to the algorithm implementation, and both are sharing just few common approaches to formal reductions.

Usually the source code optimization just applies well known formal reductions. We will not treat those formalities in this article. Instead we will take a look on some examples that I’ve collected, allowing a common reasoning about how to optimize the source code. Most ideas for source code optimizations comes from η-conversions using λ equivalences, allowing for example reductions from . Also, some optimizations without an apparent η-conversion, since most is done at low level, as those which are taken by the compiler and translated to less assembler instructions, such as function inlining and parameter reduction. Most η-conversions which implies η-reductions, and are called strength reductions. But we have also the opposed side, where we are — instead of doing reductions — populating our programs with more lines of code, including repeated code, such as the loop unrolling optimization.

inline code

C supports both macros and inlining functions. Both features are reaching the same effect, when the code is compiled, all inlined instructions are placed when we call the macro or the inline function. For example, a typical approach, is a max(a, b) function or a MAX(a, b) macro.

/* the slowest expression, compiling and running */
int
max(int a, int b)
{
	if (a > b)
		return a;
	else
		return b;
}

/* normal expression with inlining */
inline int
max(int a, int b)
{
	return ((a > b) ? a : b);
}

/* the faster expression using macros */
#define MAX(a, b)			((a > b) ? a : b)

The inline and the macro definitions both are similar. The macro places every occurrence of MAX(var1, var2) expressions with the ((var1 > var2) ? var1 : var2) expression. The inline does the same, and places every occurrence of max(var1, var2) with the result of the expression ((var1 > var2) ? var1 : var2). So, the macro usage is oriented to replace entire blocks of code, instead of inline functions which are oriented to replace calls and skip the assembler call instruction and its derivates. We may look examples of the call opcode in typical system calls on operating systems, such as FreeBSD system calls.

parameter reduction

Reducing parameters implies less assembler instructions. C calling convention typically push parameters into the stack, which implies that for each parameter a push opcode call is made. For example you can group function arguments — if they are use subsequently by a group of functions — under struct forms. Think a little, each push instruction in his family takes 2 clocks, varying to 18 clocks on x86 architecture — depending on vendor and model. For 8 arguments, it will take from 16 to 36 clocks, against 2 to 18 clocks with one argument. This approach on argument reduction may be applied to those functions which are not inlined.

/* slower version */
int
my_function(int arg1, int arg2, int arg3, int arg4)
{
	/* do something... */
}

/* faster version */
struct myargs
{
	int arg1; int arg2; int arg3; int arg4;
};

int
my_function(struct myargs *args)
{
	/* do something... */
}

practical reductions

We do reduction when we remove unnecessary steps in our functions. We can do most reductions just by thinking a little on the code, and also there are some well known reductions which can be used as optimizations.

removing else

/* with else, smaller code, but slower one */
inline int
test(int a)
{
	return a > 0 ? 1 : 0;
}

/* without else, large code but faster one */
inline int
test(int a)
{
	if (a > 0)
		return 1;
	/* implied else */
	return 0;
}

On this optimization, we are removing jmp and jxx family of instructions, where most of them takes near to 7 clocks on x86 architecture and also the required instruction to setup the proper context for the following instruction in the program sequence. This is like the spartan programming style, where most code is minimized through reductions and similar programming tasks.

bitwise operations

Bitwise operations are cheaper than other operations. For example, curiously, a shift operation and plus operations have less clocks than multiplication and division operations. The happens to logical evaluations. In other post, I’ve wrote about logical minimizations. A complement to what I’ve said, is the fact that mul instruction takes from 10 to 40 clocks and div instruction takes from 15 to 40 clocks on x86 architecture, against add, sub, shr, shl and similar instruction that are taking from 2 to 7 clocks. Remember that the number clocks used by some instruction is vendor and model dependent. For example if we do:

/* we take a call to the math.h function pow() */
n = y * pow(2.0, x);

/* we can replace it by */
n = y << x;

In other words, the x << y operation is equivalent to . You can find other math equivalences too. From this basic approach, we can reach other types of optimizations through the proper maths. You can take a look on the bit wizardry page from Jörg Arndt if you want to seek for more bitwise operations and reductions.

reversing counter loops

Usually we do one step loops for fixed set of counters. The following example shows a reduction which can be used every time we can do a reverse loop.

/* for loop coded usually */
for (c = 0; c < MAX_C_VALUE; c++) {
	/* do something... */
}

/* for reversed loop */
for (c = MAX_C_VALUE; c--; /* we do not do nothing here */ ) {
	/* do something... */
}

table lookups

A table lookup is the technique where we create an array and kind related structures to lookup for data, usually previously calculated data or simply, we are looking for concrete data that we want to use.

/* we can have a switch statement */
ourtype_t *varn = NULL;
switch (var1) {
case 0:
	varn = value1; break;
case 1:
	varn = value2; break;
case 2:
	varn = value3; break;
default:
	varn = value1;
}

/* or have an if/else statement */
if (var1 == 0)
	varn = value1;
else if (var1 == 1)
	varn = value2;
else if (var2 == 2)
	varn = value3;
else
	varn = value1;

/* so we can simply replace those values using arrays */
ourtype_t mapsvalues[] = { value1, value2, value3 };
varn = mapsvalues[var1];

Also we usually can setup reductions by creating table lookups and state machines, so we can create the proper map between certain variable data and certain function. State machines are a powerful abstraction which allows us to code different states inside data structures.

We can have predefined values in our table lookup tasks. Then we are using lazy evaluation. Every time we have a constant value which is requested and not calculated each time we work with it, we are doing lazy evaluation.

register keyword

/* using the register keyword should help creating faster code */
register int counter;

reduce data access computation

If we have deep constructed data structures (struct in C), every time we access most deep nodes, we are using pointer arithmetics, which implies basic math operations to access certain parts of our structures. Here we can do tow tasks: alias creation and usage and padding adjustment. Alias usage, means that we must use an internal pointer to access directly a structure member, so we omit pointer calculation each time we access it. Padding adjustment, is just about to create the proper data structure member order.

/* structure without the proper padding */
struct my_struct {
	char *a;
	void *b;
	int c;
	double n;
	char *x;
};

/* the same structure with the proper padding */
struct my_struct {
	double n;
	int c;
	char *a;
	char *x;
	void *b;
};

Also the data access computation is reduced on assembler level code, not the C code. Here we have an implied η-reduction and invisible one. For deeply and nested data structures, we have the same issue. We should have as rule that the structure size must have an ideal size of , with as the structure size and as the padding adjustemnt to power of two.

/* some structs with nested members */
struct a {
	int m1;
};

struct b {
	int m2;
	struct a *m3;
}

struct c {
	int m4;
	struct b *m5;
}

/* the non cheaper version to access its members */
struct c *x = some_function_returning_c();
x->m5->m3->m1 = some_other_function();
if (x->m5->m3->m1 != 0) {
	another_function(x->m5->m3->m1);
}

/* the cheaper version to access its members
   using aliases */
struct c *x = some_function_returning_c();
struct b *p;
b = x->m5->m3;
b->m1 = some_other_function();
if (b->m1 != 0) {
	another_function(b->m1);
}

loop unrolling

Each step on a loop repeats some instructions. For example if we have a fixed size array, where we must treat each element with certain function, we can use unrolled loops.

/* normal iteration over fixed size array */
for (i = 0; i < 100; i++) {
	call_some_function(my_array[i]);
}

/* applied loop unrolling and reverse loop */
for (i = 100; i--;) {
	call_some_function(my_array[i]);
	call_some_function(my_array[--i]);
	call_some_function(my_array[--i]);
	call_some_function(my_array[--i]);
	call_some_function(my_array[--i]);
}

Note that here we have a fixed size array. In other case is hard to know the array size. Also we can use our compiler optimization to unroll each loop, if our compiler has the proper option. For example GCC supports automatic loop unrolling by using the -funroll-loops flag.

loop jamming

Reusing blocks of code inside loops matters.

/* here we have two loops for something that can
   be done one loop */
for (i = 0; i < 100; i++) {
	some_function_a(my_array[i]);
}
for (i = 0; i < 100; i += 10) {
	some_function_b(my_array[i]);
}

/* here we have two loops for something that can
   be done one loop */
for (i = 0; i < 100; i++) {
	some_function_a(my_array[i]);
	if ((i % 10) == 0) {
		some_function_b(my_array[i]);
	}
}

On the example above we have the proper loop doing the same tasks with less operations, so we have reduced the steps of that loop from to — note that is just symbolic and not strict.

bit padding matters

It depends on the architecture. Processing data types shorter or larger than the register size do not have a cheaper cost than using them. For example if we use char or short, we are not using a complete register, which is more hard to handle than register length variables, such as int and long.

references

• Michael E. Lee, "Optimization of Computer Programs in C", Ontek Corporation.
• Adrian Barnetts, "C optimisation tutorial".
• Paul Hsieh, "Programming Optimization".
• Koushik Ghosh, Writing Efficient C and C Code Optimization, The Code Project.
• Optimizing C and C++ Code. EventHelix.
• Cris H. Pappas and William H. Murray, "386 Microprocessor Handbook", McGraw Hill.

Copyright © 2009 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2009. | Permalink | One comment
Post tags:

In Java, thread synchronization made using the synchronized keyword can create a heavy — with a high number of threads — overhead in two main cases, the first one is when it is used on classes and the second one when it is used in methods. Then we must think a way to create small locks, mainly if we are using design patterns like singletons.

/* BAD CASE 1, throws NullPointerException */
class BadMutexBaseClass {
	private static BadMutexBaseClass singletonInstance;
	public static BadMutexBaseClass getInstance(void) {
		synchronized (singletonInstance) {
			if (singletonInstance == null) {
				singletonInstance = new BadMutexBaseClass();
			}
			return singletonInstance;
		}
	}
}

/* BAD CASE 2, heavy overhead */
class BadMutexBaseClass {
	private static BadMutexBaseClass singletonInstance;
	public synchronized static BadMutexBaseClass getInstance(void) {
		if (singletonInstance == null) {
			singletonInstance = new BadMutexBaseClass();
		}
		return singletonInstance;
	}
}

Both cases are wrong, we must use mutex object, without touching the singleton member and allowing the program to access the singleton atomically. We can use a basic Object instance to create the proper mutex.

/* MUTEX BASED CASE */
class MutexBaseClass {
	private static Object mutex = new Object();
	private static MutexBaseClass singletonInstance;
	public static MutexBaseClass getInstance(void) {
		synchronized (mutex) { /* critical section */
			if (singletonInstance == null) {
				singletonInstance = new MutexBaseClass();
			}
			return singletonInstance;
		}
	}
}

By creating a dummy object, we can use it as mutex in our program. With a more low level perspective, this technique is similar to the one used with pthread_mutex_lock(P) techniques while we are programming with threads in C. An interesting point is the fact that non-blocking algorithms are faster than locking ones. The mutex technique is a blocking technique. Non-blocking techniques, such as lock-free and obstruction-free, can allow the user to implement faster — but more complex — code.

GCC has built-in atomic operations to allow the creation of such code, like CAS operations, allowing the use of lock-free techniques. For example in the code bellow — which is using the __sync_bool_compare_and_swap atomic operation — we can see that the CAS or Compare and Swap operation is made around the number of transactions with old value and old value plus one.

do {
	old = vproc_shmem->vp_shmem_transaction_cnt;

	if (unlikely(old < 0)) {
		if (vproc_shmem->vp_shmem_flags & VPROC_SHMEM_EXITING) {
			_exit(0);
		} else {
			__crashreporter_info__ = "Unbalanced: vproc_transaction_begin()";
		}
		abort();
	}
} while( !__sync_bool_compare_and_swap(&vproc_shmem->vp_shmem_transaction_cnt, old, old + 1) );

I hope that Java will include kind that atomic operations to allow the use of non-blocking algorithms and more faster code and since I agree with the fact that we are entering the multi-core era…

Copyright © 2009 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2009. | Permalink | No comment
Post tags:

1.3r (2009.11.30):

        Daniel Molina Wegener <dmw@coder.cl>
        * pyxser_tools.c: Removed bug concerning deserialization of
        collection elements; pyxser now can handle extension declared
        classes, such as boost::python ones --- thanks to the
        suggestions made by Daniel Filonik.
        * pyxser.c: Added /selector/ as optional argument used as
        callback to select object attributes to being serialized.
        * pyxser_string.c: Removed bug concerning unicode serialization.

        Thanks pyxser users for your feedback...

I want to bring special thanks to Daniel Filonik for his feedback.

First, thank you for a great Python module, I believe it has a lot of potential! I was never quite happy with pickle because of its cryptic format. The XML schema you use is so much nicer to read.

the selector callback

An interesting feature requested by Daniel Filonik is to allow the user to select the object attributes to being serialized. So I’ve added the attribute selector callback as optional argument called selector. As he suggest:

PyObject *selector = (PyObject *)NULL;
/* ... */
static char *kwlist[] = {"obj", "enc", "depth", "selector ", NULL};
/* ... */
ok = PyArg_ParseTupleAndKeywords(args, keywds, "Os|iO", kwlist,
                                 &input, &in_enc, &py_depth, &selector);
/* ... */
sargs.selector = &selector;

Then, I use the selector callback in pyxser to obtain a list of the required attributes for the serialization process, and the callback is applied to the given object to being serialized.

if (PYTHON_IS_NONE(select)) {
    lstItems = PyObject_GetAttrString(o, pyxser_attr_dict);
} else {
    /* selector must return NULL if there are errors... */
    arglist = Py_BuildValue("(O)", o);
    lstItems = PyObject_CallObject(select, arglist);
    Py_XDECREF(arglist);
}

And I’ve implemented the proper method to allow the instantiation of classes which are defined in Python extensions, such as boost::python, by replacing PyInstance_NewRaw() call by the default empty constructor for Python objects:

#ifdef OLDCODE
	PyObject *ndict = (PyObject *)NULL;
	/* ... */
	ndict = PyDict_New();
	unser = PyInstance_NewRaw(ct, ndict);
	Py_XDECREF(ndict);
#else
    int ctrl = 0;
    unser = PyObject_CallFunctionObjArgs(ct, NULL);
    attr_name = pyxser_ExtractPropertyName(pyxser_xml_attr_name,
                                           ron);
#endif /* !OLDCODE */

I hope that pyxser is neat serializing extension and it is usefull for your transparent and interoperable communications with other platforms in Python.

Copyright © 2009 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2009. | Permalink | One comment
Post tags:

service definition

Each service is created with the caf_msg_svc_t structure. It holds a message seed of type caf_msg_t. The message seed has the IPC key, and related IPC data, and it is used as template for other messages sent by through this facility. Each session has a u_long identifier, which is processed by the master process in the session_inc member. And both snd_inc and rcv_inc keep tracking the sent and received messages respectively. The machine pointer holds the state machine pointer, and its type is defined by the type structure member. Each error on the process of managing the messaging facility is hold on the errno_v, which allows you to know which error is produced internally on those routines, so you can check for your operating system alerts. Each session is stored in the deque member sessions.

<br /> /* this code is part of caffeine and is licensed under LGPLv3 */<br /> struct caf_msg_svc_s {<br /> /** Seed and sample message */<br /> caf_msg_t *seed;<br /> /** Session identifier increment */<br /> u_long session_inc;<br /> /** Sender identifier increment */<br /> u_long snd_inc;<br /> /** Receiver identifier increment */<br /> u_long rcv_inc;<br /> /** List of sessions */<br /> lstdl_t *sessions;<br /> /** Processing Machine */<br /> void *machine;<br /> /** Processing Machine Type */<br /> caf_msg_svc_sm_t type;<br /> /** The Packet parser using Data Packer @see caf_data_packer */<br /> caf_packet_t *parser;<br /> /** errno value for common operations */<br /> int errno_v;<br /> };<br />

The idea is to process each session — as list that is — through the lstdl_map function, which, which uses the machine structure member by passing processing session item to the packet parser by using the caf_packet_parse_machine() caffeine function, and then hold the parsed packet to next processing functions in the state machine. The only one state machine that can not be converted by caffeine from a normal state machine to a packet processing state machine is the static one, the others two machines, both the plugable and the dynamic one can be converted to packet processing ones. The difference between the state machines is the fact that the static is build on top of arrays, and this means that that each state in it would be made statically on compile time, the plugable one is created on runtime and is using a pointer instead of arrays, and the dynamic one uses a deque as state storage.

session definition

Each session has a pointer to its parent service, and it is assigned while it is created. This allows the state machines and service processing routines to setup and identify which packets can be received through the binding IPC keys. The IPC key is bounded to the session_id member, and the structure itself keeps tracking both sent and received packets count through the snd_cnt and rcv_cnt respectively. The snd_id and the rcv_id are both made to setup the sender and receiver identifiers, which both will be used as endpoints identifiers. The service will be kept receiving messages and it will identify its senders through the snd_id identifier in its sessions.

<br /> /* this code is part of caffeine and is licensed under LGPLv3 */<br /> struct caf_msg_session_s {<br /> /** Session identifier */<br /> u_long session_id;<br /> /** Sender identifier */<br /> u_long snd_id;<br /> /** Receiver identifier */<br /> u_long rcv_id;<br /> /** Messages sent counter */<br /> u_long snd_cnt;<br /> /** Messages received counter */<br /> u_long rcv_cnt;<br /> /** Session message size */<br /> size_t sz;<br /> /** Client PID */<br /> pid_t client;<br /> /** Server PID */<br /> pid_t server;<br /> /** Lock */<br /> int locked;<br /> /** Message storage */<br /> caf_msg_t *msg;<br /> /** Service holder */<br /> caf_msg_svc_t *svc;<br /> };<br />

On each created session, you must setup the client pid_t and the server pid_t for further use. Also the message size is setup when the machine is used though the parsing facility. Each time a session is being used, it is locked, through the lock member — yeah, I know, I’m using locking on it, but I’m thinking to change it by CAS operations — and the service member structure is used to hold data about the message template and useful information about the service. Since IPC communications are thinked to share small amounts of data, you can not setup messages which are too wide.

At the other side, the implementation is not completely finished but the initial algorithm is implemented. I will use it as initial algorithm to implement the networking facility implementation. As further works, I will implement ASN.1, XDR and SDXF binary formats to allow standardized data format processing for networking protocols. It will allow more interoperability with other systems. I know that there is a lot of work forward, but I’m going fine with my works on caffeine. Also, I’m not using text based data formats, such as JSON, YAML or XML, since those formats generate a huge overhead by processing them, and really is so hard to implement thread safe text parsers, and all my work on caffeine is — yet — thread safe. Since its conception caffeine is build with multicore in mind…

processing model

The processing model is quiet old, the use of state machines is. We can see them on device drivers, memory management, VFS implementations and a lot of examples more. You must worry on designing the proper packets for communicating your processes and write them through the caffeine facilities and plug those components. The steps for building an IPC messaging implementation through caffeine are as follows:

1. Define a packet parser for your implementation.
2. Create the service in the master process.
3. Copy the service in each session on fork, keep it on the master process.
4. Begin processing the service sessions.

Since it is planned as thread safe implementation — yet I must look at the "_r" suffixed POSIX and C99 functions to use — you can keep a thread working on the message processing. A similar model, but using lock-free and similar techniques is going to be implemented on the protocol stack for caffeine. And some other features will be applied on further works on it, always with multicore, distribution, parallelism and lock-free some other nice computing things in mind.

Honestly, I’m quiet anxious to implement the other features and enhance the current ones in caffeine. The main problem is time to develop it and since I have researched enough and good features for it, it keeps me more anxious…

Copyright © 2009 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2009. | Permalink | No comment
Post tags:

On the post from Eduardo Silva: "vmsplice() + SPLICE_F_GIFT flag ??", he’s asking about the problem of EINVAL errno value on calling vmsplice(2) system call on Linux systems.

does anybody know why the linux syscall vmsplice() doesn’t work with the SPLICE_F_GIFT flag ?, per manual page it should be supported but i get the following error:

My answer is as follows — seems that the blog of Eduardo is cutting the paragraphs — is quiet simple, and I’m doing the reasoning as C programmer regarding the manual page, without looking at the code.

We have the next declaration:

long vmsplice(int fd, const struct iovec *iov,
unsigned long nr_segs, unsigned int flags);

According to the manual page, an EINVAL or "Invalid Argument"
error is generated in two cases:

1. The number of chunks to supply to the kernel as nr_segs
number of blocks is greater than IOV_MAX. If IOV_MAX < nr_segs
EINVAL is thrown. Check for the value of IOV_MAX and nr_segs.

2. io_base addresses are not aligned in the iovec array:
If we have iovec[4] as:
iovec[0]->io_base equal to 0×080ed000;
iovec[1]->io_base equal to 0×080ee024; (aligned in 1024 bytes)
iovec[2]->io_base equal to 0×080ef048; (aligned in 1024 bytes)
iovec[3]->io_base equal to 0×080f0090; (aligned in 1048 bytes)*

The alignment of 1048 bytes differs from the base alignment of
1024 bytes, and EINVAL is thrown. You must ensure that the GIFT
blocks are aligned.

The complete explanation on the answer can redacted in a few lines. The declaration of the vmsplice(2) system call receives a file descriptor fd, an array of iovec structures iov, the number of iovec structures in the array nr_segs and the flags which are passed to the system call. The subject is the EINVAL, aka “Invalid Argument” errno value. For the first instance we have that nr_segs, or number of segments argument, can not be greater than IOV_MAX. If it is greater than this value, the EINVAL errrno value is raised. The number of segments is limited to IOV_MAX, we can not gift more segments than IOV_MAX permits. The IOV_MAX macro is defined in the <limits.h> header.

The second issue with the SPLICE_F_GIFT flag, is the fact that memory chunks that we gift to the kernel must be a range of byte aligned bytes with the same size and continuous blocks. In the example we have three continuous blocks of 1024 bytes, but the last block, the 4th of them, is aligned at the byte 1048 with a hole of 24 bytes between the two last blocks. This will cause EINVAL errno value to be raised. Other interesting case, are blocks of different size, they also must be byte aligned. In other case, one byte of difference in the memory alignment, will cause the EINVAL errno value to be raised.

The EBADF errno value is also interesting. You must create a pipe through pipe(2) or socketpair(2) system calls or the popen(3) C library routine. The file descriptor that those functions returns must be passed to the vmsplice(2) system call. The file descriptor fd must refer to a pipe. Finally on success, the system call returns the number of bytes transferred.

Now I’m thinking on threaded applications. What happens to those memory blocks pointed by the iovec array when those bytes are protected to other threads through the pthread_key_create(P) system call on Linux systems?. This may be an interesting subject to study. I think that I can discover interesting things on key implementation on POSIX threads on Linux systems, mainly because the Linux kernel do not have a separate implementation for threads. For the Linux kernel, everything is a procces, even if it is a user process or a process thread. I think that this issue, in conjunction with the O(1) Linux scheduler will be the matter for my next studying chapter.

Copyright © 2009 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2009. | Permalink | 2 comments
Post tags:

Today I’ve released pyxser-1.2r my Python Object to XML serializer/deserializer. To get more information about this project, visit the pyxser web page. This release has with byte string encoding for Unicode strings. This means that all Unicode strings are serialized in a human readable form, and also a better form to use pyxser with other platforms, since the encoding regards a better interoperability with byte encoded string handlers.

Another interesting issue, is the fact that the source code is more ordered and it was refactored to get better reading on it. The disadvantage is that pyxser still is using the indirect recursion to serialize certain objects. I hope that it will change in a near future, but it do not harm the stack, because it is using pointer as parameters and not enough bytes to cause a stack overflow.

useful links for pyxser:

• pyxser home page
• sourceforge project hosting
• pypi entry

I you want to participate, just mail me

Copyright © 2009 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2009. | Permalink | No comment
Post tags:

In other posts I’ve presented you some ways to detect memory leaks, conceptually rather than practically, but enough to understand them and solve those programming issues. Normally, valgrind messages. There are some posts about valgrind and well documented user guides, such as the guide written by Alex Ott. This article is about on how to interpret valgrind messages while you are using it as memory allocation debugger.

First of all, I want to share a small script with the flags that I’m using with valgrind. Here is a small script which runs valgrind on the given program and creates a log with the valgrind output. You can find the explanation on the flags in the Alex’s article and the valgrind documentation.

#!/bin/bash

if test "$#" "<" "1"; then
    echo "$0: too few arguments"
    exit 1
fi

VLG=`which valgrind`
VLG_LOG=`basename $PWD`
VLG_LOG="log.$VLG_LOG"
VLG_OPTS="$VLG_OPTS --verbose"
VLG_OPTS="$VLG_OPTS --trace-children=yes"
VLG_OPTS="$VLG_OPTS --track-fds=yes"
VLG_OPTS="$VLG_OPTS --run-libc-freeres=yes"
VLG_OPTS="$VLG_OPTS --logfile-fd=1"
VLG_OPTS="$VLG_OPTS --num-callers=100"
VLG_OPTS="$VLG_OPTS --error-limit=no"
VLG_OPTS="$VLG_OPTS --show-below-main=yes"
VLG_OPTS="$VLG_OPTS --pointercheck=yes"
VLG_OPTS="$VLG_OPTS --demangle=yes"
VLG_OPTS="$VLG_OPTS --leak-check=yes"

export VLG VLG_LOG VALGRIND_OPTS

if test -r "$VLG_LOG"; then
        echo "$0: removing $VLG_LOG"
        rm -f -v "$VLG_LOG"
fi
$VLG $VLG_OPTS $* 2>&1 | tee "$VLG_LOG"

This shell script the call to valgrind — IMO — with the best options that some programs can have to debug them. This command line will generate a large trace on your command and as I’ve posted in a previous topic, valgrind replaces the allocation system calls with his own group of system calls, wrapping the original ones, just track those calls.

invalid read of size nnnn

Your pointer arithmetics are bad. Yes, this message means that the program is accessing an invalid address by reading nnnn bytes from it. The code bellow — that seems to be ridiculous, but is not different that the error that you have in your code — would throw this message.

#include <stdio.h>
int
main (int c, char **argv) {
    char test[] = "hola mundo!";
    printf ("%s\n", (char *)(test + 20));
    return 0;
}

The message obtained is as follows.

==22111== 1 errors in context 1 of 1:
==22111== Invalid read of size 1
==22111==    by 0x3010A4E0: vfprintf (in /lib/libc.so.6)
==22111==    by 0x300F6B2F: printf (in /lib/libc.so.6)
==22111==    by 0x8048571: main (test-invalid-read.c:11)

As you, see in the backtrace, the part where it reaches the invalid address is inside the printf(3) function, but the previous step is on the line 11 of the file test-invalid-read.c.

use of uninitialised value of size nnnn

By default, there is no automatic variable initialization for certain kinds of variables, all depends on the compiler implementation and the C99 standard reveals it as undefined behavior. Since that condition, is required that you initialize all your variables to ensure code portability.

#include <string.h>
#include <stdio.h>

int
main (int c, char **argv) {
    char *tst;
    printf ("%s\n", tst);
    return 0;
}

The code above will produce the next error messages on valgrind.

==22309== 1 errors in context 1 of 6:
==22309== Use of uninitialised value of size 4
==22309==    at 0x30039DC0: memcpy (in /usr/local/lib/valgrind/vgpreload_memcheck.so)
==22309==    by 0x3010A4E0: vfprintf (in /lib/libc.so.6)
==22309==    by 0x300F6B2F: printf (in /lib/libc.so.6)
==22309==    by 0x804854B: main (test-invalid-read.c:8)
==22309==
==22309== 1 errors in context 2 of 6:
==22309== Conditional jump or move depends on uninitialised value(s)
==22309==    at 0x30039DAF: memcpy (in /usr/local/lib/valgrind/vgpreload_memcheck.so)
==22309==    by 0x3010A4E0: vfprintf (in /lib/libc.so.6)
==22309==    by 0x300F6B2F: printf (in /lib/libc.so.6)
==22309==    by 0x804854B: main (test-invalid-read.c:8)

Both errors are pointing to the same programming error. The variable tst was not initialized, and then you operate — and what makes one of the most ugliest errors on programming — using it as address reference. Also, by default gcc has automatic initialization of those pointers to null, and then you get "Null References: The Billion Dollar Mistake" pointed by Tony Hoare. Remember that variable initialization is an undefined behavior according to the current C standard. Reference verification and initialization is an important task, recently was announced a new version of Eiffel — a pure object oriented paradigm based language — that is avoiding null deferencing.

process terminating with default action of signal nn (SIGNAME): dumping core

Here the valgrind execution stops. Why? the program receives a signal and is terminated by the operating system. Take a look on the signal(3) manual page for more details.

The operating system generates a core file, from where you can get the proper information about the crashing instruction. All depends on the signal that the program was catching, a SIGBUS or bus error, means that you are writing bytes to wrong address. The code bellow shows that bug.

#include <string.h>
#include <stdio.h>

int
main (int c, char **argv) {
    char tst[10];
    const char hl[] = "hola mundo!";
    memset (tst, 0, sizeof(tst));
    memcpy (tst, hl, strlen(hl) + 1);
    return 0;
}

The memset(3) instruction sets the memory buffer pointed by tst to zero and it is OK. The size of tst is known, and since the operator sizeof is a compile time operator, there is no problem. If tst is a pointer instead of an array of bytes — I mean "char *tst" declaration — we can’t use sizeof and using it is like to get sizeof(NULL) — have noticed that every pointer has the same size?. If we extract the backtrace from the core file, we can reach the line of code which have the error. Depending on the signal, we can reach different kinds of errors, you must read your system documentation to know to what specific error refers it.

other valgrind errors

Depending on your operating system implementation, other errors can occur, if there is no proper memory protection. For example, you can have a conditional jump error and then an invalid write error. If you mix both of them and the system is not capable to protect itself of those errors, valgrind will report them. In other case, the proper signal will be thrown to your process and possibly a core file will be generated, then you must use the traditional debugging techniques to reach the bug.

conclusion

You must consider learning assembly language and computer architecture if you want to work at this level. Nowdays, by C programming most people understand a low level programming. By the years that I was studying, C programming was just another high level language, and I got the basics of programming on it, but just before I was introduced to it by reading some books — about 5 to 7 books on C programming — I was capable to build stronger programs.

Everything is important on C programming, from the basic techniques to complex data structures and algorithms. Most operating systems and user interfaces are made with C as the main programming language and also with C++. The applications of this language are not only applied to low level tasks. There are many programs implementing their algorithms in C and assembler to reach optimal performance, just look at ATLAS, BLAS and some other interesting projects.

Copyright © 2009 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2009. | Permalink | No comment
Post tags:

Emacs is a powerful editor, the most powerful and the fastest programmable editor that I know. Emacs uses a Lisp dialect to create configurations, functions and modes. My favorite mode on Emacs is ECB, a powerful code browsing one, like many IDEs that I know. The difference is that ECB, do not does code indexing until you configure it. For large C projects, I like to use external tools, but not ctags or etags, instead I prefer to use cscope. But you may ask "how can I integrate cscope and Emacs running the ECB mode?". Well, in this post I will try to explain how to integrate both, cscope and Emacs.

To know more about ECB, CEDET and related Emacs modes you can look some articles on the Internet, such as "A Gentle introduction to Cedet", by Alex Ott. All of them would help a lot on configuring both, Emacs and ECB. Also the CEDET — the base package for ECB — and ECB itself are well documented.

The cscope mode that I’m using is the cscope mode contributed to the cscope source tree. It’s called xcscope.el and comes with a small cscope indexing script called cscope-indexer.

On ECB there is a hook that runs on every time that you change your working directory by using the directory navigator. The hook is called ecb-after-directory-change-hook. For my C and C++ projects I use various indentation settings, depending on the project that I’m working on. Then, I’ve created a hook function to ecb-after-directory-change-hook that sets the indenting style depending on the project that I’m working on. I’ve created the dmw-ecb-directory-change-alist variable, and it holds the project path and the indenting style to be applied to the project.

(defvar dmw-ecb-directory-change-alist
  '(("/work/dmw/c/caffeine" caffeine-c-mode-hook)
    ("/work/dmw/c/pyxser" python-c-mode-hook)
    ("/work/dmw/cxx/caffeine++" caffeine-c++-mode-hook)
    ("/work/dmw/cxx/qstats" caffeine-c++-mode-hook)
    ("/usr/src/sys" knf)
    ("/current/usr/src/sys" knf)))

Then, I’ve created the hook to apply the indenting style.

(defun dmw-ecb-directory-change-hook (dirold dirnew)
  (dolist (prji dmw-ecb-directory-change-alist)
    (let ((prj-name (first prji))
          (prj-match (string-match (concat (first prji) ".*") dirnew)))
      (if (numberp prj-match)
          (progn
            ;; run hooks and functions related to
            ;; relative path matching on directory browsing
            (if (>= prj-match 0)
                (dolist (h (last prji))
                  (run-hook-with-args h)))
            ;; run hooks and functions related to
            ;; full path matching on directory browsing
            (if (string= prj-name dirnew)
                (let ((newpath (concat dirnew "/cscope.out"))
                      (dirnew dirnew))
                  (progn
                    (setq cscope-database-file newpath
                          cscope-do-not-update-database nil
                          cscope-use-relative-path t
                          cscope-initial-directory dirnew)
                    (cscope-index-files dirnew))))
            )))))

What does this hook? See the documentation for ecb-after-directory-change-hook. On every directory change on the directory browsing window, the hook runs passing the old directory and the new directory as it’s arguments.

Hook which run directly after the selected directory has changed. This means not only after a click onto a directory in the directory-window of ECB but it means this hook runs always when the current directory changes regardless of the trigger of this change[...]

Then, when the hook matches my C and C++ projects, it pass by the (if (>= prj-match 0) expression and setup the indenting style configured in dmw-ecb-directory-change-alist. If the directory pass by the (if (string= prj-name dirnew) expression, then it runs the block related to the new directory bellow the if sentence. What it does? It sets the cscope-database-file variable to the newpath local variable, which have the value of the new directory concatenated with "/cscope.out", also set some variable to ensure that the cscope database is updated, to enable cscope to use relative paths — this enables that cscope can lookup on different directories if the path changes — and set the initial directory to the new directory. Then, it runs the cscope-indexer. In few words, this hooks refresh the cscope database, and sets a permanent database location relative to the project that I’m working on.

Then, somewhere in your .emacs file, you should put your ECB variables as follows.

(add-hook 'ecb-after-directory-change-hook
          'dmw-ecb-directory-change-hook)

Then, just setup the keybinding to use the cscope mode — I do a lot of C programming under emacs, that’s the reason that I’ve configured the keybinding as global, and not local to the C programming mode. Those keybindings bellow are a little bit tricky, you can search for C symbols just by typing Ctrl + Meta + Down.

;; cscope keys
(define-key global-map [(control meta s)] 'cscope-set-initial-directory)
(define-key global-map [(control meta u)] 'cscope-unset-initial-directory)
(define-key global-map [(control meta f)] 'cscope-find-this-symbol)
(define-key global-map [(control meta g)] 'cscope-find-global-definition)
(define-key global-map [(control meta x)] 'cscope-find-global-definition-no-prompting)
(define-key global-map [(control meta m)] 'cscope-pop-mark)
(define-key global-map [(control meta n)] 'cscope-next-symbol)
(define-key global-map [(control meta N)] 'cscope-next-file)
(define-key global-map [(control meta p)] 'cscope-prev-symbol)
(define-key global-map [(control meta P)] 'cscope-prev-file)
(define-key global-map [(control meta c)] 'cscope-display-buffer)
(define-key global-map [(control meta C)] 'cscope-display-buffer-toggle)

I prefer cscope since it have the capability of searching symbols on many ways that other code indexer do not do, and large IDEs like eclipse with CDT gets leaked with large projects like the kernel sources — CDT takes just a few hours to index the complete kernel! — and since cscope is purely made in C and runs really fast.

Copyright © 2009 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2009. | Permalink | No comment
Post tags: