The data is lifted from its original type when it is stored as set. We can have a basic and plain example on how operates the map over the data lifting its original type. For example we have a plain text file with lines containing numbers separated by spaces and we want to extract the product for each line of those numbers. Since the data is plain text and it is not a numeric type, we need to lift those numbers and convert them to type that we can operate — like Integer types. So, map implies data extraction and data conversion, to operate that data with reduce. Following our plain text file example, we have a plain text file as follows.

4 3 1 4 5
2 1 3 2 4
3 4 4 2 1
2 1 3 4 1

Now, we can use map to make the proper data conversion to lift the input data from the plain text file and process each line obtaining lists of integers instead of string characters as follows.

module Main where

import System.Environment
import Data.String.Utils

liftLine :: String -> [Int]
liftLine l = fmap ( read . strip ) $ split " " l

main :: IO()
main = do [i] <- getArgs
          f <- readFile i
          let rows = fmap liftLine $ lines f
              in putStrLn $ show rows

In Haskell there are various map functions, here we using the most basic map function called fmap, which is implemented using Functor type class with instance types or anonymous types. So, the liftLine function extract a list of Integers from each line splitting the line using the space character, then applies strip to each tokenized string and reads its input converting it to Integer, and since it is using fmap, it will return a List of Integers. Where map is used again to lift each line, so, we are using fmap Functor as our lifting map function to extract the data as Integer from its plain text representation.

Now we need to process that lifted data using map, to obtain our required product for each integer list. So, we need to use the reduce function, which has two variants in Haskell, foldl and foldr, and for our convenience, we will use foldr, where reduce has a Monoidal behaviour due to its typed nature and can be expressed as .

module Main where

import System.Environment
import Data.String.Utils

liftLine :: String -> [Int]
liftLine l = fmap ( read . strip ) $ split " " l

reduceProduct :: [Int] -> Int
reduceProduct l = foldr (\ x y -> x * y ) 1 l

main :: IO()
main = do [i] <- getArgs
          f <- readFile i
          let rows = fmap liftLine $ lines f
              prds = fmap reduceProduct rows
              in putStrLn $ show prds

So, those are the very basic principles of MapReduce, which is used to calculate the PageRank on Google and get the site ranking on search results. So, this example is not distributed and it is a plain example of how it operates, just think on a distributed map and a distributed reduce operating with a very large amount of data to be processed. And there are various frameworks to work with MapReduce as model, because the original Google framework is not open source and you cannot work with it. As example of available frameworks supporting MapReduce as model you have Hadoop, Elastic MapReduce and Holumbus, so you can use this model with those frameworks reducing computation costs if you do not want to process your data with linear computations.

Copyright © 2012 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment
Post tags:

An example of a good placed Unix system is Mac OS X, it has several command line tools to automate the operating system and make it friendly and its native development environment called X Code is very easy to acquire, without too much problems. It is a real programmer environment. Flexible enough to be customized, with various tools that allows you to program the environment, automate tasks and make life really more easier. For example while I’m programming and I want to seek source code with log lines, I use a single command line pipe to filter log messages, as follows.

$ find . -type f -name '*.py' -exec grep -nHi 'l.debug' {} \; | sed '/exc/d;'

On Windows and similar operating systems, which are specially made for users, rather than developers, I feel really truncated because I cannot do stuff like that. So, I become less productive under Windows and I hate it. Things that should be solved quickly with few command line and editor hooks — like Emacs provides me for editing automation — are delayed under Windows because it does not support that kind of stuff. What’s better than having an editor with an embedded Lisp interpreter? What’s better than having Perl and Awk among a lot of command line tools that allows me to automate any task that I want?

The same happens on its browser. Internet Explorer does not have enough tools for Web Development like others have. On Firefox and Google Chrome we have very nice development tools, allowing us to create amazing DHTML / AJAX based applications, among other stuff like building amazing HTML 5 and WebGL applications. I find really frustrating any work made under Internet Explorer. If you want to debug a simple JavaScript implementation — which is not really JavaScript under windows and it is dialect called JScript — you need to launch a whale called Visual Studio. So, to become productive developing Internet Explorer applications, you must pay for something that will be used with a 10% of its capacity, including the licensing cost of Visual Studio.

So, for me Windows, Internet Explorer, and related technologies are a mere gaming console, rather than a developer friendly environment. That is why I dislike using Windows and developing systems for that operating system, because it is poor, overfilled with useless ornaments rather than useful tools.

Copyright © 2012 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | One comment
Post tags:

An example of requirement change management process is as follows:

1. Log the changes.
2. Perform an impact analysis on the work products.
3. Estimate the effort needed for the change requests.
4. Reestimate the delivery schedule.
5. Perform a cumulative cost impact analysis.
6. Review the impact with senior management if thresholds are exceeded.
7. Obtain customer sign-off.
8. Rework work products.

So, you should record each change in the project documentation, on the product backlog on agile projects and the project specification in structured projects. If you do not record all changes that are made to any software piece, there is not log or evidence that were made changes to the project, and you are loosing the core project specification. Any change requires analysis of all changes that are applied to the software piece. If you do not perform analysis, you are delivering software pieces without the proper structure and logical basis, even if you are working with prototypes. And finally, all changes should be measured and estimate the effort required to complete them, without an estimation of those efforts, you will not be able to handle the change set cost, deriving in a project with a higher cost than it was measured originally. Even if you are using agile or structured methodologies, you should measure changes and manage them.

So, if you are working with project managers that are not reestimating the delivery schedule with requirement changes, you are really wasting your time on that company. You can find about this topic on any modern project management book, and you will agree with the idea that a requirement change implies a change in the delivery schedule.

If you think in software pieces like products or physical products, for example like building houses, if you have some UML diagrams, like class diagrams and sequence diagrams, which are specifying how the application should work, if you receive changes over those diagrams and specifications, is the same thing as changing a house plane, you need to rebuild and refactor everything, and it has a very high cost. The same applies to those applications implementing work flows, if they are implemented using precarious if statements, instead of using state machines, any change to the work flow will lead you very hard changes on the code. So you should measure and control each change, so you must do management.

Copyright © 2012 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment
Post tags:

Once you have configured and working the django-storages package, you should fill the Date header or the x-amz-date header each time that you do a request to S3. So, the only way to do that is not to leave the module sending the date automatically, instead you should write the settings variable AWS_HEADERS.

import pytz
from datetime import datetime
from django.conf import settings

def get_aws_date(self):
    """
    Returns the server date formatted and using the server
    time zone localized to be used as Date header with
    Amazon S3.
    """
    stz = pytz.timezone(settings.TIME_ZONE)
    dtm = stz.localize(datetime.now()).strftime("%a, %d %b %Y %H:%M:%S %z")
    settings.AWS_HEADERS['Date'] = dtm
    settings.AWS_HEADERS['x-amz-date'] = dtm
    return dtm

Also, you must consider this requirement to work with S3 on all your requests. Since django-storages checks if the AWS_HEADERS has the Date or x-amz-date headers set, you must set that header each time that you make a request to S3, so you cannot use a batch read() from the storage or a batch write() to the storage, because it will use the previously sent Date header and it will fail, because it is considered inconsistent by the S3 authentication mechanism. If we use S3 as default storage, the example below will fail, because it will send file chunks with the same Date header, because the header was set previously.

from django.core.files.storage import default_storage

if default_storage.exist('test-large-file.mp3'):
    mp3file = open('test-large-file.mp3')
    s3file = default_storage.open('test-large-file.mp3')
    mp3file.write(s3file.read())
    mp3file.close()
    s3file.close()

So, you need to use small chunks to read from the storage, as the example below.

from django.core.files.storage import default_storage

if default_storage.exist('test-large-file.mp3'):
    get_aws_date()
    mp3file = open('test-large-file.mp3')
    s3file = default_storage.open('test-large-file.mp3')
    buff = s3file.read(settings.AWS_CHUNK_SIZE)
    while buff:
        mp3file.write(buff)
        try:
            buff = s3file.read(settings.AWS_CHUNK_SIZE)
        except Exception, exc:
            buff = None
    mp3file.close()
    s3file.close()

But also you must consider few issues about this. Since AWS_HEADERS is a global variable, writing to that variable will slow down your code because it lacks time on GIL usage, also it will lack your threaded application if it written twice, even if you are using locks and Django can handle parallel writes to that variable. So, be careful reading and writing large files from S3, and take a look on how the Date header is sent on each request.

def _add_aws_auth_header(self, headers, method,
                         bucket, key, query_args):
    if not headers.has_key('Date'):
        headers['Date'] = time.strftime("%a, %d %b %Y %X GMT",
                                        time.gmtime())

    c_string = canonical_string(method, bucket, key,
                                query_args, headers)
    headers['Authorization'] = \
        "AWS %s:%s" % (self.aws_access_key_id,
                       encode(self.aws_secret_access_key, c_string))

Where the %X specifier for localized machines with an environment variable LC_ALL different from C will throw the wrong date format disallowing your application to work with S3. This is a well known bug reported on this link. So, the right implementation, should be as follows.

def _add_aws_auth_header(self, headers, method,
                         bucket, key, query_args):
    if not 'Date' in headers:
        stz = pytz.timezone(settings.TIME_ZONE)
        dtm = stz.localize(datetime.now()).strftime("%a, %d %b %Y %H:%M:%S %z")
        headers['Date'] = dtm

    c_string = canonical_string(method, bucket, key,
                                query_args, headers)
    headers['Authorization'] = \
        "AWS %s:%s" % (self.aws_access_key_id,
                       encode(self.aws_secret_access_key, c_string))

Good luck using S3.

Copyright © 2012 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment
Post tags:

As example, on Python I have a list of integers and I want the power of two of each element. Someone with the basic approach of imperative programming and procedural programming will implement something like the code that follows.

nums = xrange(1, 11)
resl = list()
for n in nums:
    resl.append(n ** 2)
print(resl)

If you know a little of functional programming and you know that Python is a multi-paradigm programming language that allows you to use some functional features on it, you know that the code above have bad implementation and should be simplified just to three lines of code, because Python supports lambda expressions and higher order functions, as code the code that follows.

nums = xrange(1, 11)
resl = map(lambda x: x ** 2, nums)
print(resl)

Among other stuff, like reducing cyclomatic complexity and making more elegant code, you are not conscious about those changes that are present in other programming languages you will fall in very ugly mistakes about programming. Most programming languages are currently including support for various functional features like lambda expressions, higher order functions, closures, continuations and many other features considered functional. I do not need to name each language that has added support for them, but as examples we have Java, C Sharp, C++, JavaScript, and many others.

We have reached a point where the functional paradigm cannot be ignored any more. You must understand many concepts that comes from functional programming to understand concepts related to technologies that are currently widely used, like MapReduce, which has its conceptual basis on functional operations too, by mixing two higher order functions called map and fold — which is called reduce() in Python and (apply) in Lisp.

So, you should start learning about functional programming, and learning well, from its basis, concepts and implementations. Also, you should know that most languages used in algorithm research are functional languages. So, the next time that I see a for statement placed where should be a statement where you can use a higher-order function, I will consider your code not good as you think.

Copyright © 2012 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | 4 comments
Post tags: