There are various kinds of techniques that can be used to create thread-safe algorithms and data structures. The are two main classes called blocking algorithms and non-blocking algorithms. On a blocking algorithm, locks are used to encapsulate fragments of code where you must not do concurrent writes to avoid racing conditions, where that fragment is called critical section. On non-blocking algorithms, you do not use locks, instead you use atomic operations to switch between the used resources, for example using a technique called compare and swap or CAS.

A racing condition, is an event that occurs when two or more threads are accessing the same resource at the same time, leading to data corruption or system faults. The most classic lock techniques in Java and C, are the synchronized keyword on Java and pthread_mutex_lock() call in C. On Java we protect our critical section using synchronized as follows.

class SingletonSampleImpl extends Object {  

    private static Object singletonMutex = new Object();
    private static SingletonSampleImpl instance = null;  

    public static SingletonSampleImpl getInstance() {
        synchronized (singletonMutex) {
            /* when we instantiate the singleton
               we have a critical section to protect
               from racing conditions */
            if (instance == null) {
                instance = new SingletonSampleImpl();
            }
        }
        return instance;
    }
}

The singletonMutex object acts as mutex, where a mutex — term that comes from mutual exclusion — has a flag that indicates to the other threads if it is blocked or not to jump into the critical section and execute the code. So, a mutex or semaphore is a mere flag that indicates to the program if the resource is busy or not. Using locks or blocking-algorithms is expensive on most systems, depending on how are they used. For example is very expensive to use a blocking algorithm to protect a socket, where it must wait its chance to be written by other threads, with inherent network time wait.

On the non-blocking side, we have three types of algorithms: wait-free, lock-free and obstruction-free. Where wait free is the strongest technique, but the hardest to handle, because once you implement it you have an algorithm that do not waits for writing or reading. Lock-free usually uses CAS, so it using a compare and swap operation to ensure that the critical section is protected, without locking. Obstruction-free uses locking, but a different one called live-locking.

The atomic operations used on the non-blocking techniques usually are operating on a very low level of the processor, because they must executed without the interference of any other threads. For example to work with atomic operations using the GCC compiler we have some special extensions to work with them.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | No comment
Post tags:

You can start checking the “2011 CWE/SANS Top 25 Most Dangerous Software Errors” using your unit tests. Regarding the best programming practices, and even using style checkers and static analysis tools, you will be generating better code. So, the cycle is simple while you are testing your application:

1. Create a Test Case.
2. Check that the Test Case runs correctly.
3. Choose a well known error, for example null references.
4. Change the Test Case and add the wrong input to your code.
5. Refactor the application hardening its input.
6. Check that the Test Case runs correctly with wrong input.
7. Document your progress with source code comments or annotations.
8. Choose another error and make the Test Case fail again.

Code Hardening Testing Cycle

There is also some interesting papers and documents with some techniques that you can integrate in your development process, so you can create more stable code, mostly applied to Object-Oriented software and strongly-typed languages, which are talking about using manually created evolutionary tests to using genetic programming for those tasks. I hope that we can see the results of that research using genetic programming to automate the test handling process. The advantage of strongly-typed languages is the fact that most of them require to be statically-typed, which means that your code must follow well implemented type declarations and variable declaration using the proper types. So, the type system on those languages can be used to ensure that implementations are correctly done.

Initially, the test cluster for the given class under test is defined using static analysis. The concerned classes are then instrumented; thereby, the test goals — in our case all branches of the methods of the class under test — are collected. We follow the goal oriented approach and carry out a search for a test program for each individual test goal.

Static analysis again plays an interesting role on this approach, where is easier to handle static analysis on strongly-typed languages, rather than using static analysis on dynamically-typed languages.

When comparing evolutionary-based approaches over random testing [21] several prominent advantages arise, which include: less need for human analysis, as the evolutionary algorithm pre-analyses the software in accordance to the fitness function; the ability to automatically test combinations of suspicious parameters; and the possibility of finding combinations of inputs that lead to a more severe fault behaviour. Drawbacks include the difficulty of detecting solitary errors (“needles in a haystack”) with greater efficiency than random testing, and the impossibility of guaranteeing code coverage in black-box testing.

But if we have a lack of that kind of tools in our environment, so we must do that kind of analysis manually, and manually make our test to be tested against suspicious environment conditions, so we can use manual evolutionary unit testing. I hope that you will enjoy hardening your code ;)

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | No comment
Post tags:

We can use Python as example language, or even Java. To check the Python style, we can use the pep8 style checker, which is based on the PEP8 Python Style Guide. To make lightweight Python correctness checks, we can use PyLint and PyFlakes, which are static checkers. To build unit test suites, we can use PyUnit. All those tools will help us in our coding tasks, so we can create a better and clean code. Very close to the required standards cited by the PEP8 and The Zen of Python. I have modified a script that runs all those checkers in one command to be integrated under Emacs, and you can download it here. To integrate that script on your Emacs editor, you can use the following settings:

(defun my-python-mode-hook ()
  (setq py-indent-offset 4
        py-smart-indentation nil
        py-continuation-offset 4
        indent-tabs-mode nil
        py-pychecker-command "pycheckers.py"
        py-pychecker-command-args (quote ("")))
  (setq interpreter-mode-alist(cons '("python" . python-mode)
                                    interpreter-mode-alist))
  (eldoc-mode 1)
  (define-key py-mode-map "\C-c\C-w" (lambda ()
     (interactive)
     (command-execute 'py-pychecker-run)))
  (message ">>> done my-python-mode-hook..."))

(add-hook 'python-mode-hook 'my-python-mode-hook)

On the Java side, we have CheckStyle as style checker. As static checker, we have PMD and FindBugs. To build unit testing suites, we can use the well known JUnit. All of those tools can be integrated in your favourite Java IDE, like Eclipse or NetBeans, since you can integrate them on those IDEs using plug-ins. Java is quite different from Python, because it has several coding styles, among a wide variety of FOSS projects and company standards, so you need to choose between those styles.

Among other tasks, the code review matters. Static analysis tools and style checkers are not enough to ensure that your code is clean. You can use code hardening techniques to make safer code. But that requires that you must stay a little bit paranoid with your code, checking every execution path on it. There are some good books on code hardening techniques like “GNU/Linux Application Programming”, which explains some techniques and “Secure Coding Principles & Practice”.

You can integrate those techniques in your evolutionary testing process. Debugging tests is not a ridiculous task, I think that it is a good approach to make better code, assuming that you will modify the program variables enough to make your unit tests fail, and then refactor the code, repair possible failure and fix a harder execution path. But those activities requires the proper time assignment, so your project manager must include enough time to test and prove that your code will work well, even if the environment fails.

I hope that you will enjoy changing your mind about thinking that the code that just works is fine, switching a little and make sure that your code will work always.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | No comment
Post tags:

Certainly an application that uses a worker model, using threads, probably requires concurrent programming techniques to use the singleton pattern. That kind of applications, for example those implemented under JavaEE containers, like IBM WebSphere and Oracle WebLogic, require a good governance over the code and a good knowledge on how will be used a singleton implementation. In other environments, where static objects are not shared on the stack, you don’t need to care about singleton implementations, like PHP environments, where only one thread or process is reaching the singleton object, it just requires to be instanced once, and do not requires mutexes.

If you are good enough, you can use the singleton pattern on worker enabled environments. The well known getInstance method, should be implemented as follows in Java under JavaEE environments:

class SingletonSampleImpl extends Object {

    private static Object singletonMutex = new Object();
    private static SingletonSampleImpl instance = null;

    public static SingletonSampleImpl getInstance() {
        synchronized (singletonMutex) {
            if (instance == null) {
                instance = new SingletonSampleImpl();
            }
        }
        return instance;
    }
}

In other environments like PHP, you do not need to care about creating a mutex that protects the singleton instance to be overwritten or allocated twice, you just need to care about an object instance that is not accessed twice by various threads concurrently.

class SingletonSample {

    private static $instance = null;

    public static function getInstance() {
        if ($instance == null) {
            $instance = SingletonSample();
        }
        return $instance;
    }
}

Also you must be very careful, and never externalise the control of mutexes to external routines. A mutex must be controlled locally. The reason is simple, you just need to avoid a deadlock by double locking the mutex. If you externalise the control of a mutex, you are in risk of reaching a situation like the following problem, where the second lock will wait for the first one, creating a deadlock and killing the thread.

synchronized (singletonMutex) {
    synchronized (singletonMutex) {
        instance = new SingletonSampleImpl();
    }
}

This is just the situation that you can reach by externalising the control of a mutex, not the algorithm itself that you will implement, but the risk that you run by allowing mutexes to be locked outside of the controlling class. This is a similar situation that generates a race condition when you are programming singletons in worker environments without using the proper mutex. The wrong implementation of a singleton that is under the risk of a race condition is on the following example, where the algorithm just checks for a null value, and do not uses a lock for the concurrent threads that can instantiate the instance object.

class SingletonSampleImpl extends Object {

    private static Object singletonMutex = new Object();
    private static SingletonSampleImpl instance = null;

    public static SingletonSampleImpl getInstance() {
        if (instance == null) {
            instance = new SingletonSampleImpl();
        }
        return instance;
    }
}

So, using singletons is not dangerous. Just requires to have developers with the proper knowledge.

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | One comment
Post tags:

You can use unit tests and TDD, and double check that your algorithms are working. Also you can check your code using some static checker. Both will help to find you a little bit optimal code. This will allow you create more stable and reliable code, but not enough to be hard. When you code an aplication, you must think that the application and your code is merged within an environment, and it has dependencies. So, you need enough time to double check your code using tests and also, you need to double check your code using static checkers when they are available for your language. All is about input and output…

Your code has an input, from a service, from the user using a browser, from some data retrieval cron task, whatever, your application is subject of input errors. The typicall error is to reach null references, where your application tries to use an unallocated address space, using a reference to an object or anything related to it. The main technique against null reference, is to check the reference before you begin using it. Just to show some examples I will use Java code, I can use C++ or C code, but I prefer to use Java codes in this post:

  /***
   * Returns a list of File objects that match the given stream
   * name on the given directory.
   *
   * @param strm    Stream name.
   * @param bdir    Directory to List.
   * @return      An array of found File objects.
   */
  public static File[] getFileNamesForStream(String strm, String bdir) {
    ArrayList<File> res = new ArrayList<File>();
    if (strm == null || bdir == null) {
      return new File[0];
    }
    if (strm.trim().length() == 0 || bdir.trim().length() == 0) {
      return new File[0];
    }
    File fdir = new File(bdir);
    File[] rawdir = fdir.listFiles();
    if (rawdir == null || rawdir.length == 0) {
      return new File[0];
    }
    for (int i = 0; i < rawdir.length; i++) {
      String name = rawdir[i].getName();
      if (name.indexOf(strm) >= 0) {
        res.add(new File(bdir + File.separator + name));
      }
    }
    Collections.sort(res, new FileDateComparator());
    return (File[]) res.toArray(new File[0]);
  }

The code above does various checks. Ther first one is to check the strm and the bdir parameters to be null. If those parameters are null, the method returns an empty array — instead of returning a new null reference increasing the error probability — so the user do not process any file from the returned array, the returned object can be used transparently. Then checks for string emptyness, we cannot use empty file names, because the operating system do not allows empty file names, and again we are using a zero filled array as return object. Then we list files on the input directory bdir, and we check if there is any available file on the directory, if the directory is empty, we return a zero filled array, but never returns a null reference. The method is hardened against input and output. The example above is easy to handle, but “what happens to service enabled methods?”.

  /**
   * Initial method to handle playlist items on server startup.
   *
   * @param server      The server where to put the streams.
   */
  private void refreshPlayList(IServer server) {
    try {
      if (server == null) {
        return;
      }
      if (minutesToShift < 0 && ownServer != null) {
        synchronized (schedulerMutex) {
          minutesToShift = Long.valueOf(ownServer.
              getProperties().
              getPropertyStr("MinutesToShift", "41")).
            longValue();
        }
      }
      IVHost vhost = VHostSingleton.getInstance(server.
                         getProperties().
                         getPropertyStr("PublishToVHost", VHost.VHOST_DEFAULT));
      if (vhost == null) {
        return;
      }
      IApplication app = vhost.getApplication(server.
                         getProperties().
                         getPropertyStr("PublishToApplication", "live"));
      if (app == null) {
        return;
      }
      streamsToAdd = WowzaTools.getStreams(server.
                         getProperties().
                         getPropertyStr("StreamsToAdd", "_default_"));
      for (int si = 0; si < streamsToAdd.length; si++) {
        String streamName = streamsToAdd[si] + STREAM_SUFFIX;
        Stream stream = getStream(vhost, app, streamName);
        if (stream == null) {
          continue;
        }
        streamMap.put(streamName, stream);
        app.getAppInstance("_definst_").
          getProperties().
          setProperty(streamName, stream);
        setupStream(stream);
        IStreamActionNotify actionNotify = new
          StreamListener(app.getAppInstance("_definst_"));
        stream.addListener(actionNotify);
      }
      ScheduledItem sched = new ScheduledItem(
        new Date(System.currentTimeMillis() + 30000)
      );
      sched.start();
    } catch (Exception ex) {
      log.error(ex.getMessage());
      log.error(ex);
    }
  }

The method above is void, it do not have a return value. The first check is over the server argument, if it is a null reference, we exit from the method. Then checks for the minutesToShift and the ownServer static members, and uses a mutex to write on one of them, doing a synchronous write and doing it once. Then we check for the vhost and app variable to be null references, we exit from the method if any of them are null references. Those checks are made before using those variables. Depending on how do you handle that kind of errors, you should use an exception instead of a plain return, but it depends on how do you need to handle the error. Then, another check is done on the cycle, we check if the obtained stream is a null reference, if it is a null reference, we continue processing the loop, instead of using that null reference. On this method we depend on third party methods that can return null reference, so we must check for them, instead of being sure of their availability. If you are a good observer, you will notice that the method getPropertyStr() has a default return value when the parameter is not found, but “what happens with those methods without a default return value?”.

  public ActionForward execute(ActionMapping mapping,
                               ActionForm form,
                               HttpServletRequest request,
                               HttpServletResponse response)
         throws IOException, ServletException {
    String target = "success";
    String request_arg = request.getParameter("request_arg") == null ?
                         "0" :
                         request.getParameter("request_arg");
    if (form != null) {
      NameForm nameForm = (NameForm)form;
      String name = nameForm.getName() == null ?
                    "" :
                    nameForm.getName();
    }
    if (name == null) {
      target = "failure";
    } else {
      request.setAttribute("name", name);
    }
    request.setAttribute("request_arg",
                         Integer.valueOf(request_arg));
    request.setAttribute("target", target);
    return (mapping.findForward(target));
  }

Above we have a typicall struts ActionForward method. It handles a request argument with a default value, using the ternary operator to do that. Checks for the possible null reference that can provide the request_arg parameter. The same applies to the form attribute name returned by the getName method. Using default values, can help you a lot. Null references are like a ninja waiting patiently to dismember you when are not expecting him. Again we have a method that is a little bit hardened. The same we can apply to our unit tests, we can ensure that they will not fail, for example if we have the following test:

  /**
   * Test Video Info Handling.
   */
  public void testVideoInfo1() {
    System.out.println("testVideoInfo1() Entering ->");
    String streamNames1 = "CHANNEL_TV";
    String directoryName1 = "/home/dmw/tm";
    VideoInfo[] streamsToAdd1 = WowzaTools.getInfoList(streamNames1, directoryName1);
    for (int i = 0; i < streamsToAdd1.length; i++) {
      VideoInfo current = streamsToAdd1[i];
      String msg = "Name: " + current.getVideo().getName()
        + "; Date: " + current.getTime()
        + "; Duration: " + WowzaTools.secondsToMinutes(current.getDuration());
      System.out.println(msg);
    }
    System.out.println("testVideoInfo1() Exiting >-");
  }

On this method we are testing the getInfoList() method. Since it is a testing class, it is easy enough to change the arguments passed to the getInfoList() method. We can pass null references, incorrect arguments and anything we think that can generate an error. We must not trust in the user input. Is easy to change some lines to streamNames1 = null;, streamNames1 = “xxx”;, directoryName1 = null;, directoryName1 = “/dev/null”;, or whatever can help you to detect possible failures in your code. You will be hardening your code if you can handle more errors which the typicall user can generate. Also, you must request enough time to your project manager to complete code hardening tasks and debugging sessions, which are slow, since you need to trace the code various times. Is not enough to write unit tests, you need to run your tests and debug them, and change every dependent value to ensure that your code is free of flaws. I do not know if project managers are considering code hardening or testing time, usually I just see enough time only to develop software, not to ensure that it will not fail. If you do not have enough time to do those tasks, because your project manager is not measuring that time as part of the development process, your project manager sucks…

Copyright © 2011 Daniel Molina Wegener
Atribución-No Comercial-Sin Derivadas 2.0 Chile

© Daniel Molina Wegener for coder . cl, 2011. | Permalink | No comment
Post tags: