The architectural pattern should be simple. Most people is thinking on the MVC architectural pattern as the most proper architectural for any application. But really the MVC pattern was designed with very old basis and mostly with the idea of generalizing the user interface of desktop applications. Currently, Web Applications are build on top of many technologies, not RDBMS engines only, so you should start using the proper architectural pattern, where technology diversity should follow the maximum cohesion principle, rather than coupling application components.

We have as key principle of design «minimum coupling, maximum cohesion», but I would like to add «proper generalization». So, finally we should have «minimum coupling, maximum cohesion & proper generalization» as design basis. If we observe, generalizations are older than software, we can stage that the term comes from Maths, where a generalization is a concept that can be applied to multiple problems and solve them using that concept. Cohesion is the principle of independent components that can work together, and coupling is a design which cannot separate its components. One of the best and classical examples of maximum cohesion on its design is Unix, where there are many small tools that can work together without coupling its functionalities.

A good generalization comes with from a good model, and any good generalization usually is able to solve multiple problems at once. Object Oriented programming provides a rich interface for abstractions, where the generalization is one of the most powerful ones, but in words of real generalization, I can proudly say that Monads are one of the best abstractions and mostly generalization that I ever seen.

• Haskell Programmer: Hey Java programmer, you know how just about everything is represented by a class?
• Java Programmer: Yea…
• HP: Even stuff no business being a class
• JP: Well, it is our fundamental abstraction for anything larger than a function. It may be clumsy, but it can be used to implement any other design pattern we need.
• HP: Well Monads serve the same role for us.
• JP: Nulls?
• HP: Maybe Monad
• JP: I/O?
• HP: IO Monad
• JP: Exceptions and error handling?
• HP: We have several ways, but mostly Maybe Monad
• JP: List comprehension?
• HP: Again, Monads
• JP: Mutable data structures?
• HP: That would be the State Monad_[2]

With the proper generalization you can make your design to be consistent, robust and reliable. But you must leave the generalization concept just as hierarchy problem. It should be stated globally as computational problem rather than design problem only. A generalization as computation can be applied to multiple problems, where applying computational generalizations we can find mapping functions that can be globally used. This can be reached combining the IPO model and Object Oriented models properly, working together.

Software design is not easy. An UML class model is not enough, where it can only display structures, probably with a sequence models and communication models you can reach a good IPO design, but not without the proper knowledge about the implementation. So, any analyst who does not knows how to program, is just wasting your money and time, with the obvious further reimplementation and redesign of the software product.

And you should start adding the proper generalizations as computations to your design, working together with your hierarchical generalizations, so they can bring you the proper quality along the time.

[2] This is an excerpt from a discussion on a programming forum.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

There are various kinds of techniques that can be used to create thread-safe algorithms and data structures. The are two main classes called blocking algorithms and non-blocking algorithms. On a blocking algorithm, locks are used to encapsulate fragments of code where you must not do concurrent writes to avoid racing conditions, where that fragment is called critical section. On non-blocking algorithms, you do not use locks, instead you use atomic operations to switch between the used resources, for example using a technique called compare and swap or CAS.

A racing condition, is an event that occurs when two or more threads are accessing the same resource at the same time, leading to data corruption or system faults. The most classic lock techniques in Java and C, are the synchronized keyword on Java and pthread_mutex_lock() call in C. On Java we protect our critical section using synchronized as follows.

class SingletonSampleImpl extends Object {  

    private static Object singletonMutex = new Object();
    private static SingletonSampleImpl instance = null;  

    public static SingletonSampleImpl getInstance() {
        synchronized (singletonMutex) {
            /* when we instantiate the singleton
               we have a critical section to protect
               from racing conditions */
            if (instance == null) {
                instance = new SingletonSampleImpl();
            }
        }
        return instance;
    }
}

The singletonMutex object acts as mutex, where a mutex — term that comes from mutual exclusion — has a flag that indicates to the other threads if it is blocked or not to jump into the critical section and execute the code. So, a mutex or semaphore is a mere flag that indicates to the program if the resource is busy or not. Using locks or blocking-algorithms is expensive on most systems, depending on how are they used. For example is very expensive to use a blocking algorithm to protect a socket, where it must wait its chance to be written by other threads, with inherent network time wait.

On the non-blocking side, we have three types of algorithms: wait-free, lock-free and obstruction-free. Where wait free is the strongest technique, but the hardest to handle, because once you implement it you have an algorithm that do not waits for writing or reading. Lock-free usually uses CAS, so it using a compare and swap operation to ensure that the critical section is protected, without locking. Obstruction-free uses locking, but a different one called live-locking.

The atomic operations used on the non-blocking techniques usually are operating on a very low level of the processor, because they must executed without the interference of any other threads. For example to work with atomic operations using the GCC compiler we have some special extensions to work with them.

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

You can start checking the “2011 CWE/SANS Top 25 Most Dangerous Software Errors” using your unit tests. Regarding the best programming practices, and even using style checkers and static analysis tools, you will be generating better code. So, the cycle is simple while you are testing your application:

1. Create a Test Case.
2. Check that the Test Case runs correctly.
3. Choose a well known error, for example null references.
4. Change the Test Case and add the wrong input to your code.
5. Refactor the application hardening its input.
6. Check that the Test Case runs correctly with wrong input.
7. Document your progress with source code comments or annotations.
8. Choose another error and make the Test Case fail again.

Code Hardening Testing Cycle

There is also some interesting papers and documents with some techniques that you can integrate in your development process, so you can create more stable code, mostly applied to Object-Oriented software and strongly-typed languages, which are talking about using manually created evolutionary tests to using genetic programming for those tasks. I hope that we can see the results of that research using genetic programming to automate the test handling process. The advantage of strongly-typed languages is the fact that most of them require to be statically-typed, which means that your code must follow well implemented type declarations and variable declaration using the proper types. So, the type system on those languages can be used to ensure that implementations are correctly done.

Initially, the test cluster for the given class under test is defined using static analysis. The concerned classes are then instrumented; thereby, the test goals — in our case all branches of the methods of the class under test — are collected. We follow the goal oriented approach and carry out a search for a test program for each individual test goal.

Static analysis again plays an interesting role on this approach, where is easier to handle static analysis on strongly-typed languages, rather than using static analysis on dynamically-typed languages.

When comparing evolutionary-based approaches over random testing [21] several prominent advantages arise, which include: less need for human analysis, as the evolutionary algorithm pre-analyses the software in accordance to the fitness function; the ability to automatically test combinations of suspicious parameters; and the possibility of finding combinations of inputs that lead to a more severe fault behaviour. Drawbacks include the difficulty of detecting solitary errors (“needles in a haystack”) with greater efficiency than random testing, and the impossibility of guaranteeing code coverage in black-box testing.

But if we have a lack of that kind of tools in our environment, so we must do that kind of analysis manually, and manually make our test to be tested against suspicious environment conditions, so we can use manual evolutionary unit testing. I hope that you will enjoy hardening your code ;)

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

We can use Python as example language, or even Java. To check the Python style, we can use the pep8 style checker, which is based on the PEP8 Python Style Guide. To make lightweight Python correctness checks, we can use PyLint and PyFlakes, which are static checkers. To build unit test suites, we can use PyUnit. All those tools will help us in our coding tasks, so we can create a better and clean code. Very close to the required standards cited by the PEP8 and The Zen of Python. I have modified a script that runs all those checkers in one command to be integrated under Emacs, and you can download it here. To integrate that script on your Emacs editor, you can use the following settings:

(defun my-python-mode-hook ()
  (setq py-indent-offset 4
        py-smart-indentation nil
        py-continuation-offset 4
        indent-tabs-mode nil
        py-pychecker-command "pycheckers.py"
        py-pychecker-command-args (quote ("")))
  (setq interpreter-mode-alist(cons '("python" . python-mode)
                                    interpreter-mode-alist))
  (eldoc-mode 1)
  (define-key py-mode-map "C-cC-w" (lambda ()
     (interactive)
     (command-execute 'py-pychecker-run)))
  (message ">>> done my-python-mode-hook..."))

(add-hook 'python-mode-hook 'my-python-mode-hook)

On the Java side, we have CheckStyle as style checker. As static checker, we have PMD and FindBugs. To build unit testing suites, we can use the well known JUnit. All of those tools can be integrated in your favourite Java IDE, like Eclipse or NetBeans, since you can integrate them on those IDEs using plug-ins. Java is quite different from Python, because it has several coding styles, among a wide variety of FOSS projects and company standards, so you need to choose between those styles.

Among other tasks, the code review matters. Static analysis tools and style checkers are not enough to ensure that your code is clean. You can use code hardening techniques to make safer code. But that requires that you must stay a little bit paranoid with your code, checking every execution path on it. There are some good books on code hardening techniques like “GNU/Linux Application Programming”, which explains some techniques and “Secure Coding Principles & Practice”.

You can integrate those techniques in your evolutionary testing process. Debugging tests is not a ridiculous task, I think that it is a good approach to make better code, assuming that you will modify the program variables enough to make your unit tests fail, and then refactor the code, repair possible failure and fix a harder execution path. But those activities requires the proper time assignment, so your project manager must include enough time to test and prove that your code will work well, even if the environment fails.

I hope that you will enjoy changing your mind about thinking that the code that just works is fine, switching a little and make sure that your code will work always.

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Certainly an application that uses a worker model, using threads, probably requires concurrent programming techniques to use the singleton pattern. That kind of applications, for example those implemented under JavaEE containers, like IBM WebSphere and Oracle WebLogic, require a good governance over the code and a good knowledge on how will be used a singleton implementation. In other environments, where static objects are not shared on the stack, you don’t need to care about singleton implementations, like PHP environments, where only one thread or process is reaching the singleton object, it just requires to be instanced once, and do not requires mutexes.

If you are good enough, you can use the singleton pattern on worker enabled environments. The well known getInstance method, should be implemented as follows in Java under JavaEE environments:

class SingletonSampleImpl extends Object {

    private static Object singletonMutex = new Object();
    private static SingletonSampleImpl instance = null;

    public static SingletonSampleImpl getInstance() {
        synchronized (singletonMutex) {
            if (instance == null) {
                instance = new SingletonSampleImpl();
            }
        }
        return instance;
    }
}

In other environments like PHP, you do not need to care about creating a mutex that protects the singleton instance to be overwritten or allocated twice, you just need to care about an object instance that is not accessed twice by various threads concurrently.

class SingletonSample {

    private static $instance = null;

    public static function getInstance() {
        if ($instance == null) {
            $instance = SingletonSample();
        }
        return $instance;
    }
}

Also you must be very careful, and never externalise the control of mutexes to external routines. A mutex must be controlled locally. The reason is simple, you just need to avoid a deadlock by double locking the mutex. If you externalise the control of a mutex, you are in risk of reaching a situation like the following problem, where the second lock will wait for the first one, creating a deadlock and killing the thread.

synchronized (singletonMutex) {
    synchronized (singletonMutex) {
        instance = new SingletonSampleImpl();
    }
}

This is just the situation that you can reach by externalising the control of a mutex, not the algorithm itself that you will implement, but the risk that you run by allowing mutexes to be locked outside of the controlling class. This is a similar situation that generates a race condition when you are programming singletons in worker environments without using the proper mutex. The wrong implementation of a singleton that is under the risk of a race condition is on the following example, where the algorithm just checks for a null value, and do not uses a lock for the concurrent threads that can instantiate the instance object.

class SingletonSampleImpl extends Object {

    private static Object singletonMutex = new Object();
    private static SingletonSampleImpl instance = null;

    public static SingletonSampleImpl getInstance() {
        if (instance == null) {
            instance = new SingletonSampleImpl();
        }
        return instance;
    }
}

So, using singletons is not dangerous. Just requires to have developers with the proper knowledge.

© dmw for coder . cl, 2011. | Permalink | One comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

You can use unit tests and TDD, and double check that your algorithms are working. Also you can check your code using some static checker. Both will help to find you a little bit optimal code. This will allow you create more stable and reliable code, but not enough to be hard. When you code an aplication, you must think that the application and your code is merged within an environment, and it has dependencies. So, you need enough time to double check your code using tests and also, you need to double check your code using static checkers when they are available for your language. All is about input and output…

Your code has an input, from a service, from the user using a browser, from some data retrieval cron task, whatever, your application is subject of input errors. The typicall error is to reach null references, where your application tries to use an unallocated address space, using a reference to an object or anything related to it. The main technique against null reference, is to check the reference before you begin using it. Just to show some examples I will use Java code, I can use C++ or C code, but I prefer to use Java codes in this post:

  /***
   * Returns a list of File objects that match the given stream
   * name on the given directory.
   *
   * @param strm    Stream name.
   * @param bdir    Directory to List.
   * @return      An array of found File objects.
   */
  public static File[] getFileNamesForStream(String strm, String bdir) {
    ArrayList<File> res = new ArrayList<File>();
    if (strm == null || bdir == null) {
      return new File[0];
    }
    if (strm.trim().length() == 0 || bdir.trim().length() == 0) {
      return new File[0];
    }
    File fdir = new File(bdir);
    File[] rawdir = fdir.listFiles();
    if (rawdir == null || rawdir.length == 0) {
      return new File[0];
    }
    for (int i = 0; i < rawdir.length; i++) {
      String name = rawdir[i].getName();
      if (name.indexOf(strm) >= 0) {
        res.add(new File(bdir + File.separator + name));
      }
    }
    Collections.sort(res, new FileDateComparator());
    return (File[]) res.toArray(new File[0]);
  }

The code above does various checks. Ther first one is to check the strm and the bdir parameters to be null. If those parameters are null, the method returns an empty array — instead of returning a new null reference increasing the error probability — so the user do not process any file from the returned array, the returned object can be used transparently. Then checks for string emptyness, we cannot use empty file names, because the operating system do not allows empty file names, and again we are using a zero filled array as return object. Then we list files on the input directory bdir, and we check if there is any available file on the directory, if the directory is empty, we return a zero filled array, but never returns a null reference. The method is hardened against input and output. The example above is easy to handle, but “what happens to service enabled methods?”.

  /**
   * Initial method to handle playlist items on server startup.
   *
   * @param server      The server where to put the streams.
   */
  private void refreshPlayList(IServer server) {
    try {
      if (server == null) {
        return;
      }
      if (minutesToShift < 0 && ownServer != null) {
        synchronized (schedulerMutex) {
          minutesToShift = Long.valueOf(ownServer.
              getProperties().
              getPropertyStr("MinutesToShift", "41")).
            longValue();
        }
      }
      IVHost vhost = VHostSingleton.getInstance(server.
                         getProperties().
                         getPropertyStr("PublishToVHost", VHost.VHOST_DEFAULT));
      if (vhost == null) {
        return;
      }
      IApplication app = vhost.getApplication(server.
                         getProperties().
                         getPropertyStr("PublishToApplication", "live"));
      if (app == null) {
        return;
      }
      streamsToAdd = WowzaTools.getStreams(server.
                         getProperties().
                         getPropertyStr("StreamsToAdd", "_default_"));
      for (int si = 0; si < streamsToAdd.length; si++) {
        String streamName = streamsToAdd[si] + STREAM_SUFFIX;
        Stream stream = getStream(vhost, app, streamName);
        if (stream == null) {
          continue;
        }
        streamMap.put(streamName, stream);
        app.getAppInstance("_definst_").
          getProperties().
          setProperty(streamName, stream);
        setupStream(stream);
        IStreamActionNotify actionNotify = new
          StreamListener(app.getAppInstance("_definst_"));
        stream.addListener(actionNotify);
      }
      ScheduledItem sched = new ScheduledItem(
        new Date(System.currentTimeMillis() + 30000)
      );
      sched.start();
    } catch (Exception ex) {
      log.error(ex.getMessage());
      log.error(ex);
    }
  }

The method above is void, it do not have a return value. The first check is over the server argument, if it is a null reference, we exit from the method. Then checks for the minutesToShift and the ownServer static members, and uses a mutex to write on one of them, doing a synchronous write and doing it once. Then we check for the vhost and app variable to be null references, we exit from the method if any of them are null references. Those checks are made before using those variables. Depending on how do you handle that kind of errors, you should use an exception instead of a plain return, but it depends on how do you need to handle the error. Then, another check is done on the cycle, we check if the obtained stream is a null reference, if it is a null reference, we continue processing the loop, instead of using that null reference. On this method we depend on third party methods that can return null reference, so we must check for them, instead of being sure of their availability. If you are a good observer, you will notice that the method getPropertyStr() has a default return value when the parameter is not found, but “what happens with those methods without a default return value?”.

  public ActionForward execute(ActionMapping mapping,
                               ActionForm form,
                               HttpServletRequest request,
                               HttpServletResponse response)
         throws IOException, ServletException {
    String target = "success";
    String request_arg = request.getParameter("request_arg") == null ?
                         "0" :
                         request.getParameter("request_arg");
    if (form != null) {
      NameForm nameForm = (NameForm)form;
      String name = nameForm.getName() == null ?
                    "" :
                    nameForm.getName();
    }
    if (name == null) {
      target = "failure";
    } else {
      request.setAttribute("name", name);
    }
    request.setAttribute("request_arg",
                         Integer.valueOf(request_arg));
    request.setAttribute("target", target);
    return (mapping.findForward(target));
  }

Above we have a typicall struts ActionForward method. It handles a request argument with a default value, using the ternary operator to do that. Checks for the possible null reference that can provide the request_arg parameter. The same applies to the form attribute name returned by the getName method. Using default values, can help you a lot. Null references are like a ninja waiting patiently to dismember you when are not expecting him. Again we have a method that is a little bit hardened. The same we can apply to our unit tests, we can ensure that they will not fail, for example if we have the following test:

  /**
   * Test Video Info Handling.
   */
  public void testVideoInfo1() {
    System.out.println("testVideoInfo1() Entering ->");
    String streamNames1 = "CHANNEL_TV";
    String directoryName1 = "/home/dmw/tm";
    VideoInfo[] streamsToAdd1 = WowzaTools.getInfoList(streamNames1, directoryName1);
    for (int i = 0; i < streamsToAdd1.length; i++) {
      VideoInfo current = streamsToAdd1[i];
      String msg = "Name: " + current.getVideo().getName()
        + "; Date: " + current.getTime()
        + "; Duration: " + WowzaTools.secondsToMinutes(current.getDuration());
      System.out.println(msg);
    }
    System.out.println("testVideoInfo1() Exiting >-");
  }

On this method we are testing the getInfoList() method. Since it is a testing class, it is easy enough to change the arguments passed to the getInfoList() method. We can pass null references, incorrect arguments and anything we think that can generate an error. We must not trust in the user input. Is easy to change some lines to streamNames1 = null;, streamNames1 = “xxx”;, directoryName1 = null;, directoryName1 = “/dev/null”;, or whatever can help you to detect possible failures in your code. You will be hardening your code if you can handle more errors which the typicall user can generate. Also, you must request enough time to your project manager to complete code hardening tasks and debugging sessions, which are slow, since you need to trace the code various times. Is not enough to write unit tests, you need to run your tests and debug them, and change every dependent value to ensure that your code is free of flaws. I do not know if project managers are considering code hardening or testing time, usually I just see enough time only to develop software, not to ensure that it will not fail. If you do not have enough time to do those tasks, because your project manager is not measuring that time as part of the development process, your project manager sucks…

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

list implementation

As we have seen, the playlist container uses a List implementation to store PlaylistItem objects. The problem with that implementation is that you can’t execute the clear() method properly, because it maintains the list size. So, each time that I was adding items to the playlist, and every time that a reschedule was required I was using the addToPlaylist() method before the clear() call, expecting the list to be emptied and resized to zero. That was not true. The trick was just to create a new playlist on each reschedule event.

Playlist playlist = new Playlist(stream.getName());
playlistMap.put(stream.getName(), playlist);
long offset = WowzaTools.getTimeShiftOffset(videoList, minutesToShift);
synchronized (streamMutex) {
    for (int pi = 0; pi < videoList.length; pi++) {
        VideoInfo cs = videoList[pi];
        if (pi == 0) {
            stream.addToPlaylist(stream.getPlaylist().size() + 1,
                                 cs.getVideo().getName(),
                                 (int) offset, cs.getDuration());
        } else {
            stream.addToPlaylist(stream.getPlaylist().size() + 1,
                                 cs.getVideo().getName(),
                                 0, cs.getDuration());
        }
    }
}
playlist.open(stream);

required mutexes

When the stream recording cuts the stream recording twice — caused by network problems — the onPlaylistItemStop() is triggered twice too. This can cause a race condition, and causing the server to stop some stream transmissions, and even can cause the lost of the stream recording. Events are not triggered sequentially, and they seems to be using some asynchronous kind of execution mechanism. So, the playlist rescheduling mechanism has to deal with that condition. To solve that issue, I’ve created two mutexes. One mutex for the event scheduler and other mutex for the stream treating mechanism. That is ensuring — in some manner — that event handlers are executed sequentially.

synchronized (schedulerMutex) {
    if (item.getIndex() == (stream.getPlaylist().size() - 1)) {
        synchronized (streamMutex) {
            setupStream(stream);
        }
    }
}

The previous algorithm ensures scheduler events are triggered sequentially and even stream treating methods are treating the streams blocking them on each event. Why I don’t used the stream itself as mutex, is because the stream can be blocked by other threads, and probably it can cause a deadlock, so I don’t have blocked server side objects. Probably, instead of using a synchronized mutex on Java for certain cases is less elegant than using rwlocks in POSIX threads, but the Java API do not offers rwlocks, and you can deal only with synchronized mutexes. I’ve missed rwlocks in this project.

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

One of my current projects is to implement a time shifted stream scheduler for the Wowza Media Server. The difference between the original stream and the time shifted stream is large enough to flood the hard drive with recorded data if you leave configured the live-record stream type. Then I’ve coded a custom stream recorder. The API is very nice to do such task. I’ve created an application side module to record streams under certain conditions.

<Property>
        <Name>MediaFormat</Name>
        <Value>1</Value>
</Property>
<Property>
        <Name>Append</Name>
        <Value>false</Value>
</Property>
<Property>
        <Name>VersionFile</Name>
        <Value>true</Value>
</Property>
<Property>
        <Name>StartOnKeyFrame</Name>
        <Value>true</Value>
</Property>
<Property>
        <Name>RecordData</Name>
        <Value>true</Value>
</Property>
<Property>
        <Name>RecordStreams</Name>
        <Value>TV_STRM_CH_8,TV_STRM_CH_5,TV_STRM_CH_2</Value>
</Property>

The MediaFormat property allows you to record flv media when it’s set to 1 and mp4 media files when it’s set to 2. Otherwise, it records flv media files by default. If the Append property is set to true it must be used with VersionFile, StartOnKeyFrame and RecordData enabled. The Append property allows you to create one unique recorded media files for each stream. If you set Append to false, the module creates recorded media files with a timestamp suffix with the format yyyyMMddHHmm. And since the time shift module requires stream metadata information, I’m using Append as false and setting the full stream information with the other metadata information properties enabled too.

The RecordStreams property allows you to record only those configured streams, instead of recording all streams — by using the live-record stream type — and allowing you configure which information will be distributed along the media files.

The main trick to record streams is to use not early events. First I’ve tried using onAppStart() event method and then onStreamCreate() event method, but they didn’t work. Now I’m using the IMediaStreamActionNotify2 interface with the proper events to handle the stream creation and the stream metadata, such as stream name, stream media type and related stuff. Probably the disadvantage of using custom media recorders is the fact that the API only provides LiveStreamRecorderMP4 and LiveStreamRecorderFLV recording classes and if you want an open media format, such as OGG enabled streams, this will not work.

The stream recorder module uses the ModuleBase abstract class, the time shifter publisher implements the IServerNotify interface. You have a wide variety of classes and interfaces that can be used on different tasks, including some weird such as LDAP enabled authentication mechanisms. You just need to think and to architect your server extension a little.

© dmw for coder . cl, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

non cycled media files

If you have non-cycled media files you can’t the timestamp on files to cycle them. You need create a dynamic playlist, reading the media file metadata each time that the playlist finishes to transmit its contents. Fortunately, the Wowza Media Server API provides an event handling method called onPlaylistItemStop on the IStreamActionNotify interface. To reconstruct the playlist on that method, you just need to call a playlist refreshing method.

try {
    log.error("---> Scheduled: onPlaylistItemStop() -> Enter");
    if (item.getIndex() == (stream.getPlaylist().size() - 1)) {
        stream.getPlaylist().clear();
        setupStream(stream, pl);
    }
    log.error("---> Scheduled: onPlaylistItemStop() -> Exit");
} catch (Exception ex) {
    log.error(ex.getMessage());
}

Once we clear the playlist, we call the setupStream method — which must be implemented by you — and it will reconstruct the playlist.

The arguments that I bring to the application through the WowzaScheduler module — which is a private development, so I can’t release the module, and even the code — are as follows:

<Property>
        <Name>StreamsToAdd</Name>
        <Value>CH2_TV,CH3_TV,CH4_TV</Value><!-- dmw: stream names to add -->
</Property>
<Property>
        <Name>MinutesToShift</Name>
        <Value>30</Value><!-- dmw: 30 minutes -->
</Property>

This will allow you to add only the streams CH2_TV, CH3_TV and CH4_TV, with a timeshift of 30 minutes.

major problems found

Major problems found while I was developing the extension are a little bit hard to handle. The main one is the fact that the API do not provides publicly any class or method that allows you to read metadata information about the media files. We used some external tools to handle that issue.

Another problem is that Stream.getPlaylist method provides an instance to the playlist in the form of a standard List<PlaylistItem>. So, we have some problems with that, since we don’t have any media type related methods and the PlaylistItem interface do not provides media file metadata information methods too. We are using a kind of catalog to handle media processing tasks. Probably we need to create a daemon to handle media processing, something done in a low level language, like C, and third party FOSS libraries.

You can find problematic some issues related to the streams storing on the media server. For example if the server is very busy, and you do a server restart or server stop, and the operating system requires to kill the process instead of doing a normal shutdown, some media files will be truncated, and certain data blocks for those media files will not be flushed to the hard drive, with the risk of lost recording seconds.

For now, the stream scheduler and stream time shifter is working fine. It passed many tests, but still we are testing the module. I hope that we will finish this project very well, as it is working now…

© dmw for coder . cl, 2010. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

cycled media files

Media Files stored on the hard drive, were encoded using the MP4 codec. Those files were cycling in ranges of 10 to 20 files and updated dynamically, like rotation logs, but with video file names being reused along the Media Server play list. In other cases, the Media Server play list is very dynamic, and the list of media files changes during the playback recording. I’ve created a very simple algorithm which lists the files and sorts them. The plug-in must handle that list of files on the cycle and realign the file order to use the last modified file according to the given time of minutes.

/***
 * Returns a list of File objects that match the given stream
 * name on the given directory.
 *
 * @param strm      Stream name.
 * @param bdir      Directory to List.
 * @return          An array of found File objects.
 */
public static File[] getFileNamesForStream(String strm, String bdir) {
    ArrayList<File> res = new ArrayList<File>();
    if (strm == null || bdir == null) {
        return new File[0];
    }
    if (strm.trim().length() == 0 || bdir.trim().length() == 0) {
        return new File[0];
    }
    File fdir = new File(bdir);
    if (fdir == null) {
        return new File[0];
    }
    File[] rawdir = fdir.listFiles();
    if (rawdir == null || rawdir.length == 0) {
        return new File[0];
    }
    for (int i = 0; i < rawdir.length; i++) {
        String name = rawdir[i].getName();
        if (name.indexOf(strm) >= 0) {
            res.add(new File(bdir + File.separator + name));
        }
    }
    Collections.sort(res, new FileDateComparator());
    return (File[]) res.toArray(new File[0]);
}

Getting the file list sorted is quite easy. And reorder the the files according to the cycle and the given timestamp is very easy too.

/**
 * Returns the cycled list of File object to be scheduled on the
 * Wowza Server.
 *
 * @param strm          input stream File name pattern.
 * @param bdir          input base directory.
 * @param mins          minutes elapsed.
 * @return              a list of files to cycle in the Wowza Server.
 */
public static File[] getCycledFileList(String strm, String bdir, long mins) {
    if (mins <= 0) {
        return new File[0];
    }
    File[] lst1 = WowzaTools.getFileNamesForStream(strm, bdir);
    ArrayList<File> lst2 = new ArrayList<File>();
    ArrayList<File> lst3 = new ArrayList<File>();
    ArrayList<File> retLst = new ArrayList<File>();
    for (int i = 0; i < lst1.length; i++) {
        if (WowzaTools.getMatchElapsedTime(lst1[i], mins)) {
            lst2.add(lst1[i]);
        } else {
            lst3.add(lst1[i]);
        }
    }
    for (int i = 0; i < lst2.size(); i++) {
        retLst.add((File)lst2.get(i));
    }
    for (int i = 0; i < lst3.size(); i++) {
        retLst.add((File)lst3.get(i));
    }
    return (File[]) retLst.toArray(new File[0]);
}

Probably the trick is in the timestamp comparing routine:

/***
 * Looks if the file is immediate before the given range
 * of minutes.
 *
 * @param inf       input file.
 * @param mins      minutes.
 * @return          true if the file matches.
 */
public static boolean getMatchElapsedTime(File inf, long mins) {
    long ct = System.currentTimeMillis();
    long ct_d = ct - (mins * 60000);
    long modtm = inf.lastModified();
    return (modtm >= ct_d);
}

the matching timestamp

The Wowza Media Server recordings are stored sequentially on the <StorageDir/> directory. Since the recordings are dynamically generated, and opening each stream to get his time is a little bit hard to the Media Server, I’ve used the File modification time to get information about the time sequence of each file. Once the files are sorted, timestamps looks as follows:

if (filesToAdd.length > 1) {
    File prev = filesToAdd[(filesToAdd.length - 1)];
    for (int pi = 0; pi < filesToAdd.length; pi++) {
        File cs = filesToAdd[pi];
        int secs = (int)((prev.lastModified() - cs.lastModified()) / 1000);
        secs = secs > 0 ? secs : secs * -1;
        if (pi == 0) {
            playlist.addItem(cs.getName(), ft, secs);
        } else {
            playlist.addItem(cs.getName(), 0, secs);
        }
        prev = cs;
    }
}

The algorithm looks very simple. But the ft is the key argument for first list item: . Where is the current time, is the modification time and is the time shifting argument. So, the difference is used to calculate the first file on the playlist time shifting to use the proper offset in seconds. The solution is simple, and can be used in other tasks on the Wowza Media Server. I think that you can avoid many complex tasks using the modification time on media files.

Now I’m thinking on the creation of commercial extensions for the Wowza Media Server ;)

© dmw for coder . cl, 2010. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)