On this case, I will iterate over the list of units on the scheduler — where each unit is a composite device — it will assign a new work state on each device, calling the run method on the device object. The run method will launch the child state machine of the required action, but the parent state machine only has few states storing the device status, so the processing scheduler knows which event should be triggered on each scan cycle. The required state transitions on the parent state machine, which are informational rather than hardware related, will be useful handling nested event handling.

Each cycle will determine whether read or write actions should be triggered, even if they are manual or automatic operations on each device, and only then the program will be able to call read and write once they are required. There are some actions not using the communication channel, so I can avoid calling read and write, enhancing the application performance a little, but still I will need to check the device state to check for some automatic reads, required to display the device state and ModBus holding registers.

You can find several models of hierarchical state machines, but I prefer to write one using events bound to class methods, where each class method drives an action on the device. This approach is simpler than others using signals and similar event handling methods, but I need to keep the source code as simple as possible, because increasing the complexity should increase the development time. Finally, the unit class will be only a gateway to the each hardware device, where each state will be triggered by the scheduler. Before this model, I was using a priority queue using messages, very similar to message queues, but is required on this system, to have direct access to the device state, and that message queue is not so synchronous as it is required, where a direct feedback from the device should be placed on each state transition.

Now I have finished the scheduler and the state machine processor, I just need to complete the integration between each component and I will deliver the new device management application. Also, the HMI has changed a little, and the new model allows some direct actions like resetting the device and its state.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

The state machine model includes on each transition a basic model defined as { s’, s”, g, f, h }, with s’ as the current state, s” the next state, g a boolean function evaluating the environment, f as guard boolean function preventing errors and h as action function. Where g is similar to f, but it covers the feasibility of the state change, because it evaluates if the device state (offline, online + automatic, online + manual), and f evaluates if the request is feasible according to the current composite device state (pump + valve + manometer), where each operation request was written as follows.

struct fcu_request_s {
    bool     channel;
    bool     done;
    bool     nobreak;
    int      errors;
    int      cycle;
    fcu_io  *device;
    pwr_tUInt16          operation;
    fcm_task_prio_t      priority;
    struct timespec      start_t;
    struct timespec      current_t;
    struct timespec      end_t;
    fcu_state_machine_t  *machine;
};

Where every boolean member is evaluated using the boolean function g, integer members are evaluated using the boolean function f, and if both function are not blocking the action, the action h is executed.

HMI (the image was modified to be shown).

Finally the state machine is mutable, because there are dynamic states which are triggered by boolean functions, and every transition is not strict, they are only a behaviour template rather than strict states to be used on each transition. From the processing node, each HMI signal is checked against the boolean functions, and each state transition is strictly evaluated before executing the action. This is more representative than single state transitions from the original state machine model, allowing better control. Where we are using dynamic programming to hold state transition tables and previously evaluated boolean functions which are common to the composite device state.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Another interesting fact is how you can integrate both C and C++ libraries and programs without too much effort. For example for the lower level of abstractions in our ProView systems, we are using C interfaces to reach the hardware layer, but that layer has an abstraction layer that allow us to see each device as a C++ class, so we can get device status, device errors, device data among other interesting operations as one single object instance, rather than using device descriptors or data structure pointers. So, we can operate each device instance, even if that device is a composite device that drives multiple sub-devices, as one single object instance with its own data and methods. That makes our life much easier than handling devices as data structures and plain C routines.

For example we have named a composite device PCU or Pump Control Unit, which consist on a pump control, an actuator control, a valve control and two manometers. It is very nice to see that PCU as one device, using plain methods over the PCU rather than using multiple plain C calls over their C data structures and pointers using each sub-device to control the unit. Even if they are using different protocols, we can operate each device transparently, thanks to every abstraction layer that we have placed on the system.

Choosing between C and C++ is not a religious choice. Mainly because I am a language agnostic programmer, and I work with limited set of programming languages where I can find better programmers, like C, C++, Python, Haskell and Lisp. Also I can choose Haskell if I want, because it supports low level function calling thanks to the Foreign Function Interface, where I can create low level interfaces for hardware devices and operate them in higher level using Haskell. For your surprise, I can create complex control algorithms in Haskell, migrate those algorithms to Agda, and create the proper proofs on those controlling algorithms. The result is a formally verified control algorithm. I can do the same with Coq and Haskell program extraction. A simple example for FFI code, is the GeoIP library interface that I have made on my ApacheLogRev project, and you can see the source code here.

So, is much better to be a language agnostic programmer than be linked to programming languages as religions…

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Now just remains operation details, such as state machine definitions for each operation that is required for the machinery operations that system will handle. The system has two main kinds of operations, automatic operations and manual operations, where manual operations have a higher priority than automatic ones, also automatic operations can be interrupted by manual operations, so the state machine that is running an automatic operation can be interrupted by any manual operation. Each state transition is monitored checking the system state, looking if a requested a manual operation should replace the current automatic operation, or the automatic operation should be interrupted by another automatic operation with a higher priority.

The algorithm — now that is implemented and not placed as diagram on the whiteboard — looks much simpler, and the basic algorithm looks as follows.

req = dev_get_request(MAIN_CONFIG, NUM_DEVICES);
if (req != NULL) {
    queue_task(req);
}

while (!channel && (req = check_queue())) {
    if (req->done) {
        break;
    }

    req->device->run(req);
    if (!req->channel) {
        continue;
    }

    sts = io_write(io_ctx);
    if (EVEN(sts)) {
        errh_Fatal("ra_fcm: io_write error, %m", sts);
        fcm_io_alarm(req->device, FCM_IO_WRITE);
    }

    sts = io_read(io_ctx);
    if (EVEN(sts)) {
        errh_Fatal("ra_fcm: io_read error, %m", sts);
        fcm_io_alarm(req->device, FCM_IO_READ);
    }

    if (req->nobreak) {
        channel = false;
    } else {
        channel = true;
    }
}

The dev_get_request method checks if there is a pending request, automatic or manual, if there is a pending request, it is being queued, and immediately the process starts processing the queued items, where they can be finished in one step or not, and the device runs the state machine referenced by the request, if the state machine transition uses the I/O channel, the ProView methods io_write and io_read are called, then, if the transition requests a no-break operation to make a continuous usage of the communication channel, it keeps running the state machine. Otherwise the state machine breaks its operation and seeks another operation request in the queue. The check_queue method seeks the proper prioritized request on the queue, where its priorities are changing inside the state machine on the run method.

But the state machine runner — placed on the device as run method — as an abstraction of the managed device by the priority queue, where there are several devices to manage, schedules their tasks using a simple algorithm. The real complexity is placed on the operation of each device, and the algorithm itself is an abstraction of the generalized state machine that we are running.

if (dev_state == DEV_ST_INACTIVE) {
    dev_state = DEV_ST_ONLINE;
}

request->nobreak = false;
machine->channel = false;
request->channel = false;

for (c = 0; c < machine->num_states; c++) {
    tr = &(machine->states);

    if (tr->curr_st != dev_state) {
        continue;
    }

    if (!tr->guard(tr, machine, request, this)) {
        break;
    }

    if (!operate(tr, machine, request)) {
        break;
    }

    errh_Info(st_msg_fmt, fcu_state, tr->next_st, tr->curr_st);
    dev_state = tr->next_st;

    if (tr->next_st == machine->final_state) {
        request->done = true;
        break;
    }

    if (tr->channel) {
        chn = true;
        request->nobreak = tr->nobreak;
        machine->channel = true;
        request->channel = true;
        break;
    }

}

return chn;

The guard checks environmental variables, and if there are good environmental conditions to operate, then it starts operating the device according to the current transition, that holds the action as enum type, with a simple switch statement to check the action that should be taken on the state transition. On a successful operation and the proper prioritization, the queue is reordered outside the state machine runner, allowing queued operations to trigger alarms if they are not finished, among other similar error checking and environment checking routines. So, the system keeps working, alerting the user through the HMI if there is any condition that never meets the required operation. ProView has several ways to alert users about the system conditions, and we are using almost everything that we can. I think that this system is very well structured and now running dynamic priorities is even more stable than using plain task requests, mainly due to environmental variables. This scheduler looks preemptive, but it is not.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

During the state machine execution, each state transition has a guard ensuring that the step that will take place on the transition is guaranteed to be executed depending on the environment, if not the step is reserved to the next PLC scan cycle — do you remember that ProView operates C and C++ applications like PLC programs?. The same guard ensure that the operation on the pump will take the required minutes to deactivate the pump, meeting the task calendar. Once the pump is active, its task priority — which is hold on the request data structure — changes its priority, and using the top() and pop() methods of the STL std::priority_queue, is extracted from the queue, the priority is changed, and it is pushed again on the queue using the push() method.

All requests that are using the communication channel have the higher priority, because we have only one communication channel, so we must ensure some concurrency on the channel usage, and we call the io_read() and io_write() ProView functions only once it is required, ensuring that the communication channel is used by one device at time. I think that this is the biggest problem of the system, because the communication channel is very small, and shared between several devices, and it must be used concurrently. But locks are not an option. If the channel blocks due to its usage, the system should continue its work and must keep its system review — reviewing channel status and signal status, and reviewing the error status for several devices.

Since several requests are not using the communication channel, the queue never gets saturated, and every transition is executed. It only blocks on communication channel usage, but it calls the io_read() and io_write() ProView functions rather than using some kind of lock, like semaphore or similar one. Also, errors are not allowed, the code should not have any programming error, and it is being traced by strace(1) and valgrind(1). Also, we are thinking to check the model using spin(1), I have started some Promela code related to the processing model, at least it looks well for now. It will probably get more complexity on the future, but the model looks well, and the code seems to be really good.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

On a recent project, the project manager has requested to me to code a state machine. Its initial design is not generalized. Along two design sessions and some sample code and constructs I think that I have finally made an abstract state machine, with various reductions, based on the model designed by the project manager on the design sessions, structured mostly as automaton, with very simple state transitions, but I have added error and wait conditions, allowing more friendly error processing and wait events. Mainly because as ProView application it has several similarities to PLC programs, where each cycle can operate one time due to the protocol restrictions, but also it allows multiple operations in one cycle because it can send larger packages with multiple requests.

Generalized State Machine

On each cycle, once the I/O layer is initialized, the device error state is checked, on failure it reports errors using ProView alarms, waits the next cycle and then operates again checking the device state. If no error is detected, it operates immediately once it can do it according to its cycle. And the state machine above is a generalized or abstract state machine, considering CST and OST as checking and operating states, where it is has a normalized transition between both states with the proper error handling states.

typedef struct fcu_sm_trans_s fcu_sm_trans_t;

struct fcu_sm_trans_s {
    fcu_op_state cst;
    fcu_op_state nst;
    fcu_op_state fst;
    bool alarm;
    bool wait;
};

/* pump and valve transtion table */
fcu_sm_trans_t AUTO_FCU_TRANSITION[] = {
    { FCU_ST_OFFLINE             , FCU_ST_DISCONNECTED      , FCU_ST_OFFLINE      , false, wait  }
    , { FCU_ST_DISCONNECTED      , FCU_ST_COPY_READ_POS_P   , FCU_ST_ERROR        , true , false }
    , { FCU_ST_COPY_READ_POS_P   , FCU_ST_COPY_WRITE_POS_P  , FCU_ST_ERROR        , true , false }
    , { FCU_ST_COPY_WRITE_POS_P  , FCU_ST_COPY_READ_POS_V   , FCU_ST_ERROR        , true , false }
    , { FCU_ST_COPY_READ_POS_V   , FCU_ST_COPY_WRITE_POS_V  , FCU_ST_ERROR        , true , false }
    , { FCU_ST_COPY_WRITE_POS_V  , FCU_ST_READ_POS_P        , FCU_ST_ERROR        , true , false }
    , { FCU_ST_READ_POS_V        , FCU_ST_READ_POS_V        , FCU_ST_ERROR        , true , false }
    , { FCU_ST_READ_POS_V        , FCU_ST_READ_BURN         , FCU_ST_ERROR        , true , false }
    , { FCU_ST_READ_BURN         , FCU_ST_READ_PRES         , FCU_ST_ERROR        , true , false }
    , { FCU_ST_READ_PRES         , FCU_ST_WRITE_POS_P       , FCU_ST_ERROR        , true , false }
    , { FCU_ST_WRITE_POS_P       , FCU_ST_WRITE_POS_V       , FCU_ST_ERROR        , true , false }
    , { FCU_ST_WRITE_POS_V       , FCU_ST_READ_POS_P        , FCU_ST_ERROR        , true , true  }
    , { FCU_ST_ERROR             , FCU_ST_OFFLINE           , FCU_ST_ERROR        , true , true  }
};

And here you have sample transition table for one the controlled devices — which is not the final transition table, because we still do not have the final model implemented, but we have the proper abstraction to implement it — with error handling and wait conditions. The model is very similar to functional guards, where there are some conditions checked, including the error condition, and then the state is processed passing to the next state in the next cycle. Seems that the original model for this kind of processing is present in this paper, and seems that it is fully meeting an IPO model.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Is really nice to have such type of graphical objects, because we can mix different types of animations, user interface behaviour and XTT commands, allowing almost any type of control to be created, including custom sub-graph objects, which can be linked with custom C or C++ programs. You just need to remember the $object and $local object references once your are instantiating the sub-graph object, mainly once they are pointing to logical plant nodes. Thanks to this flexibility, you can manage for example a simple PushButton, and make it work as SwitchButton, and make it work as ControlSwitch to toggle a digital input value.

Rotate Animation Configuration

For example we can use a PumpAnim2 sub-graph object and change its dynamics to be a rotating animation, rather than using its default, we just need to ensure that the dynamic is linked to the proper object — for example an analog signal on rotate animations — and configure each dynamic attribute correctly, as it was done using the rotate facto on the case above.

Toggle Button

You have many options to use in your HMI once you are building your ICS applications using ProView. You can read the Design Guide, the Ge Reference Manual, the SubGraph Reference Manual and the Operator Guide. The Operator Guide mainly contains the list of XTT commands, if you need to use any XTT script based commands to work the logical plant. This is really nice, because you can trigger — with a proper design — custom actions inside your rt_appl based application if you need make something complex, like distributing some task in a period, like moving a pump for 20 minutes and monitoring how filled is a tank, blocking its pumping action if it is required. Tasks like that are complex to implement using PLC programming, so a good C or C++ application can do that.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

To abstract plant nodes you have two kinds of object attributes that you can use. The $IoConnect and the $SigChanCon will bring you the proper plant node reference to control the signal related device. For example if you use the gdh_NameToObjid() GDH call on that attribute reference, you will get the reference to the card object, on the card object you can get the reference to the rack object and set the send request operation to true on the next scan cycle. For example the SendReq on the PnWriteReq Profibus device object or the SendOp attribute on the Modbus module object. So, you do not really need to know the path of the plant node or create a direct reference to the plant object that you need to control.

sts = gdh_NameToObjid(nm.c_str(), &oid);
if (EVEN(sts)) {
        errh_Error("prov_io::get_sendop_link() -> 1. Error Reading %s", nm.c_str());
        return nm;
}

ss.str("");
ss.clear();
ss << nm.substr(0, nm.find_last_of(".")) << ".IoConnect";
ionm = ss.str();

sigcon_a = reinterpret_cast<pwr_taddress>(&sigcon);
sts = gdh_GetObjectInfo(ionm.c_str(), sigcon_a, sizeof(sigcon));
if (EVEN(sts)) {
        errh_Error("prov_io::get_sendop_link() -> 2. Error Reading %s", ionm.c_str());
        return nm;
}

memset(par_dev_name, 0, SEND_OP_BUFF_SZ);
sts = gdh_AttrrefToName(&sigcon, par_dev_name, SEND_OP_BUFF_SZ, cdh_mName_volumeStrict);
if (EVEN(sts)) {
        errh_Error("prov_io::get_sendop_link() -> 3. Error Reading %s", ionm.c_str());
        return nm;
}

sts = gdh_NameToObjid(par_dev_name, &devid);
if (EVEN(sts)) {
        errh_Error("prov_io::get_sendop_link() -> 4. Error Reading %s", par_dev_name);
        return nm;
}

For example the code above displays an example on how to get the plant node using the $IoConnect attribute. Finally gets the Object ID of the connected plant node, allowing real control. Also, always remember to use the cdh_mName_volumeStrict flag to guarantee that the full volume path of the node will be used, allowing GDH to get the real path of the object without wasting time on finding mounted volumes and trying to make a difference between them.

With this approach, I am thinking to write some API helpers, something with the same behavior of the XML XPath standard, to allow finding nodes and node references without too much effort. I think that this will help a lot with open(), close() and scan() methods on the rt_appl class, allowing faster C and C++ development. This framework is really nice. I would like it to be more open, and I think that the authors would create a public repository and create a centralized development model like is seen in other platforms like FreeBSD.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

But we are not using modelling tools. The problem with modelling tools is the fact that you can make many mistakes and repair them easily, so you do not get very conscious and concentrated in the problem. Designing systems, and mainly complex systems like this one, which is an Industrial Control System requires to be very concentrated. I have two desk, one to study and solve problems and another with my workstation. If you do not have two desks, you can use your dining room, mainly when you are alone or search a comfortable place to think.

The current task in the project is to design a generalized state machine for one the entire nodes to be managed by the system. Which will be reporting statuses to its parent node. So, we have made it analogous to how are device drivers working on many real-time kernels. This means one cycle means one state transition.

System Design Desk

So, I have finally got the final design working after various drafts. With two main state transition streams, one for device initialization and another for device operation, that should work on all levels. This brings me a wider perspective, allowing me to think properly on each device class as C++ class and state machines and transitions as the proper C++ data structures. Also, how the context will be shared between devices, to notify each devices about its environment, allowing the optimal operation for each node. But I must recognize that it is looking simple, but that is great, because the device organization is not so simple as it looks on the diagram, so the simplicity goal have been reached.

The amount of devices to be managed and how are those devices streaming information in the system is the most complex structure in the system. Mainly because there are many devices to manage from the processing station and there is only one communication channel, which is not handled by an Ethernet based protocol, making harder to share that channel between devices, but we cannot use locks on this system, and we must use states rather than locks to allow more fluent behaviour. I really prefer to think using paper first and then pass some models to the workstation. The same applies to other systems, for example once I am doing β-substitutions replacing some functions with their closed form.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

The I/O layers are represented by an I/O Context, Agents, Racks, Cards and Channels. An I/O Context, is an I/O virtual space for I/O operations, where one or more Agents can work separately, but sharing some data, where you can use only on I/O Context from each Application. An Agent is a protocol layer, where each Agent can work as protocol handler, for example there is a Profibus and Modbus agents, where each Agent connects to one or more endpoints. Then comes one or more Racks where they are groups of Cards, and each card is grouping Channels.

ProView I/O Layers

Also the ProView system has a class hierarchy implementation, where each Agent, Rack and Card class is allowing sub-classing. This allows some custom devices to be configured easily over some other generic devices, for example those which are handled by Profibus and Modbus protocols. For example we have created our own classes to handle some very specific devices. So, ProView is great for building custom ICS systems. For example it has support for a Siemens ET200S-2AI device as Card and some other devices, but it is so customizable that you can build your own classes for any level.

Siemens ET200S

We currently have made some custom classes for our project, almost on all levels, this is really nice. You just need to read the documentation and have a deep knowledge on C and C++ programming languages. Java is supported, but it is not so good to be used in soft real-time systems, as we have configured our processing stations. Also we cannot use PLC programming due to the algorithm complexity, like priority queues and task scheduling. Also it has some very nice features, like mapping between nodes and plant nodes automatically using a single IoConnect attribute on the mapped component, to allow automatic connections between channels on device classes and signals in I/O components. There are many useful features around the I/O system in ProView that can make it a very good framework for ICS systems. For example I am very sure that you will love to work with GDH, because it is really one of the best abstraction layers that I have seen.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)