We are currently using Hierarchical State Machines, and once a state requires channel usage, the state machine stops its executions, runs the I/O calls io_read and io_write and continues its execution on the next scan cycle. Device synchronization is using single boolean variables holding read/write states as semaphores and reviewed on each task request. The entire framework is designed as it is running a PLC program, so you have a scan → read → write cycle. All state machines are pausing its execution on channel requests because they must not use the communication channel concurrently and we cannot use a locking mechanism.

We have logical composite modules to hold the device structure, where a device uses the same physical levels as rack, card and channel, but with its logical level as rack, card and signal. This allows a better organization and binding of devices along the project structure. Modularity is important, so we are using device classes to hold each device structure. Channels remain untouched, and we are writing and reading data exclusively from signals, despite we know some calls to access channels directly.

The device organization is pretty clear. Since we are using ModBus, we organize signal groups according to register groups in the device for each register block that we want to read or write. Even more for coil registers, where the are almost organized in groups, thanks to the class BaseComponent, which allows that kind of abstraction, rather than using ModBus modules and cards directly.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

On this case, I will iterate over the list of units on the scheduler — where each unit is a composite device — it will assign a new work state on each device, calling the run method on the device object. The run method will launch the child state machine of the required action, but the parent state machine only has few states storing the device status, so the processing scheduler knows which event should be triggered on each scan cycle. The required state transitions on the parent state machine, which are informational rather than hardware related, will be useful handling nested event handling.

Each cycle will determine whether read or write actions should be triggered, even if they are manual or automatic operations on each device, and only then the program will be able to call read and write once they are required. There are some actions not using the communication channel, so I can avoid calling read and write, enhancing the application performance a little, but still I will need to check the device state to check for some automatic reads, required to display the device state and ModBus holding registers.

You can find several models of hierarchical state machines, but I prefer to write one using events bound to class methods, where each class method drives an action on the device. This approach is simpler than others using signals and similar event handling methods, but I need to keep the source code as simple as possible, because increasing the complexity should increase the development time. Finally, the unit class will be only a gateway to the each hardware device, where each state will be triggered by the scheduler. Before this model, I was using a priority queue using messages, very similar to message queues, but is required on this system, to have direct access to the device state, and that message queue is not so synchronous as it is required, where a direct feedback from the device should be placed on each state transition.

Now I have finished the scheduler and the state machine processor, I just need to complete the integration between each component and I will deliver the new device management application. Also, the HMI has changed a little, and the new model allows some direct actions like resetting the device and its state.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

The state machine model includes on each transition a basic model defined as { s’, s”, g, f, h }, with s’ as the current state, s” the next state, g a boolean function evaluating the environment, f as guard boolean function preventing errors and h as action function. Where g is similar to f, but it covers the feasibility of the state change, because it evaluates if the device state (offline, online + automatic, online + manual), and f evaluates if the request is feasible according to the current composite device state (pump + valve + manometer), where each operation request was written as follows.

struct fcu_request_s {
    bool     channel;
    bool     done;
    bool     nobreak;
    int      errors;
    int      cycle;
    fcu_io  *device;
    pwr_tUInt16          operation;
    fcm_task_prio_t      priority;
    struct timespec      start_t;
    struct timespec      current_t;
    struct timespec      end_t;
    fcu_state_machine_t  *machine;
};

Where every boolean member is evaluated using the boolean function g, integer members are evaluated using the boolean function f, and if both function are not blocking the action, the action h is executed.

HMI (the image was modified to be shown).

Finally the state machine is mutable, because there are dynamic states which are triggered by boolean functions, and every transition is not strict, they are only a behaviour template rather than strict states to be used on each transition. From the processing node, each HMI signal is checked against the boolean functions, and each state transition is strictly evaluated before executing the action. This is more representative than single state transitions from the original state machine model, allowing better control. Where we are using dynamic programming to hold state transition tables and previously evaluated boolean functions which are common to the composite device state.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Now just remains operation details, such as state machine definitions for each operation that is required for the machinery operations that system will handle. The system has two main kinds of operations, automatic operations and manual operations, where manual operations have a higher priority than automatic ones, also automatic operations can be interrupted by manual operations, so the state machine that is running an automatic operation can be interrupted by any manual operation. Each state transition is monitored checking the system state, looking if a requested a manual operation should replace the current automatic operation, or the automatic operation should be interrupted by another automatic operation with a higher priority.

The algorithm — now that is implemented and not placed as diagram on the whiteboard — looks much simpler, and the basic algorithm looks as follows.

req = dev_get_request(MAIN_CONFIG, NUM_DEVICES);
if (req != NULL) {
    queue_task(req);
}

while (!channel && (req = check_queue())) {
    if (req->done) {
        break;
    }

    req->device->run(req);
    if (!req->channel) {
        continue;
    }

    sts = io_write(io_ctx);
    if (EVEN(sts)) {
        errh_Fatal("ra_fcm: io_write error, %m", sts);
        fcm_io_alarm(req->device, FCM_IO_WRITE);
    }

    sts = io_read(io_ctx);
    if (EVEN(sts)) {
        errh_Fatal("ra_fcm: io_read error, %m", sts);
        fcm_io_alarm(req->device, FCM_IO_READ);
    }

    if (req->nobreak) {
        channel = false;
    } else {
        channel = true;
    }
}

The dev_get_request method checks if there is a pending request, automatic or manual, if there is a pending request, it is being queued, and immediately the process starts processing the queued items, where they can be finished in one step or not, and the device runs the state machine referenced by the request, if the state machine transition uses the I/O channel, the ProView methods io_write and io_read are called, then, if the transition requests a no-break operation to make a continuous usage of the communication channel, it keeps running the state machine. Otherwise the state machine breaks its operation and seeks another operation request in the queue. The check_queue method seeks the proper prioritized request on the queue, where its priorities are changing inside the state machine on the run method.

But the state machine runner — placed on the device as run method — as an abstraction of the managed device by the priority queue, where there are several devices to manage, schedules their tasks using a simple algorithm. The real complexity is placed on the operation of each device, and the algorithm itself is an abstraction of the generalized state machine that we are running.

if (dev_state == DEV_ST_INACTIVE) {
    dev_state = DEV_ST_ONLINE;
}

request->nobreak = false;
machine->channel = false;
request->channel = false;

for (c = 0; c < machine->num_states; c++) {
    tr = &(machine->states);

    if (tr->curr_st != dev_state) {
        continue;
    }

    if (!tr->guard(tr, machine, request, this)) {
        break;
    }

    if (!operate(tr, machine, request)) {
        break;
    }

    errh_Info(st_msg_fmt, fcu_state, tr->next_st, tr->curr_st);
    dev_state = tr->next_st;

    if (tr->next_st == machine->final_state) {
        request->done = true;
        break;
    }

    if (tr->channel) {
        chn = true;
        request->nobreak = tr->nobreak;
        machine->channel = true;
        request->channel = true;
        break;
    }

}

return chn;

The guard checks environmental variables, and if there are good environmental conditions to operate, then it starts operating the device according to the current transition, that holds the action as enum type, with a simple switch statement to check the action that should be taken on the state transition. On a successful operation and the proper prioritization, the queue is reordered outside the state machine runner, allowing queued operations to trigger alarms if they are not finished, among other similar error checking and environment checking routines. So, the system keeps working, alerting the user through the HMI if there is any condition that never meets the required operation. ProView has several ways to alert users about the system conditions, and we are using almost everything that we can. I think that this system is very well structured and now running dynamic priorities is even more stable than using plain task requests, mainly due to environmental variables. This scheduler looks preemptive, but it is not.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

During the state machine execution, each state transition has a guard ensuring that the step that will take place on the transition is guaranteed to be executed depending on the environment, if not the step is reserved to the next PLC scan cycle — do you remember that ProView operates C and C++ applications like PLC programs?. The same guard ensure that the operation on the pump will take the required minutes to deactivate the pump, meeting the task calendar. Once the pump is active, its task priority — which is hold on the request data structure — changes its priority, and using the top() and pop() methods of the STL std::priority_queue, is extracted from the queue, the priority is changed, and it is pushed again on the queue using the push() method.

All requests that are using the communication channel have the higher priority, because we have only one communication channel, so we must ensure some concurrency on the channel usage, and we call the io_read() and io_write() ProView functions only once it is required, ensuring that the communication channel is used by one device at time. I think that this is the biggest problem of the system, because the communication channel is very small, and shared between several devices, and it must be used concurrently. But locks are not an option. If the channel blocks due to its usage, the system should continue its work and must keep its system review — reviewing channel status and signal status, and reviewing the error status for several devices.

Since several requests are not using the communication channel, the queue never gets saturated, and every transition is executed. It only blocks on communication channel usage, but it calls the io_read() and io_write() ProView functions rather than using some kind of lock, like semaphore or similar one. Also, errors are not allowed, the code should not have any programming error, and it is being traced by strace(1) and valgrind(1). Also, we are thinking to check the model using spin(1), I have started some Promela code related to the processing model, at least it looks well for now. It will probably get more complexity on the future, but the model looks well, and the code seems to be really good.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

On a recent project, the project manager has requested to me to code a state machine. Its initial design is not generalized. Along two design sessions and some sample code and constructs I think that I have finally made an abstract state machine, with various reductions, based on the model designed by the project manager on the design sessions, structured mostly as automaton, with very simple state transitions, but I have added error and wait conditions, allowing more friendly error processing and wait events. Mainly because as ProView application it has several similarities to PLC programs, where each cycle can operate one time due to the protocol restrictions, but also it allows multiple operations in one cycle because it can send larger packages with multiple requests.

Generalized State Machine

On each cycle, once the I/O layer is initialized, the device error state is checked, on failure it reports errors using ProView alarms, waits the next cycle and then operates again checking the device state. If no error is detected, it operates immediately once it can do it according to its cycle. And the state machine above is a generalized or abstract state machine, considering CST and OST as checking and operating states, where it is has a normalized transition between both states with the proper error handling states.

typedef struct fcu_sm_trans_s fcu_sm_trans_t;

struct fcu_sm_trans_s {
    fcu_op_state cst;
    fcu_op_state nst;
    fcu_op_state fst;
    bool alarm;
    bool wait;
};

/* pump and valve transtion table */
fcu_sm_trans_t AUTO_FCU_TRANSITION[] = {
    { FCU_ST_OFFLINE             , FCU_ST_DISCONNECTED      , FCU_ST_OFFLINE      , false, wait  }
    , { FCU_ST_DISCONNECTED      , FCU_ST_COPY_READ_POS_P   , FCU_ST_ERROR        , true , false }
    , { FCU_ST_COPY_READ_POS_P   , FCU_ST_COPY_WRITE_POS_P  , FCU_ST_ERROR        , true , false }
    , { FCU_ST_COPY_WRITE_POS_P  , FCU_ST_COPY_READ_POS_V   , FCU_ST_ERROR        , true , false }
    , { FCU_ST_COPY_READ_POS_V   , FCU_ST_COPY_WRITE_POS_V  , FCU_ST_ERROR        , true , false }
    , { FCU_ST_COPY_WRITE_POS_V  , FCU_ST_READ_POS_P        , FCU_ST_ERROR        , true , false }
    , { FCU_ST_READ_POS_V        , FCU_ST_READ_POS_V        , FCU_ST_ERROR        , true , false }
    , { FCU_ST_READ_POS_V        , FCU_ST_READ_BURN         , FCU_ST_ERROR        , true , false }
    , { FCU_ST_READ_BURN         , FCU_ST_READ_PRES         , FCU_ST_ERROR        , true , false }
    , { FCU_ST_READ_PRES         , FCU_ST_WRITE_POS_P       , FCU_ST_ERROR        , true , false }
    , { FCU_ST_WRITE_POS_P       , FCU_ST_WRITE_POS_V       , FCU_ST_ERROR        , true , false }
    , { FCU_ST_WRITE_POS_V       , FCU_ST_READ_POS_P        , FCU_ST_ERROR        , true , true  }
    , { FCU_ST_ERROR             , FCU_ST_OFFLINE           , FCU_ST_ERROR        , true , true  }
};

And here you have sample transition table for one the controlled devices — which is not the final transition table, because we still do not have the final model implemented, but we have the proper abstraction to implement it — with error handling and wait conditions. The model is very similar to functional guards, where there are some conditions checked, including the error condition, and then the state is processed passing to the next state in the next cycle. Seems that the original model for this kind of processing is present in this paper, and seems that it is fully meeting an IPO model.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Is really nice to have such type of graphical objects, because we can mix different types of animations, user interface behaviour and XTT commands, allowing almost any type of control to be created, including custom sub-graph objects, which can be linked with custom C or C++ programs. You just need to remember the $object and $local object references once your are instantiating the sub-graph object, mainly once they are pointing to logical plant nodes. Thanks to this flexibility, you can manage for example a simple PushButton, and make it work as SwitchButton, and make it work as ControlSwitch to toggle a digital input value.

Rotate Animation Configuration

For example we can use a PumpAnim2 sub-graph object and change its dynamics to be a rotating animation, rather than using its default, we just need to ensure that the dynamic is linked to the proper object — for example an analog signal on rotate animations — and configure each dynamic attribute correctly, as it was done using the rotate facto on the case above.

Toggle Button

You have many options to use in your HMI once you are building your ICS applications using ProView. You can read the Design Guide, the Ge Reference Manual, the SubGraph Reference Manual and the Operator Guide. The Operator Guide mainly contains the list of XTT commands, if you need to use any XTT script based commands to work the logical plant. This is really nice, because you can trigger — with a proper design — custom actions inside your rt_appl based application if you need make something complex, like distributing some task in a period, like moving a pump for 20 minutes and monitoring how filled is a tank, blocking its pumping action if it is required. Tasks like that are complex to implement using PLC programming, so a good C or C++ application can do that.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

To abstract plant nodes you have two kinds of object attributes that you can use. The $IoConnect and the $SigChanCon will bring you the proper plant node reference to control the signal related device. For example if you use the gdh_NameToObjid() GDH call on that attribute reference, you will get the reference to the card object, on the card object you can get the reference to the rack object and set the send request operation to true on the next scan cycle. For example the SendReq on the PnWriteReq Profibus device object or the SendOp attribute on the Modbus module object. So, you do not really need to know the path of the plant node or create a direct reference to the plant object that you need to control.

sts = gdh_NameToObjid(nm.c_str(), &oid);
if (EVEN(sts)) {
        errh_Error("prov_io::get_sendop_link() -> 1. Error Reading %s", nm.c_str());
        return nm;
}

ss.str("");
ss.clear();
ss << nm.substr(0, nm.find_last_of(".")) << ".IoConnect";
ionm = ss.str();

sigcon_a = reinterpret_cast<pwr_taddress>(&sigcon);
sts = gdh_GetObjectInfo(ionm.c_str(), sigcon_a, sizeof(sigcon));
if (EVEN(sts)) {
        errh_Error("prov_io::get_sendop_link() -> 2. Error Reading %s", ionm.c_str());
        return nm;
}

memset(par_dev_name, 0, SEND_OP_BUFF_SZ);
sts = gdh_AttrrefToName(&sigcon, par_dev_name, SEND_OP_BUFF_SZ, cdh_mName_volumeStrict);
if (EVEN(sts)) {
        errh_Error("prov_io::get_sendop_link() -> 3. Error Reading %s", ionm.c_str());
        return nm;
}

sts = gdh_NameToObjid(par_dev_name, &devid);
if (EVEN(sts)) {
        errh_Error("prov_io::get_sendop_link() -> 4. Error Reading %s", par_dev_name);
        return nm;
}

For example the code above displays an example on how to get the plant node using the $IoConnect attribute. Finally gets the Object ID of the connected plant node, allowing real control. Also, always remember to use the cdh_mName_volumeStrict flag to guarantee that the full volume path of the node will be used, allowing GDH to get the real path of the object without wasting time on finding mounted volumes and trying to make a difference between them.

With this approach, I am thinking to write some API helpers, something with the same behavior of the XML XPath standard, to allow finding nodes and node references without too much effort. I think that this will help a lot with open(), close() and scan() methods on the rt_appl class, allowing faster C and C++ development. This framework is really nice. I would like it to be more open, and I think that the authors would create a public repository and create a centralized development model like is seen in other platforms like FreeBSD.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Is nice to have that kind of configurations, we have several classes for almost all layers and node types — both physical and logical nodes — allowing a higher level of abstraction and avoiding repeated configurations. We just need to setup the UnitId property on most objects — due to its Modbus base classes.

Device Classes

Also, as graphic component, we have configured some generic nodes — for logical nodes and I/O classes — where we have some nice interfaces as samples because the definitive HMI is not really defined yet. Still we are defining some algorithmic behavior of the controlling program, which is being developed using C and C++. Rather than defining an IoConnect reference attribute, you should create an object reference attribute and all sub-graph objects should be connected using $object as base reference, as it is being done using IoConnect object references.

Sample HMI

As this happens on the ProView side, the same happens on the C and C++ programming side. The code base has all device classes defined as they are ProView classes with their custom I/O methods and controlling logic. Also there are some signal transformations as it is being done by PLC programs but implemented in C, just to ensure that some signal objects have the proper representation as analog devices — like the percent needle.

So, finally is nice to work with this framework, it is very flexible, so the graphic object is being connected to the device node in the following picture.

Device Connection

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

The I/O layers are represented by an I/O Context, Agents, Racks, Cards and Channels. An I/O Context, is an I/O virtual space for I/O operations, where one or more Agents can work separately, but sharing some data, where you can use only on I/O Context from each Application. An Agent is a protocol layer, where each Agent can work as protocol handler, for example there is a Profibus and Modbus agents, where each Agent connects to one or more endpoints. Then comes one or more Racks where they are groups of Cards, and each card is grouping Channels.

ProView I/O Layers

Also the ProView system has a class hierarchy implementation, where each Agent, Rack and Card class is allowing sub-classing. This allows some custom devices to be configured easily over some other generic devices, for example those which are handled by Profibus and Modbus protocols. For example we have created our own classes to handle some very specific devices. So, ProView is great for building custom ICS systems. For example it has support for a Siemens ET200S-2AI device as Card and some other devices, but it is so customizable that you can build your own classes for any level.

Siemens ET200S

We currently have made some custom classes for our project, almost on all levels, this is really nice. You just need to read the documentation and have a deep knowledge on C and C++ programming languages. Java is supported, but it is not so good to be used in soft real-time systems, as we have configured our processing stations. Also we cannot use PLC programming due to the algorithm complexity, like priority queues and task scheduling. Also it has some very nice features, like mapping between nodes and plant nodes automatically using a single IoConnect attribute on the mapped component, to allow automatic connections between channels on device classes and signals in I/O components. There are many useful features around the I/O system in ProView that can make it a very good framework for ICS systems. For example I am very sure that you will love to work with GDH, because it is really one of the best abstraction layers that I have seen.

© Daniel Molina Wegener for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)