International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

1. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS” Tbilisi, Georgia 6-9 Jun 2013 Dynamic Applications on Complex Distributed Systems and Machine Learning algorithms • D. Metafas1, M. Rangoussi1, B. Meparishvili2, G. Goderdzishvili2 • (1) Department of Electronics Eng., TEI Piraeus, Greece • (2) Faculty of Informatics and Management Systems, Georgian Technical University, Georgia International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

2. Outline • Internet of Things (IoT) • IoT: Focus on Critical Infrastructures • SCADA and Wireless Sensors and Actuator Networks (WSANs) • Review of existing middleware for networked robots and Wireless Sensor Networks • The need of a WSAN middleware • WSAN middleware: Focus on autonomous decision-making • Machine Learning International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

3. Internet of Things • Internet of Things (IoT) • “the next step in ‘always on’ computing … promising a world of networked and interconnected devices.” – ITU 2005 • “Machine-to-Machine”, “smart systems”, “web of things”, “sensor web”, … • “The real (i.e., physical), virtual, and digital worlds are converging, thanks to an ever proliferation of connected (smart) sensors and objects, ubiquitous wireless networks, communications standards and the activities of humans themselves”– Economist 2010 International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

4. Internet of Things • The vision of IoT is changing from emerging “realizations”, via e.g., RFIDs and WSANs, to include Internet-connected objects, whereby “objects” will include any possible “physical” entity that could be classified according to size, mobility, power, connectivity, automation, physical/logical type, etc. • Internet-connected objects is a central part within International/European strategies/roadmaps.This change is characterized as “metamorphosis of objects” in EU Cluster of European Research Projects on the IoT. A key enabler/prerequisite for the proliferation of internet-connected objects is the widespread deployment of IPv6. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

5. IoT: Focus on Critical Infrastructures International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

6. IoT: Focus on Critical Infrastructures • Our main focus is Critical Infrastructures (CIs), i.e., those facilities that provide essential services for everyday life (such as energy, food, water, transport, communications, health). • Protecting Critical Infrastructures (CIs) against disruption of any kind is deemed vital to the national/international security, public health and safety, as well as economic prosperity. • PCS (Process Control Systems)/SCADA (Supervisory Control and Data Acquisition) systems are largely used for data acquisition and control over large and geographically distributed infrastructures; these systems are critical elements in every aspect of literally every CI (e.g., Oil/Gas, Electricity, Transportation). International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

7. Critical Infrastructures: SCADA systems • SCADA systems range from relatively simple networks that monitor environmental conditions of a given location to incredibly complex systems that monitor all the activity in a power plant or a municipal water system. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

8. SCADA and Wireless Sensors and Actuator Networks (WSANs) • SCADA systems are moving from expensive, complex, proprietary technologies to affordable, standards-based, open solutions. One of the promising (in cost and usage) technologies for use in SCADA systems includes IEEE 802.15.4-based Wireless Sensors and Actuator Networks (WSANs) such as ZigBee, WirelessHART and ISA100.11a. • but .. • Wireless systems are vulnerable to in terms of cyber attacks. • CIs (like oil and gas and power grid) are attractive target for cyber-attacks. • The use of WSANs in CIs results in a deviation from the WSAN assumption of densely deployed nodes. This is not the case when they are used for SCADA of CIs, and a non-communicating node can have much more significant effect than the standard WSAN scenario. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

9. WSAN middleware • Most current middleware solutions for WSAN systems fail to take into account two key aspects: robustness of the solution and support for heterogeneous configurations. This is especially important in environments where their complexity makes it absolutely necessary to have different types of techniques and monitoring equipment that needs to interact to project CIs. • For example, the case where oil pipelines are to be monitored require the cooperation and interaction of sensors, actuators and mobile devices (UAVs, robots, etc.) that interact in a seamless way to achieve a common goal. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

10. Dynamic Applications for CIs and the need of a high level middleware • Dynamic applications for CIs require complex distributed systems consisting of a number of embedded hardware and software components. • Since these components are heterogeneous, it is necessary to find ways to mask this heterogeneity and offer some innovative solutions to develop the applications. • In addition, advanced and efficient techniques for cooperation and collaboration among them are required to achieve the desired goals. Therefore, there must be new software services (middleware) that act as the glue to link everything together in an efficient manner, supporting concurrency-intensive operations, enhancing collaboration, and insuring efficiency and robustness. • How to get efficient communication links, design better ways to exchange and use information over these links and manage dynamic work environments are some of the common issues. • Proposed middleware approaches for specific technologies such as ad-hoc networks, wireless sensor networks and robotics are addressing a number of issues but clearly in the broader scope of Dynamic applications for CIs there is a need of a widely accepted high-level middleware addressing all open issues and meet the design and implementation of different challenges. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

11. Dynamic Applications for CIs and the need of a high level middleware (2) • Objective: To design a distributed middleware and communications framework that will “equip” objects/nodes with advanced functionalities and abilities to understand their environment for collaborative and autonomous decision-making. • To prescribe all the components of the distributed middleware, which shall enable cooperation, intelligence and autonomy among heterogeneous, spatially distributed nodes/objects, while providing self-X capabilities (self-configuration, self-healing, self-management, self-optimization). • To develop a mission-to-system translation methodology allowing for a systematic decomposition of high-level mission requirements, to low-level system primitives and parameters. The methodology will allow for reusability across various application areas, potentially reducing the effort and the development time. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

12. Reviewing the middleware for Networked Robotsand WSNs • Networked Robots middleware: • Concerning the communication model: MIRO, PEIS Kernel, Orca (CORBA but also non standard models) • Concerning the Interoperability of the software components: Marie, Middle Layer • Concerning the Automatic Discovery, Configuration and Integration support: PEIS Kernel, UPnP • Concerning the Specific/Expandable Services: Sensory Data Processing Middleware, the AWARE data-centric model, MIRO, Marie • Concerning the communication performance issues (reliability, availability or QoS): RSCA (Robot Software Communication Architecture) • Concerning the support of Low Resources devices: PEIS Kernel • WSN middleware: • Moteview, ScatterViewer, Hourglass, SenseWeb, jWebDust, GSN, TinyDB, Hood, SNACK, Kairos, … International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

13. WSAN middleware: Focus on autonomous decision-making • Given the model of the WSAN system .. • Train the system using different scenarios, conclude to AI brains and provide them to the AI agents. • An innovative idea we are exploring is the use of Q-Learning algorithms and model the Q as a Model Tree.

14. Machine Learning • Classification: • Given • Training data • Learn • A model for making a single prediction or decision xnew Training Data (x1, y1) (x2, y2) (x3, y3) … Classification Algorithm Model ynew Source: Lisa Torrey, Univ. of Wisconsin – Madison (2009) International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

15. Machine Learning • Procedural Learning: • Learning how to act to accomplish goals • Given • Environment that contains rewards • Learn • A policy for acting • Important differences from classification • You don’t get examples of correct answers • You have to try things in order to learn International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

16. Machine Learning • We don’t know all the effects of our decisions so, the procedural learning drive us to the Reinforcement Learning (RL) (acting and observing the environment - or the model of it in our case - ). • What kind of RL ? Model-free: learn a policy without a strict model. Use Temporal difference methods (TD). • What kind of Model-free RL ? Q-Learning International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

17. Q-Learning Current state: s Current action: a Transition function: δ(s, a) = sʹ Reward function: r(s, a) Є R Policy π(s) = a Q(s, a) ≈ value of taking action a from state s • The basic update equation • With a discount factor to give later rewards less impact • With a learning rate for non-deterministic worlds International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

18. Q(s, a) The Q (state, action) table is huge .. What we could do ? Approximate the Q with a non-linear function. What kind of function ? A Neural Net or a Model Tree. Let’s test the Model Tree. If its application is successful we could have a much more comprehensive and flexible set of rules for our AI agents. Model trees are similar to decision trees, but instead of predicting discrete classes, they predict real valued functions. Model trees recursively partition the feature-space into regions, by choosing one attribute as a split criterion at every level of the tree. In the leaves of the trees, regression models are trained to predict a numerical value from the features of an instance. What algorithm ? A variant of the Quinlan's M5 algorithm (Quinlan 1992). This algorithm first grows a tree by selecting splitting criteria so as to minimize the target variable's variance, and then builds linear regression models in the nodes. Finally the tree is pruned, which means that sub-trees are replaced with leaves, as long as the prediction error of the resulting tree does not exceed a certain threshold. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

19. Decision Trees (ID3) International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

20. M5 algorithm • Example International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

21. The problems .. • The main disadvantage in using model trees for value function approximation is that there is currently no algorithm for online training. To refine the predictions of a model tree, we must therefore rebuild it from scratch, using not only the new training examples, but also the old ones, that were used for the previous model trees, i.e. anew model tree approximation of the Q-functions (more than one because of the hierarchical model trees) has to be built from the whole updated training set, and these trees are then used to “play” the next set of training data. • Even though there is not experience with using model trees in reinforcement learning we believe that it is a promising approach. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

22. Next steps .. • To prescribe the components of the distributed middleware, which shall enable cooperation, intelligence and autonomy among heterogeneous, spatially distributed nodes/objects. • To develop a mission-to-system translation methodology allowing for a systematic decomposition of high-level mission requirements, to low-level system primitives and parameters. • If the experiments of the proposed approach are successful, the methodology could be based on hierarchical Model Trees generated AI agents, trained for a number of scenarios. International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”

23. Thank you! International Conference “NANOSENSORY SYSTEMS AND QUANTUM SENSORY MATERIALS”