A Generic Architecture for Large-Scale Distributed Simulations

1. A Generic Architecture for Large-Scale Distributed Simulations Stephen J. TURNER School of Computer Engineering, Nanyang Technological University, Singapore 639798 ASSJTurner@ntu.edu.sg

2. Overview • Parallel and Distributed Simulation • Example Application Areas • Research Issues • A Generic Architecture • Federating Parallel Simulators • HLA vs. Customized Protocol • Hierarchical Federations Architecture • Conclusions and Future Research

3. Parallel and Distributed Simulation • Parallel Discrete Event Simulation • Aims to reduce the execution time of large discrete event simulations. • The simulation model is partitioned into a number of Logical Processes (LPs) that are executed in parallel. • As each LP has its own event list, a synchronization protocol is required to preserve causality: • A conservative protocol strictly avoids the occurrence of any causality error. • An optimistic protocol detects and recovers from causality violations.

4. Parallel and Distributed Simulation • Distributed Simulation • Provides a way of linking simulation components (federates) of various types at possibly different locations to create a common virtual environment (federation). training federate: real-time execution constructive federate: time-stepped execution live component: real-time execution with hard deadlines constructive federate: event driven execution

5. Parallel and Distributed Simulation • Causality Violation real world simulated world event message “fire” Simulator A (artillery unit) the observer should see the artillery unit fire before the target is destroyed. Simulator B (target) “target destroyed” Simulator C (observer) Time (wallclock time)

6. Example Application Areas • Battle Simulation • Links different types of forces at multiple physical locations to create a realistic and complex virtual world. • Multi-player Internet Games • Requires massive multi-player (~10,000) virtual world. • Air Traffic Control • Simulates airports and airspace sectors to provide “faster than real-time” simulation for alternative scenario analysis. • Supply Chain Management • Covers the planning and management of material and information flow, from the manufacturer through the distributors to the customer.

7. Inventory information Wafers Example Application Areas • Supply Chain Management • With the globalization of markets, factories and distribution centres in a supply chain simulation may be dispersed across many different countries. Shipment to Customers Wafer Fab 1 ICs Wafer Fab 2 Assembly & Test Wafer Fab 3

8. Research Issues • Fast Execution • The execution time of large-scale simulations may be unacceptably large due to the detail and complexity. • Fast simulation is required for “what if” and alternative scenario analysis. • Reuse and Interoperability • Large-scale simulations are constructed by linking together existing simulation models to form a simulation federation. • These component models may have been implemented using different languages or packages and developed for different hardware platforms.

9. Research Issues • Geographical Distribution • A large-scale simulation may involve linking a number of simulation components that are geographically distributed. • Scalability • As the number of simulation components and the size of the network increases, the run-time system should be able to handle the communications effectively. • Data Security • A group of simulation components may need to share some sensitive information with each other while hiding that data from other simulation components in the federation.

10. A Generic Architecture • Generic Architecture • A generic architecture for large-scale distributed simulation is being developed to investigate these research issues. • As these research issues are common to many application areas, the architecture is not restricted to any particular application. • The research issues are addressed by: • Federating Parallel Simulators • High Level Architecture • Hierarchical Federations Architecture

11. Federating Parallel Simulators • Hybrid Distributed/Parallel Simulation • This is a distributed simulation architecture where one or more simulation components (federates) is itself partitioned into LPs which are executed in parallel. • Addresses research issue of fast execution. workstation Our distributed supply chain simulation can be speeded up by executing the Assembly and Test facility as a parallel federate workstation workstation multiprocessor multiprocessor

12. HLA vs. Customized Protocol • Option 1: High Level Architecture (HLA) for Simulation • HLA is designed to support reuse and interoperability of simulation models through its rules, interface specification and object model template.

13. HLA vs. Customized Protocol • Features of High Level Architecture • Each federate has a simulation object model (SOM) that defines the data it is willing to share with other federates. • The federation (set of federates) has a common federation object model (FOM). • With its capability defined by its SOM, a federate may be reused in different federations. • HLA is designed to support distributed simulations linking the federates of a federation over a LAN or the Internet. • Time Management can be used to ensure the correct ordering of events. • HLA is an IEEE (1516) and OMG standard.

14. HLA vs. Customized Protocol • Option 2:Customized Distributed Simulation Protocol • As each federate has its own simulation time, a synchronization protocol is required to preserve causality. • A customized distributed simulation protocol can be developed based on an existing protocol such as the conservative “null message” approach. • A standard message passing library such as MPI can be used for communication over a LAN or Internet. • Encapsulation of information within the federate can be achieved by specifying the interactions and data that can be sent as external events.

15. HLA vs. Customized Protocol

16. Simulation over the Internet • Simulation using HLA-RTI between Singapore and UK • Three Sun workstations at NTU + SGI & Sun at Oxford.

17. Hierarchical Federations Architecture • Cluster Based Architecture • In many distributed simulation applications, the individual federates are found to be organized into groups. • Communication traffic within a group is generally higher than that between groups, due to closer physical and logical proximity. • Related federates can therefore be grouped into clusters, where each cluster is supported by a high-speed communication link. • Each cluster has its own RTI session, with an application gateway connecting it to other clusters.

18. Hierarchical Federations Architecture

19. Scalability • Gateway Data Filtering • The transmission of irrelevant data between clusters can be avoided since data filtering algorithms can be implemented efficiently at the individual gateways. • Gateway Packet Bundling • Packet bundling techniques can be implemented at the gateways to reduce the bandwidth requirements. • Time Management • Hierarchical federations can provide more efficient time management as the federates are more loosely synchronized and lookahead restrictions may be relaxed.

20. Reuse and Interoperability • Heterogeneous Federations • A cluster based approach can support heterogeneous federations, each with its own FOM: • Different federations/clusters may have FOMS at different levels of resolution. • Heterogeneous FOMS allow interoperability with legacy simulations, where it is infeasible to develop new FOMS. • Heterogeneous RTIs • Different RTI implementations may be used for different clusters within the hierarchy: • A single RTI might not support all the hardware platforms used. • Some clusters may benefit from specialized RTIs.

21. Data Security • Information Hiding • A gateway can provide information hiding, by filtering out sensitive data that should not be transmitted to other clusters. • While the HLA allows data to be encapsulated within the SOM of a single federate, it does not provide a mechanism whereby a subset of federates may share information. • Hierarchical federations allow a group of federates within a cluster to share sensitive information without making this visible to federates outside the group. • For information hiding, a hybrid gateway/proxy architecture has been developed.

22. Conclusions • Federating Parallel Simulators • Can provide a solution to the requirement of fast execution. • High Level Architecture • Facilitates reuse and interoperability of component models and supports geographically distributed simulations. • Hierarchical Federations Architecture • Improves the scalability of distributed simulations through bandwidth reduction and improved time management. • Increases the reusability and interoperability through heterogeneous federations and RTIs. • Provides data security through information hiding.

23. Future Research • Consistency • Maintaining a consistent view of the virtual world. • Characterization of inconsistency and the development of techniques for latency hiding. • Time Management • Efficient simulation even with zero or very small lookahead. • Development of alternative mechanisms, e.g. causal ordering. • Reuse and Interoperability • Tools to support reuse and interoperability at the semantic level. • Automatic generation of gateways in hierarchical federations. • Verification and validation of hierarchical federations.

24. The End Thank You! HLA-RTI Internet Questions? Further Information htttp://www.ntu.edu.sg/home/ASSJTurner