High Performance Cluster Computing with PHOTOMOD HPC Edition

1. 11th International Scientific and Technical Conference From Imagery to Map: Digital Photogrammetric Technologies High Performance Cluster Computing with PHOTOMOD HPC Edition Mikhail Drakin Head of Software Development Department Racurs September 2011, Tossa de Mar, Spain

2. In Brief • Distributed processing of remote sensing data – the key to performance and efficiency • Parallelizing trends in modern hardware • Distributed processing in “conventional” PHOTOMOD:pros and cons • PHOTOMOD HPC Edition for computing clusters • PHOTOMOD Conveyor – the solution for on-line satellite monitoring

3. RSD – Ideal for Parallel Processing Most operations on remote sensing data satisfy the conditions for effective parallelization – can be easily split into numerous tasks, which: • Are similar in nature and/or • Take source data bits independent from other tasks and/or • Produce data bits independent from output of other tasksand/or • Are performed automatically

4. RSD – Ideal for Parallel Processing Common operations, implemented in automatic distributed mode in PHOTOMOD: • Tie points measurement • DTM calculation (pickets and “dense” DEM) • Orthorectification • Mosaics including sequence of distributed task sets: • Cutlines creation • Tie points searching • Final mosaic building + a bunch of service operations (image format conversion, pan-sharpening, DEM reprojection, etc). Parallel processing helps saving precious time, reducing days of work to hours

5. Parallel Hardware Trends The same trend at all scales: performance gained by increasing number of identical computing units. This allows building applications scalable over a wide range • Increasing number of cores in off-the-shelf CPUs • Larger local networks • Supercomputers as clusters of tens to thousands similar nodes

6. “Conventional” PHOTOMOD Distributed processing first implemented in version 4.4. Each new version supports several new operations in distributed mode. • Suited for medium-scale parallelization: • Distribution of tasks within single workstation to effectively utilize allprocessor cores • Distribution of tasks between several workstations to employ LAN resources of a department • Ideology: “one server – several workstations”

7. “Conventional” PHOTOMOD Drawbacks of distributed processing in standard PHOTOMOD: • Targeted at interactive work (flexible, but multiple GUI options complicate automation) • Not optimized for centralizedadministering • Implies storing configurationdata, logs, etc. locally on the workstation Standard PHOTOMOD does not rely on consistent LAN structure and fixed hardware configuration, thus sacrificing some performance for flexibility.

8. Hardware Side Sophisticatedcomputingalgorithms Intensivedataexchange Hugestored datavolume CPU Net Storage x K units? x M Gbps? x N TB? Massive photogrammetric processing involves: Hardware configuration must be carefully planned and balanced to avoid bottlenecks

9. PHOTOMOD HPC Edition: Key Features PHOTOMOD HPC Edition – next level of distributed processing in PHOTOMOD • Ideology: “one workstation – multiple servers” • Oriented at coherent high-end hardware and software complex • Centralized administering, “lessmessage boxes” • No modifications on computing nodes – all installation, configuration, logs storage, etc organized on a dedicated server Customization for particular customer needs and computing environment is implied.

10. Applications Fully automatic or semi-automatic solutions based on PHOTOMOD HPC Edition can pay off in processing of any kind of digital imagery, including: • Conventional aerial survey and rapidly emerging UAV imaging projects, with constantly increasing productivity and positioning quality, which allows fully automatic processing, especially with properly marked ground control points. • Rapid creation of quality orthomaps from HR and VHR satellite images for online monitoring (emergency management in situation centers, reconnaissance, etc).

11. Counterparts Examples of counterpart products: • VisionMap A3 Ground Processing System • PCI GeoImaging Accelerator (GXL) • Spot Image Pixel Factory • Photogrammetric systems with distributed processing (LPS, Inpho, etc) “Counterparts” here is not exactly equal to “Competitors”: each high-end product has specific niche.

12. PHOTOMOD HPC Edition: Distinctions • Distributed processing subsystem is native for the platform andconstantly improved for specific applied needs • The system can be easily adopted to any hardware meetingbasic requirements, not tied to some specific set of devices. The simplest cluster, which can bring significant productivity benefits with PHOTOMOD HPC, is probably 4 computing nodes (Windows HPC Server) with carefully designed storage system (e.g. FreeBSD-based), starting from $30-40 thousand • A computing cluster with HPC Edition can be tightly integrated with “conventional” interactive PHOTOMOD workstations, taking all advantages and eliminating drawbacks mentioned earlier • The system is initially positioned as a platform for building customized solutions

13. Testing in Kazakhstan December 2010: PHOTOMOD HPC was successfully tested on KazNTU computing cluster • 10.9 TFLOPS • 128 computing nodes • NetApp storage system • Windows HPC Server OS Image courtesy of SmartCom • The experiment: • Orthomosaic creation • Output GSD 0.16 m • Area 1289 sq. km • DEM cell size 1.6 m • Breaklines 858 • Total output files 156 GB • Processing time: 3 hours

14. PHOTOMOD Conveyor Example of HPC Edition customization for specific application Collaborative project by Racurs and CTT Group (Russia) for heavy-duty mosaic production from VHR satellite data: hardware + software complex solution +

15. PHOTOMOD Conveyor Fully automatic workflow, including: • Creation of the processing project from source satellite products • Adjustment of the project (by RPC) • Orthorectification by existing DEM • Exclusion of images with undue cloud coverage • Building cutlines • Searching for tiepoints to eliminate residual discrepancies along seams • Radiometric balancing • Final stitching into seamless mosaic, saved into standard sheets set 1 000 000 sq. km per day of 1:25 000 mosaic from GeoEye, DG or Alos

16. Future • Distributed GPU computing in cluster environment • Extensive set of automatic functions for data processing and enhancement • Library of fully automatic workflows for UAV processing, DTM creation and filtering, etc. • Advanced control over scheduling for convenient sharing of a single cluster between several workgroups • Integration with GIS systems, databases and geoportals • Any modifications and customizations requested by customers

17. Thank you for attention!