Modular Design in Robots for Under-Vehicle Inspection

1. Modular Design in Robots for Under-Vehicle Inspection Cheng Qian Imaging, Robotics, & Intelligent Systems Laboratory University of Tennessee Spring 2004

2. Outline • Modular concept: “Three primitives motivation ”, Modular architecture • Module: Structure, • Data storage, • Work thread, • Synchronization, • Priority • Implementation • A realistic system based on modular design. • 9 DLL Structure • 10 Sensing DLL Programming

3. Modularity Concept: Three primitives 1. A robot is composed of three basic primtives: Sensing, Mobility, Computing Sensing Mobility Mechanical & Power support Information Strategy Computing Robot ( CMU Millibot ) 2. Every primitive is further decomposed into sub-modules

4. Modularity Concept: Modular Architecture Mobility Sensing Power Control circuits ON Sensor … Sensor N Sensor1 Sensor2 Sensor platform Processor Virtual hardware control (Resampling&recalibration ) QOS Lighting Filter (noise filter & feature enhancement ) Motors Wheels Modality conversion Communicator Source Compression&Coding Monitoring sensor Channel Coding Protocol Computing Reactive inspection behaviors Data fusion Reactive navigation behaviors Display Data Interpretation Sense1 Sense1 Sense1 Strategy planning Sense1 Deliberative inspection Deliberative navigation Act1 Act1 Act1 Act1 Operator

5. Fresh frame of data *The three steps above are executed sequentially, hence, a Module = a Thread. *It is a real-time system, and the sensors are scanning frame by frame, all the time, hence Every round, the module only reads the fresh frame, process the fresh frame, and write the fresh results, and Size of buffer = Size of a frame. Module: Structure In under-vehicle system, the data is passed, one module after one module, from the start ( sensing ) to the end ( mobility ). So every module is modeled as : • Read the fresh data from the front modules into the Inbuffer. • Process the fresh data. • Write the Processed data into the Outbuffer, waiting for back modules to pick up. Module X InBuffer Process OutBuffer

6. ――Add buffers to backup the previous frames. The whole data ever read are kept in a temporary file Store: If N Previous frames is frequently needed New Buffer ( N* size of Frame) Move data in the BackupBufferN to Temp file Move data in BackupBuffer i to BackupBuffer i+1, i=1~N-1 Move data in the InBuffer to BackBuffer1 Read data into the Inbuffer Module X InBuffer ( current frame) BackupBuffer 1 ( Previous Frame 1) BackupBuffer 2 ( Previous Frame 2) : BackupBuffer N ( Previous Frame N) Retrieval: Search InBuffer, If failed, Search BackBuffer i, i=1~N If failed again, Search Temp file * Temp File Module: Data Storage The InBuffer and OutBuffer only store one frame of fresh data and a frame of fresh processed data. The size of the buffer in the every module is equal to the size of a frame , and every module is only processing the data of the fresh frame. But what if the previous frames are needed in the processing of the module?

7. i.e. Display Wait Display data in backup buffer New work (displaying) thread is created while the module is waiting for fresh data to come Display data in temp file Display data at low resolution Module: Work Threads Normally, a module only has one work thread. If it is a leaf module without downlinked modules, and without critical real time requirements, the processing in the module is first partially finished, if time permits, other processing is continued. 1. Initialize the resolution Res = min ( minimum resolution required, ½*Res of last round ) 2. Display data at Res 3. Store 2*Res data into backup buffer 1, ……, 2N Res data into backup buffer N, and store other data into temp file 4. While Waiting for data of next round, another threading of displaying the data in backup buffer and temp file sequentially is created, and is to be terminated by the termination of waiting

8. 1. Done-Event from a uplinked modulei: →Reading data from that module i into inbuffer i is activated ModuleiM Modulei1 …… → Then ACK is sentto the uplinked module i 3. The readings from all the uplinked modules are finished ….. →Then resources are caught and Process is started →Then resources are freed and check wait for ACK from downlinked module. Wait 5. ACK from all downlinked modules → Writing the processed data into Outbuffer is activated → Then Done-Event is sent to all the downlinked modules and wait for Done-Event from uplinked modules to start a new round. Module: Synchronization Activated by Event, and Terminated by Event 2. The reading from module i is finished Module X InBuffer 1 Wait InBuffer M Wait ….. … Process…. 4. Process is finished OutBuffer 6. Writing is finished Done-Event ACK Data

9. Suspended 1. If module Mi causes its back module suspended, —— priority of Mi increases by Ei:Set of moduleMk suspended due toMi, Tk: Suspended Time ofMk Pk : Current priority ofMk , Pi : Current priority ofMi MII Too slow MI Mi Suspended by MI, MII Mk 2. If Mi is suspended, —— priority of Mi decreases by Ti: Suspended Time ofMi, Pi : Current priority ofMi Fi:Set of module MI suspendsMi, PI : Current priority ofMI Suspended due to Mi Module: Priority Every module, as a single thread, is executed independently, and to make full use of the computation capability of the computer, the Priority is scheduled as: Fixed, i.e, The priorities of the reactive behavior and data acquisition, are always high. Flexible: Updated when suspending happens

10. Definition of Attributes & Behavior CLASS CThreadApp MFC Activation THREAD (Module) Void Module X :: Run { Loop: Wait for Done-Events from uplinked modules,( and Create other work thread ) If Wait returns, then (terminate the work thread ) Read data into input buffer Send ACK back to Uplinked modules Catch Resource …. Process…… Free Resources Wait for ACK from back modules If Wait returns, then write data into downlinked buffer Set Done-Event Endloop } Module X: Public CThreadApp { Public: // interface attributes BYTE InBuffer [ SizeFrame ], BYTE OutBuffer [SizeFrame ] MODULE * UpLink [ ], * DownLink [ ] EVENT * DoneEvent, * ACK Protected: // Internal attributes BYTE BackupBuffer [ ][SizeFrame] FILE * TempFile HANDLE * Resource [ ] Private: // Behaviors virtual BOOL Init ( ); virtual VOID Run( ); } Module: Implementation • Modularity Object-Oriented Programming

11. = Laptop + Application = Sensor + Plug-in Driver ( DLL ) = Mobile Platform + Plug-in Driver (DLL) Sensing Computing Mobility A Realistic System Computing Sensing Mobility Application and Drivers are modular-based developed

12. DLL Structure Mobility DLL : Strategy Explanation & Parsing Strategy Vector Creator1 Strategy Vector Creator2 Strategy Vector CreatorN Communication Port 1 Communication Port 2 Communication Port N … Sensing DLL: Data Acquisition Serial Processing Data Formatting How to build three sensing DLL ? ModalitySensor Sensor working mode Data format Range Sick Scanner Continuous (Step up once) continuous points ( 3D Pos ) Texture Line CCD camera Setup before capture sequence of Line ( RGB ) Thermal Area thermal Setup before capture sequence of images (relative temperature)

13. Sensing DLL Diagram Raytheon_ThermalCamera SickScanner Adaptive_HLRCLineCCD Thermal_Raytheon.LIB Range_SickScanner.LIB Texture_HLRCLineCCD.LIB SDK Programming SDK Programming Parallel communication Serial communication Parallel communication Sensor Protocol Sensor Protocol Frame grabber Protocol Sensor Protocol Read FIFO … PAK1 PAK2 PAKN Read FIFO Frame grabber Protocol Setup … FRM1 FRM 2 FRM N Virtual Setup Virtual Setup Serial Proc (Color Calibration) Serial Proc (Noise filtering) Serial Proc (Temperature Normalization) Formatting Formatting Formatting Write FIFO Write FIFO Write FIFO … … PAK1 PAK2 PAKN Line1 Line 2 Line N … FRM1 FRM2 FRMN Range Data Calibration parameters Texture Data Calibration parameters Application Thermal Data Calibration parameters

14. Done-Event ACK Done-Event ACK Done-Event “Serial Communication “Programming Serial communication Scanner Protocol Write Command ( ) { ……. Wait (DataContinue) Move data to Wbuffer SetEvent( DataToIOPort ) } Read Commamd( ) { ……. Wait ( DataforPickup ) Move data from Rbuffer SetEvent( DataMoved ) } RS232 IOPort WriteThread Loop { Wait ( DataToIOPort ); WriteFile ( * Wbuffer ) If ( Wbuffer is empty and writing is finished ) SetEvent ( DataContinue ) } ReadThread Loop { Wait ( DatafromIOPort ) If ( Rbuffer is refillable ) ReadFile( * Rbuffer) If ( Rbuffer is full or reading is finished ) SetEvent ( DataforPickup ) }

15. Inverse Problem in Thermal Imaging Cheng Qian Imaging, Robotics, & Intelligent Systems Laboratory University of Tennessee Spring 2004

16. Outline • Inverse problem • PSF building • Inverse Results

17. unknown Inverse problem g : blurred image PSF: blurring operator fopt: restored image : Regularization parameter : Regularization norm Tikhonov Regularization: TV Regularization Lasso Regularization Favor smooth f Favor f of large variance Favor f of large variance, and ease the computation

18. Use this PSF to restore thermal images Experiment setting Average over 10 continuous capture and over 10 experiment settings Image of a point heat source Approximated by Gaussian function PSF building Find a image of a point heat source Small and isotopic

19. Inverse Results Tikhonov Regularization Original image TV Regularization Lasso Regularization

20. THE END Thanks