Introduction to HIPE and the HSA Applet

1. Introduction to HIPE and the HSA Applet David Shupe, NHSC on behalf of the Herschel Science Ground Segment Consortiumhttp://herschel.esac.esa.int/DpHipeContributors.shtml

2. This presentation is intended asan overview and “quick start” • It sets the stage for hands-on work • Some features will not be covered • Read also the “HIPE Owners Manual” • Included with your installation

3. HIPE is designed to easily handleyour Herschel data It handles Herschel data types It includes routines to go from raw data to publishable results It places the official pipeline software on your desktop It is modern and actively developed

4. Introduction to HIPE • Key Data Concepts • A Visual Tour of the HIPE Interface • Help and Documentation • A First Script – Unpacking the Workshop Data

5. Introduction to HIPE • Key Data Concepts • Objects and Data Products • Contexts • Memory management • A Visual Tour of the HIPE Interface • Help and Documentation • A First Script – Unpacking the Workshop Data

6. Wrapping data as objectslets HIPE do more for you • Sensible defaults for “double-click” • Usually a viewer • A module or function works on any object of a given class • Use the same methods or interfaces • The system can suggest the right task or option • Less for you to remember

7. Contexts allow data to be organizedin tree structures • Members can be “named” or simply “numbered” • Data items are loaded only when they are accessed(“lazy loading”) • Access is the same,wherever the data are

8. The Observation Context consolidatesall the different types of data ObservationContext Auxiliary logObsContext Calibration Level0(Raw Data) Points to everything that was usedfor processing Easy to choose what to look ator ignore

9. Computer memory is automaticallymanaged for you • The system allocates memory whenever a new object is created • The system runs “Garbage Collection” as needed • Reclaims memory • Bar in lower right cornershows status

10. Introduction to HIPE • Key Data Concepts • A Visual Tour of the HIPE Interface • Views and Perspectives • The Welcome! Perspective • The Work Bench Perspective • The Product Browser Perspective • The Herschel Science Archive Perspective & Applet • Help and Documentation • A First Script – Unpacking the Workshop Data

11. Views are windows or regionswith specialized functions Views can be resized or minimized Views can be endlessly rearranged Views don’t shut down when closed Views are accessible by menu

12. A Perspective is a specific collectionof view windows Welcome! Product Browser HSA Work Bench Reset Perspectives are pre-defined Your re-arrangement of views is “sticky” You can reset the arrangement to the default

13. The Welcome! perspective is a map to the major parts of HIPE

14. Most of the action takes placein the Work Bench perspective

15. The Variables view provideseasy access to each data item Delete all variables Delete selected variable Double-click for default (usually a viewer) Right-click or control-click for a quick list of options

16. The Outline view showsdetails and structure of a data item Red = not loaded yet Black = loaded in session For any item: Double-click for default (usually a viewer) Right-click or control-click for a quick list of options

17. Jython commands are executedin the Console window Log window contains system messages History tab keeps a record of your commands

18. The Editor view containsscripts, viewers, and task interfaces Script controls (see tooltips) Script window

19. The Observation Viewer breaks outall the pieces of your observations

20. The Tasks view enables quick startupof applicable modules simpleFitsWriter task • Double-click to launch • Drag-and-drop variables into parameter slots • The “Applicable” tab shows all the available tasks for a selected variable

21. The Navigator view enablesbrowsing of your filesystem • Double-click to load a script or a FITS file • Refresh • Show Hidden • Create “user area”

22. The Product Browser perspectiveprovides powerful search and retrieval

23. Interact directly with the archiveusing the HSA perspective Log in using your Herschel credentials Start the HSA Applet

24. Query the archive by obsid,target name, instrument, public status If OBSID is known… Set to “Public” to browsereleased data Search by name Specify instrument

25. The HSA Applet can send datadirectly to your HIPE session Select “Send to External Application” then “All”

26. For multiple downloads,use “Retrieve” to receive tarfiles Select “Retrieve” then “All”

27. The Shopping Basket will collectseveral observations …or reprocess with last released version You can retrieve thecontents at one time…

28. Introduction to HIPE • Key Data Concepts • A Visual Tour of the HIPE Interface • Help and Documentation • Starting the Help system • User Guides, Tutorials and How-Tos • Detailed Developer Documentation • A First Script – Unpacking the Workshop Data

29. The Help and Documentationare accessed in your web browser • Start the help system by 1 of 3 ways: • Menu “Help” -> Help Contents • “show_help” in the app directory of HIPE • Right-click on variable • Help in URM (Users Ref. Manual) • Help in DRM (Developers Ref. Manual) • Only one help system can be running

30. The Help system includesuser guides, tutorials and how-tos

31. The Help system includes allthe detailed developer documentation

32. Introduction to HIPE • Key Data Concepts • A Visual Tour of the HIPE Interface • Help and Documentation • A First Script – Unpacking the Workshop Data • Quick introduction to scripting • Where your local data are stored • Edit and run the UnpackWorkshopData.py script

33. Variables are simply namesthat point to data items No ‘type’ or declaration needed Assignment creates the variable:a = 1b = 2 Strings can use single or double quotes:c = “hello world”e = ‘hi there’ Reference: Scripting and Data Mining, Sec 1.2

34. Comments, continuations, and printinghave a simple syntax The comment character is the pound sign# this is a comment The continuation character is the backslashx = a + b + \ c * d * e A formatted string uses C-style format characters and the percent signprint “integer = %d, real = %f” %(j,x)

35. if-then-else blocksare denoted by indentation Reference: Scripting and Data Mining, Sec 1.8.1 Syntax:if condition1:block1elif condition2:block2else:block3 Notice that blocks are denoted by indentation only Example:if (0 <= x <= 10): y = y - 10print “Value is in range [0,10]”elif (10 < x < 20): print “Value is in range [10,20]”else: print “Value not in range [0,20]”

36. for loops are not just for integers Reference: Scripting and Data Mining, Sec. 1.8.2 Syntax of a for loop:for var in sequence:block The sequence can be any list, array, etc. Examples:for petin [“cat”, “dog”, “bird”]: print petfor i in range(10): # [0, 1, …, 9] print i # prints numbers 0-9 Therangefunction returns a list of integers. In general range(start,end,stepsize)where start defaults to 0 and stepsize to 1.print range(5)# [0, 1, 2, 3, 4]

37. Your “local pools” are stored in the “lstore” directory • Defaults: • Macs /Users/[userid]/.hcss/lstore • Linux /home/[userid]/.hcss/lstore • Windows C:\Documents and Settings\[userid]\.hcss\lstore • You can change this using “Edit” and then “Preferences” then “Local Store”

38. The UnpackWorkshopData.py scriptwill unpack the workshop data • Get the script from agenda webpage • Follow link at top of PACS section • Unpacks all .tgz files in a directory • …which you downloaded previously • These are special compressed pools • Not the format of HSA tarballs

39. Edit Line 6 of UnpackWorkshopData.pyand run script to unpack …then put cursor in between “”on Line 6, and ctrl-V or cmd-V to paste HIPE tip: Select data directory, then copy with ctrl-C or cmd-C… Navigator Editor Console output:

40. Backup: Alternate ways of unpacking .tgz datasets to your lstore Courtesy of Phil Appleton

41. Data downloads DOWNLOADS OF gzipped-tarred (tgz) example datasets should already have been done before coming to the workshop If you forgot its https://nhscsci.ipac.caltech.edu/sc/index.php/Workshops/Feb2011WorkshopDownloads for the full instructions and https://nhscsci.ipac.caltech.edu/workshop/Feb_2011_NHSC_DPWorkShop_Repository/ DataDownload/ for the data repository

42. Data Pools are stored in a lstore directory on your computer: It lives in the .hcss directory Macs /Users/[userid]/.hcss/lstore Linux /home/[userid]/.hcss/lstore windows GOTO folder C:\Documents and Settings\[userid]\.hcss\lstore (to make it : then pulldown file>>new>>folder lstore)

43. OLD WAY OF COPYING FROMCOMMAND LINE (Script given earlier is best)

44. Safest to move the pool to the .hcss/lstore BEFORE UNCOMPRESSING .../.hcss/lstore Data (pool) (already copied from WebDav) DO NOT UNTAR or UNZIP YET 1342184579.tgz move to lstore without uncompressing yet Once in lstore you can untar or unzip .../.hcss/lstore/1342184579 THIS IS A POOL

45. On the Mac the .hcss directory is "hidden" so best do move with a command line instruction For UNIX users (Mac or LINUX) * move the dataset to the HIPE local store location >mv dataset.gz ~/.hcss/lstore THEN UNCOMPRESS For WINDOWS * the easiest thing is to drag and drop the subdirectory dataset.gz or zip into the location of the .hcss/lstore directory located for example in C:\Documents and Setting\[user]\.hcss\lstore THEN UNCOMPRESS

46. How do you know if it worked? >cd ~/.hcss/lstore (e.g. /Users/phil/.hcss/lstore on Mac) >ls 1342184579 >cd 1342184579 (This directory contains many sub-directories) that look like this >ls herschel.ia.dataset.Product herschel.ia.dataset.image.SimpleImage herschel.ia.obs.ObservationContext herschel.ia.obs.QPLog herschel.ia.obs.auxiliary.AuxiliaryContext herschel.ia.obs.auxiliary.missingtm.MissingTmProduct herschel.ia.obs.auxiliary.ool.OolProduct .......