Overview of Boston University’s Scientific Computing Facility

1. Introduction to Boston University’s Scientific Computing Facility Doug Sondak sondak@bu.edu Scientific Computing and Visualization

2. Information Services & Technology Outline • Introduction • Hardware • Software • Account Information • Tutorials • Some Basic Unix Commands • How to Run Batch

3. Information Services & Technology Introduction • BU’s Scientific Computing Facility (SCF) addresses high-performance computing and visualization needs of the BU community • Large resource requirements • CPU time • Memory • One group used 3 centuries of CPU time last year!

4. Information Services & Technology Overview of SCV • Scientific Computing and Visualization • Computing • high-performance, parallel computers (SCF) • support • parallelization • optimization • tutorials • Visualization • support to create graphics, from publication figures to interactive 3D displays and movies • 15’x8’ tiled display for 3D visualization • tutorials

5. Information Services & Technology Current Hardware - BladeCenter • IBM BladeCenter • katana • currently 272 processors • heterogeneous mixture of blades • Intel Xeon and AMD Opteron • 2.4-3.0 GHz • blades contain 4 or 8 processors • May use up to 32 processors at a time • parallel or serial • Two blades each have 8 processors and 96 GB memory

6. Information Services & Technology Current Hardware – Blue Gene • IBM Blue Gene • 1024 nodes, each with 2 processors • PowerPC 440 processors • 700 MHz • 32-bit • only runs MPI codes • each processor is relatively slow, so you need good scalability to make it worthwhile • May use 1024 processors during the day, all 2048 processors at night

7. Information Services & Technology Current Hardware - pSeries • IBM p655 • shared-memory machines • Power4 processors • 9 machines, each with 8 processors • 6 machines 1.1 GHz, 16 GB memory • 3 machines 1.7 GHz, 8 GB memory • login node is twister

8. Information Services & Technology Software • All machines run under Unix • C, C++, Fortran, Java, Perl, Python, Tcl • Matlab • all machines except Blue Gene • PCT (Parallel Computing Toolbox) • Maple • all machines except Blue Gene • Mathematica • all machines except Blue Gene

9. Information Services & Technology Software (cont’d) • Gauss • blade center, pSeries • R • all machines except Blue Gene • VTK • Paraview • Maya

10. Information Services & Technology Accounts and Policies • Accounts and allocations are based on “projects” • Must be faculty or research staff to apply for projects • Project PI then adds accounts for students, collaborators, etc. • May assign post-doc or admin. staff as “administrative contact” • Apply for project at http://www.bu.edu/tech/accounts/special/research/accounts/applications/ • Click on “Boston University faculty and research staff may apply for a new project.” • If you’re not sure how much time to ask for, 1000 katana hours is considered modest • The form has fields for requesting time on each machine, but the time is actually all in one pot, and can be spent on any machine • This will give you accounts on all machines except for Blue Gene

11. Information Services & Technology Accounts and Policies (cont’d) • For Blue Gene account, once you are awarded a “regular” account go back to • http://www.bu.edu/tech/accounts/special/research/accounts/applications/ • There is a paragraph starting with “Blue Gene accounts.” Click on the “SCF User Information Page” link. • You will be requested for your user name and password • Click on the “Update Personal Information” link

12. Information Services & Technology Disk Space • The disk space in your home directory is minimal • Most project PI’s request a “project” directory • project directory contains disk space allocated to the specific project • once your account has been activated go back to http://www.bu.edu/tech/accounts/special/research/accounts/applications/ and click “request a Project Disk Space allocation.” • a request of a few GB will be rubber-stamped • large requests (hundreds of GB) will require stronger justification

13. Information Services & Technology “Buy-In” Nodes • Departments/groups may purchase their own BladeCenter nodes • alternative to buying and administering their own machines • Group that owns node(s) gets priority for batch runs • can be restricted to specific users if desired • May also have log-in node(s) for interactive use • log-in node restricted to specified groups/users

14. Information Services & Technology Services • SCV has 3 components: • system administration • excellent administrators! • can often accommodate special requests • visualization • can help with creating visualizations, including animations, for presentations, publication, and display wall • wall has 3D capability

15. Information Services & Technology Services (cont’d) • high-performance computing • code parallelization and optimization • MPI • OpenMP • PCT (Matlab) • help with any large-scale computing issues • algorithms • coding • There is no charge for SCV services

16. Information Services & Technology Tutorials • SCV offers tutorials on a variety of computing and visualization topics • If you have interest in an area that we don’t presently cover, let us know! • Current offerings are in three areas • Programming • High-Performance Computing • Visualization

17. Information Services & Technology Tutorials (cont’d) • Programming • Intro to Matlab • Intro to Fortran • Intro to C/C++ • High-Performance Computing • Code Tuning (C, Fortran) • Tuning Matlab Codes • Intro to MPI • Parallelization with OpenMP • Matlab Parallel Computing Toolbox (PCT)

18. Information Services & Technology Tutorials (3) • Visualization • Intro to Scientific Visualization • Scientific Visualization using Paraview • Scientific Visualization using VTK • Scientific Visualization using Matlab • Graphics Programming in C/C++: OpenGL and OpenSceneGraph • Graphics and Images for Presentation and Publication

19. Information Services & Technology Some Basic Unix

20. Information Services & Technology Login • From a terminal (e.g., Xwin-32, putty) ssh katana • this requests a “secure shell” on katana • you’ll be prompted for your username and password • You’re now logged on to the blade server “katana” • You’ll see something like “katana:~ %” • this is a Unix prompt, which is where you type in commands

21. Information Services & Technology Directories • “directories” are analogous to PC “folders” • delimited by / rather than \ as on a PC • when you first log in, current directory is called your “home” directory • directory where you are located at any given time is your “working” directory • pwd means “print working directory” • type pwd • to create a subdirectory, use mkdir (make directory) • type mkdir sub1, where sub1 is the subdirectory name

22. Information Services & Technology File Management • ls lists all files and directories in your current directory • type ls • you should see your new directory listed • some shorthand directory names: ~ home directory . current directory .. one level above current directory • cd dirname(change directory) moves you to dirname • type cd sub1 • typepwd

23. Information Services & Technology File Management (cont’d) • type ls~ • lists everything in your home directory • type ls .. • since you’re one directory below your home directory, this is the same as ls ~ • cp from to will copy a file • from is the file name you’re copying from • to is either a file name or a directory • if it’s a file name, the copy will be given that name • if it’s a directory, the file will retain the old name and be placed in the specified directory

24. Information Services & Technology File Management (3) • copy the file /scratch/sondak/emacs_file to your sub1 directory • cp /scratch/sondak/emacs_file . • we will use it later • a path is a full set of directories leading to a file or directory • e.g., when you type pwd, it shows the path to your current directory • file suffixes generally don’t have any special meanings as they do under Windows • there are some cases where they do have meaning, such as .m for a Matlab program

25. Information Services & Technology Emacs • text files are created with an editor • kind of like a fancy “notepad” • two most common Unix editors are vi and emacs • since I use emacs, that’s what we’ll use here • we’ll introduce some emacs commands using the file emacs_file • two ways to enter commands • menu • shortcuts

26. Information Services & Technology Emacs Exercise • make sure you’re in your sub1 directory • type emacs emacs_file • the file will appear in an emacs window • try migrating around the file with the arrow keys • you can also navigate by clicking on the desired location • click on the 0 in 0.282 • to delete a character use Delete button • hit Delete 3 times to delete 0.2 • type 1.3 • you’ve now changed 0.282 to 1.382

27. Information Services & Technology Emacs Exercise (cont’d) • Highlight 0.288 with the mouse • Don’t include spaces before or after • Edit -> Cut will delete the highlighted characters • can also use CTL-w • Oh no, we made a mistake! Edit -> Undo undoes the previous command. Try it; 0.288 should reappear. • can also use CTL-x u • can keep repeating undo, and it will undo last command, second-to-last command, third-to-last command, etc.

28. Information Services & Technology Emacs Exercise (3) • The point is the location on the left side of the cursor; i.e., between the character on which the cursor resides and the character to its left. • The mark is similar to the point, but it’s location is set explicitly (i.e., doesn’t move with cursor).

29. Information Services & Technology Emacs Exercise (4) • Suppose we want to delete all the characters between (inclusively) 0.288 and 0.407. • Set the cursor on the 0 in 0.288. • To set the mark, CTL-spacebar • The note “Mark set” will appear at the bottom of the screen. • Move the cursor to the right of 0.407 • We place it to the right of the 7 rather than on it because the point is always to the left of the cursor • Edit -> Cut will delete all characters between the mark and the point • shortcut is CTL-w

30. Information Services & Technology Emacs Exercise (5) • For now, let’s put the characters back in by using Edit -> Undo (or CTL-x u) • Move the cursor to the start of the current line using CTL-a • (CTL-e moves to end of current line) • Delete (“kill”) the line with CTL-k • Another CTL-k deletes the newline character

31. Information Services & Technology Emacs Exercise (6) • “Meta” key is the Esc key • We will use M to refer to the Meta key • The Meta key is hit prior to the subsequent key(s), not simultaneously as with CTL • Place the cursor at the top of the file • M-> will move the cursor to the bottom of the file (don’t confuse notation with arrows I’m using for menu notation; this is “Meta greater-than”) • M-< will move it back to the top

32. Information Services & Technology Emacs Exercise (7) • Whatever we deleted last is available in a buffer • Move the cursor to the beginning of the “M1rel” line • Edit -> Paste “yanks” the current buffer • can also use CTL-y

33. Information Services & Technology Emacs Exercise (8) • CTL-h gets you into the Emacs help function • at prompt type ? to list types of help • I often use “a” (apropos) • Type CTL-h • Thentype a at the prompt • Then type “kill” at the prompt • The window will split, listing all Emacs commands that have something to do with the word kill

34. Information Services & Technology Emacs Exercise (9) • Type CTL-x 1 to get you back to the un-split screen • Split screens can be very handy, especially for cutting from one file and pasting in another • Type CTL-x 2 to split screen horizontally • can open different files in each half of the screen • toggle across split with CTL-o (“other”) • CTL-x 0 eliminates the half screen in which the cursor currently resides (try it) • this is a zero, not an “oh”

35. Information Services & Technology Emacs Exercise (10) • To save the modified file, File -> Save (current buffer) • can also use CTL-x CTL-s • A note will appear at the bottom of the window saying the file has been saved • To save it under a new name, File -> Save Buffer As… • can also use CTL-x CTL-w • You’ll be prompted for the name at the bottom of the screen • Note that when you get these kinds of prompts, you can edit them using emacs commands • Type a file name and then move back and forth with arrow keys • File -> Exit Emacs (or CTL-x CTL-c) to quit • Previous version will appear with ~ suffix

36. Information Services & Technology Katana Batch System

37. Information Services & Technology Katana Batch System • If your run will take less than 10 minutes of CPU time, you can run interactively • Longer runs are submitted to the batch system • Sun Grid Engine • May use up to 32 processors at a time • can be 32 serial runs • can be one 32-processor parallel run • can be combination • Serial runs are limited to 24 hours • Parallel runs are limited to 5 hours

38. Information Services & Technology Katana Batch System (cont’d) • Job Control File (JCF) • this is file you submit to batch system • line starting with #! defines Unix shell • lines starting with #$ are batch system commands • lines starting with # are comments • Here’s a “minimalist” JCF for a serial run: #!/bin/tcsh # run time limit hr:min:sec #$ -l h_rt=2:00:00 mycode

39. Information Services & Technology Katana Batch System (3) • JCF must have execute permission • chmod 755 myjcf • qsub myjcf submits your job to the queue • qstat –u username checks status of all your jobs

40. Information Services & Technology Katana Batch System (4) • Create a trivial Matlab code with some output, e.g., • I called mine “dumbcode.m” • Create a JCF a = 3; disp([‘a = ‘ int2str(a)]); #!/bin/tcsh # run time limit hr:min:sec #$ -l h_rt=0:02:00 matlab –r –nodisplay dumbcode

41. Information Services & Technology Katana Batch System (5) • Submit to batch system • When it’s done 2 files should be in your working directory • dumbcode.e###### • dumbcode.o###### where ###### represents a 6-digit number • The .e file contains errors • The .o file contains output

42. Information Services & Technology References • General SCV/SCF information • http://www.bu.edu/tech/research/ • Blade Server • http://www.bu.edu/tech/research/computation/linux-cluster/katana-cluster/ • Account Application and Maintenance • http://www.bu.edu/tech/accounts/special/research/accounts/applications/ • Slides from Live SCV Tutorials • http://www.bu.edu/tech/research/training/presentations/list/ • Web-based SCV Tutorials • http://www.bu.edu/tech/research/training/tutorials/list/ • Katana Batch System • http://www.bu.edu/tech/research/computation/linux-cluster/katana-cluster/runningjobs/

43. Information Services & Technology Human Help • If you ever have any questions, feel free to contact us • Doug Sondak, sondak@bu.edu • Kadin Tseng, kadin@bu.edu