nsr_shutdown Changes in NetWorker 7.3

1. nsr_shutdown Changes in NetWorker 7.3 nsr_shutdown’s Redesign for Scalability Bryce Tompkins (tompkins_bryce@emc.com)

2. Preliminaries • nsr_shutdown is a UNIX shell script provided with the NetWorker Client package to properly shutdown NetWorker services. • In this TOI, service, parent process and daemon can be used interchangeably.

3. Highlights • Completely re-written • Primary design goals include safety and scalability • Default behavior and options have changed

4. nsr_shutdown’s Legacy Behavior • Target services were hard-coded into nsr_shutdown script. • nsr_shutdown indiscriminately targeted all listed daemons at once. • Shutdown sequence consisted of ten iterations of 5 second delimited SIGTERMs to each process. • After ten iterations, nsr_shutdown would SIGKILL all remaining alive processes. • No support for process dependencies (shutting down one process before the other). • Not scalable. nsr_shutdown requires editing to add processes to its target list.

5. New nsr_shutdown Features • Supports process dependencies. For example, nsrd depends on nsrexecd for proper operation. As such, nsr_shutdown will not target nsrexecd until nsrd is down. • Scalable. Target services are specified outside of nsr_shutdown’s source code. • Safe. By default, nsr_shutdown will not SIGKILL processes unless explicitly told to do so. • Informative. nsr_shutdown’s -l flag lists NetWorker services and process IDs in a heirarchical display. • Specific. User can target individual services for shutdown.

6. New nsr_shutdown Design • The new nsr_shutdown makes use of run files to determine which processes to shutdown and in what order they need to be shutdown. • Run files are created under /nsr/run by each NetWorker service or parent process: nsrd, nsrexecd, savegrp, etc. • Run files contain three pieces of information: • The process name. Eg. nsrexecd • The process ID. Eg. 12345 • Prerequisite services to shutdown.Eg. nsrd and savegrp • Example run file: /nsr/run/nsrexecd.12345 nsrd savegrp

7. nsr_shutdown’s New Behavior • NetWorker service or parent process creates a run file at startup. • When nsr_shutdown is launched, it creates a list of running processes and a dependency file for each process based on their run files. • For each process in the list, nsr_shutdown will recursively attempt to kill, via SIGTERM, all prerequisite services found in its dependency file. • When all prerequisites are killed, nsr_shutdown will signal the process for termination. • After a process has successfully shutdown, it is removed from nsr_shutdown’s process list and its run file is deleted.

8. nsr_shutdown’s New Behavior (cont…) • Unlike legacy nsr_shutdown, the redesigned nsr_shutdown only signals a process once and only the parent process or process group leader. For example, during shutdown, nsr_shutdown will send one SIGTERM to nsrd to terminate it and all its child processes (nsrmmd, nsrindexd, etc). The shutdown notification is propogated by nsrd to its children. • If a process group leader aborts/crashes and orphans its children to init, nsr_shutdown can still be used to shutdown the process heirarchy using the aborted process group leader’s run file.

9. When nsr_shutdown Fails…. • By default, nsr_shutdown uses a time period of 120 seconds to shutdown all specified services. If nsr_shutdown fails to terminate all processes successfully after 120 seconds, it will error with the following message:nsr_shutdown: Failed to shutdown service: nsrexecd (12345) • The default timeout can be changed using the -t <timeout> flag. • Alternatively, users can forcefully shutdown NetWorker services after the timeout period has elapsed using the -f flag. This flag will instruct nsr_shutdown to send a SIGKILL to each remaining NetWorker process and their children. This is NOT recommended.

10. When Things Really Go Wrong…. • nsr_shutdown requires valid run files for each process it is asked to shutdown. If, for whatever reason, run files are deleted prematurely, users will have to resort to the “ps and kill” method of terminating processes: • Use ps to find the PID of the process you want to terminate: • # ps –ef | grep nsrexecd • Use kill to terminate the process: • # kill <PID>

11. Additional nsr_shutdown Features • The -l flag will list running NetWorker services: > sudo nsr_shutdown -l Password: +--o lgtolmd (3231) +--o nsrd (3234) +--o nsrmmd (10215) +--o nsrindexd (3250) +--o nsrntfyd (3251) +--o nsrjobd (3252) +--o nsrmmdbd (3245) +--o nsrexecd (3228) • Individual services can be specified on the command line:# nsr_shutdown nsrd

12. Questions?

13. Demo

14. More Information • d1647: SDS nsr_shutdown Redesign for Scalability • nsr_shutdown man page (NetWorker 7.3) • tompkins_bryce@emc.com