Slc-aware Ioc

1. Slc-aware Ioc SLC Executive Message Service Diane Fairley 11/11/2004

2. SLC Executive • Slc-aware IOC Initialization • The SLC Executive Thread • Startup Process • Shutdown Process • Some General Thread Issues • Utilities

3. Overall Architecture

4. slcIocInitialization • The saIocMain.cpp file • Epics initialization via st.cmd file • Read environment variables • slcIoc name (like the microname: XL01, XL02, … ) • Proxy IP address (ip address of PX01 for now) • Load epics databases • Currently a Restart/Stop subroutine record • start cmlog client • iocInit() • slcStart() • iocsh(NULL)

5. slcStart() • Initializes slc globals • Starts the slcExec thread • Executes the Restart function

6. SLC Globals • slcThreads[ thread enum] • Array of thread information and global resources • discovered_name • Name string, from environment variable • proxy_ip_address • from environment variable • The ip address string in xxx.xxx.xxx.xxx format • Db downloadEvent semaphore • saIocJobs[jobcode] = thread enum • converts micro job code to an slcIoc thread enumeration • slcSockets[ socket enum] • Array of socket descriptors • dbExists

7. The SLC ExecutiveThread • Manages the slc startup, creating all other slc threads in order • Supervises threads and globals • Manages a clean shutdown of slc threads • Responds to slcRestart (IPL a micro) • Responds to slcStop (RESET a micro) • SlcExec thread initialization • slcCmlogOpen • Create a message queue • Set slcExec global active=True • Go into loop waiting for messages

8. slcRestart • Commanded via IPL on VMS control system • TST_SLC_NOTIFY message “BOOT” • Received by msgRecv, forwarded to msgHdlr • Commanded from ioc shell • >slcRestart • Commanded from channel access via subroutine record • Sets CSTR:<micro>:1:REST.A=0, • EDM display can set this • Commanded from slcStart() initialization

9. SLC Restart

10. slcRestart • Send TEST_START message to slcExec • From msgHdlr, or • from slcRestart iocsh command, or • from subRecord • slcExec process: Phase I • If threads are running, call slcStop • Wait for all threads to stop • Create downloadEvent semaphore for threads to wait on • Create all threads except msgRecv, dbRecv threads • Wait for these threads to set their global active=True

11. Restart: Phase I

12. slcRestart • Restart Phase II • Create the msgRecv and dbRecv threads • Wait for the Recv threads to set their global active=True • Send a db_download_req message to dbSend message queue. • Goes back to its loop waiting for messages and checking on thread active flags

13. Restart: Phase II

14. slcRestart • Restart Phase III • Database Service downloads the database • Database Service sets the dbExists flag to true • Database Service signals the downloadEvent semaphore to allow threads to proceed.

15. Ready to go

16. General Thread Initialization • slcCmlogOpen • Create resources • Some will create a message queue • Recv threads create a socket • Async threads may create a timer • Create any other thread specific resources • Set active=True • Perform dblists (optional) • Wait on database event • Go into loop waiting for messages or wake-up

17. slcStop • Commanded via RESET on VMS control system • TST_SLC_NOTIFY message “RSET” • Received by msgRecv • Commanded from the ioc shell • >slcStop • Commanded from channel access via subroutine record • Sets CSTR:<micro>:1:REST.A = 1 • EDM display can set this

18. slcStop • Send TEST_STOP message to slcExec • From msgHdlr, or • from iocsh command, or • from subRecord • slcExec process • Set all the thread global stop=True; • Send TEST_STOP message to all threads with message queues • Wait for all thread global “active” flags to be set false • Wait for all threads to stop • Delete the global db downloadEvent • Set dbExists to false; • Set all thread global “stop” flags back to False; • Could re-initialize all globals, do final global resource cleanup…. • go back to loop waiting for messages

19. General Thread Shutdown • Release resources • msg queues, • sockets, • heap memory, • timers, • semaphores • slcCmlogClose() • set active=False • Set tid=NO_THREAD • Return

20. General slcIoc thread assumptions • There are three basic thread types • ‘socket’ threads that wait at sockets • ‘message queue’ threads that wait at queues • Async threads that wake up periodically • Non-message queue threads must poll the stop flag! • No ‘kill a thread’ function in epics thread library • Allows for orderly clean-up in all threads at all times • ‘waits’ should have a timeout • Possibly, message queue threads may wait indefinitely at the queue because they can receive a TEST_STOP message. • Sockets should be non-blocking or interruptable

21. Some Utilities • Thread Utilities • slcThreadNameEnum getThreadEnum(epicsThreadId tid); • slcThreadNameEnum getThreadEnumByName(const char* name); • char* getMicroname(void); eg: ‘XL02’ • char* getProxyIPString(void); eg: ‘142.67.94.25’ • some new wait(…) utilities • To be used by threads when waiting (with a timeout) on a flag or socket or …. • slcCmlog Utilities • int slcCmlogOpen(void); • int slcCmlogClose(void); • int slcCmlogLogMsg (…,.,..,…., ...); • Message queue utilities • epicsMessageQueueId createMsgQ( slcThreadNameEnum thrd ); • vmsstat_t sendToMsgQ ( slcThreadNameEnum thrd, void* msg, int size );

22. slcIoc Message Service • Threads overview • Message structure • Some Assumptions • Message Flow • Utilities

23. Message Service

24. Slc Message Service • The msgRecv thread • “owns” the msg service socket; creates and destroys it • Receives all incoming messages at the msg service socket • Sends the incoming message to the proper slc Hdlr thread • Uses the slcIocJobs[job code] global • The msgSend thread • Sends all outgoing message at the message service socket • “uses” the message service socket • When the connection fails, waits for the msgRecv thread to re-connect. • Checks for a valid socket in slcSockets[sdMsg] • The msgHdlr thread • Handles messages with the TEST function code • Handles the new TST_SLC_NOTIFY messages

25. Message Structure NOTE: this structure is shared with VMS and micros

26. Message Service Assumptions • Sockets are non-blocking or interruptable • They must be able to see the global thread stop flag • Large messages will need heap memory allocations • Message queues do not (efficiently) allow variable size msgs • Use a memory pool to avoid memory leaks, fragmentation • Messages from the VMS control system are little-endian, VMS format, and packed. • The message service threads are responsible for converting the header portions of the message • The other ‘job’ threads are responsible for converting the function specific portion of the message (reqdata)

27. Message Service message flow • The msgRecv thread • Receives the message into a local inMsg variable using a getBuffer utility. • interprets the fwdheader • Allocate memory to store a multi-packet message. • converts/unpacks the msgheader into native format • replaces timestamp in msgheader • Translates the incoming job code to an slcIoc thread id • sends the entire message on to the correct queue. • Message queue send actually copies the message • Large messages: reqdata portion would include a pointer

28. Message Service message flow • The job thread • Reads the message into an inMsg variable. • Ignores the fwdheader portion • Reads the message data length from the msgheader • converts/unpack the incoming function specific data • Releases large message memory when it can, if necessary • uses the data – unidentified requirements here. • converts/packs the function specific reply into an outMsg variable. • Allocates memory for large response message • fills in the new msgheader. (leaves it in native format) • The outMsg reply is sent to the msgSend queue.

29. Message service message flow • The msgSend thread • Reads the message into a msg variable • converts/packs the msgheader • fills in the fwdheader • sends reply to VMS control system • Creates and sends multiple buffers for large messages • De-allocates memory used by large messages

30. Converting and packing/unpacking • Forward header and Msgheader • The forward header follows networking standard in VMS and micros – uses hton and ntoh • Jobs don’t need to see the forward header • Conversion and unpacking handled in by Message Service • Function Specific Data (requests and replies) • Typically a structure is defined for the data and shared by the VMS, micros and slcIocs • See nfs_ref/inc_c for xxx_reqreply_struc.* • To unpack, must memcpy data field-by-field from incoming msg.reqdata[] to a variable of this structure, then convert. • To pack, do the reverse, convert then memcpy field-by-field from structure to outgoing msg.reqdata[] • Utilities will be written for each shared structure.

31. Conversion Utilities • void sswap(const unsigned short* s); • void lswap(const unsigned long* l); • void swap_int2u(void* pOut2u, const void* pIn2u); • void swap_int4u(void* pOut4u, const void* pIn4u); • void cvt_vms_to_ieee(void* pOutR4, const void* pInR4); • void cvt_ieee_to_vms(void* pOutR4, const void* pInR4); • int getVMSTime(void* pDest); • int epicsTimeToVMSTime(void* pDest, const epicsTimeStamp* eTime); • long int diffVMSTimes_10usec(void* pRef, void* pLater); • long int diffVMSTimes_10msec(void* pRef, void* pLater);

32. TCP Utilities • SOCKET getConnection(unsigned short port, unsigned short socket); • Gets a connection with the proxy • Registers with the proxy • Reports error conditions • vmsstat_t getBuffer(SOCKET sd, fwd_hdr_ts *pFwd, void *pBuf, int bytesReqd, int *pBytesRcvd); • Reads incoming message • Allocates memory and re-creates full message from multi-packet • Reports error conditions