Real-Time Design Guidelines and Rules of Thumb Embedded Systems Conference Boston 2011 ESC-309

1. Real-Time Design Guidelinesand Rules of ThumbEmbedded Systems Conference Boston 2011ESC-309 Dave Stewart, PhD Director of Research and Systems Integration InHand Electronics dstewart@inhand.com www.inhand.com Embedded Systems Conference Boston, September 2011

2. Class Objective • To provide some guidelines and rules of thumb to quickly make the right decisions when designing and implementing real-time systems, even if you don’t have an in-depth theoretical or practical real-time systems experience. • Guideline: Generally a good practice to follow • Rule of Thumb: quick decision or approximation • Caution: Common Pitfall to Avoid Embedded Systems Conference Boston, September 2011

3. Contents • Choosing the Right Environment • Software Decomposition • Practical Real-Time Scheduling • Interprocess Communication • Implementation Solutions Embedded Systems Conference Boston, September 2011

4. I. Choosing the Right Environment • Hard vs. soft real-time • Predictable systems • Preemption • Operating systems vs. executives Embedded Systems Conference Boston, September 2011

5. Terminology • Hard Real-Time • Timing requirements must be met precisely • Failure to meet requirements leads to significant failure • catastrophic (system is destroyed) • damage to environment or injury to people • economic (product fails in the market place) • Soft Real-Time • There is some flexibility in the timing constraints Embedded Systems Conference Boston, September 2011

6. Hard or Soft? • Are these applications Hard or Soft real-time? • Flight Control System • Bottling Plant Production Line • Anti-Lock Braking System • DVD Playback • Airline Reservation System • Internet Video • Cellular Phone RF Reception Embedded Systems Conference Boston, September 2011

7. Hard or Soft? • Are these applications Hard or Soft real-time? • Flight Control System Soft • Bottling Plant Production Line Hard • Anti-Lock Braking System Soft • DVD Playback Hard • Airline Reservation System Soft • Internet Video Soft • Cellular Phone RF Reception Hard My Opinion Any Surprises? Embedded Systems Conference Boston, September 2011

8. Soft Real-Time as a Spectrum • Almost all applications can be soft real-time • Sometimes, one or more timing constraints are hard • Right-most end of spectrum • Rarely are all timing constraints hard • Few applications are non-real-time • Left-most end of spectrum Soft Not RT Hard Computer Simulation User Interface Internet Video Temperature Control Flight Control Wireless Communication Electronic Engine Embedded Systems Conference Boston, September 2011

9. Real vs. Artificial Constraints • Real • There is no flexibility in the timing constraint. If it is changed or relaxed, the system will fail. • Eg. 150 msec window for filling bottle on conveyor belt. • Artificial • Timing constraint was determined somewhat arbitrarily. • E.g. 40 Hz rate for cruise control. • Important to distinguish between Real and Artificial: • Artificial constraints can be refined, possibly improving other design factors (like cost & functionality) • Real constraints must be met at any cost. Embedded Systems Conference Boston, September 2011

10. Terminology • Predictable • The system operates within known bounds. • Exact order of operations is not always known. • Deterministic • A special (extreme) case of predictable. • The system operates with exact order of operations known at all times. • A Real-Time System’s timing behavior needs to be Predictable, but seldom Deterministic Embedded Systems Conference Boston, September 2011

11. Example • Dice: Predictable or Deterministic? Embedded Systems Conference Boston, September 2011

12. How do we make a System Predictable? • Maximize periodic activities • Minimize aperiodic (random) activities • Avoid unnecessary preemption • Choose appropriate scheduling strategies • Eliminate interrupts • Use deferrable or sporadic server for aperiodic tasks • Use only “guaranteed” soft real-time tasks • Measure execution time to monitor resource usage • Implement proper timing error detection and handling Embedded Systems Conference Boston, September 2011

13. Terminology • Periodic • Code that executes at regular intervals • Rate refers to how often task executes: • E.g. 50 Hz (50 times per second) • Period = 1/Rate. • E.g. 1/(50Hz) = 20 msec. • Aperiodic • Code that executes irregularly or randomly. • Minimum inter-arrival time is shortest amount of time between two successive aperiodic events. Embedded Systems Conference Boston, September 2011

14. Periodic • In Real-Life, is it easier to schedule? • 1hr meeting at the same time each day? • 1hr meeting at the same time each week? • 4hr meeting at the same time each week? • 1hr meeting twice a week, but time changes? • Real-Time is like real-life. It is easier to schedule: • Tasks that are periodic instead of irregular • Less frequent tasks • Shorter tasks Embedded Systems Conference Boston, September 2011

15. KISS Principle • From the Jargon Dictionary: • KISS Principle/kis' prin'si-pl/ n.“Keep it short and simple” • Periodic Tasks help KISS. • Execute each task as slow as possible • Execute each task for as little time as possible • Anything else adds complexity Embedded Systems Conference Boston, September 2011

16. Preemption • Do you need it? • Is it helping or hurting your application? Embedded Systems Conference Boston, September 2011

17. Preemption: When it Helps • Wide variety of periods and execution times • If freq of fastest task < 100 * freq of slowest task, then the task set is likely schedulable without preemption • Multiple applications sharing the processor • RT and non-RT tasks sharing a processor • Hard and soft RT tasks sharing the processor • More tolerant to design-time mistakes • But this leads to laziness at design time Embedded Systems Conference Boston, September 2011

18. Preemption: What it Costs • Reduces predictability • Increases overhead (scheduling, context switch) • Possible race conditions • Introduces possibility of priority inversion • Significant increase in overhead to avoid race conditions and priority inversion • Reduces schedulable bound if blocking is allowed • More difficult to test and debug • Potential for deadlock if blocking occurs • Estimate at least 25% increase in CPU usage for a preemptive system, compared to non-preemptive. Embedded Systems Conference Boston, September 2011

19. Preemption: Do You Need It? • Use KISS to decide: • If your application needs it, preemption can be a tremendous benefit to reduce complexity. • If you don’t need it but use it, your application will be much more complex than necessary. • The best compromise might be limited preemption. • more later Embedded Systems Conference Boston, September 2011

20. Real-Time Operating Systems (RTOS) • Every RTOS provides preemption. • Why don’t any provide the option of non-preemption? Embedded Systems Conference Boston, September 2011

21. Non-Preemptive RTOS • Called “Real-Time Executive” (RTE) • There are no COTS RTEs because they can be implemented in 5 to 10 lines of code. • You can’t sell 10 lines of code for $3000. • I don’t need preemption, so I don’t need an RTOS? • Maybe • I need preemption, so I need an RTOS? • Yes COTS: Commercial Off-The-Shelf Embedded Systems Conference Boston, September 2011

22. COTS RTOS • When do I use an RTOS? • You determine preemption is necessary • Lots of COTS input/output devices, like VGA monitor, USB serial ports, touchscreen, ethernet, etc. • You are using advanced CASE tools to design and implement software CASE: Computer-Aided Software Engineering Embedded Systems Conference Boston, September 2011

23. COTS RTOS • Roll my own RTOS vs. COTS RTOS • If you don’t have a degree in RTOS design, always use COTS • Roll your own only if you are an expert in the field, and there are extenuating circumstances requiring it. E.g. • Using RT mechanisms not available in commercial RTOS • Domain-specific certifications require 100% control of source code • If you roll your own, use a “free” RTOS as a starting point (eg uCOS, RTLinux) • You are in the business of selling RTOS technology • Licensing costs: • Must compare cost of license(s) to cost of labor. • More often than not, cost of labor is substantially higher Embedded Systems Conference Boston, September 2011

24. Real-Time Executive • When do I use a Real-Time Executive? • You determine preemption is not necessary • Most or all of the I/O is custom • You need 100% control of all code on the processor for software certification • Extreme processor resource constraints • RAM or ROM < 10 KBytes • CPU speed < 10 MHz Embedded Systems Conference Boston, September 2011

25. Real-Time Executive • Roll my own? • Yes, because there are no commercial ones • Use a Commercial or Free IDE that provides the compiler, standard libraries, and debugging tools • Do NOT hard-code the process flow. That’s not an RTE, its just hacked code. • An expert should design the core RTE engine, so that user tasks simply “plug in” to the framework. IDE: Integrated Development Environment Embedded Systems Conference Boston, September 2011

26. II. Software Decomposition • Circular Dependencies are BAD • Use a Data Flow Architecture • Model of a Real-Time Task Embedded Systems Conference Boston, September 2011

27. Circular Dependencies are BAD • Circular Dependencies are the primary cause of “spaghetti code” • Use Dependency Trees to spot and remove circular dependencies • Eliminating circular dependencies provides the following benefits: • Component reusability • Fault confinement • Incremental testing • Modularized performance and timing analysis Embedded Systems Conference Boston, September 2011

28. abc abc abc abc ghi def def def def ghi ghi ghi ghi jkl jkl stu stu jkl stu jkl stu jkl stu jkl mno mno mno mno mno pqr pqr pqr pqr pqr uvw uvw uvw uvw uvw pqr uvw xyz xyz xyz xyz xyz xyz Dependency Tree Example jkl A circular dependency changes the tree into a graph. Avoid circular dependencies at (almost) all costs. Embedded Systems Conference Boston, September 2011

29. Avoiding circular dependencies • Never have a single “globals.h” or equivalent file with all header information. • Instead, one .h file per “module” • Only put exported declarations in the .h file. • Data structures that are private should be defined in the .c file, not in the .h file. • If a module is dependent on another, only then should it include the .h file of the other module. Embedded Systems Conference Boston, September 2011

30. Breaking Circular Dependencies replace with ghi jkl ghi jkl shared This is not always easy. It will force careful reconsideration of the design. Re-designing to satisfy dependency requirements will result in software that is substantially easier to maintain. Additional details is beyond the scope of this class. Embedded Systems Conference Boston, September 2011

31. Use a Data-Flow Architecture • Every task is independent • Use the dependency tree to verify this • Only use explicit Interprocess Communication to exchange data between tasks Bad Good task A task B task A task A task B task B task A task B ipc Embedded Systems Conference Boston, September 2011

32. Use a Data-Flow Architecture • Independent tasks enable use of real-time scheduling • Without real-time scheduling, timing constraints cannot be guaranteed Embedded Systems Conference Boston, September 2011

33. f ra accel engine accelerator pedal force to f engine de + + f rb – split-pm brake – = f dw wheels brake pedal U force to f r d wheels ccui control v z v r cruise control user interface x z dx/dt position dashboard wheel position speedometer odometer resolver Data Flow Architecture Example(Cruise Control System) Start with outputs then produce diagram right to left. Embedded Systems Conference Boston, September 2011

34. Configuration Constants : k k 1 p x y 1 real-time 1 IPC Input : : IPC Output task x y m m r r : 1 p Sensor Inputs and Control Outputs Model of a Real-Time Task Embedded Systems Conference Boston, September 2011

35. Model of a Real-Time Task Each task has a main loop that does the following: For periodic tasks, event is time-based. For other tasks, event could be an interrupt, message arrival, semaphore wakeup, or any other signal. Wait for Event Read Inputs/Events Do Processing Write Outputs Embedded Systems Conference Boston, September 2011

36. Proper Implementation of Periodic Tasks • DO (use a framework and “wait until” to ensure no clock skew) nextstart = clock(); while (1) { nextstart += period; wait until period // wait for timer event call task->cycle() } • DON’T (implement timing inside task, and use sleep that causes clock skew) taskMain() { laststart = clock(); while (1) { delta = period – (clock() – lasttime); sleep(delta); // wait for timer event do stuff } } • Additional examples when real-time scheduling examples are presented. Embedded Systems Conference Boston, September 2011

37. Proper Implementation of Aperiodic Tasks • Use Aperiodic Servers • More details near end of this talk. Embedded Systems Conference Boston, September 2011

38. III. Practical Real-Time Scheduling • Your Life is a Real-Time System • Cyclic executives • Rate monotonic algorithm • Dynamic scheduling Embedded Systems Conference Boston, September 2011

39. Practical Real-Time Scheduling • Now: Hard Real-Time Periodic Tasks • Non-Preemptive • Preemptive • Fixed Priority • Dynamic Priority • Later: • Interrupts • Aperiodic servers • Soft real-time tasks • Limited preemption Embedded Systems Conference Boston, September 2011

40. Your LifeIS a Real-Time System Embedded Systems Conference Boston, September 2011

41. Real-Life Scheduling • If you have 10 tasks to do, what order to you do them in? • If you go to the bank, in what order are you served? • If you need to catch a flight at 5pm, how does that affect your schedule? • If it is lunch time and you have a report due at 3pm, do you stop for lunch? • If the report is due at 7pm, do you stop for lunch? • These same kinds of decisions must be made for each task in a real-time system. That’s real-time scheduling. Embedded Systems Conference Boston, September 2011

42. A B C A B C A B C A B C 0 20 40 60 time (msec) Basic Cyclic Executive • Everything executes at same rate • Example: 3 Tasks • Each task executes every 20 msec (TA=TB=TC=20) • Note: Ti=Period of Task i • Execution time constraints: • SCi < TA • Note:Ci=Worst Case Execution Time of Task i • Suitable for simple control loops. A B C Embedded Systems Conference Boston, September 2011

43. A B C A B C A B C A B C 0 20 40 60 time (msec) Basic Cyclic Executive Implementation See paper for details on implementing the timer. typedef void (*funcptr)(void); void A(void) { // B and C defined similarly //code goes here } funcptr cyclic[] = { A, B, C}; const int cycliclength = sizeof(cyclic)/sizeof(funcptr); cyclicexec(funcptr cyclic[],int cycliclength, int cycletime) { int i; intrSetPeriod(20msec); while (1) { intrWait(); for (i=0;i<cycliclength;++i) cyclic[i](); } } A B C Embedded Systems Conference Boston, September 2011

44. A B A C A B A C 0 20 40 60 time (msec) Multi-Rate Cyclic Executive • Tasks execute at different rates • Example: 3 Tasks • A executes every 20 msec (TA=20) • B and C execute every 40 msec (TB=TC=40) • Execution time constraints: • CA+CB < TA && CA+CC < TA Embedded Systems Conference Boston, September 2011

45. A B A C A B A C 0 20 40 60 time (msec) Multi-Rate Cyclic Executive Implementation typedef void (*funcptr)(void); void A(void) { // B and C defined similarly //code goes here } funcptr cyclic[] = { A, B, NULL, A, C, NULL}; const int cycliclength = sizeof(cyclic)/sizeof(funcptr); cyclicexec(funcptr cyclic[],int cycliclength, int cycletime) { int i; intrSetPeriod(20msec); while (1) { for (i=0;i<cycliclength;++i) { if (cyclic[i]==NULL) intrWait(); else cyclic[i](); } } Embedded Systems Conference Boston, September 2011

46. Multi-Rate Cyclic Executive • Rates for each task are constrained: • Periods of each task must be harmonic • To setup schedule, place fastest task in schedule first • Then place next fastest, and so on. • Execution time is constrained: • SCifor tasks executing each cycle must be less than period of interrupt. Embedded Systems Conference Boston, September 2011

47. Harmonic Task Set • Period of Task K is multiple of every faster task • "k>1, Tk=nk*Tk-1, k, nk are integers • Examples of Harmonic: • 10, 20, 40, 80, 160 • 10, 30, 120, 480 • 20, 100, 2000 • Examples of Non-Harmonic: • 10, 25, 50, 75 • 10, 30, 100, 500 • 20, 120, 2200 Embedded Systems Conference Boston, September 2011

48. Dynamic Multi-Rate Cyclic Executive • Relaxes constraints on period and execution time • Uses Earliest-Deadline First (EDF) Algorithm • Task with earliest deadline executes next • Deadline of current cycle is time task is ready for next cycle • Rate/Period constraints: • none • Execution Time constraints: • S(Ui) <1 (i.e. CPU not overloaded) Note: Ui=Ci/Ti=Utilization of Task I • Cifor each task is < T1/2 • i.e. period of fastest task dictates max execution time for every other task • Cost of using this method: • Increased RTE complexity • Requires more thorough analysis to guarantee scheduling feasibility Embedded Systems Conference Boston, September 2011

49. Implementation ofDynamic Multi-Rate Cyclic Executive while (1) { do { etimeClock(&now); // Move newly awoken tasks from PauseQ to ReadyQ while (!taskEmptyQ(pauseQ) && etimeCompare(taskSchedTime(pauseQ),now) <= 0) { taskptr=taskDequeue(&pauseQ); etimeAdd(taskptr>schedtime, taskptr->period); taskEnqueue(&readyQ,taskptr); } } while (readyQ.next == NULL); // Head of readyQ has earliest-deadline task taskDequeue(&readyQ, taskptr); taskptr->func(args); taskEnqueue(&pauseQ, taskptr); } (note, some detail has been omitted) Embedded Systems Conference Boston, September 2011

50. Preemptive Scheduling • Always execute task with highest priority, even if it means preempting a task with lower priority. Embedded Systems Conference Boston, September 2011