Operational Profiles

1. Operational Profiles Material taken from “Operational Profiles in Software Reliability Engineering”, John Musa, IEEE Software, March 93. Special Issue devoted to Operational Profiles. When the article was written, John Musa was supervisor of software reliability engineering at AT&T Bell Labs, New Jersey.

2. Operational Profiles’ Underlying Premises • Software-based reliability is a function of how the customer uses it • A good reliability estimate depends on testing as if the product were in the field

3. What’s an Operational Profile? • Top-level view: a quantitative characterization of how a system will be used. • Assists in allocation of resources for development, review, testing, etc. on the basis of expected use. • Aids performance analysis, architectural design, user manual design, ...

4. A Brief History of the World • Developed by John Musa et al at AT&T to guide testing. • An AT&T PBX switching system combined an operational profile with other quality improvement techniques. • Adopted by HP to re-organize system-test process for multi-processor operating system. • First published results, 1987; active use since.

5. Some benefit/cost results • The PBX switching system experienced reductions in: • customer-reported problems by factor of 10 • maintenance costs by factor of 10 • system-test phase length by factor of two • product-introduction phase by 30% • Experienced no (count ‘em!) serious service outages in first two years of use

6. Some more cost/benefit results • The HP system-test process revision reduced system-test time by 50% (Note: this HP process-revision also used automated test and failure recording)

7. What does an O.P. cost to create? • AT&T defines “average” project • 10 developers • 100,000 source lines • 18 month development time • Effort to construct the O.P. for an average project is “about one staff month” • Increase in effort with increase in project size is less than linear.

8. OK, OK! Enough marketing hype! Fundamental thesis: if we knew how the product will be used, we could focus resources to develop/test faster and meet customer needs better. How do you create an Operational Profile for a product?

9. Who develops an O.P.? • Developed by: • systems engineers • high-level designers (architecture) • test planners • Strong participation by: • product planning • marketing professionals • key customers, if available

10. Overall creation process • A progressively narrowing perspective from customers down to operation • At each step, quantify how often each of the elements in that step will be used; convert to probabilities • Process has been refined many times. Most AT&T applications have been real-time telecommuni-cations systems.

11. Definition of a profile • A set of disjoint alternatives and the probability that each will occur • On the way to creating the operational profile, several intermediate profiles will be produced

12. Usage Information • If this is for an existing product, the data may be available. If not, it can be estimated – better than you may believe. • The usage data is not a profile until you add the probability info

13. Usage data example • Example: • 100 X -type transactions an hour • 500 Y-type transactions an hour • 300 Z-type transactions an hour • Interesting but not useful until you know total number of transactions per hour so you can compute probabilities • 100/2000 = .05; 500/2000 = .25; 300/2000 = .15

14. Example continued • Completeness check: do the probabilities add up to 1? • .05 + .25 + .15 = .45 Missing some. • (Note: Can use raw data to re-create appropriate traffic levels in test.) • How accurate does this (combination of data and probabilities) need to be?

15. Degree of Accuracy Required • What is the economic gain expected from better decisions resulting from more accurate data? (classic “risk management” question!) • In practice, often use “informed engineering judgment” rather than formal economic analysis • Emphasis on the word informed

16. Is emphasis on use sufficient? • Infrequently executed functions of a highly critical nature ARE important, e.g., • pilot ejection from cockpit • overheating nuclear reactor shutdown procedures • Must incorporate notion of criticality as well as use of operations . We’ll get to that...

17. Worst case: 5 steps to create the operational profile • customer profile • user profile • system-mode profile • functional profile • operational profile (based on the above)

18. The O. P. Triangle Customer groups User groups System-modes Functional Operational

19. Examples of each for retail store market • Customer groups: large retail stores, small chains, grocery chains • user groups: cashiers, marketing analysts, I-S specialists • system-modes: I-S specialists do database cleanup and also report generation

20. Retail Market example continued • functions: each mode has several functions (e.g., various reports in report generation mode) • Note use of word function is from user perspective, i.e. user task • operations: • user functions are mapped onto the software product’s operations

21. Some Steps are Unnecessary • customer profile -- not needed if you only have one customer or all customers use the product in the same way; go on to user profile. • functional profile -- not needed if requirements specification details how users will execute the system to do their tasks

22. When Are Steps Conducted? • Possible to determine customer, user, system-mode, and functional profile during requirements definition phase. • May not be able to determine operational profile until design (may be modified in implementation) is complete • Note: article written prior to growing emphasis on incremental development. Now, consider growing the operational profile incrementally.

23. Uniformity of Detail Is Not Required • What is the net economic gain? • Analysis for very important customers may be more detailed than for some other customers • May try out the process of creating an operational profile by doing it for one major customer

24. Customer Profile • Customer is the entity acquiring the product • Customer group is set of customers who use the system in the same way • Customer profile is the complete set of customer groups and their associated occurrence probabilities -- number of customers in each group divided by total number of customers

25. Occurrence probability of a customer group? If earlier version already released, how many copies of the product are installed? Take number installed for customers in that group and divide that number by total number of installations. For potential customers?

26. Occurrence probability for potential customers? • Marketing data from related systems, modified by marketing estimates • Business case usually includes expected customer base

27. User Profile • System’s users not identical to customers • User is entity that employs/uses the system (not “acquires”) • User Group is set of users who use the system in the same way • User Profile is set of user groups and their occurrence probabilities

28. User Profile Is Refinement of Customer Profile • Look at each customer group and determine what user groups exist • Combine same user groups found in different customer groups

29. Occurrence Probability of a User Group • Within a customer group: use the proportion of customer group’s usage that the user group represents • If can’t determine usage, use the number of users as proportion of the total users in that group • What if multiple customer groups?

30. User Group Probability If Multiple Customer Groups • If you have multiple customer groups, multiply the user group’s probability times the customer group probability in which the user group occurs • If a user group is combined across customer groups, determine probability of that user group within each customer group, then add the results to get the overall prob. of that user group.

31. Example of User Groups • For an AT&T PBX customer: • telecomm users (people who make calls or send data) • attendants (internal operators who answer the main number) • system administrator (manages the system, controls user data base) • maintenance personnel (test the system periodically, diagnose and correct problems)

32. System-Mode Profile • System mode is set of functions that are grouped for convenience when analyzing execution behavior • Examples: normal ops, diagnostic mode, system initialization, system shutdown • System-mode profile is set of system modes and their occurrence probabilities

33. System Mode Proliferation? • For each system mode profile, you develop an operational profile (and maybe a functional profile first) • Note that the same operation can occur in different system modes • How many system modes do you define? (it’s that “net economic gain” answer again)

34. Suggestions for Characterizing System Modes • User group -- administration mode, maintenance mode, caller mode • Significant environment conditions -- overload; initialization; continuous operation; system location • Operational architecture -- online sales; after-hours billing • Criticality • User experience -- novice mode; expert mode • Hardware component

35. Important Point About System Modes • So that the probabilities add up to 1, you must define all of the modes that make up a given characterization • Operational profile for each system mode in that characterization • MAY decide to do more than one characterization which would ultimately need to another SET of operational profiles

36. Functional Profile • For each system mode, create a list of the functions used in that mode • Determine each function’s occurrence probability • A function is a part of the overall work to be done as viewed by the system engineer and high-level designer

37. How to develop a function list • Construct work-flow chart showing overall process, including software, hardware, and people. • The work flow shows the context and suggests necessary functions • Usually done during requirements phase

38. Number of Functions • At AT&T, “Average project” had from 50 to several hundred functions • Two tasks are different functions if processing is so different that you manage their development with different priorities and resource allocations • ...especially if they differ a lot in frequency of use or criticality • “Command X” may represent more than one function

39. One command, many functions • Say X has parameters A and B • Parameter A can have values a1 or a2; Parameter B can have values b1, b2, or b3 • If B has little effect on which code is executed, we have two functions: X a1 B and X a2 B • The paper suggests ways to avoid unnecessary function proliferation

40. Function differentiation • Basic requirements definition: ensure that almost all important input values (commands, their variables, global data) and environment variables are covered by the defined functions. • Function differentiation is independent of that. • The more refined the differentiation, the more detailed profile you obtain.

41. Implicit Function Profile • If key input variables are independent w.r.t. the occurrence probabilities of their values, you can develop an implicit profile. • Requires fewer elements • May miss testing some seldom-used but critical operations • Sets of key input variables’ values with associated occurrence probabilities

42. Explicit Function Profile • can get very large • one enumerated set where each element in the set is a group of possible values for the key input variables: [ {c1, d1}, {c1, d2}, {c1, d3}, {c2, d1}, {c2, d2}, {c2, d3}] • Selection of input states is methodically direct -- easy to record and to recognize

43. Initial Function List • Focuses on features which are functional capabilities of interest and value to users • System requirements are best source. • if cannot identify functions from reqs probably the requirements are vague or incomplete • User input is vital -- uncover problems with proposed process • Will be incomplete without...

44. Environmental variables • describe conditions that affect the way the program runs but do not relate directly to features • which control paths it takes • which data it accesses • source: brainstorming by experienced designers • examples: hardware configuration, traffic load

45. Final Function List • Examine key environmental AND feature variables and look for dependencies • Number of functions in final list =: (number of functions in initial list) x (number of env’l variable values) - (combinations of init functions and env variables that do not occur)

46. Occurrence Probabilities of the Functions! • Usage measurements taken on latest release, similar system, or the manual function that is being automated • Maybe available in machine-readable system logs • these are usually operations, not functions • add when one function maps to more than one operation • Adjust for new functions, new env’l variables, etc.

47. New Functions • Estimate usage in combination with existing functions • Less accurate but total number of new functions usually small • Overall functional profile accuracy will remain good • If all new system -- • functional profile may be very inaccurate • but it’s best picture of customer use you have • must be based on customer interaction which indicates relative value

48. At last! Operational Profile • Functional profile is user-oriented and shows functions, not the operations that implement them • You test operations, not functions • An operation is a system task as viewed by the people who will run the system (and as viewed by testers) • Must be available when begin test planning

49. Where do operations come from? • Functions evolve into operations as one develops the way the user will invoke operations (indirectly) to accomplish functions • Italicized phrase is called operational architecture • Op arch is not the same as the system architecture which is about how modules and subsystems combine

50. Mapping from functions to operations • Not straightforward • Example: an administrative function (in the PBX system) is to relocate a telephone. • implement by two operations: removal and installation because these tasks are assigned to different work groups • Usually, there are more ops than functions • An operation may represent an associated sequence of actual procedure calls.