Semantic Structuring in Analyst Acquisition and Representation of Facts in Requirements Analysis

1. Semantic Structuring in Analyst Acquisition and Representation of Facts in Requirements Analysis • George Marakas and Joyce Elam • ISR 1998, vol 9., no.1 • Presented for EEL 5881 • Virginia Ilie • November 5, 2002

2. Overview • Importance of the RA phase • Purpose of the Study • Research Questions • Information Gathering in Requirements Determination • Hypotheses • Methodology • Analysis and Results • Implications • Potential Limitations • Future Directions for Research

3. “One of the greatest problems the system designer must face is the need to completely define the requirements of the system under development” • (Zmud et al 1993)

4. Importance of the RA phase • Most IS failures traced back to a lack of specific and correct information requirements (Davis 1982, Kydd 1989) • Many IS successes traced to the proper identification of information requirements early in the design process (Boehm 1976, 1981)

5. Importance of the RA phase • Over 50% of the systems reviewed had problems necessitating return to the RA phase (Jenkins et al 1984) • Relative cost of correcting an error increases exponentially with the project phase error is detected (Boehm 1973)

6. Purpose of the Study • Investigate the effects of a semantic structuring process of inquiry (SSPI) on interview-derived information gathering and the subsequent overall correctness of a derived logical model • Specifically, investigate particular question types used by system analysts and the role their semantic construction played in understanding the information flows in a business system

7. Definitions • Semantic Structuring - the application and use of specific question types for obtaining specific interview-derived information

8. Research Questions • What specific question types are most effective in gathering direct interview-derived information related to the construction of an accurate model representing the logical data flows within an organizational or business subsystem ? • Can an inquiry process composed of specific question types lead to obtaining greater accuracy in the collection of relevant direct interview-derived information related to business subsystem processes and data flows ?

9. Premise • The use of a SSPI: • For the analyst: enhance his/her ability to elicit user requirements • For the user: better explicate his/her requirements • For both: more effectively and accurately communicate with each other

10. Information Gathering in Requirements Determination • Direct interview of the user remains one of the primary means of obtaining information (Holtzblatt&Beyer 1995)

11. Unstructured vs. Structured Interviews • Without some type of format or structure, the value and accuracy of information is greatly diminished (Dorsey & Koletzke 1997) • Unstructured communications manifested in dysfunctional characteristics and costly errors in design (Salaway 1987)

12. Unstructured vs. Structured Interviews • Structured interviews: method to improve the quality of information gathering during the questioning • Structured interviews found to be more efficient than unstructured interviews (Agarwal & Tanniru 1990)

13. The Psychology of Questioning • Loftus & Palmer’s experiment (1974): The film of a traffic accident • The wording of the questions and the modification of a single word within the body of a question altered significantly respondent’s recollection of an event • Very little is known about the role of questionning processes as they relate to information requirements (Lauer et al 1992)

14. Hypotheses • H1: The use of a semantic structuring model of inquiry •  greater accuracy in the construction of logical representationsover the use of unstructured inquiry

15. Hypotheses H1a. The use of a semantic structuring model of inquiry greater accuracy in the construction of logical representations by low experience analysts (over the use of unstructured inquiry) H1b. The use of a semantic structuring model of inquiry greater accuracy in the construction of logical representations for high-experience analysts (over the use of unstructured inquiry) H1c. The relative degree of improvement resulting from the use of the semantic structuring model of inquiry in the correctness score of a logical representation will be greater for low experience analysts than for high experience analysts.

16. Hypotheses • H2a. The number of feasible but nonexistent elements mapped in the construction of a logical representation will be significantly greater for those using no specific inquiry strategy rather than for those making use of the semantic structuring model of inquiry • H2b. The number of feasible but nonexistent elements mapped in the construction of a logical representation will be significantly greater for high experience subjects than for low experience subjects regardless of the inquiry method employed.

17. Hypotheses • H3. The presence of a question inquiring about a specific data element or flow will be significantly related to the probability of identifying it correctly regardless of the inquiry method employed • H4. Subjects using no specific inquiry strategy that are successful in constructing an accurate logical representation will exhibit semantic questioning patterns similar to other successful subjects • H5. Subjects using no specific inquiry strategy that are successful in constructing an accurate logical representation will exhibit questioning patterns similar to the SSPI

18. Methodology • Lab experiment • Two control groups (low experience and high experience analysts using an unstructured elicitation approach) - CL and CH • Two experimental groups (low experience and high experience analysts using a semantic model of inquiry) - EL and EH • Low experience subjects recruited from an undergrad MIS program at a southern univ. • The high experience subjects recruited from professional system analysts and software designers from several major corporations

19. Methodology • Pretest questionnaire • Subjects randomly assigned to one of the two subject groups based on their experience level • Subjects received a copy of an abridged description of an existing business system • Their task was to develop a thorough understanding of the system through interviewing a “key employee” of the organization • Following the interview, each subject was asked to construct a logical DFD from the information obtained

20. Analysis and Results: H1 • H1: SUPPORTED • Method: ANOVA and post-hoc pairwise comparisons • Comparison of DFD scores from the groups using no specific inquiry model (CL and CH) and the ones using the semantic model of inquiry (EL and EH) •  The use of a specific semantic process model of inquiry has a significant effect on the accuracy of the logical representation (the DFD) over the use of no specific model of question construction

21. Analysis and Results: H1a • H1a: SUPPORTED • Method: ANOVA • Comparison of the DFD scores obtained from the low experience control and treatment groups only (CL vs EL)  significant differences exists • Low experience analysts benefited from the use of the specific model of inquiry over use of no specific inquiry process

22. Analysis and Results: H1b • H1b: SUPPORTED • Method: ANOVA • Comparison of DFD scores from the high experience analysts (CH vs EH) •  High experience analysts derived value through the application of the semantic process of inquiry

23. Analysis and Results: H1c • H1c: NOT SUPPORTED • Method: t-test – ascertain the significance of the differences between groups CL and EL and CH and EH • No support regarding the degree of improvement associated with the use of the specific inquiry process over the unstructured process

24. Analysis and Results: H2a • H2a: SUPPORTED • Method: t-test – compare the difference between the number of feasible but non-existent data flows mapped by the combined control groups and that of the combined treatment groups • Feasible data flow: it it could theoretically exist within the specified context but but in fact it does not • Any element deemed feasible but non-existent counted and a group mean calculated

25. Analysis and Results: H2b • H2b: NOT SUPPORTED • Method: t-test – compare the difference between the high experience and low experience subjects regarding feasible but non-existent data flows • The difference is not significant

26. Analysis and Results: H3 • H3: SUPPORTED • Method: chi-square test for difference in probabilities • The correct identification of a data element in the DFD must be associated with at least one question resulting in a response that either establishes or confirms its existence • The creation of an accurate logical representation of the business process is significantly associated with the inquiry itself (in addition to any experiential or inferential skills)

27. Analysis and Results: H4 • H4: SUPPORTED • Method: K-means cluster analysis – It produces clusters that best represent the primary similarities among the cluster items and minimize variation within each cluster • 3 clusters emerged – Balanced Pattern, Micro-Inquiry and • Confirmatory Pattern • 7 of 9 high score subjects fell under Balanced Pattern • The majority of high-score control group subjects displayed a similar questioning pattern (even though no specific pattern was imposed

28. Analysis and Results: H5 • H5: SUPPORTED • Method: K-means cluster analysis • 7 of the 9 control group high-score control subjects - Balanced approach • 13 of the 20 experimental treatment group – Balanced approach High score control group subjects classified into a Balanced approach exhibited a questioning pattern similar in structure to the semantic model

29. Implications • The SSPI plays an important part in obtaining accurate interview-derived information during the RA phase • There is a definite relationship between certain semantic constructions (the type of question) and the accuracy of logical representations • Application of a SSPI may be independent of the experience level of the analysts • SSPI may serve as a source of reduction of errors associated with guessing or inference based on experience or logic • Application of the SSPI may reduce the number of technically feasible errors associated with the design of a DFD • Accurate representations of business processes may be closely related to the method of inquiry itself than with any given level of experience

30. Potential Limitations • Lab experiments cannot capture certain realities existent in the real world • Student subjects • Using the DFD as a single method of logical system representation

31. Future Directions for Research • Examine the applicability of the semantic structuring across various information gathering contexts (RAD, OOD) • Is the semantic structuring of an interview effective in the gathering of physical data elements for the construction of an ERD ? • Can semantic structuring be understood such that once the context of an information gathering activity is established, a model can be designed to maximize interviewer’s efforts ?

32. Questions ?