Applying the Churchman/Ackoff Value Estimation Procedure to Spatial Modeling

1. Applying the Churchman/AckoffValue Estimation Procedure to Spatial Modeling Susan L. Ose MGIS Capstone Presentation Penn State University - World Campus 17 June 2008

2. GIS Models - A Refresher • GIS models provide • Decision Support • Prediction • Cost assessment, etc. • Weighting values • Emphasize/De-emphasize impact of inputs • Model adjusted via tweaking weights • How does an analyst determine weights? Figure source: Joseph K. Berry, University of Denver

3. Experts ? Compile Results Questionnaire Discussion Estimating Weights - Gathering Expert Opinions via Delphi Process • Anonymous Questionnaires • Statistical aggregation • Controlled feedback • Avoids groupthink • Consensus determines number of rounds

4. Questionnaire Format • The ideal questionnaire: • Guides the user through the weight estimation process • Focuses on expert knowledge - not statistics • Emphasizes relative not numeric values • Software module • Stores and compiles results • Allows easy distribution to expert group • Can be used in online meeting

5. Churchman/Ackoff Procedure • Focuses on relative value of layers • Compares layer against combination of other layers • Uses recursive procedures • Breaks down larger groups of layers for easier assessment • Recommended for group decision making • Project adapted method to browser-based software module

6. Churchman/Ackoff ProcedureStep 1: Rank Layers Layers are dragged and dropped into desired order

7. Churchman/Ackoff ProcedureStep 2: Assign Initial Weights Group 1 Layers divided subdivided into equal groups of no more than four. Control layer randomly chosen, assigned value of 50, added to each group Expert inputs initial estimate of weight Group 2

8. Churchman/Ackoff ProcedureStep 3: Judge Importance Expert chooses one of the three conditions and clicks on it. Algorithm adjusts values accordingly.

9. Churchman/Ackoff Procedure :Results Screen

10. Procedure Test Case • Potential for rain-fed agriculture in Liberia taking into account cost to market • Expert panel consisted of MDA Federal, Inc. employees experienced in modeling • Used two Delphi rounds • Compiled statistics • Ran model using average, high, and low values

11. Model Diagram

12. First Round Results • Reference layers received lowest scores • Proximity to water and land layers received highest scores • Model output using average score and 8 highest weighted layers consistent with results obtained using different methodology • Discussion focused on experience with procedure • Panel agreed on top 7 layers to include in final model

13. Second Round Results • Results reflected post-first round discussion • Access to water considered key, water layers grouped near top of ranking • Land cover shows potential land to be transformed to agriculture, thus higher score than previous • Higher variance in scores than in first round • Probably due to score distribution among less layers • First round lower variance may not have happened if expert could discard layers • Model more definitive around water features

14. Round Comparison

15. Model Results - Average 1st round 2nd round

16. Does the method work? • Model result viable • Considering one vs. many values more difficult than considering one vs. one (pairwise method) • Deliberately designed that way to challenge one's opinions • May frustrate participants - "weighting fatigue" • Random groupings confused some participants • Difficult to recall original ranking, weight • However randomness focuses judgment on subgroup • Can assist individual in examining own conclusions

17. What happens when someone joins the group later? • One expert did not attend discussions, used tool with minimal guidance • First round results changed slightly • Average Z-score highest of the group

18. Second Round with Additional Person • Pronounced difference in results • Greater difference in layer order in mid to lower layers • Average Z-score still highest of the group • Demonstrates importance of group discussion

19. Further Development/Study • Improve the software tool • Rework the math algorithm that calculates the weights • Offer opportunity to discard layers • Provide comparison of start/finish results • Provide progress bar • Application of the method • Categorical values within layers need to be rated first • Orientation meetings important • Conduct an experiment using both this method and a pairwise approach and compare results

20. Acknowledgements • Dr. Gregory Koeln, President, MDA Federal Inc. for sponsoring this project • Michael Schreiber, Webmaster, MDA Federal Inc. for transforming the procedure from paper to software • Dr. Todd Bacastow and Dr. Douglas Miller, Penn State World Campus, for their advice and guidance • Dr. Douglas Way, MDA Federal Inc. for his invaluable guidance • David Cunningham, Dr. Anna Oldak, Dr. François Smith, and Dr. Andrew Ralowicz, MDA Federal Inc. for their expertise • Gregory A. Ose, for his unfailing support