Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process

1. Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process Calvin Kam (PI) - Civil and Environmental Engineering Martin Fischer (PI) - Civil and Environmental Engineering Sadri Khalessi (PI) - Statistics DeviniSenaratna (RA) - Statistics

2. A set of statistical models for innovative: benchmarking, decision-prioritization and prediction of performance and adoption, in Virtual Design and Construction projects

3. Problem Statement Why?

4. “What Gets Measured, Gets Done” –Peter Drucker What do AEC firms need to “Get Done” using VDC? Minimal waste in the process Shorter total duration Tight synchronization between design & fabrication Last responsible moment benefit from VDC More alternatives evaluated earlier

5. AEC may not see the value in KPIs for VDC Ref: McGraw and Hill (2012): The Business Value of BIM Ref: Kam, McKinney, Xiao & Senaratna (2013): The Formulation and Validation of the VDC Scorecard

6. AEC firms monitor KPIs for VDC less than other industries VDC in AEC: 40% Apparel industries: 80% Ref: Esin, et al. (2009) Are KPIs and Benchmarking actively used among organizations of the Swiss apparel industry to assure revenue? Ref: Kam, McKinney, Xiao & Senaratna (2013): The Formulation and Validation of the VDC Scorecard

7. What information do AEC firms want from the KPIs?

8. AEC project teams want to know where they stand relative to other AEC project teams Frequency of Integrated Meetings (Monthly) Where do we stand relative to other projects? Extent of Stakeholder Involvement (% of Stakeholder involved in VDC) One data-driven benchmark that combines KPIs?

9. AEC project teams want to quantify and understand interdependencies among KPIs for VDC Frequency of Integrated Meetings How much more involved would the stakeholders be? How much more frequently would the stakeholders attend the integrated meetings Extent of Stakeholder Involvement

10. Our on-going research using the VDC Scorecard Tool A survey based benchmarking tool 108 cases and counting 50+ indicators 13 countries 11 types of facilities • More cases for statistical modeling • Adding information to existing cases • Data-driven benchmarks • Data-driven prioritization of KPIs What more do we need? Ref: Kam, McKinney, Xiao & Senaratna(2013)

11. VDC Scorecard measures many types of KPIs 1 Score VDC Scorecard 4 Areas Planning Adoption Technology Performance 10 Divisions Objective Standard Preparation Process Organization Quantity Quality Maturity Coverage Integration Documentation Guideline Budget Project Stakeholders Stakeholder Survey Reduced Design Errors Depth Level of Detail Communication Metric Benchmark Tool Broader Context Training Post Occupancy S. Reduced Labor Breadth Model Use Life Cycle Interoper- ability Etc. Etc. Etc. Etc. 50+ Measures

12. Decisions based on anecdotal learning’s only won't be as strong as making decisions that are also supported by statistically established insights Stakeholder satisfaction Benchmark Breadth Meetings Model Use Life Cycle Reduced Design Errors Involvement Project Safety Depth Reduced Labor Stakeholders 1542 Measure Combinations

13. “What gets measured, gets done “How can AEC firms benefit from this advantage ? Raw Data (KPIs) = Statistical analysis of KPIs + VDC expert opinion Information

14. What we propose What?

15. We will benchmark with a variant of Principal Component Analysis (PCA) Picks up the most important signals Few KPIs for benchmarking and ranking Many KPIs Many Features Few Features Ref: neural.cs.nthu.edu.tw

16. Clustering to find features about “projects like mine” Features of projects that are different to mine Features of projects that are similar to mine

17. Clustering groups of projects with similar KPIs together Used in Recommendation Engines Methods: K-means, Hierarchical, Two-Step Clustering Projects with similar features will group together Feature Combination Score 2 Numerical statistical measures built using a collection of KPIs Feature Combination Score 1

18. Observation for Cluster Groups 1 and 3: GOOD Technological Coverage but POOR Technological Integration • Common Drawbacks : • Interoperability problems • Inconsistent Level of Detail for model uses • 3. Only native formats used for model exchange Performance Adoption

19. Statistical predictive models to identify interdependencies • Factor Analysis • Structural Equation Modeling • Classification Trees • Canonical Correlations • Correspondence Analysis For Prediction and Prioritization

20. Structural Equation Modeling will model interdependencies Example of a Structural Equation model for Job Dis-Satisfaction Ref: Lee, Kleinman, (2003): Statistical choices and apparent work outcomes in auditing

21. How are we going to do it? How?

22. Timetable • Test data • Using expert opinions • Descriptive Analysis and assumption checks • Statistical model based on SEM, FA, CHAID • Data cleaning and cross checking • Organizing the data for analysis • Surveying108 cases (Online – Web survey) • Addition of KPIs to Scorecard • Testing of statistical methods on sample data

23. Dissemination of Findings • Publications • New findings to be posted on vdcscorecard.stanford.edu • Journal paper publication Two Industry Workshops (Kam, Khalessi, Fischer) • November CIFE workshop • February CIFE workshop Contributions towards: • Statistical KPI wiki for AEC • CIFE members (AEC firms) • National BIM Standards and McGraw Hill Smart Market Report

24. The data-flow process of the proposed study Statistical Model Prioritized, Customized feedback based on KPIs Model Analyses KPIs and their Correlations VDC Scorecard AEC Professionals

25. Goals Short Term Goal: Statistical analysis of KPIs for VDC decision-making Long Term Goal: Automation of this process for faster decision-making; and to make decision-making more consistent across an organization

26. References • Abeysekera, S. (2006) Multivariate methods for index construction • Alarcón, L., Mourgues, C., O’Ryan, C., and Fischer, M. (2010) Designing a Benchmarking Platform to select VDC/BIM Implementation Strategies • Bloom, N. and Van Reenen, J. (2006) Measuring and explaining management practices across firms and countries • Bloom, N. and Van Reenen, J. (2007) Why do management practices differ across firms and countries? • Esin, C., Von Bergen, M., and Wüthrich, R. (2009) Are KPIs and Benchmarking actively used among organisations of the Swiss apparel industry to assure revenue? • Kam, C., McKinney, B., Xiao, Y., Senaratna, D. (2013) • Kam, C., Xiao, Y., McKinney, B., Senaratna, D. (2013) • McGraw and Hill (2012) The Business Value of BIM. • Lee, Kleinman, (2003): Statistical choices and apparent work outcomes in auditing • Li, W., Fischer, M., Schwegler, B., Bloom, N., Van Reenen, J. (2012) • Zevenbergen, J., Gerry, J., and Buckbee, G., (2006) Automation KPIs Critical for Improvement of Enterprise KPIs.

27. Don't you want to find out about which VDC practices will give you the best results?

28. Thank You!