Surveying eValues: Experiences and Challenges

1. Surveying eValues:Experiences and Challenges Panel at the Bled Conference Monday June 5, 2006 Ronald Batenburg Institute of Information and Computing Sciences Utrecht University The Netherlands Bled panel Surveying e-values

2. Agenda for this panel 14:00-14:20 • Ronald Batenburg, Utrecht University • Georgios Karageorgos, European Commission, DG Enterprise & Industry (not present) 14:20-14:40 • Fernando Reis, European Commission, DG Eurostat 14:40-15:00 • Hannes Selhofer, Empirica Gesellschaft für Kommunikations- und Technologieforschung 15:00-15:20 • Vasja Vehovar, University of Ljubljana 15:20-15:30 • Closing remarks an discussion Bled panel Surveying e-values

3. Introduction • Why this panel? • My recent survey experience • The STILE project and beyond • My three main problems to tackle in the next 5 years Bled panel Surveying e-values

4. Example: evaluating the Dutch‘Go Digital’ program • Goal of this policy program: improving the ICT/e-business maturity of Dutch SMEs to meet the ‘European’ level • Sponsored/funded by the Dutch Ministry of Economic Affairs • Executed by intermediary organizations, providing (free) IT advise, consultancy, workshops, etc • Budget: € 40 Mio over 2001-2005, app. 20,000 Dutch SMEs participated Bled panel Surveying e-values

5. Evaluation questions • Was the program executed efficiently and effectively? • Did the program made a significant difference to the ICT-maturity of Dutch SMEs? • (how) to continue with Go Digital program for Dutch SMEs in the near future? Bled panel Surveying e-values

6. Research questions • How to measure ICT/e-business maturity and effectiveness in SMEs? • How to conclude if the program was successful, i.e.: • How to make a ‘ fair’ comparison between large, small and non-participants of the Go Digital program? • How to control for other determinants of ICT maturity and developments over time? Bled panel Surveying e-values

7. Result 1: Go Digital participantsare very ICT mature % yes Bled panel Surveying e-values

8. Result 2: Go Digital participantsare successful ICT users as well Fully disagree Fully agree Bled panel Surveying e-values

9. Reflection (1) • Standardized questions on ICT/e-business maturity were available by Dutch Statistics • Standardized questions on the added value, effectiveness and success ICT/e-business were not available, or ‘under construction’ • Hence: • Two new surveys among Dutch SMEs and ICT were conducted (n=900) • ‘Acceptable’ response rates (30%) among SMEs were realized only after major investments • The validity, reliability and potential future use of the new surveys are to be explored Bled panel Surveying e-values

10. Reflection (2) • The Ministry of Economic Affairs (and its policy partners) have little incentives to monitor survey methods and measurements • Dutch Statistics has little incentives to interfere or advise in survey methods and measurements of other survey projects • We as (academic) researchers do have an incentive to conduct new and customized survey methods and measurements • Hence: • Users, sponsors, researchers and statisticians of surveys are loosely coupled, not optimally cooperating or integrating • New ICT/e-business/Go Digital evaluations will lead to new survey projects Bled panel Surveying e-values

11. The STILE project 2001-2004 Goals: “to provide innovative methodologies and content for the statistical monitoring of the European labor market in the eEconomy” “a critical assessment of the European statistical system and to increase awareness in the international statistics and research community because of the current lack of measurement accuracy” Research partners: • Cork Telework Centre (CTC) and Central Statistical Office (CSO), Ireland • Institute for Employment Studies (IES), United Kingdom • CAMIRE Estadística y Análisis, S.L. (CAMIRE), Luxembourg • Institut für Arbeitsmarkt- und Berufsforschung (IAB), Germany • Istituto di Ricerche Economiche e Sociali (IRES), Italy • Organisatie voor Strategisch Arbeidsmarktonderzoek (OSA), The Netherlands • Institute of Sociology-Hungarian Academy of Sciences (ISB), Hungary Products: • Ramioul M., Huws U. & Bollen A. (eds.), Measuring the Information Society, STILE report, HIVA-K.U.Leuven, Leuven • www.stile.be • Questionnaire database • Digital toolkit Bled panel Surveying e-values

12. The 11 STILE enterprise surveys • Longitudinal or panel • Telephone, web, F2F, postal • Sample size 1,000-23,000 • Stratified Bled panel Surveying e-values

13. The STILE survey and questionnaire database Very informative overviews, however … • What would be the ‘best’ survey questions? • Or: when to apply which questions? • What would be the ‘best’ way to sample and approach enterprises for surveys? • Or: when to apply which approaches? • What would be the ‘best’ method to conduct survey enterprise questionnaires? • Or: when to apply which methods? Bled panel Surveying e-values

14. Again: the actors involved(derived from STILE project) Statisticians (academic) Researchers 1 3 2 Policy-makers Respondents Bled panel Surveying e-values

15. My Research Agenda Three main problems to be tackled in the next five years: • To improve the use of questionnaire and survey standards, by researchers and research organizations • To improve the monitoring of questionnaire and survey standards, by statisticians • To improve the awareness of questionnaire and survey standards, by policy makers Bled panel Surveying e-values

16. European Commission, DG Enterprise & Industry e-Business W@tch in the context of e-business related activities of DG Enterprise and Industry Georgios KARAGEORGOS 19th Bled eConference, Surveying eValues Panel Bled, 5 June 2006

17. Overview of the presentation • Background info and policy context(slides 3-5) • What is the e-Business W@tch(slides 6-13) • Lessons learned and future plans(slides 14-16) • Support / Background info (slides 17-20)

18. Main lessons learned • Fulfilling existing, real need=> Great interest from stake-holders (even university from Mexico)+support to continue; • Project unique (internationally?) in combining statistics (going one step further than ‘traditional’ surveys), qualitative analysis (incl. case studies) and policy recommendations; • Strong points = Timeliness, quality and transparency! • ‘Victims of success’: Growing expectations, demands and involvement BUTdifferent interests => Can NOT satisfy all! • Continuous trade-offs, e.g. increase sectoral focus => reduced survey’s capacity; • Increased burden for flexibility and coordination capacities.

19. Future Plans: a ‘new’ eBW (within the policy context described in slide 5) • Insights from on-going project, interim evaluation (by IDEA) and Cambridge conference(http://ec.europa.eu/enterprise/ict/conferences/cambridge.htm); • Emphasis on 3 themes: Impacts, impacts and impacts! • Focus on specific sectors, companies’ size classes & regions; • Approach: Maintaining quantitative (data) and qualitative (case studies) information collection, combined with high quality analysis, leading to identification of policy challenges; • ‘Common’ data coming from available official statistics (mainly Eurostat), complemented by purpose-made surveys; • More flexibility of implementation arrangements.

20. Thank you for your attention! • For moreinformation: • http://europa.eu.int/comm/enterprise/ict/index.htm • http://www.ebusiness-watch.org • entr-innov-ict-ebiz@cec.eu.int & info@ebusiness-watch.org • georgios.karageorgos@cec.eu.int