Innovation data collection: methodological procedures

1. Innovation data collection:methodological procedures ECO - UIS Regional Workshop on Science, Technology and Innovation (STI) Indicators Tehran, Iran 8-10 December 2013 Luciana Marins, UIS

2. Ch. 8 OM - Survey procedures • Guidelines - collection and analysis of innovation data; • Comparableresults over time and across countries; • Particular circumstances may require other methodology comparability.

3. The survey approach • The “subject” approach: • Innovative behaviour and activities of the firm as a whole; • The “object” approach: • Specific innovations (“significant innovation” of some kind, firm’s main innovation).

4. Populations (1) • The target population: • Innovation activities in the business enterprise sector (goods-producing and services industries); • Minimum: all statistical units with at least ten employees; • Classification by size: • Small: 10-49; • Medium: 50-249; • Large: 250 and above.

5. Populations (2) • Statistical unit: • Size cut-off point: • Source: 2012 UIS Innovation Metadata Collection

6. Populations (3) • The target population (cont.): • Classification by main economic activity: • (National industrial classification system); • ISIC; • NACE.

7. Populations (4) • The frame population: • Units from which a survey sample or census is drawn; • Based on the last year of the observation period for surveys; • Ideal frame = up-to-date official business register NSOs; • If the register forms the basis for several surveys (innovation, R&D, general business), the information can be restricted to innovation.

8. Survey methods (1) • Mandatory surveys increase response rates; • Census or sample surveys? • Sample surveys - representative of target population (industry, size, region) stratified sample; • Census - costly but unavoidable in some cases.

9. Survey methods (2) • Completion: • Survey type: • Source: 2012 UIS Innovation Metadata Collection

10. Survey methods (3) • Domains (sub-populations): • Subsets of the sampling strata; • Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active; • Guidelines: • Same statistical units and classifications; • Consistence of the methods for results calculation; • Documentation of deviations in data treatment or differences in the quality of the results from the domains.

11. Survey methods (4) • Sampling techniques: • Stratified sample surveys: size and principal activity; • Sampling fractions should not be the same for all strata; • Cross-sections: standard approach - new random sample for each innovation survey; • Panel data: alternative/supplementary approach.

12. Survey methods (5) • Suitable respondents: • Methods: e.g., postal surveys, web-based questionnaires, personal interviews; • Unit’s most suitable respondent - very specialised questions that can be answered by only a few people; • Try to identify respondents by name before data collection starts.

13. Survey methods (6) • Data collection method: • Source: 2012 UIS Innovation Metadata Collection

14. Survey methods (7) • The questionnaire: • Pre-test; • Simple and short; • Order of the questions; • Questions on qualitativeindicators - binary or ordinal scale; • International innovation surveys - attention to translation and design; • Short-form questionnaires - units with little/no innovation activity previously reported.

15. Survey methods (8) • Combination of Innovation and R&D surveys: • Reduction in the overall response burden; • Scope for analysing the relations between R&D and innovation activities; • Increase in the frequency of innovation surveys; • Country experiences - it is possible to obtain reliable results for R&D expenditures; • Longer questionnaire; • Units not familiar with the concepts of R&D and innovation may confuse them; • Different frames for the two surveys.

16. Survey methods (9) • Survey combination: • Source: 2012 UIS Innovation Metadata Collection

17. Survey methods (10) • Guidelines for conducting combined surveys: • Questionnaire: two distinct sections; • Smaller individual sections; • Comparison of results from combined and stand-alone surveys should be done with care - surveying methods should be reported; • Samplesextraction from a common business register.

18. Estimation of results (1) • Weighting methods: • Weighting by the inverse of the sampling fractions of the sampling units, corrected by the unit non-response; • If a stratified sampling technique with different sampling fractions is used, weights should be calculated individually for each; • Based on the number of enterprises in a stratum; • International and other comparisons:same weighting method.

19. Estimation of results (2) • Non-response: • Unit non-response: reporting unit does not reply at all; • Item non-response: response rate to a specific question - % of blank or missing answers; • Disregarding missing values + applying simple weighting procedures based on the responses received assumes that respondents andnon-respondents are distributed in the same way biased results; • Possibility: imputation methods.

20. Estimation of results (3) • Item non-response: • Unit non-response: • Source: 2012 UIS Innovation Metadata Collection

21. Presentation of results • Descriptive analysis: no generalisation of results; • Inferential analysis: conclusions about target population; • Variance for the results: (average) values for innovation indicators and their coefficients of variation and/or confidence intervals; • Results presentation:metadata (data collection procedure, sampling methods, procedures for dealing with non-response, quality indicators).

22. Frequency of data collection • Every 2 years; • If not economically feasiblefrequency of 3 or 4 years; • Specify an observation period; • The length of the observation period for innovation surveys should not exceed 3 years nor be less than 1 year.

23. Annex A - 5. Methodological issues for developing country contexts (1) • Information system specificities: • Relative weakness of statistical systems: • Absence of linkages between surveys and data sets; • Lack of official business registers; • Involvement of NSOs; • When lacking, basic variables about firms’ performance can be included in the innovation survey.

24. Annex A - 5. Methodological issues for developing country contexts (2) • General methodological considerations: • Survey application: • In-person; • Trained personnel; • Questionnaire design: • Sections can be separated to allow different persons in the firm to reply them; • Guidance/definitions; • Language and translation of technical terms.

25. Annex A - 5. Methodological issues for developing country contexts (3) • General methodological considerations: • Frequency: • Every 3 to 4 years (e.g., timed to CISrounds); • Update a minimum set of variables every year; • Purpose of surveys; • Clear questions; • Adequate legislative base; • The results should be published and distributed widely.

26. Basic innovation indicators:examples

27. Howdo we measure innovation? (1) • Indicators - definition: • Statistics and data, often gathered through specialised surveys, are the building blocks from which indicators are constructed; • An indicator can be defined as something that helps us understand where we are, where we are going and how far we are from a specific goal. Therefore it can be a sign, a number, a graphic; • An indicator quantifies and simplifies phenomena and helps us understand complex realities. Source: International Institute for Sustainable Development / Adapted from Blakley, W. (2012). Providing and calculating innovation indicators. Cape Town, South Africa. ASTII/HSRC/UIS Workshop. (PowerPoint Presentation)

28. How do we measure innovation? (2) • Indicators - definition: • Basic indicators: based on “one question”; • Composite indicators: combine answers to several questions in order to examine a number of policy-relevant factors and better capture the diversity of innovative firms.

29. Innovation indicators - examples (1) • Product or process innovation: • % of firms that implemented product innovation • % of firms that implemented process innovation • % of firms that implemented product or process innovation (innovative firms) • % of firms that developed in-house product or process innovation • % of firms that implemented new-to-market product innovation

30. Innovation indicators - examples (2) • Product or process innovation: • % of firms that implemented product innovation

31. Innovation indicators - examples (3) • Product or process innovation: • Source: 2011 UIS Pilot Data Collection of Innovation Statistics

32. Innovation indicators - examples (4) • Marketing or organisational innovation: • % of firms that implemented marketing innovation • % of firms that implemented organisational innovation • % of firms that implemented marketing or organisational innovation

33. Innovation indicators - examples (5) • Marketing or organisational innovation: • % of firms that implemented marketing or organisational innovation

34. Innovation indicators - examples (6) • Inputs: • Total expenditures on innovation (as a % of total turnover) • Expenditure on innovation by type of expenditure (as a % of total expenditure on innovation) • % of firms that performed R&D • % of firms that performed R&D on a continuous basis

35. Innovation indicators - examples (7) • Inputs: • % of firms that performed R&D

36. Innovation indicators - examples (8) • Key policy-relevant characteristics: • % of firms that were active on international markets • % of firms that co-operated with foreign partners on innovations • % of firms that co-operated with universities or other higher education institutions • % of firms that received public financial support for innovation • % of firms that applied for one or more patents • % of R&D-performing firms that co-operated with other institutions

37. Innovation indicators - examples (9) • Key policy-relevant characteristics: • % of firms that co-operated with universities or other higher education institutions

38. Innovation indicators - examples (10) • Key policy-relevant characteristics: co-operation • Source: 2011 UIS Pilot Data Collection of Innovation Statistics

39. Innovation indicators - examples (11) • Key policy-relevant characteristics: • % of R&D-performing firms that co-operated with other institutions

40. Final remarks • Data collected with innovation surveys are a important component of comparativestudies about countries’competitiveperformance; • Strategically important for policy-makers; • Data confidentiality; • Data reliability.

41. Thank you! http://www.uis.unesco.org l.marins@unesco.org