Planning and Running Research and Development Projects

1. Planning and Running Research and Development Projects (IFI8109.DT) Practices of preparing research project proposalsSpecifics of development projects

2. Plan for 16.03 • General comments on the research project proposals. • General quality criteria in deciding on a research (and development) project. • Ranking of quality criteria for R&D projects. • Discussion of CALLCORP project proposal. • Specifics of development projects.

3. General comments on the research project proposals

4. Major problems • The aim/objective can not be “to examine/study/analyse”. • Inconsistency in using terminology: aim/objective/goal, child/student etc. • Importance of the study is not explained enough/convincingly. • Previous work of the applicant is not explained (“participated in a project” does not reveal the content of work). • List items should be numbered, not bulleted (for referencing purposes). • Avoid research questions that assume yes/no answers.

5. Major problems (cont.) • Budget items are not explained. If there are mobility costs, then a concrete conference should be indicated. • Adequacy of selected methodology is not grounded. • Some parts of texts are in wrong sections: background in Summary, analysis of previous studies in Methodology etc. • Volume of work (amount of empirical data, number of interviews etc) is not described. • Wording is too general (“preparatory works”) or too specific, understandable for a small group of experts. • Low level publications.

6. General quality criteria in deciding on a research (and development) projectConditions that should be satisfyied before initiating a R&D project.

7. The basic principle • Support is given to the applicants that already have achieved a high (international) level, not for reaching this level. • Workable strategy for junior researchers: • Start as a member in a group of high level researchers, conducting joint studies and publishing joint articles.

8. Quality criteria • Satisfaction of formal qualification requirements. Why? • Dealing with a (possibly widely) acknowledged problem. • Existence of a novel original idea for solving the problem. • Project acceptance probability is large enough, i.e. the quality criteria compared to potential competitors are well met. Example: PUT, http://www.etag.ee/rahastamine/uurimistoetused/personaalne-uurimistoetus/put-taotlusvoor-2015/2015-a-taotlusvooru-tulemused/, no ICT, almost no humanities! (in Estonian). NB! In research projects – differently from development projects - risk analysis is in general not required (highly innovative research is almost always risky).

9. Ranking of quality criteria

10. Ranking of factors • Existence of high level publications (level and number). Why? • The value of h-index: the largest natural n, such that the author has n 1.1 level publications and each on them is cited at least n times. NB! Google Scholar (http://scholar.google.com/) can be used for finding the h-index. • Actuality (topicality) of the research problem, elaboration of the project (including the analysis of prior research, correctness and readability of the text), and the relevance of the methodology. • Experience in completing similar projects. • Supervision of graduate students.

11. Ranking of factors of research proposal – subjective view • Quality of research questions and/or hypotheses. • Expected amount and level of new knowledge, for example: • Answering previously raised open/unsolved problems • Elaboration of a new framework, model, methodology or algorithm • Improvement of some framework, model, methodology or algorithm • Identifying relations between different phenomena • Showing inaccuracy of some previous results (of other researchers). • Methodology. • Consistency of the research field with institutional and national development plans/strategies. • Earlier cooperation between the project team members. • International and/or internal cooperation.

12. Initation of a research project – recommendations • Be a (co-)author in as many as possible high level articles and a member of project teams. The position in the authors list counts. • Follow strictly the formal requirements set for project proposals if there are any. • Before applying a research project, check the competence indicators of previous successful applicants.

13. Additional factors of development project proposal • Impact of the project – both nationally and internationally – and the width of the application area of the project’s outcomes. • Sustainability of the project, including the market demand, export potential and social impact. • Technological level and innovativeness of a project, including conformity of the project’s conception to the objective. • Project management, including reality and measurability of project’s objective/outcome, quality of the action plan and budget, experience of the team in the project area.

14. Additional factors of development project proposal • Aspects related to implementation of the project’s outcome, including for example: • capability of implementation • Intellectual property issues. • Indirect/directly not measurable impact (for example on economic development or on the development of sectoral technologies).

15. Methods of taking into account the quality criteria

16. Consideration of formal criteria • Check out the documentation of the measure and follow them strictly. • Check out the evaluation criteria and take them into account in composing the project proposal. Ask if necessary! Example of a failure: Humboldt Foundation. • Determine the critical success factors of the project and your bottlenecks (biggest problems). • Pay special attention to the bottlenecks when composing the project proposal, involving additional competence if necessary.

17. Consideration of non-formal quality criteria • Try to get clarity on current assessment practice: • Consult with members of the decision-making bodies (in particular about the aspects that have greater weight in evaluation). Example: the age of PI-s. • If possible, study the previous project proposals and their reviews. • If possible, participate in decision-making bodies. • Identify your greatest strengths (in comparison with other potential applicants) and how the project can benefit of it. • Base on your strengths, integrating their description and their need for completing the project in the application text.

18. (Research based) development projects

19. Definition of a development project • We define development projects as projects which: • are based on some research, • are aimed at satisfying some needs of a wider target group. • Examples: • Development of a tool that bases on a new concept / framework or that has some completely new features (example – Dippler). • Development of a new method of use of some tool (example – usage of tablet PC in the classroom). • Development of learning exersices and / or materials for application of some new methodology (example – e-textbook). • Examples of a multi-level development. • NB! Testing some methodology does not fit under this definition.

20. Discussion of CALLCORP project proposal • Name 3 factors why this project belongs to the category of (research based) development projects. • Name 3 major strengths of the project proposal. • Name 3 major weaknesses of the project proposal. How would you deal with the weaknesses? Summative assessments of reviewers:

21. Planning development projects – specific aspects

22. Development projects – Description of the outcome • Expected outcome should be described as specifically as possible, both in quantitative and qualitative terms. It should be possible to assess the adequacy of project proposal (including resources and the risks), based on the description of the outcome. • Sector-specific, according to the terminology used in the sector. • The outcome of software development projects is normally described by the list of requirements; in the case of larger projects the description of architecture or some other elements (for example, any description of the design elements) may also be included. • Examples: Dippler University textbook

23. Development projects – Impact of the project outcome • Impact can be: • Economic, measured in money or eficiency. • Social, measured, for example, by the rate of involvement of retired people in social networking. • Political (example – the percentage of citizens participating in elections). • Educational (example – the percentage of school leavers). • By certain universal indicators. • What else? • NB! Evaluation of the outcome is more influenced from personal understanding and preferences of reviewers.

24. Development projects – Cooperation with the target groups • The purpose of cooperation: ensure that the outcomes of the project meets the needs of potential users, and will subsequently be widely used. • Principles: • Representatives of potential users are involved in the project. Why? • The outcomes of the project will be given to the representatives of users for testing as early as possible. • Foresee the possibilities to change the outcome if necessary.

25. Development projects - Intellectual property • The purpose: establish the conditions for usage of the project outcome. • Often the financier does not prescribe the rights and conditions for usage of the outcome; these should just be explicitly stated. • NB! For public sector funded projects the conditions of use are normally given (example: EU funded BeSt programme for development of learning objects and e-courses).

26. Development projects - risks • Risk is a feature of an outcome of an activity (the potential of an undesirable outcome); it describes the level of uncertainty for obtaining a predefined outcome. Risk is a probabilistic measure: the simplest formula – M/N. • The objective of risk management: keep the realization probability of risks cost-effectively on an acceptable level. • Some possible techniques for determination of risks: • analysis of documents (including web based) • brainstorming • Delphi-technique • interviewing customers • SWOT-analysis • diagrammatic analysis (analysis of cause-and-effect diagrams)‏ • NB! Usually people are the biggest sources of risks, then information/data (non-adequate, missing, leaking etc) and then infrastructure. • .

27. Development projects – universal indicators • Universal (cross-cutting) indicators concern the overall objectives of the society, such as the effect of the project on: • environmental protection • promotion of equal opportunities (by sex, race, language etc) • information society promotion • regional development promotion • civil society development promotion • the intensification of cooperation in the region (for example in the Baltic states). • NB! So far, the role of the universal indicators has been relatively small in assessing the projects.

28. Home assignment • Study the LearnMix project application (sent indvidually). • Compose a form based LearnMix evaluation, according to the Guide. • The form – Evaluation_form_application_project.doc • The guide – Evaluation_guide_application_project.doc. • Team work. Specify the aim and outcome of the development project: • Determine the general aim of the project. • Determine the main target groups of the project. • Conduct the needs analysis. • Conduct the market analysis (availability of similar products/services). • Describe the expected outcome.

29. Next class: Wednesday, March 30, at 14.15Topics: Case analysis: Learning Layers (T.Ley)Reviewing development projectsPractice of running R&D projects

30. Example – the position in the authors list • Tomberg, V., Laanpere, M., Ley, T., Normak, P. (2013). Sustaining Teacher Control in a Blog-Based Personal Learning Environment. International Review of Research in Open and Distance Learning. 14(3): 109-133. • Normak, P., Pata, K., Kaipainen, M. (2012). An Ecological Approach to Learning Dynamics. Journal of Educational Technology & Society, 15(3), 262-274. • Not significant in earlier years: • Knauer, U., Normak, P. (1990). Morita duality for monoids, Semigroup Forum, Vol. 40, 39-57.

31. Speech of Estonian children – almost missing disjunction Logic operators and quantifiers at age 10 to 11 Erika Matsak, 2007

32. Example: multi-stage development projects • Research in development of intellectual abilities of children • Development of teaching methods for enhancing intellectual abilities of children • Development of teaching materials used in applying teaching methods • Conducting a pilot training (and an action research).

33. Example: training athletes • Development of a formal model of a kind of sport (sprint, swimming, skiing, for example). • Interpreting the model in a real environment and development of analysis software. • Building a dynamic model of human muscles activated in sport activity. • Development of training program to train selected muscles.

34. Example: incompetent evaluation of a development project • The objective: Development of a web-based and competence-driven learning environment for personal learning which is open to all educational institutions in Croatia. • Reviewer 1: This is an extremely innovative solution to make teaching more effective and more personal, the application could also be used in workplace learning. Particularly worthwhile is the possibility for automatic generation of feedback in assessment (the score – 4). • Reviewer 2: It is not clear achievement of what learning outcomes the tools support: focusing on the assessment of forgetting indicates that the outcomes of the project aim primarily at increasing the learners knowledge, not so much at increasing their ability to apply their knowledge, and ability to analyze and synthesize new knowledge from existing one (the score – 1,5).

35. Summative assessments of CALLCORP • Reviewer 1 – grade 5: • Excellent, with respect to the role of the Estonian language in Estonia. Moreover, the proposal and teams are very well connected with other European efforts about natural language treatment. • Reviewer 2 – grade 3: • Due to the development of language technology distinguishes positively from most other proposals. Innovativeness is automatically secured because development is possible only in Estonia. If the results will be implemented in the broader language learning and will allow to build commercial services upon it, then the project is fully justified.

36. Example: requirements for DIPPLER • Use of Semantic Web and Web 2.0 technologies (RSS, folksonomies, web services, widgets, embedding, semantic annotation, ontologies). • Modularity and open source code (GNU LGPL or BSD license), which would allow community based development. • Different levels of use: suitable for both beginners and advanced users. • Compatible with learning technology standards and specifications (LOM, QTI, SCORM, LD, CommonCartridge etc). • Interoperability with other TEL tools (blogs, wikis, repositories). • Support for Single Sign-on and authentication with OpenID and ID card. • Multilingual and easily localized. • Possibility to add individual tools and services, personalize the look and feel. • Possibility to apply different business models. • Powerful Learning Analytics possibilities. • Compliance to general software requirements: scalable, secure, well-documented, adaptable, easy installation and administration.

37. Example: the architecture of DIPPLER/IVA2

38. Example: cooperation with the target groups • An IT department of a university was asked to develop a software allowing students to register over the Internet to the courses. As the Study Information System of the university did contain all the necessary databases the task seemed to be relatively easy and this was given to a novice developer without any list of requirements from academic departments. After launching huge amount of problems emerged, including for example the following: • as not all time-tables were available in the internet,the students had to check the time-tables in academic departments; • As only users of the university computer network had opportunity to register over the web (authentication!), the departments were forced to introduce a parallel registration; • no control was performed whether all prerequisite courses are already completed. • Only 1,5% of the students used this service. An additional analyst was involved and the project was repeated from scratch.

39. Example: university textbook • Determination of the target group. • Description of possible usage (textbook/web support/training). • General principles of composing the textbook (sh innovation). • The structure of the textbook and description of the chapters. • Description of the structure of chapters and principles of their composition. • Example of a completed chapter or a part of it. • Description of web support (both content and functionality). • The approximate volume of a textbook.

40. Example: impact to the economic development • The need for determining the level of Estonian proficiency level is extensive: knowledge of Estonian – the only official language in Estonia – is included in a large part of job requirements (in the public sector almost exclusively). It concerns a relatively large proportion of the population, because 31,3% of people living in Estonia are non-Estonians (2011). • The software that will be developed during the project can also be used for development of different services supporting, for example: 1) unsupervised learning of Estonian, 2) composition and running courses of Estonian for immigrants, 3) testing Estonian proficiency level in specific contexts or business areas. • Relating each proficiency level with concrete requirements (vocabulary and constructions) would make learning Estonian more targeted and effective.

41. Example: risks NB! Risks are changing in the course of a project and need for regular review. More examples: http://www.headfirstlabs.com/books/hfpmp/hfpmp_ch11.pdf

42. Example: universal indicators (a program for learning Estonian) • Environmental protection: “green IT” (paper-free learning). • Promotion of equal opportunities: knowing Estonian (the only official language in Estonia) increases employment opportunities for non-Estonians. • Promotion of information society: using the learning program increases ICT skills of users. • Promotion of regional development: The unemployment rate in Ida-Virumaal is by far the highest in Estonia (20,3%in 2011). Knowing Estonian would improve the opportunities to find a job, and integrate the region better into the common economic space of Estonia. • Promotion of development of civil society: knowing Estonian supports better involvement of citizens in societal processes, and therefore in development of civic society.

43. Example: Estonian classification of research publications • Categories that are counted for doctoral students: • 1.1 – Scholarly articles indexed by Thomson Reuters Web of Science and/or published in journals indexed by ERIH (European Reference Index of the Humanities) categories INT1 and INT2 and/or indexed by Scopus. • 1.2 – Peer-reviewed articles in other international research journals with an ISSN code and international editorial board, which are circulated internationally and open to international contributions; articles of ERIH category NAT. • 3.1 – Articles/chapters in books published by the publishers listed in Annex (including collections indexed by the Thomson Reuters Book Citation Index, Thomson Reuters Conference Proceedings Citation Index, Scopus). https://www.etis.ee/Portal/Classifiers/Index.

44. Example: trivial hypothesis • The field of pedagogical pattern language is broad and well-developed, but the proposal gives little indication as to even the kinds of patterns that might be suitable for the language they wish to create. It is not much of a scientific hypothesis to say that "Pedagogical pattern language is an effective, valid and reliable instrument for analysing and formalizing e-learning processes."

45. Example: calculation of h-index (Google Scholar) • On equalizer-flat and pullback-flat acts, 54 citations • Purity in the category of M-sets, 26 citations • Monoids over which all flat cyclic right acts are strongly flat, 21 cit. • Soft ontologies, spatial representations and multi-perspective explorability, 18 citations • An Ecological Approach to Learning Dynamics, 13 citations • Flatness properties of monocyclic acts, 12 citations • On Noetherian and finitely presented polygons, 9=7+2 citations • Topological S-acts: preliminaries and problems, 7=5+2 citations • Morita duality for monoids, 7=6+1 citations • Congruence Compact Acts, 6 citations

46. How to measure impact? • In monetary terms (tools for cheaper production). • Time savings (quicker production). • New functions/procedures possible. • Greater simplicity • Higher quality • Broader target/user group • …