COSC2148/2149/2150 Research Methods Reading and Assessing Research Literature

1. COSC2148/2149/2150 Research MethodsReading and Assessing Research Literature James Harland james.harland@rmit.edu.au

2. Generating PDF from Word Despite the lab exercise, I believe the easiest way is: • Download and install Open Office • Open .doc file with Open Office • Save as PDF • Close .doc file

3. Overview • Reading is a fundamental research activity • Assessing what is read can be difficult • Assessment is required of active researchers • 1 paper published = 3 papers reviewed, on average • Need to know where and when published • May need to investigate publication details • Need to know publishing culture • Be rational, skeptical and humble

4. Publication Types • Book • Journal • Conference • Workshop • Technical Report • Web page • Thesis • Manuscript • ...

5. Book Published because someone believes many copies will be sold. • Author/s detailed development of ideas • Gathering of scattered research results into one volume • Collection of papers by various authors from a meeting • `Handbook on X' series by invited experts • Commemorative volume dedicated to an individual • PhD thesis published as a book • Not necessarily just for the academic market

6. Book • Often become standard references. • Publication can be slow. • Reviews are often written by other experts. • Good sources of basic concepts and background material. • Rarely have up-to-the-minute results. • Refereeing process varies from rigourous to virtually none.

7. Journal Published by major companies exclusively for academia. • Viewed generally as the most important kind of publication • Often seen as an archive of research • Published at regular intervals (monthly, quarterly) • Available in most (good) libraries • No bounds on number of papers • No real-time refereeing constraints • Increasingly publishing special issues on topic X • Main customers are university libraries

8. Journal • Rigourous refereeing by (at least) two experts • Only accepted for publication when editor is satisfied • Can take many iterations to succeed • Generally no bounds on pages or time to (re-)submit • Anything that is appropriate will eventually be published • Can sometimes have good but unimportant papers • Any good paper should eventually appear in a journal

9. Conference Proceedings published for distribution to attendees, and sometimes more widely. • Probably the most common form of publication • Not always refereed (but generally are in CS) • Proceedings sometimes published well after conference • Tend to have up to date material • Quality more variable than journals • Worldwide, regional, national, local, ... • Main papers, poster abstracts, short papers, system demonstrations, invited papers, panel sessions, ...

10. Conference • Generally 3-5 referees, not all from the programme committee • Often use numeric scores to simplify the process • Real-time refereeing constraints exist • ``One-shot'' chance at publication • Programme committee selects best-scoring papers • No chance to re-evaluate papers later • Reports generally not as detailed as for journals • Common for acceptance rate to be under 50%

11. Workshop • More informal than conferences • Deadlines closer to actual meeting • More specialised audiences • Refereeing generally more lightweight • Often self-selecting • Usually has ``cutting edge'' material • Quality again more variable • Proceedings often not generally available

12. Technical Report • Earliest of all publications • Not refereed in any sense • Used to disseminate information quickly • Often same paper submitted to a conference or journal • Trend away from these per se to links on personal Web pages A typical evolution is TR to Workshop to Conference to Journal. Any of these steps may be missing.

13. Manuscript • Sometimes you may get a personal copy of some material not widely available. • This is generally from a trusted source only. • Examples including incomplete material from colleagues or working notes from a supervisor. • Generally does not include justification, background or discussion.

14. Theses • Paper (or book) submitted by a student towards a degree • PhD theses can be hundreds of pages • MSc theses around one hundred pages • Honours/minor thesis around fifty pages • Read in detail by at least two examiners • Designed to show that a student is ``qualified'' • Often read and criticised in greater depth than other publications • Possibly more arcane material than journal or conference publications • Don't have to be important, just good

15. Reading Papers • Read the abstract first • If still interested, read the introduction and conclusion • If still interested, read the rest of the paper • If still interested, look for follow-up papers • It is not necessary to understand everything in detail to get some benefit

16. Evaluation • What? • When? • Where? • How did I hear about it? • What's new? • What is/is not obvious? • What is/is not justified? • What confidence do I have in my judgement?

17. Evaluation Approaches you could use: • Discuss the paper with colleagues • Ask your supervisor about it • Work through techniques from paper • Reproduce results from the paper • Ask the author/s about the paper • Look up references mentioned in the paper