50 likes | 201 Vues
Desiderata for evaluation. Nancy Green, Kathy McCoy, David McDonald, Cecile Paris, Donia Scott. For any evaluation. It’s a scientific enterprise … Hypothesis-driven (appropriate methodology will fall naturally from this) Should position itself wrt the big picture
E N D
Desiderata for evaluation Nancy Green, Kathy McCoy, David McDonald, Cecile Paris, Donia Scott
For any evaluation • It’s a scientific enterprise … • Hypothesis-driven (appropriate methodology will fall naturally from this) • Should position itself wrt • the big picture • known results from related fields (e.g., human communication sciences) …. and should include a place for qualitative evaluation (vs. f-measures) • Replicability/rigour is important • Clarity of inputs/outputs, methods, materials, resources, constraints, limitations etc. • Important to identify the benefits of the outcomes • scientific • commercial • engineering
What we get for free … • Easier to publish if you can show that you’re doing (clear) evaluations • Will educate reviewers (and each other) on what a good NLG evaluation is • Given particular long-term goals, short(er) term steps/subtasks can also be identified, addressed and evaluated
Sharing: (good) Effect on the sociology of NLG • Shared anything (e.g., frameworks, resources) • will give students a place to start that isn’t a clean slate • establishes benchmarks (which in turn encourages people to go ‘one step more’ than they otherwise might have) • Shared tasks: can generate a buzz and some excitement
Shared tasks: (bad) Effect on the sociology of NLG • GALE is an example of the worst case …. • If it’s any good it has to have narrow focus, but this will bring limitations on broader progress (e.g., interactive narrative). • Marginalizing research areas/groups that don’t play, or other aspects of NLG (psycholinguistics, content-planning, etc.) is a danger to be recognised. • Needs to be convincing regarding whether it really address important problems and not just low-hanging fruit • DESIDERATA: Clarity of scope (what it includes/excludes) must be made explicit. • Competition is inevitable. • DESIDERATA: we need to ensure that we don’t get swept away with this and lose sight of the big picture.