A preliminary analysis of AAAI-99 submissions

1. A preliminary analysis of AAAI-99 submissions Devika Subramanian Rice University

2. Distribution of 400 submitted papers by area

3. Trends in submissions • Multi-agent/agents, KR, reasoning/UAI, planning, learning are the largest constituencies. • Robotics papers are coming into AAAI; however these are mobile robot papers. • Neural computation papers are not a large part of the submissions.

4. Overall acceptance statistics • Before SPC meeting: • Accepts (105) • Undecided (50) • Rejects (255) • After SPC meeting • Accepts (104) • Conditional accepts (5) • Rejects (290) • 15 papers accepted with 1 A/2M or R. • 3 papers rejected with 2 A/1 strong R. • Every other accepted paper had two As. • 32 accepted papers had 3 As! (just under 30%)

5. Distribution of 109 accepted papers by area

6. Trends in accepted papers • Search, planning, neural computation, and information extraction/information retrieval have higher than average acceptance rates. • Data on information extraction/information retrieval and neural computation unreliable because of small sample size.

7. Other trends in papers • AAAI still attracts the best work in planning, constraint satisfaction, search, multi-agent systems and KR.

8. Other trends in papers • The best work in machine learning, uncertainty, KDD, neural computation, natural language and Web agents (not including multi-agent systems) is not being submitted to the conference. • The work in mobile robotics submitted to the conference is not competitive with the work represented at the top vision/robotics conferences..

9. Any new ideas in the papers? • Proverb: a system that solves NY Times crosswords. A tour-de-force integration of ideas in AI. • Hybrid approaches to collaborative filtering. • New extensions to Graphplan and Satplan. • Active learning: analysis and implementation. • Integrating the fields of constraint satisfaction and classical planning.

10. Remarks gathered from SPC • Number of NLP papers submitted to AAAI is increasing; they are of much better quality than in years past. Still not the very best papers (which tend to go to ACL), but solid work at the intersection of statistics/machine learning and NLP. • Neural computation papers should be reviewed on a special track as they were this year (with two special SPC members Giles and Sun) even though the actual number of submissions from that community was small this year.

11. More remarks from the SPC • Need to develop mechanisms to get the best work in machine learning, UAI, agents, KDD, robotics and neural computation to be submitted to conference. • Many were concerned about the fact that most accepted papers were incremental advances. • signs of a maturing field? Or self-selection among submissions because of reputation as archival conference? • Mechanisms to detect and encourage revolutionary work among the submissions not working well. • reflects fundamental split in community on what a significant result is, and about the extent of evaluation needed to “prove” that an idea works.

12. Some SPC statistics Acceptance rates vary widely among SPC. Partially explains variation in acceptance rates across sub-areas. Mean=26.37 stdev=10.22

13. 110 PC members

14. SPC statistics for AAAI-98 Acceptance rates vary widely. Mean=30.95 stdev=12.33

15. 204 PC members

16. Issues to consider for AAAI-2000 and beyond • Size of SPC and PC to get consistent reviewing standards. • Ways of attracting “work-in-progress” to AAAI and to set good evaluation guidelines for them. • Reconsider need for an SPC meeting. • Paper assignment to reviewers benefits from a manual component. Consider providing electronic access to paper abstracts and to assignment software so chairs can teleconference and do reviewer assignment. • Consider accepting few papers for plenary presentation (say 20) and have all papers presented at poster sessions. • How to exist and cooperate/compete with speciality conferences.