Chinese and American Forum on Legal Information and Law Libraries 中美法律信息与法律图书馆论坛

1. Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛 Using Technology to Track and Assess Student Learning Outcomes and Associated Performance Criteria July 10, 2019 Amy Emerson Ariel Scotese https://tinyurl.com/y65uwm7j

2. Ariel Scotese Access & Research Services Librarian, Assistant Director of the Legal Research Clinic, Adjunct Professor of Law Cornell University Law School Amy Emerson Director of the Law Library and Assistant Professor of Law Villanova University Charles Widger School of Law Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

3. Agenda • The ABA Standards • The Cornell Legal Research Clinic and its outcomes • Case Management Software • Cornell partnership with LegalServer • Clinic assessment survey • Alternative technology Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

4. ABA Standard 301 - Objectives of Program of Legal Education • A law school shall maintain a rigorous program of legal education that prepares its students, upon graduation, for admission to the bar and for effective, ethical, and responsible participation as members of the legal profession. • A law school shall establish and publish learning outcomes designed to achieve these objectives. Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

5. ABA Standard 302 - Learning Outcomes • A law school shall establish learning outcomes that shall, at a minimum, include competency in the following: • (a) Knowledge and understanding of substantive and procedural law; • (b) Legal analysis and reasoning, legal research, problem-solving, and written and oral communication in the legal context; • (c) Exercise of proper professional and ethical responsibilities to clients and the legal system; and • (d) Other professional skills needed for competent and ethical participation as a member of the legal profession.  • Interpretation 302-1: For the purposes of Standard 302(d), other professional skills are determined by the law school and may include skills such as, interviewing, counseling, negotiation, fact development and analysis, trial practice, document drafting, conflict resolution, organization and management of legal work, collaboration, cultural competency, and self-evaluation. • Interpretation 302-2: A law school may also identify any additional learning outcomes pertinent to its program of legal education. Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

6. ABA Standard 314 – Assessment of Student Learning A law school shall utilize both formative and summative assessment methods in its curriculum to measure and improve student learning and provide meaningful feedback to students. Interpretation 314-1 Formative assessment methods are measurements at different points during a particular course or at different points over the span of a student’s education that provide meaningful feedback to improve student learning. Summative assessment methods are measurements at the culmination of a particular course or at the culmination of any part of a student’s legal education that measure the degree of student learning. Interpretation 314-2 A law school need not apply multiple assessment methods in any particular course. Assessment methods are likely to be different from school to school. Law schools are not required by Standard 314 to use any particular assessment method. Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

7. ABA Standard 315 – Evaluation of Program of Legal Education, Learning Outcomes and Assessment Methods The dean and the faculty of a law school shall conduct ongoing evaluation of the law school’s program of legal education, learning outcomes, and assessment methods; and shall use the results of this evaluation to determine the degree of student attainment of competency in the learning outcomes and to make appropriate changes to improve the curriculum. Interpretation 315-1 Examples of methods that may be used to measure the degree to which students have attained competency in the school’s student learning outcomes include review of the records the law school maintains to measure individual student achievement pursuant to Standard 314; evaluation of student learning portfolios; student evaluation of the sufficiency of their education; student performance in capstone courses or other courses that appropriately assess a variety of skills and knowledge; bar exam passage rates; placement rates; surveys of attorneys, judges, and alumni; and assessment of student performance by judges, attorneys, or law professors from other schools. The methods used to measure the degree of student achievement of learning outcomes are likely to differ from school to school and law schools are not required by this standard to use any particular methods. Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

8. https://law.library.cornell.edu/lrc Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

9. Cornell Law School Learning Outcomes Upon completion of the program of legal education, Cornell Law School graduates will: Possess knowledge of the substantive and procedural law required for effective participation in the legal profession. Engage in legal research, analysis, and problem-solving. Graduates will be able to independently and critically analyze common law and statutory authority to… Communicate effectively in both oral and written form as counselors and advocates. Possess the practical skills fundamental to exceptional lawyering and client representation. Conduct themselves with the highest moral and ethical standards. Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

10. Cornell Legal Research Clinic Learning Outcomes • Conduct research in a real-world context • Gain familiarity with a variety of resources and research methods • Interview clients to ascertain legal issues • Become adept at analyzing, interpreting, and applying the law • Exercise legal and ethical judgment • Counsel clients through written and oral work product • Assist the local community through pro bono service • Understand obstacles barring access to information and access to justice Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

11. Case Management Software Traditional Functions • Managing Records • Contact Information • Notes • Correspondence • Deadlines • Etc. • Storing Documents • Conducting Conflict Checks • Recording Time • Billing Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

12. Abacus Law PracticeMaster Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

13. New Functions Needed • Capacity to capture an assessment of student learning outcomes • Faculty Assessment • Student Assessment • Ability to generate reports • External - for ABA • Internal - to track trends • Ability to archive content Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

14. Assessment in the Legal Research Clinic Clinical education involves developing “soft skills” in addition to understanding specific concepts Client interaction Advocacy Legal Research Assessing soft skills in the clinic is challenging Goal: Develop a tool that would allow the instructor to conduct ongoing assessment, as well as allow the students to easily engage in self reflection Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

15. Developing an Assessment Survey Decided to develop an assessment survey on LegalServer Survey would allow students to independently reflect on the questions Clinics are using LegalServer for timekeeping, case management, etc. Instructors can prepare reports Developing the Survey Questions Reviewed the Cornell Law School Learning Outcomes Reviewed the 26 Lawyering Effectiveness Factors Effectiveness Factors created from a study that created 26 indicators and 8 umbrella categories to predict the success of law school students when they practice law Provided evidence-based case for developing soft skills used in clinics Provided detailed factors for Cornell Law School Learning Outcomes Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

16. The Result • Survey of 26 questions divided into 8 categories • Categories are the 8 umbrella categories from the 26 Lawyering Effectiveness Factors • The questions synthesized from the Cornell Law School Learning Outcomes and the Effectiveness Factors • Students have 3 possible responses to the questions • I engaged but did not improve on this • I engaged with and improved on this • No opportunity to engage with this in the course • Conducted as mid-semester survey (formative assessment) and final survey (summative assessment) • Reporting Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

17. The Result – Mid-Semester Survey

18. The Result – Final Survey

19. The Result – Final Survey

20. Live Demo https://clsclinics.legalserver.org/ Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

21. The Result – Reports, Final Survey, Intellectual and Cognitive Goals (Category 1), Problem Solving (Question 3) Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

22. The Result – Reports, Final Survey, Research and Information Gathering (Category 2), Researching the Law (Question 1) Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

23. Benefits of the Survey Short Term Assess areas where students need additional instruction/assistance More detailed recommendations Provides additional opportunities for feedback Easy for students to complete Long Term Trends Course evaluation that is more detailed than self-reporting Grant writing Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

24. Application to Other Research Contexts • Use case management software to practice and discuss • The amount of time research takes, and how that impacts clients • Importance of naming conventions and organizing documents • Practical matters (e.g., billing, conflicts checks) • Questions can be developed using a similar method with the learning outcomes for the class • While LegalServer was appropriate in this context, could use other platforms to create and distribute the survey • Want a platform where students could submit forms individually • Report building functionality is useful for identifying trends • Primarily useful for semester-long research courses Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

25. Alternative Technology Google Forms https://www.google.com/forms/about/ Templates > Education > Assessment https://docs.google.com/forms/d/e/1FAIpQLSeumAc1yxRGXgu8tmqOP0As1SXPhVQQmBJ-3JtOefXxS74bJQ/viewform?vc=0&c=0&w=1 Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

26. References Marjorie M. Shultz and Sheldon Zedeck, Predicting Lawyer Effectiveness: Broadening the Basis for Law School Admissions Decisions, LAW & SOCIAL INQUIRY, JOURNAL OF THE AMERICAN BAR FOUNDATION, Vol.36, Issue 3, 620, 661 (2011) UNC School of Law, 26 Lawyering Effectiveness Factors available online at http://www.law.unc.edu/career/handbook/effectiveness.aspx Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛

27. Thank you! Amy Emerson Director of the Law Library and Assistant Professor of Law Villanova Law Library at Charles Widger School of Law 610-519-7023 amy.emerson@law.villanova.edu Ariel Scotese Access and Research Services Librarian, Associate Director of the Legal Research Clinic, Adjunct Professor of Law Cornell University Law Library 607-255-0980 aae42@cornell.edu Chinese and American Forum onLegal Information and Law Libraries中美法律信息与法律图书馆论坛