(19a) Engaging Faculty In Assessing and Improving Students' Critical Thinking

This session will provide information about a new tool for assessing students’ critical thinking skills.  The CAT instrument is the result of a ten year nation-wide project (funded by NSF) to refine and disseminate an interdisciplinary instrument to assess critical thinking/real-world problem solving skills.  Our efforts have been focused on (1) developing an instrument to assess critical thinking that has high validity for faculty across disciplines and for experts in learning science, (2) creating an assessment tool that involves faculty in the scoring process to help them understand student weaknesses, and (3) creating an assessment system that supports broad-based efforts to improve student learning.  The session will provide an overview of the development process, research findings, and the unique ways the CAT instrument supports faculty involvement in program improvement efforts.  The session will provide numerous opportunities for discussion and exploring issues related to assessing and improving students’ critical thinking.

National polls indicate over 90% of the faculty in this county think critical thinking is the most important part of undergraduate education (Derrick Bok, president emeritus Harvard University, 2005).  In fact, colleges and universities across the country are looking for better ways to evaluate student learning in this area in response to state or national mandates for accountability.  Although the acquisition of factual knowledge is important, most would argue that an undergraduate education should prepare individuals to do more than recite facts, it should prepare people to think critically, solve problems and effectively communicate ideas.  Our institution has been exploring existing methods and developing new tools (with support from the National Science Foundation NSF) to assess these broader thinking skills for the past nine years. 

A diverse group of faculty worked in teams and as members of a larger group to identify important critical thinking skills and develop questions/materials that would measure those skills.  It was agreed that the test would involve mostly essay answers to help assess communication skills and leave opportunities for creative answers to questions that don’t always have a single correct response.  The essay format would also involve faculty in the scoring of exams and hence promote more interest in improving critical thinking skills. 

In 2004, NSF funded Project CAT at TTU.  This project has allowed us to work with seven other diverse institutions across the country to use, evaluate, and help refine the CAT instrument (the University of Hawaii, the University of Texas, the University of Southern Maine, the University of Colorado, the University of Washington, Howard University, and Madisonville Community College).  The results of our work with these institutions were extremely encouraging and indicate the CAT instrument has high face validity for faculty across the country. 

The CAT instrument has been tested and refined through repeated administrations over a six-year period across a variety of institutions.  The test has exhibited good criterion validity when compared to other performance measures and correlates with several key measures from the National Survey of Student Engagement.  The test has also exhibited good scoring reliability and is sensitive to changes that occur from freshmen to senior level students and individual courses. 

In 2007, NSF funded a three year national dissemination grant for the CAT instrument that focused on training trainers from other institutions to use the instrument and conduct scoring workshops on their own campuses.  We will be discussing these training workshops, our experiences working with other institutions, and the unique features of the CAT instrument that lend itself to closing the loop between assessment and the improvement of student learning.  The workshop will also stimulate discussion on broader issues related to the assessment of student learning.

In 2010, NSF funded a four year broader dissemination of the CAT instrument that provides opportunities for broad institutional involvement across the country and that engages currently funded research projects in evaluating the effects of new educational practices on improving students’ critical thinking.  These efforts are expected to provide synergies that facilitate the identification and dissemination of effective practices.    

National polls indicate over 90% of the faculty in this county think critical thinking is the most important part of undergraduate education (Derrick Bok, president emeritus Harvard University, 2005).  In fact, colleges and universities across the country are looking for better ways to evaluate student learning in this area in response to state or national mandates for accountability.  Although the acquisition of factual knowledge is important, most would argue that an undergraduate education should prepare individuals to do more than recite facts, it should prepare people to think critically, solve problems and effectively communicate ideas.  Our institution has been exploring existing methods and developing new tools (with support from the National Science Foundation NSF) to assess these broader thinking skills for the past nine years. 

A diverse group of faculty worked in teams and as members of a larger group to identify important critical thinking skills and develop questions/materials that would measure those skills.  It was agreed that the test would involve mostly essay answers to help assess communication skills and leave opportunities for creative answers to questions that don’t always have a single correct response.  The essay format would also involve faculty in the scoring of exams and hence promote more interest in improving critical thinking skills. 

In 2004, NSF funded Project CAT at TTU.  This project has allowed us to work with seven other diverse institutions across the country to use, evaluate, and help refine the CAT instrument (the University of Hawaii, the University of Texas, the University of Southern Maine, the University of Colorado, the University of Washington, Howard University, and Madisonville Community College).  The results of our work with these institutions were extremely encouraging and indicate the CAT instrument has high face validity for faculty across the country. 

The CAT instrument has been tested and refined through repeated administrations over a six-year period across a variety of institutions.  The test has exhibited good criterion validity when compared to other performance measures and correlates with several key measures from the National Survey of Student Engagement.  The test has also exhibited good scoring reliability and is sensitive to changes that occur from freshmen to senior level students and individual courses. 

In 2007, NSF funded a three year national dissemination grant for the CAT instrument that focused on training trainers from other institutions to use the instrument and conduct scoring workshops on their own campuses.  We will be discussing these training workshops, our experiences working with other institutions, and the unique features of the CAT instrument that lend itself to closing the loop between assessment and the improvement of student learning.  The workshop will also stimulate discussion on broader issues related to the assessment of student learning.

In 2010, NSF funded a four year broader dissemination of the CAT instrument that provides opportunities for broad institutional involvement across the country and that engages currently funded research projects in evaluating the effects of new educational practices on improving students’ critical thinking.  These efforts are expected to provide synergies that facilitate the identification and dissemination of effective practices.