 |
Assessment of this project will be multi-faceted, done both internally by the PI's as well as by external experts and panels, and founded on scientifically based evidence. Three research questions were formulated in the project goals (section 1.2), and the assessment methods used allow this project to address these questions.
Question 1: How does emphasizing student development rather than knowledge acquisition affect student learning, and what is the impact on effectiveness, engagement, efficiency, and retention?
To answer this question, student learning, retention, and transfer will be measured at different levels of Bloom's Taxonomy. In order to assess the impact of the developmental model, courses will be taught by faculty for at least one semester using the teaching methods currently employed, then transitioned to the developmental model. Assessment will be done before (pre) and after (post) transitioning to the developmental model using three different techniques:
To assess whether students master shallow levels of learning outside of class, the performance of students on formative quizzes (1) built into the development model (section 2.2, 4a) will be measured.
To assess achievement on the higher levels of Bloom's Taxonomy, portfolios [12, 77] created by students (2) that contain both individual and team work will be analyzed by faculty both internal and external to the project using a rubric. The rubric will let evaluators score how well students are able to apply concepts to a problem, analyze data, and evaluate their own understanding. The rubric will be based on an existing critical thinking rubric developed by the OSU Assessment Office, and the director of this office will coordinate the reviewers. Stipends for external reviewers are budgeted.
To assess students' ability to integrate concepts (near transfer) a case analysis test (3) will be given as part of the final exam to measure how well students are able to transfer what they learned to different context. The results of the case assessment test will be assessed externally by a graduate student supervised by Dr. Lundeberg at Michigan State University (see letter of support).
Pre-post analysis of this data will determine how effective the developmental model is compared to lecture courses and how student development changes as more courses are added to the program. The performance on formative quizzes (1) will be compared to performance on examinations (3) and portfolios (2) to determine how competence on different levels is related to student development.
To determine changes in student engagement, the project will administer a version of the National Survey of Student Engagement (NSSE) being developed at the National Academy of Engineering (4) to test engagement of student engineers [78]. Student retention across the curriculum, enrollments, and grade distributions (5) will be tracked throughout the project. An existing alumni survey administered by the OSU Assessment Office will track students who leave the program. To assess student confidence in their own learning gains a Student Assessment of Learning Gains (SALG) [13] survey (6) designed in cooperation with Dr. Lundeberg will be given at the conclusion of each course. To ensure SALG results are accurate, student responses will be corroborated by transcripts from student focus groups run by psychology graduate students. The results of the engagement survey (4) will be tracked over time and correlated with retention and enrollment data (5) to determine how the changes to the program impact engagement and retention. Retention data (5) will be compared with results from summative examinations (3,5) and engagement surveys (6,4). Data on what classes students take (5) will be tracked over the project to determine changes in the time to a degree (efficiency).
Question 2: How does removing legacy material from the curriculum impact student learning?
Retention of content knowledge will be measured using a comprehensive test (7) covering material from all required ECEN courses. This test, scored on a rubric, will include short answer questions, conceptual diagnostic questions [79], and case analysis questions. Scores on questions from different classes will be compared to determine how reducing content coverage affects different aspects of learning compared to lecture courses. To relate curricular changes and metacognition, a Motivated Strategies for Learning Questionnaire (8) (MSLQ) [80, 81] will be given in the sophomore and senior years. To determine how students' self-perception and confidence relates to the material learned the results from the comprehensive examination (7) and formative quizzes (1) will be correlated with SALG data (6) to determine shifts in learning over the course of the project.
Question 3: What are the changes in students' learning in classes taught by graduate students who use the developmental model, and what is the impact on graduate students' professional development?
To measure how students' learning changes when taught by graduate students instead of faculty, assessment data from formative (1) and summative (7) examinations as well as students self-perception of learning gains (6) will be compared for courses taught by graduate students and by faculty. To determine changes in graduate students professional development, the portfolios created by graduate students (section 3.2) at three levels of development (no experience, course only, course + teaching experience) will be compared by Dr. Turns, an external consultant, to measure the effectiveness of the best practices course and actual teaching experience. A letter of support is attached.
To assess the impact this project has on preparing students for a broad spectrum of careers, a panel of faculty and business leaders will be formed and supported by the dean of the engineering college (see attached letter of support). An annual review of this project (9) will bring together this panel, students, and the participating faculty to provide feedback on whether the curriculum changes meet the broader goals of the university, and those who employ our graduates. This panel will report to the CEAT dean and ECEN department head (Teague)
Figure 7: The nine assessments used give a comprehensive picture of this project.
Coordination of assessment will be managed by Dr. Bryant in OSU's College of Education. Surveys (6,4) and focus groups (6) will be conducted and/or analyzed by a graduate student working for Dr. Bryant. This graduate student will prepare assessment reports each semester for the participating faculty. Tracking enrollment data (5) is currently done by an ECEN staff member as part of her regular duties. The summative examination (7) will consist of questions contributed by all ECEN faculty, and the exam will be given by Dr. Cheville and assessed by Dr. Bryant. Development of formative quizzes (1) and final exams (3) will be guided by individual faculty in consultation with Dr.s Bryant and Lundeberg, while creation and implementation of the formative quizzes will be handled by OSU's Faculty Support Center (Elliott) to minimize the faculty time investment needed. The PI's received approval from OSU's Internal Review Board for this assessment program during the first phase of this grant, and Dr. Bryant will be responsible for ensuring this certification remains up to date and compliance with the humans subject section of the Grant Proposal Guide.
These assessment metrics used give a comprehensive assessment of the this project span the spectrum of student vs. program and summative vs. formative as shown in figure 7. Each assessment metric is identified by the number referred to previously, and the symbol surrounding the number indicates the primary type of information it provides.
|
