2004
(Last Revised Summer
2007)
Omnibus
Assessment Plan:
Documenting
a Culture of Learning
|
The
primary purposes of assessment are to determine whether St. Ambrose University is
currently meeting its goals and objectives for teaching and learning,
and to improve the quality of teaching and learning in the future.
At
times, students will be asked to participate in the assessment process
by completing specialized assessment activities. These assessment
activities can be completed in a variety of settings (such as the
classroom, at home, or at a testing center) as well as in a variety of
ways (such as online, paper-and-pencil, in small or large groups)
depending upon the activity. All students, regardless of class level or
enrollment status, are asked to assist with this important process.
(Approved by the St. Ambrose University Educational Policies Committee
Fall 2002) |
|
|
Academic Assessment Plan Evaluation
Proposed Institutional Activities
Annual Departmental Assessment Activities
Assessment of Non-academic/Co-curricular Institutional Departments
Appendix A: Possible Annual Assessment Report
Form
Appendix B: Information on Standardized Instruments for Institutional
Assessment
Appendix C: Alumni Survey
Appendix D: General Education
Philosophy, Goals, and Objectives
St. Ambrose University has been involved in the process of assessing student
learning outcomes for many years. Historical
data indicate that the institution has been collecting some systematic data
concerning our students since, at least, as early as the late 1940s.
Archival data has recently been discovered that documents that St.
Ambrose University participated in the National College Sophomore Testing
Program from 1947 to 1957 and that the University was testing first-year
students as early as 1950.
Unfortunately, it is not clear how these data were used to improve student
learning outcomes.
Activities
related to assessment began in a more coordinated way beginning in 1991 with
the formation by the Provost of a task force on mission values and assessment.
Committees that were active in examining how best to assess our
students are identified immediately below.
These activities culminated in the St. Ambrose University 1995 Academic
Assessment Plan (see http://web.sau.edu/assessment/1995plan.htm
for a reproduction of much of this report), which was approved by North
Central Association of Colleges and Schools Commission on Institutions of
Higher Education in 1995.
Many
of the activities that were identified in the 1995 assessment plan were, and
continue to be, carried out. At
the same time, many of the activities have been modified and it had been some
time since we have taken a serious look at our assessment plan for the
institution. Re-envisioning
assessment at St. Ambrose University was timely, as 1) in order to ensure and
document educational excellence, we needed to know whether our students were
attaining the outcomes that we had identified for them both in and out of the
classroom, 2) we were beginning the process of planning for our visit from the Higher Learning Commission of the North Central Association of
Colleges and Schools during the 2007/2008 academic year, and 3) we wanted to
help ensure institutional integrity and excellence.
This latter reason for re-envisioning assessment was deemed critical, as we
were beginning the process of assessing not only academic outcomes, but
also non-academic and co-curricular areas of the University.
St.
Ambrose University is doing what it can to ensure that assessment is not a
stagnant process, and this ongoing revision of the assessment plan was an additional
step. The St. Ambrose University
assessment web site (see http://web.sau.edu/assessment/) is used to help
ensure that timely communications are made with the University and outside
communities and that assessment information is as up-to-date as possible.
Dr. Paul Koch, then Dean of the College of Arts and Sciences, was
appointed Assistant Vice President of Academic Affairs for
Assessment fall 2004. Dr. Koch was
appointed Associate Vice President for Assessment and Institutional Research
July 1, 2005. Dr. Koch works
with Mr. Bradley Thiessen, Instructor in the Math Department and Assessment
Coordinator, and the Assessment Task Force in helping to champion, design, and
carry out assessment activities with input from those impacted by such
assessment.
Although assessment activities can benefit from individual champions, it is ultimately up to the faculty, staff, and students at St. Ambrose University to ensure that we are achieving and measuring our educational and organizational goals and objectives. We need to continually remind ourselves that it is all about learning, and that should not be necessarily thinking about just an effective Culture of Assessment, but an effective Culture of Learning. Learning is an ever changing and dynamic process. As such, it is intended that the assessment plan will be a dynamic process that will change as necessary to help ensure an effective learning environment for all at St. Ambrose University.
Return to Table of Contents
Dr. Ann Freeberg, former Vice President
for Institutional Research, brought a group of interested faculty members
together approximately 3 years ago to begin the process of examining
assessment process at the institution. This
task force served in an advisory capacity to Dr. Freeberg. Dr. Koch reconstituted the task force during the 2002/2003
academic year. Members of the task force
during the 2003/2004 academic year included: Jeremy Burke, Brenda DuBois,
Rhys Fullerlove (student member), Hope Gardina, Martin Hansen, Michael
Hustedde, Paul Koch, Brenda Peters, Ann Preston, Jeanne Roller, Lew Sanborne,
Mary Taylor, Brad Thiessen, and Jim VanSpeybroeck.
Mr. Bradley Thiessen, the Assessment Coordinator, began chairing this task force with the
2004/2005 academic year, and will continue to define the role of this task
force.
The previous mission of the task force was
changed from the follwing.
Mission: The mission of the ad hoc St. Ambrose University Assessment
committee is to evaluate current university-wide assessment activities; prepare a
systematic and institutional model for university-wide assessment; and implement
a systematic
university-wide assessment program.
to:
Title:
1) serving as a consultative body to the Office of Assessment and Institutional Research,
2) sharing resources and models about assessment and evaluation with the University community, and
3) evaluating the status of university-wide assessment and evaluation activities.
Members of the Advisory Board are appointed by the Associate Vice President for Assessment and Institutional Research in consultation with the Assessment Coordinator and will be chaired by the Assessment Coordinator. Current members include both faculty and staff members from various areas of the campus.
Return
to Table of Contents
In
its “Statement on Assessment of Student Academic Achievement,” the North
Central Association of Colleges and Schools states:
“The
program to assess student learning should emerge from and be sustained by a
faculty and administrative commitment of excellent teaching and effective
learning; provide explicit and public statements regarding
the institution’s expectations for student learning; and use the information
gained from the systematic collection and examination of assessment data both
to document and improve student learning. A strong assessment program is
founded on a plan that is widely accepted and routinely updated, it is
ongoing, and it is related to other planning and budgeting processes.” (Handbook of Accreditation, 1997)
This statement was used to guide an evaluation of assessment at St. Ambrose conducted by Mr. Thiessen in 2004/2005.
From this statement, one can extract the following criteria for evaluating St.
Ambrose University’s 1995 Academic
Assessment Plan:
A
good assessment plan:
1)
is linked to the university mission and goals,
2)
specifies the logistics of the assessment process,
3)
is integrated into university operations,
4)
demonstrates a high level of commitment to the assessment process, and
5)
will lead to improvement.
Criteria #1: The assessment plan
is linked to the university mission and goals
Evidence:
Appendix A in the Assessment Plan summarizes the development of SAU’s mission and
beliefs statements. Appendix D
describes the process used to create general education goals and objectives
based on the mission statement. Appendix
C documents departmental mission statements, goals, and objectives.
Strengths:
General Education goals are specified for student attitudes/values as
well as the traditional academic domain (skills and content understanding).
Each set of goals seems to align with the university mission and belief
statements. The goals are
comprehensive and manageable in number. The
results of a 1993 General Education Goals Survey demonstrate the high-level of
input faculty had in developing these goals.
The departmental mission statements and goals align with the university
mission statement; yet remain unique for each department. The vast majority of departments have a manageable number of
goals and objectives (most have between 4-8 goals/objectives).
Areas
for Improvement: As they are
stated in the Assessment Plan, the
university goals are somewhat nebulous. Explicit
objectives under each goal do a better job of defining the expected student
outcomes. These objectives, while
not stated in the Assessment Plan,
are documented on the university website (see
http://web.sau.edu/GeneralEducation/goalsobjectives.htm
for the most recent and approved version).
Criteria #2: The
assessment plan specifies the logistics of the assessment process
Evidence:
Table 3 provides guidelines for the development of departmental
assessment plans and a schedule of program reviews.
Table 4 lays out the process for implementing these assessment plans.
Tables 5 and 6 provide summaries of assessment measures and methods
used by departments. Table 8
displays the process of implementing the general education assessment plan.
Table 9 summarizes the methods used to assess general education goals.
Table 10 displays a timeline that specifies who is responsible for
collecting assessment data, to whom the results are distributed, and intended
uses of assessment results. Appendix
B provides a description of assessment resources available to each department.
Appendix C displays the assessment methods used by each department
along with plans for collecting, analyzing, and using the assessment results.
Appendix E contains the General Education Assessment Action Plan.
Strengths:
The guidelines for developing departmental assessment plans are clear
and comprehensive, as evidenced by the uniformity of the plans in Appendix C.
The flowchart in Table 4 clearly displays the assessment process –
from data collection and analysis to the use of results for improvement. The assessment process is clearly tied to departmental
program reviews. For general
education assessment, those responsible for data collection and dissemination
are identified in Table 10 along with the intended uses of assessment results.
That same information is stated for each department in Appendix C.
Areas for Improvement: In evaluating the logistical information contained in the Assessment Plan, two areas for improvement emerge. First, more information needs to be provided about the assessments that will be administered to students. While Table 9 does display the assessment methods used in the General Education Program, it does not provide any specific information about these assessments. The plan could be improved by including a brief description of each assessment along with a rationale for selecting each measure (as several departments included in their plans in Appendix C). The plan would also be improved by providing some information on how the selected assessments align with university goals and objectives. [in process]
The assessment plan states that the technical quality (reliability and
validity) of the information obtained from the assessments is enhanced by the
use of multiple measures. Table 5
seems to support that statement, showing that every department implements at
least three methods of assessment. Closer
examination of the assessments used by each department in Table 6 uncovers
that while departments are using multiple assessments, many departments are
using only one or two modes (or types) of assessment. Instead of using two or more assessments that are very
similar to each other, departments should use a greater variety of direct and
indirect measures to improve the quality of information obtained from the
assessment process.
The second area for improvement relates to the details of the General
Education assessment plan. The
plan does not specify which students are to be assessed, who exactly is
responsible for the administration of the assessments and the
analysis/communication of the assessment results.
The timeline should include more detailed information on which
assessments will be administered each year along with the resources needed to
implement the assessment process each year.
Criteria #3: The
assessment plan is integrated into university operations
Evidence:
The discussion of the Assessment Plan on pages 6-18 provides evidence that the assessment
plan is integrated into university operations.
Table 4 shows the link between departmental assessment and program
reviews. Appendix C shows the
specific plans for each department.
Strengths: By tying
departmental assessment plans to program reviews, departments are encouraged
to make assessment an ongoing process rather than a special activity.
The integration of assessment into regular university operations is
also strengthened by the decision to keep assessment-related decisions in the
existing faculty committee structure (Educational Policies, University-wide
Planning, and General Education Program Committees). The plan also demonstrates its integration into university
operations through its extensive use of existing sources of information (much
of the information can be obtained through in-class activities) and networks
for communication (Assessment Committee updates and the assessment page on the
university website).
Areas for
Improvement: While
the plan is tied closely to the program review, there is no clear annual
check-up on departmental assessment plans.
This could lead departments to implement their assessment plans only
before their scheduled program reviews (making assessment an event rather than
a process). Also, many
departments have listed an unrealistic number of assessments, which will
always make assessment seem to be an extra burden.
Departments should only use assessments if they will be able to
collect, analyze, and use the results for improvement.
The university needs to institutionalize the assessment process,
ensuring departmental assessment processes are ongoing and creating a climate
of assessment that is integrated into normal university operations.
Criteria #4: The
assessment plan demonstrates a high level of commitment to the assessment
process
Evidence:
Table 1 provides a Chronology of Assessment Activities at SAU from
1991-1995. The process of
developing departmental assessment plans is described on page 15, while the
development of the General Education plan is described on page 21.
Appendix A details the process used to analyze the university mission
and belief statements. Appendix C
provides the assessment plans developed by each department.
Appendix D describes the process used to specify General Education
Program goals and objectives.
Strengths: The evidence
listed above demonstrates that faculty members had a high-level of input in
the development of both the General Education and departmental assessment
plans. The Strategic Plan Action Team, Faculty Development
Committee, Educational Policies Committee, General Education Assessment Task
Force, Mission Values Committee, and the Provost’s Task Force on Mission
Values and on Assessment all provided leadership in the development of the Assessment
Plan. Faculty members who
provide input into the development of the Assessment
Plan will most likely have a higher level of commitment to the assessment
process.
Areas
for Improvement: The
low response rate to the 1993 General Education Goals Survey (22 responses)
indicates a low-level of faculty involvement in developing university goals
and objectives. To ensure a
high-level of commitment, the implementation of departmental assessment plans
should be evaluated more frequently.
Criteria #5: The
assessment plan will lead to improvement
Evidence:
Table 4, a flowchart of the departmental assessment process, shows the
process departments use to discuss assessment results and plan program
improvements. Table 8
demonstrates how assessment information is used to plan improvements in the
General Education Program. Table
10 lists intended uses of the results of the General Education assessments.
Table 11 displays the relationship between academic assessment and
strategic planning. Many departments explicitly stated the intended uses of
assessment results in Appendix C.
Assessment activities, along with the communal wisdom of the faculty,
have led to some change in the General Education Program at St. Ambrose
University. See
http://web.sau.edu/GeneralEducation/
for a summary of changes that have occurred in the General Education Program
over time.
Strengths:
An intended use has been stated for each assessment used to track
progress towards General Education goals and objectives.
These intended uses include value-added measures (tracking progress
over time) as well as snapshot measures (measuring attainment of goals at one
point in time). Information from the assessments is used in planning,
budgeting, program review, and faculty development.
The plan specifies how assessment information will be aggregated and
disseminated to faculty. The plan
also provides faculty with opportunities to respond to recommended program
improvements.
Areas for Improvement: While the General Education process seems manageable, many of the departmental assessment processes seem overly ambitious. If too many assessments are used, fewer resources are available for a thoughtful analysis of assessment results (a good amount of data may not be used). Departments may want to use fewer assessments, keeping only the assessments that will provide the most meaningful and useful information. The assessment plan may also want to indicate how assessment results will be shared with students. There is a trend for current departmental assessments to be more doable (see http://web.sau.edu/assessment/deptassessments.htm for current departmental assessments).
Return
to Table of Contents
March 2000, updated March 1, 2002, the Higher
Learning Commission provided a new framework for examining the assessment of
academic achievement (Assessment of Student Academic Achievement: Levels of
Implementation; these guidelines are likely to be updated by fall 2004).
Levels of Implementation is a tool to assist institutions and site
visit teams in understanding and strengthening an institution’s assessment
activities.
The following provides a brief outline (without
specific patterns of characteristics) of Levels of Implementation. Institutions
can rate themselves within each category identified below as being at 1) LEVEL
ONE: Beginning Implementation of Assessment Programs, 2) LEVEL 2: Making
Progress in Implementing Assessment Programs, and 3) LEVEL 3: Maturing Stages
of Continuous Improvement. We may
find it useful to compare our assessment plan against the patterns of
characteristics provided for each level, and this task will be started by the
Assessment Coordinator and the Assistant Vice President for Academic Affairs
summer 2004. Information will be
reported to the Assessment Committee and the Faculty during the 2004/2005
academic year.
I.
Institutional Culture
A.
Collective/Shared Values
B.
Mission
II.
Shared responsibility
A.
Faculty
B.
Administration and Board
C.
Students
III.
Institutional Support
A.
Resources
B.
Structures
IV. Efficacy of Assessment
Return to Table of Contents
The following three standardized instruments will
be administered on a 3-year rotating cycle beginning with the 2004/2005
academic year. By administering
these instruments on such a cycle, we not only get comparisons with national
norms and cross-sectional data within a year, but we will also get
longitudinal data for some of our students who persist from their first-year
to graduation. Administration of
these instruments will be the responsibility of both the Assistant Vice
President of Academic Affairs for Assessment (Paul Koch - Associate Vice
President of Assessment and Institutional Research as of July 1, 2005) and the
Assessment Coordinator (currently Brad Thiessen).
Results will be analyzed the summer following administration, summary
papers will be written, and results will be disseminated to university
constituents the following fall.
It is the belief of those coordinating assessment
activities that it is the conversation among faculty members and others
regarding assessment results that will be of some of the greatest benefit to
St. Ambrose University. If
warranted, additional assessment activities, such as focus groups or tailored
surveys, will be conducted to probe further into the results that are obtained
from these standardized instruments. See
Appendix B for more detailed information about these instruments.
A concerted effort has been made over the last
several years to ensure that the results of assessments were distributed to
faculty members and to engage the St. Ambrose University faculty in a
conversation about the results. It
is clear, however, that less has been done to communicate with our students
about the results of assessment activities.
Informal and formal conversations with student leaders indicate that
students share many of the concerns that faculty and staff members have about
the outcomes that the University has identified.
As such, efforts will be made to enhance communication with students
using the student government association as a vehicle.
Academic Profile – Short Form
2004/2005 Academic Year
150-200 first-year students during Welcome Week
100-150 graduating students during the fall
semester
National Survey of Student Engagement
(administered; see
http://web.sau.edu/assessment/NSSE_Benchmarking_Presentation_2006_Data_Web.pdf
for a summary of results from 2006)
2005/2006 Academic Year: Random sample selected and administered via
mail/online by test publishers
Faculty Survey of Student Engagement (administered)
Beginning College Survey of Student Engagement
(new instrument administered fall 2005; see
http://web.sau.edu/assessment/BCSSE%202005-06%20Selected%20Results.pdf
for a summary of the results)
NSSE will be administered again during the 2008-2009 academic year.
Student Satisfaction Inventory (see http://web.sau.edu/assessment/St.AmbroseUniversity4-2005.html for a summary of the results from 2003; see http://web.sau.edu/assessment/St.%20Ambrose%20University%20-%20Composite%20-%205-2007.html for a summary of the results from 2007)
Institutional Priorities Survey
(see
http://web.sau.edu/assessment/St%20Ambrose%20University%20(IPS)%20-%204-2007.html
for a summary of results from 2007)
Adult Student Priorities Survey (see http://web.sau.edu/assessment/St.%20Ambrose%20University%20(ASPS)%20-%204-2007.html for a summary of results from 2007)
2006/2007 Academic Year
Administered in selected classes - this was actually completed online instead of paper-and-pencil.
The SSI will be administered again during the 2009/2010 academic year.
One area of assessment that St. Ambrose University
had not paid much attention to was assessing alumni of our undergraduate
programs. Graduate programs,
especially those with specialized accreditation, have a longer track record of
assessing their alumni. As such,
a new alumni assessment instrument (see Appendix C) was piloted fall 2003 with
our 2002/2003 graduates (undergraduate only).
This instrument was adapted with permission from an instrument used by
the University of St. Thomas in St. Paul/Minneapolis, MN. It intended that this instrument will be used at both 1-year
and 5-year intervals, and that follow-up assessments will be conducted when
warranted.
Multiple years of alumni results can
now be found at
http://web.sau.edu/assessment/ReportsPage.htm.
Additional
Areas for Assessment:
There are many
other areas that the institution will have to develop assessment practices for
including, but not limited to…
Writing
Learning
Health and Wellness
Information/Technology Literacy
In addition to the use of standardized tests, St. Ambrose has
developed and implemented, as of spring 2007, an additional technique for
assessing student learning outcomes in the the content area of the
humanities, social sciences, natural sciences, and creative arts. The
humanities were involved spring 2007, and it is anticipated that the other
content areas will be assessed on an rotating annual basis with the natural
sciences participating spring 2008.
Return to Table of Contents
As indicated in the evaluation of the assessment plan above, there was no annual check up annual departmental assessment activities. Such a check up is relatively simple to complete, and allows several things to be accomplished: 1) it reminds departments that assessment is an ongoing process, and 2) it allows the institution to track assessment activities to identify potential weaknesses in the institutional assessment process, and 3) modify assessment activities where necessary. Such an annual check up could potentially use a form such as is found in Appendix A (exact details to be worked out, but the intent is to keep this to one page). Note: the annual check has been implemented as of summer 2006.
Return to Table of Contents
Although many of the administrative/co-curricular
institutional departments currently engage in a range of informal and formal
evaluation processes, it was time to institutionalize these processes in order
to ensure organizational excellence. It
was recommended that each of these administrative/co-curricular areas engage in a
five-year program review that is modeled after the academic department
five-year program review. All of
these areas should be required to submit an evaluation plan to the appropriate
University committee no later than fall 2005, with the 2004/2005 academic year
used to assist departments in developing these plans (largely completed). Many of these departments will find results from other
assessment instruments used at St. Ambrose useful in obtaining assessment
evidence at the institution.
It is important to not duplicate the efforts of existing committees. As such, it was recommended that the Academic support committee review the assessment plans and program reviews for the following areas, as recommended by the Faculty Handbook Committee: Admissions, Bookstore, Career Center, Financial Aid, General Accounting, Information Technology, O'Keefe Library, Records and Registration, Student Accounts, and the Student Success Center. It was recommended that the Student Life Committee perform a similar function for Student Services. A structure, perhaps already existing, will have to be determined for areas such as housekeeping, maintenance, academic advising (perhaps this latter area would report to Academic Support).
See http://web.sau.edu/assessment/deptassessments.htm for copies of evaluation plans for co-curricular and administrative support areas.
Return to Table of Contents
Academic departments have filled out a general education grid as part of their five-year program review for approximately the last 5-6 years. The intent of this grid was to document where and what general education goals and objectives were being taught to, both in and out of courses designated as fulfilling general education requirements. An informal analysis of the data collected from this grid suggests that the institution is teaching to a broad range of general education goals and objectives, in a broad range of courses at many different levels. The question must be asked, however, whether the grid is still fulfilling a useful purpose for the institution. As such, a more formal analysis of this system will be conducted by the Director of General Education beginning summer 2004 and the Director will make recommendations about whether we should continue this practice.
Return to Table of Contents
The St. Ambrose University faculty may want to give consideration to an annual assessment day or half day directed at the course, program/departmental, and institutional levels. We are not proposing that classes be canceled for such a day, as the experience at other institutions has suggested that activities the night before a “day off” such as this lead to less than satisfactory participation and performance. Rather, we are suggesting that this time be used to educate students about the importance of assessment activities and as a time when they can participate. Proposal for such an event will require broad university participation. (Not accomplished as of fall 2007.)
Return to Table of Contents
Possible Annual Assessment Report Forms (see http://web.sau.edu/assessment/AnnualAssessmentUpdate.htm for the currently used form)
Option
1
St. Ambrose University Annual Departmental Assessment Update
(Due no later than May 31 of each year)
Academic Year:
Department:
Person Reporting:
Position:
Assessment Activities Engaged in During the Academic Year:
1.
2.
3.
4.
5.
Changes Made During the Academic Year as a Result of Assessment Activities:
1.
2.
3.
4.
5.
Changes Anticipated During the Next Academic Year as a Result of Assessment Activities:
1.
2.
3.
4.
5.
Option 2
St. Ambrose University Annual Departmental Assessment Update
(Due no later than May 31 of each year)
|
Intended Student Academic Achievement or Other Outcomes |
Assessment Criteria and Measurement Procedures |
Results |
Action Taken on Results |
|
|
|
|
|
Return to Table of Contents
Information on Standardized Instruments for Institutional Assessment
The
technical merits of a standardized test are evaluated by examining evidence of
the validity and reliability of scores obtained from the test.
Reliability
The
reliability of a set of test scores is evaluated by examining the consistency
of those scores under four conditions:
1.
Consistency across time
If
students were to take the same test at two different times, we would expect
their scores to remain consistent. Evidence
of this type of consistency can be obtained by administering the same exam to
a group of students twice and calculating the correlation between the scores.
2.
Consistency across
parallel forms of the test
A
test attempts to measure student performance on a general set of skills
through a specific set of questions. The
questions that appear on the test are not the only questions that could have
been used to measure those skills. Reliable
test scores would remain consistent if students had been administered a
different set of questions measuring the same skills.
Evidence of this type of consistency can be obtained by administering
two equivalent exams to the same group of students and calculating the
correlation between the scores.
3.
Internal consistency
If
a test is designed to measure one skill, student responses to the questions on
the test should be consistent. The
reliability coefficient provides a measure of the homogeneity of a test –
the degree to which all test items are measuring the same skill.
A reliability coefficient near 1.0 indicates the items on a test are
consistent.
4.
Decision consistency
The
final type of test score reliability concerns the consistency of decisions
made from the test scores. Consistent
decisions are more likely to be made if the tests are objectively scored and
the scores are carefully analyzed.
Validity
The
validity of a test refers to the appropriateness, meaningfulness, and
usefulness of the scores obtained from the test.
Tests need to measure what they intend to measure (and not measure
things they are not intended to measure).
Tests must be fair and test scores must be reliable and useful. Therefore, the validity of all instruments used for
assessment activities at St. Ambrose will ultimately be determined by the
utility of the results for improving student learning outcomes.
Academic
Profile – Abbreviated
Form (see http://www.ets.org/hea/acpro/index.html
for additional information on the Academic Profile)
The
following information is directly from the Educational Testing Service web
site version of the Academic Profile User’s Guide; only portions were
selected for inclusion in this report (http://www.ets.org/hea/acpro/pdfs/userguide.pdf).
The Academic Profile was developed to assist in the assessment of the outcomes of general education programs, to improve the quality of instruction and learning. Because of the diversity of general education across and even within institutions, no single examination or assessment approach can meet every definition of general education outcomes (emphasis added). The Academic Profile focuses on the academic skills developed through general education courses, rather than on the knowledge acquired about the subjects taught in these courses. It tests college-level reading, college level writing, critical thinking, and mathematics skills.
College-level reading questions measure students’ ability to
interpret the meaning of key terms
recognize the primary purpose of a passage
recognize explicitly presented information
make appropriate inferences
recognize rhetorical devices
College-level writing questions measure students’ ability to
recognize the most grammatically correct revision of a clause, sentence, or group of sentences
organize units of language for coherence and rhetorical effect
recognize and reword figurative language
organize elements of writing into larger units of meaning
Critical thinking questions measure students’ ability to
distinguish between rhetoric and argumentation in a piece of nonfiction prose
recognize assumptions
recognize the best hypothesis to account for information presented
infer and interpret a relationship between variables
draw valid conclusions based on information presented
Mathematics questions measure students’ ability to
recognize and interpret mathematical terms
read and interpret tables and graphs
evaluate formulas
order and compare large and small numbers
interpret ratios, proportions, and percentages
read scientific measuring instruments
recognize and use equivalent mathematical formulas or expressions
The reading and critical thinking questions measure students’ ability to read carefully and think critically about issues and arguments from the humanities, social sciences, and natural sciences. The emphasis is on measuring the skill. Test takers are not expected to have specific knowledge about the content; all the subject knowledge required to answer each question is contained in the question itself or in the stimulus materials that accompany the question. The assumption is that a student who has successfully completed general education courses in the humanities, social science, and natural sciences will be better able to read the materials and answer the questions than a student who has not taken such courses.
What Information Does the Academic Profile Give Colleges about Their Students?
The Academic Profile provides two kinds of information about students’ skills: norm-referenced scaled scores and criterion-referenced proficiency classifications. The scaled scores are useful for comparing different groups of students, or for comparing an individual student with a group. The proficiency classifications are useful for describing students’ performance in terms of specific skills.
Scaled scores. The scaled scores include ...
A total score,
four skills subscores,
Critical Thinking
Reading
Writing
Mathematics
and three context-based subscores
Humanities
Social Sciences
Natural Sciences
The total score is reported on a scale that extends from 400 to 500. The skills subscores and the context-based subscores are reported on a scale that extends from 100 to 130.
Although the possible range (100 to 130) is the same for all the subscores, different subscores are not intended to be comparable to each other. For example, a Reading score of 120 does not necessarily have the same meaning as a Writing score of 120. However, scores on the abbreviated form are comparable to the corresponding scores on the standard form.
Proficiency Classifications and Proficiency Level Statistics
Proficiency levels. The skills measured by the Academic Profile are grouped into three skill areas. There is a single skill area that includes both reading and critical thinking, a separate skill area for writing, and a separate skill area for mathematics. Within each of these three skill areas, the specific skills tested by the Academic Profile are classified into three proficiency levels, identified simply as Level 1, Level 2, and Level 3. Each proficiency level is defined in terms of a set of specific competencies expected of students. (See document Academic Profile Proficiency Levels) Each question on the Academic Profile is associated with a particular proficiency level in a particular skill area (e.g., Level 2 in writing).
Proficiency classifications on the abbreviated form. A student taking abbreviated form of the Academic Profile answers only four or five questions at Levels 1 and 2 in reading, and only three questions at each proficiency level in writing and in mathematics. Therefore, the abbreviated form does not provide sufficient information for reliable proficiency classifications of individual students. However, it does provide sufficient information for estimating the percentages of the students who would be classified as proficient, marginal, and not proficient, if the students took the standard form. The estimated percentages can be reliable even when the individual classifications are not, because in a sizable group of students, classification errors tend to cancel each other out.
The percentages of students at each proficiency level in reading, writing, and mathematics are estimated by computing a proficiency classification for each student. This process uses information not only from the questions at that particular proficiency level, but also information from questions at other proficiency levels. The percentages computed from these individual classifications are reported, but the individual classifications are not.
All institutions receive a “Summary of Proficiency Classifications” report for each group of students whose answer sheets are submitted as a separate batch. The report shows the percentages of the students classified as proficient, marginal, and not proficient, at each proficiency level in each skill area.
Reliability
In the world of educational testing, reliability means consistency, or reproducibility. A student’s score on a test is a very specific piece of information. It describes the student's performance on a particular set of test questions. But the questions on the test the student took are not the only questions that could have been used to measure those skills. If different questions had been used to measure those skills, how different would the student’s score have been? It is not possible to answer this question for an individual student, unless the student actually took two different versions of the test containing different questions – and even then, the answer would apply only to those two versions. But it is possible to estimate, for a large group of students, statistics that indicate how closely their scores on an alternate version, containing different questions, would match the scores they actually earned on the test.
The statistics most commonly used to describe the reliability of test scores are the reliability coefficient and the standard error of measurement. The reliability coefficient is an estimate of the correlation between students’ scores under two different conditions of testing, e.g., different questions on the test. It describes, on a scale of .00 to 1.00, the extent to which the students who scored high under one condition (e.g., the set of questions they actually took) would also score high under the other condition (e.g., another set of questions measuring the same skills). It can vary substantially from one group of students to another. The reliability coefficient tends to be large for groups of students whose scores differ greatly and small for groups of students whose scores are similar.
The standard error of measurement (abbreviated SEM) is expressed in terms of points on the test (in this case, scaled-score points). The SEM indicates how much the students’ scores tend to differ, on the average, from the scores they would get by taking many different versions of the test (containing different questions) and averaging the results. The more reliable the students’ scores, the closer their actual scores are to the scores they would get under this averaging process, and the smaller the SEM. In a large group of students, about two-thirds of the students will have scores within one SEM of the scores they would get under this averaging process. Unlike the reliability coefficient, the SEM tends to be similar for different groups of students taking the same test.
The table below presents reliability statistics estimated from the responses of 11,111 students who took the standard form of the new Academic Profile during academic year 2001-2002.
Reliability Statistics for Individually Reported Norm-Referenced Scores.
|
Form of |
Score name |
Number of |
Score |
Reliability |
SEM |
||
|
Academic Profile |
questions |
scale |
coefficient |
||||
|
Standard |
Reading |
27 |
100 to 130 |
0.80 |
3.34 |
||
|
Standard |
Critical Thinking |
27 |
100 to 130 |
0.73 |
3.31 |
||