Standard 2 Assessment System

2a. Assessment System

1. How is the unit assessment system evaluated and continuously improved? Who is involved and how? The Center for Education unit assessment system consists of unit data, program data, individual data and data about the operations of the unit and programs are aligned with the Center for Education Conceptual framework (academic excellence, collaboration, diversity, lifelong learning and commitments to best practices and integration of emerging technologies. Assessment elements are indexed to NCATE categories of knowledge, skills and dispositions, as well as with the belief that all children can learn.

The assessment system is continually reviewed and revised. It is evaluated and improved through interaction with a variety of faculty committees, administrative committees and stakeholder groups who are vested in the quality of Center for Education candidates, its programs and its operations. Center for Education faculty as a committee of the whole participated in a series of discussions regarding the revamping of existing assessment systems early in AY 2007. The Center for Education Assessment Committee meets monthly during the academic year to discuss assessment and the assessment system with the NCATE Coordinator. These discussions target assessment concerns which are then brought to the faculty as agenda items for faculty meetings; or they are the subject of assessment retreats which at this time are scheduled to occur semi-annually in Fall and Spring semesters. The NCATE Coordinator also brings assessment concerns to the Center for Education Certification Committee; or may act as an assessment consultant when the committee has identified the need for a new assessment. Other stakeholder groups that have been involved with unit evidence and the unit assessment system include the Center for Education Advisory Committee and The Alumni Committee. Administrative groups who are directly involved in the improvement of the unit assessment system include the Office of Certification and its Director, as well as the Office of the Director of the Center for Education. The NCATE Coordinator since 2006 has improved the process of continuous assessment by creating new systems for acquiring, tracking and analyzing assessment evidence, and has identified a number of areas where existing data could be more efficiently integrated into electronic systems that would make evidence readily accessible and efficiently studied and analyzed. The NCATE Coordinator also identified areas where evidence for unit assessment was lacking and designed new data-bases that would integrate academic performance and proficiencies of candidates in advanced programs.

Improvements to the assessment system Prior to Academic Year 2006/7 the unit collected a variety of evidence related to both candidates and to its operations; however these activities were not systematic, nor were they integrated. The evidence included survey information such as: results from graduate and undergraduate surveys that were conducted by the Office of Institutional Research and disseminated to local units, including the Center
for Education; surveys of alumni; surveys of employers; and surveys of graduates. Several different offices in the Center for Education collected evidence, recorded evidence on EXCEL spreadsheets, and stored the evidence on personal office computers. Because of changes in administration (e.g., the resignations of 2 assistant directors), the exact location of the evidence, file names, and the nature of the data was not clearly understood. In Spring 2007 a systematic effort was made to locate and integrate these data. Since the surveys mentioned above were not initially designed to conform to a common conceptual structure, the data they yielded could not be logically integrated electronically. However, after 2007 the unit assigned official responsibility for collecting, storing, safe-guarding, and analyzing the evidence; and the conceptual framework of the Center for Education became the framework for analyzing and interpreting the information. In AY 2003 an ACCESS relational database was designed to house unit-level evidence about candidates in initial teacher certification programs. Early in AY 2007, a Center for Education assessment meeting of faculty was convened to discuss shortcomings of the 2003 database. The following shortcomings of design were reviewed, such as: the inability to disaggregate data by programs; inconsistent coding of variable values; a large number of empty fields created by the inclusion of candidates who planned to teach states other than the Commonwealth of Pennsylvania; evidence stored in YES/NO and TRUE/FALSE formats that obviated the possibility of studying candidate progress as related to NCATE Standards 1 and 3; evidence that had no relevance to candidate performance at transition points; and missing evidence. Much evidence related to candidate performance was entered at the time a candidate became a completer. This led to many instances where evidence was incomplete, and evidence was not used to track or monitor candidate performance in a systematic way. In AY 2007 the ACCESS relational database was re-designed. The new database standardizes the input of evidence through the use of clickable pull-down menus for most categories of evidence. Evidence is organized according to transition points. New field were created to include evidence about field experiences prior to student teaching and clinical practice.

In AY 2006 evidence related to Other School Personnel (graduate level candidates) was not systematically collected, stored, analyzed or organized. Again, in AY 2007 the Center for Education faculty addressed the need for integrated data collection for candidates in advanced degrees and certificate programs. A parallel database was created. Evidence that tracks the proficiency of graduate candidates for certificates such as reading specialist, technology facilitator, home and school visitor, school counselor, school nurse, principal, supervisor and superintendent is indexed to transition points at admissions, mid-program, completion and exit from a certification program. In the fall of 2007 adopted a new e-portfolio system, TaskStream. Once this EP system was tried by the initial certification programs using the rubrics already developed, it became apparent that all of the Center for Education’s unit and program assessments could be integrated. INTASC principles could continue to be used to evaluate candidate artifacts according to certification transition points. Programs could also use standards that were electronically available within the TaskStream platform, or faculty could develop assessment rubrics themselves. A retreat, sponsored by the Certification Committee, in December 2007 demonstrated how to use TaskStream for program assessment to the faculty in certificates for specialist and advanced certificates and programs. Now, rather than serving as a program assessment only, the electronic portfolio has emerged as a unit assessment. Though each of the program rubrics are different, the same processes for developing rubrics and rubric validation, can be followed regardless of discipline. Because each of the program rubrics are an integration of both SPA criteria and the Center for Education’s conceptual framework, data can disaggregated for programs, analyzed for unit purposes, and used to provide continuous feedback to individual candidates.

The Assessment Committee. The Center for Education Assessment Committee plays a pivotal role in the assessment system. The committee meets monthly. The NCATE Coordinator is a member of the Assessment Committee and communicates assessment concerns. In response, the committee determines a course of action: i.e., the committee determines how other faculty committees will be involved in assessment. In AY 2007/8 the committee planned and conducted a number of retreats involved the unit in the review of the following topics: e-portfolios as assessment strategies; candidate assessment and the TaskStream e-portfolios; unit review of program evidence; weighting evidence from work samples in the determination of course grades; unit review of PRAXIS II outcomes; unit review of field experience and clinical practice evidence.

2. Transition Points Please refer to Table 5 in the Institutional Report. An additional link to a chart of transition points can be viewed at this link Transitions Points.

3. How does the unit ensure that the assessment system collects information on candidate proficiencies outlined in the unit's conceptual framework, state standards, and professional standards? All assessments are aligned with the Center for Education Conceptual Framework, Commonwealth of PA General Standards for the Preparation of Professional Educations, and professional standards where appropriate.

Since this is a first accreditation, all the assessments that have been planned at unit, program and individual candidate levels were by design aligned with the Conceptual Framework as an initial step. The themes of the framework were operationally defined by listing general unit proficiencies that demonstrate candidate proficiencies in each of the areas of academic excellence, collaboration, diversity and life-long learning. These proficiencies were a blueprint for deciding how assessments that were developed later could be focused. See Exhibit 2a.3 for a description of the Center for Education collects, records, maintains and reports assessment evidence in a systematic way.

In its early stages the assessment system consisted mostly of measures of content knowledge -- i.e., PRAXIS scores and course and program grade point averages. As additional assessments were developed to assess field experiences, student teaching and clinical practice assessments for other school personnel, and assessments for key courses in licensure and nonlicensure programs, they became part of the assessment system. The development of these assessment tools always began with alignment. INTASC principles are the basis for the evaluation of clinical practice. Formal field evaluation forms are aligned with INTASC and the more specific proficiencies associated with standards of professional organizations are embedded within the INTASC principles. Therefore, INTASC principles and professional standards are fully integrated with each other and with the conceptual framework.

Rigorous alignment with professional standards at every level of assessment has been they keystone of assessment planning and assessment discussions for the unit and its programs. Assessments, rubrics and scoring guides, as well as other forms of assessment information (e.g., course grades, surveys, and PRAXIS evidence) are aligned with the strategic goals of Widener University and the elements of the Center for Education Conceptual Framework; alignment with the Commonwealth of Pennsylvania DOE General Standards for the Preparation of Professional Educators and with professional standards for specialized professional associations may apply as well.

For initial certification programs, rubrics used to assess candidates' artifacts are aligned with INTASC.Artifacts are also individually assessed using standards of various specialized professional associations. Beginning in AY 2003 the Assessment Committee carefully planned the rubric development process using a template that entailed a matrix aligning the themes of the conceptual framework against four levels of achievement: distinguished, proficient, emergent, and insufficient for each INTASC principle. Faculty began by establishing what an artifact would have to demonstrate in order to be deemed “distinguished” (4) on the rubric in terms of each of the themes of the conceptual framework. Next, faculty established what an artifact would be lacking in order to be deemed “insufficient” (1) on the rubric. As they worked to establish what an artifact would have to include in order to be deemed “proficient” or “emergent,” faculty lowered the components/qualities of the selected INTASC principle as well as the levels of learning (Bloom’s cognitive and affective domain). Faculty then compared the completed rubric to other INTASC principles and completed INTASC rubrics to be sure that it reflected the essence of the INTASC principle in question. They evaluated each rubric in terms of its clarity,conciseness, and measurability. Faculty considered how these rubrics would be used both formatively and summatively. Each rubric had to be usable across transition points in each program. Finally by August 2006, the INTASC rubrics were piloted using portfolios developed by student teachers that spring. Revisions were based on issues of clarity, conciseness, measurability, fairness, accuracy and utility. Further refinements of INTASC rubrics then occurred. These have also been discussed in our response to Standard 2.

For candidates' in specialist and educational leadership certificates, rubrics are aligned with the Center for Education Conceptual Framework, the Commonwealth of Pennsylvania DOE General Standards for the Preparation of Professional Educators and with professional standards for specialized professional associations whenever applicable. For example, technology facilitator assessments are aligned with Commonwealth standards, assessments of content knowledge are aligned with both NETS and ISTE, professional skills and dispositions are aligned with ISTE. The reading specialist assessments are aligned with IRA professional standards throughout all of the assessments that embedded into key courses having signature assessments. Moreover, the reading specialist assessments are continuous in their application; the assessment of candidates is an ongoing process culminating in a capstone assessment at program completion. Some specialist certificates are not associated with specialized professional associations: i.e., Counselor Education (elementary and secondary school counselor), Home and School Visitor, and School Nurse. Program assessments are primarily aligned with Commonwealth standards; however Counselor Education has taken the extra step of aligned assessments with CACREP. The Home and School Visitor certification program is unique in the way it is offered at Widener University. Candidates enter through either the Center for Education or through the Center for Social Work Education. Those who enter in the joint MSW/HSV program are assessed according to standards of ASWA.

4. How does the unit ensure its assessment procedures are fair, accurate, consistent, and free of bias? Alignment is the primary strategy for assuring that assessments are valid, and therefore will provide feedback to candidates and to faculty that is useful and that contributes to further professional development of candidates. The development of the rubrics used to evaluate e-portfolio artifacts demonstrates how alignment and faculty revisions occurred, bringing the INTASC based rubrics to their final form.

The NCATE Coordinator and the Undergraduate Program Coordinator carefully planned the rubric development process by asking participating faculty to follow a set of guidelines that included a specific set of steps and a template for rubric design.

A template for the rubric entailed a matrix that aligns the themes of the conceptual framework against four levels of achievement: distinguished, proficient, emergent, and insufficient per INTASC principle. The Faculty members were asked to begin by establishing what an artifact would have to include in order to be deemed “distinguished” (4) on the rubric in terms of each of the themes of the conceptual framework. Next, faculty established what an artifact would be lacking in order to be deemed “insufficient” (1) on the rubric. As they worked to establish what an artifact would have to include in order to be deemed “proficient” or “emergent,” faculty were asked to consult the same information used in preceding steps, this time lowering the components/qualities of the selected INTASC principle as well as the levels of learning (Bloom’s cognitive and affective domain).

The Faculty then compared the completed rubric to other INTASC principles and completed INTASC rubrics to be sure that it was distinguishable and that it truly reflected the essence of the INTASC principle in question. They were asked to evaluate each rubric in terms of its clarity, conciseness, and measurability. While working through the actual rubric design, faculty members were also asked to consider how these rubrics would be used both formatively and summatively. If the electronic portfolio was to document progress over time, then the rubric must also be usable across decision points in each program.

The initial rubrics were designed by a six person faculty task force. Teams of two faculty members were assigned a few principles and associated conceptual framework themes and asked to develop a rubric. The teams then switched rubrics for revision and clarification purposes to ensure fairness, accuracy and utility. In August 2006, the INTASC rubrics were ready for a field test using portfolios developed by student teachers that spring. After using the rubrics to evaluate student artifacts, further revision occurred as faculty were able to closely examine the consistency of each of the rubrics’ criterial statements and artifacts selected by students. Revisions were based on issues of clarity, conciseness, measurability, fairness, accuracy and utility. The results of this pilot study led to further refinements in the rubrics. The process of refining this rubric and its administration continues.

Surveys are validated using pilot studies. For example, the advising survey was first tested with a randomly selected group of students. They were able to supply information that led to clarifying items, adding items that were key to student advising experiences, and clarifying directions for taking the survey. A similar study to validate the Office Systems and Support survey was conducted. Most rubrics and field-based performance assessment tools are field-tested themselves by establishing first their content validity, and then through inter-judge agreement studies.

An Advising Survey is used to explore strengths and weaknesses of candidate advising experiences. An Office Systems and Support Survey is used to evaluate the quality of communication regarding policies and resources. Students rate the quality of office systems and support on a 10 item survey. The Undergraduate/Graduate Student Survey investigates the quality of the broader experience of candidates at Widener: the registrar's office and registration procedures, financial aide policies and procedures,dining, safety, library resources and policies, parking and residence life (for undergraduates). The Employer Survey is administered to administrators in local school districts in Pennsylvania, New Jersey and the northern tip of Delaware, where our candidates are employed as teachers and other school personnel. This survey provides information about the competence of our graduates in terms of their content knowledge, their professional skills, and professional dispositions.

Please refer to the Exhibit for Standard 2: Transition Points in the Institutional Report. An additional link is provided to view a chart of transition points for the convenience of the BOE team. Exhibit 2a.3 Assessment System: Collection, Recording, Maintaining and Reporting Assessment Evidence

Standard 2 Assessment System home