Evaluation and Assessment

These strategic objectives reflect campuswide efforts to increase diversity in myriad forms, offer equity for all, and ensure a vibrant culture of inclusiveness. The qualitative and quantitative data yielded year over year will be used as a basis for decisions on programming and resource allocation to achieve and sustain desired outcomes.

Progress is communicated regularly across campus through multiple channels, with opportunities to engage in discussion and continued planning to build on our efforts over time.

Tracking and reporting occurs at university, school, college and unit levels. Progress on plan-related action steps such as implementation and participation in programs, utilization of services, and awareness of DEI are tracked on an annual basis.

Longer-term measures being tracked over time include the demographic composition of our campus, shifts in climate and indicators of equity across populations.

The U-M Office of Diversity, Equity and Inclusion (ODEI) leads and provides primary support for evaluation and assessment of the U-M DEI Strategic Plan implementation.

ODEI leads DEI data development efforts for our large, decentralized campus. As a central entity on campus, ODEI is developing an institutional domain to conduct DEI evaluation and assessment and to archive this data for reference and re-use.

Our work in evaluation and assessment of our DEI Strategic Plan implementation is guided by the following goals:

• Evidence: Strengthen evidence base for informed decision-making, not only for what we want to hear, but what we don’t know and what we are missing, and for what our community thinks and feels. Evidence will garner influence to create action.
• Influence: Translate and disseminate policy-relevant and decision-relevant research and data to key groups and decision-makers to inform and guide the infusion of DEI into the fabric of our institution.
• Action: Design and implement best research practices to achieve impact and goals.
• Transparency: Communicate about what we are doing and how we are doing it in a transparent way, so that others trust that we are being objective and accurate with the way that data is collected, prepared and shared. This supports accountability for the work we are doing.

All U-M students, staff and faculty and all campus units stand to benefit from the DEI evaluation and assessment work that results from and drives our DEI Strategic Planning and Implementation process.

DEI Leads in participating campus units are a specific audience in terms of the central reporting process and for unit-based metrics and climate survey reports produced centrally.

At the Unit level, evaluation and assessment is essential for data-driven decision making. Executive Officers, Deans and Directors are another key audience in terms of their need to access and use relevant data and reports in the context of their work both internal and external to the institution.

To effectively conduct evaluation and assessment related to the DEI Strategic Plan implementation, the Office of Diversity, Equity and Inclusion (ODEI) collaborates with multiple campus offices, including the Office of Budget & Planning, the Institute for Social Research (ISR), U-M Human Resources (UMHR), and Information Technology Services (ITS).

For example, to conduct a campus climate survey, we consulted with ISR and coordinated with OBP, UMHR and ITS on specific aspects of the project. Meanwhile, we partnered with an external survey research vendor to conduct our campus climate survey, providing an additional layer of assurance for the confidentiality of survey responses and data handling.

As you plan for evaluation and assessment of your campus DEI efforts, then, determine key audiences and stakeholders who will help gather or provide feedback and data, who will have a vested interest in seeing progress documented in different areas, and who will utilize the results to inform decisions and future plans.

In addition, take stock of your organizational structure, resources, and goals. Identify different entities on campus involved with data, evaluation and assessment, data systems, reporting and research administration to determine your points of contact and collaboration.

The year prior to the 2016 launch of the DEI Strategic Plan began with an assessment and community engagement phase.

During this time, students, staff and faculty across the campus participated in more than 200 community engagement events, including campuswide activities and many events hosted by schools, colleges and units.

Participants contributed ideas and feedback, which were used along with other existing diversity, equity and inclusion-related data to inform the unit plans.

At the conclusion of the 2015-16 academic year, participating campus units had articulated their DEI Strategic Plans, and building on these plans, the central level plan was established as well.

This initial assessment and planning period is essential in that it involves many people – not just a few high-level administrators – in evaluating needs and priorities.

As part of the central campus plan, we confirmed the following set of metrics to help build accountability against goals for diversity, equity and inclusion.

These metrics will be tracked and will be assessed at the end of the five-year implementation period. We anticipate continued tracking and effort on these and other metrics beyond the current strategic planning window.

Initial Categories of Metrics for Continued Tracking

Progress on plan action steps and initiatives:

• Activity toward implementing initiatives
• Participation rates
• Action-specific outcomes (e.g. learning outcomes from training, etc.)

Demographic diversity:

• Undergraduate students —Demographic composition (e.g., race/ethnicity, sex), enrollment status (class level, entry status), 4- and 6-year graduation rates
• Graduate students — Demographic composition (e.g., race/ethnicity, sex), degree level (masters, doctoral, professional)
• Faculty — Demographic composition (e.g., race/ethnicity, sex, age), tenure status or job family
• Staff — Demographic composition (e.g., race/ ethnicity, sex, age)

Climate indicators:

• Student, faculty and staff 12-month satisfaction with the overall campus or school/ college or unit climate/environment, depending on constituency
• Student, faculty and staff assessment of aspects of the general climate and DEI climate of overall campus or school/college or unit, depending on constituency
• Student, faculty and staff assessment of institutional commitment to diversity, equity and inclusion
• Student, faculty and staff feelings of sense of affirmation and academic or professional growth, depending on constituency
• Student, faculty and staff feelings of discrimination in the prior 12 months

To plan and implement our DEI Strategic Plan evaluation and assessment efforts, the U-M Office of Diversity, Equity and Inclusion (ODEI) involves multiple professional staff members and collaborates with key campus offices.

Essential to this work is a dedicated, full-time research analyst, whose primary responsibilities include providing services in collecting, discovering, cataloging, manipulating, and transforming data as well as consulting on the proper use of data.

Other responsibilities include developing reports, making presentations and providing training on the use, cataloging, and manipulation of data to University staff and other constituents, as needed.

The core activities we have engaged as part of our program of evaluation and assessment include:

Campuswide Climate Surveys:

As our process of implementation began, we conducted a comprehensive evaluation of each unit, aggregated campuswide, with respect to:

• How diverse unit/campus is as a community
• The degree to which individuals experience the unit/University as inclusive
• Whether individuals have equitable opportunities to be successful

Diversity, Equity & Inclusion (DEI) Metrics Reports and Dashboard:

Reports are generated from this dashboard and provided annually to campus units.

As the DEI Strategic Plan implementation unfolded, a Diversity, Equity and Inclusion Metrics (DEI Metrics) Dashboard was developed through a collaborative effort between the Office of Diversity, Equity and Inclusion, Office of Budget and Planning, and Information and Technology Services.

DEI Metrics will track and report on shifts up or down, trends, and other indicators representing progress and outcomes that enhance diversity, equity, and inclusion on campus (see list of metrics being tracked in the planning segment of this guide). At regular intervals, the campus community will be updated on DEI Metrics results at the university as well as the School, College, and unit levels.

Reporting and Assessment of Annual Progress:

We work to demonstrate accountability in terms of progress toward unit and campuswide goals, while identifying new opportunities and challenges to provide a basis for informed decision-making. Essential steps in an annual progress and reporting process include:

• Gather data on progress towards unit DEI Plan goals and actions to enhance diversity, equity and inclusion along with the specified timetables for execution
• Include unit-level DEI assessment data as part of unit plan among implementation domains (as applicable): retention, recruitment, and development; inclusive climate; education and innovative scholarship; research or scholarly activity; and service or community engagement.
• Provide a memorandum to each unit involved including a progress assessment that provides feedback, identifies strengths and suggests areas for further attention.
• Issue overall campuswide progress report. This could be a formal report, an interactive website, a set of infographics, or other formats and approaches. At U-M, we use all three of these approaches to help communicate our DEI Progress, as well as video-based stories of progress and impact, and access to a searchable dataset of DEI actions and completion status across all unit DEI Strategic Plans.

Revising DEI Plans Based on Reporting and Feedback Efforts:

Each year we engage our DEI Leads across all units to review and update DEI Strategic Plans and in doing so, to reflect, examine, and learn from the outcomes of research and assessment efforts (i.e., climate survey reports, DEI metrics reports, annual reporting assessment memorandum).

Data Development, Stewardship, and Support of Institutional Research:

We have developed and continue to maintain relationships, partnerships, and collaborations with key stakeholders, improve our workflow with data, documentation and communication, and continue to design and implement best research practices.

Education, Training, and Oversight:

We uphold high standards for research and data sharing. This involves participation in research oversight administered by the campus IRB (Institutional Review Board) to protect human subjects, and as appropriate, to prompt participation in the campus Program for Education and Evaluation in Responsible Research and Scholarship (PEERS), the university’s online resource providing regulatory and responsible conduct of research (RCR) training courses for employees and students.

Even our evaluation and assessment process needs to be evaluated! Our collaborative approach led to many inputs from multiple groups – including faculty, staff and students – regarding decisions about climate surveys, key metrics and annual progress reporting.

ODEI continues to seek feedback makes adjustments in the content and process of our DEI evaluation and assessment efforts. We continue to strive for transparency and accountability with our work to document and communicate progress in our DEI Strategic Plan implementation.

Here are just a few of the many ways we have used feedback to inform decisions and changes in our evaluation and assessment work:

• Input from DEI Leads informs annual updates to the DEI Strategic Plan Progress reporting tool and process which has made reporting more efficient and effective.
• Community input led to a decision to use a third-party vendor to conduct our campuswide climate survey for additional assurance of confidentiality.
• Feedback from students led us to create a set of infographics as well as an interactive digital progress update to better communicate about progress in key areas and initiatives.

• Include planning for evaluation and assessment as you develop your DEI Strategic Plan; this is a critical function that is essential to the process and outcomes of your efforts.
• Engage multiple stakeholders, experts and constituents in your planning and decision-making with regards to evaluation and assessment.
• Build evaluation and assessment into your budget, including – and especially – for experienced, professional staffing that will enable you to conduct strong, effective, and sound practices with regards to evaluation and assessment.
• Ensure that you are following best practices in the collection, reporting, storage, sharing, and re-use of data; again it is key to involve data experts to guide this work.
• Ensure that your process complies with federal, state and local law and/or university policies or agreements that require the university to implement specific privacy and security safeguards (i.e., ITAR, EAR, HIPAA, FISMA).
• Communicate with your campus about how you are conducting evaluation and assessment as well as about the results of your evaluation and assessment.
• Be willing to share about areas that are not progressing as planned or that need additional attention; this transparency is important in terms of building trust and being authentic in this work.

• UM Campus Climate Survey (Campuswide and Sampling Surveys)
• ICPSR 2016 U-M Campus Climate Survey Archive Data
• U-M DEI Progress Reports
• Database of Campuswide DEI Actions and Completion Status