OMNI Institute’s Advancing the
Evidence Framework

+ Literature Review and Assessment to Support Students with Disabilities in Colorado

Students with disabilities are at a higher risk for school discipline than other students. This can lead to students with disabilities missing important instructional time, repeating a grade, dropping out of school, and experiencing a range of other challenges. This risk is even greater for African American/Black and Latinx students with disabilities. The Colorado Developmental Disabilities Council (the Council) partnered with OMNI to identify currently available resources and tools recognized as promising or best practices for individuals with disabilities and their families, with a particular focus on resources and support for communities of color. This research consisted of a literature scan, as well as focus groups and key informant interviews with community members impacted by the issues.

OMNI presented the Council with recommendations from this work to inform their efforts to better support students with disabilities in Colorado. Recommendations included ideas to promote the development of culturally responsive resources, opportunities for the Council to engage in systems-level changes that will better serve students of color with disabilities, concrete information that families and stakeholders would like to see to make resources more user-friendly and easy to access, and strategies that promote peer-to-peer support and empowerment.

+ Evaluation Planning to Help the Rocky Mountain MicroFinance Institute Examine Its Programming on Entrepreneurs' Economic and Social Mobility

For many, entrepreneurship is a key path to achieving economic and social mobility. However, too many individuals, especially women and people of color, face significant barriers to business ownership due to a lack of financial, social, and/or human capital. To address these challenges, the Rocky Mountain MicroFinance Institute (RMMFI) supports entrepreneurs and entrepreneur-hopefuls in the Denver community to cultivate their business ideas, effectively launch (or relaunch) their businesses, and successfully grow and thrive in business ownership.

RMMFI engaged OMNI to better understand how their three-phased approach to programming was helping participants improve economic mobility, social mobility, and community connections. OMNI hosted a series of facilitated meetings with RMMFI staff to develop and refine an Evaluation Action Plan that included: what outcomes mattered, when and how they should be measured, and the infrastructure needed to support evaluation. By harnessing the input of staff and evaluation best practices, the Evaluation Action Plan created a clear path that RMMFI could follow to understand fidelity and results, with the ultimate goal of validating a return on investment model. Most importantly, it ensured that the time and resources invested in evaluation will yield meaningful and actionable findings for years to come.

+ Developing a Culturally Informed Survey to Evaluate How Refugees Integrate into Colorado Communities

Entering a new country as a refugee is a difficult adjustment. That is why refugee service providers and resettlement agencies support individuals and their families to successfully integrate into their host communities. The Colorado Refugee Services Program (CRSP) needed a brief tool to measure key aspects of successful integration, such as social bridging and social bonding. No such tool currently exists, and they needed one that could easily be incorporated into their broader assessment framework. To help CRSP develop the tool, OMNI emphasized a culturally informed approach to survey development. Our research team conducted a literature review to find best practices for existing scales and items that have been validated in different refugee and host communities and obtained feedback from subject-matter experts who work in refugee communities. Once a draft tool was developed, OMNI conducted cognitive interviews with individuals who came from the same countries of origin as many Colorado refugee populations to understand how they interpreted and understood survey questions. Feedback from the interviews was used to further refine the survey questions. The result of the project was a 9-question survey (plus two additional questions for youth) to be pilot-tested with individuals served by refugee service providers in Colorado.

+ Evaluating the Quality Teacher Recruitment (QTR) Grant Program’s Efforts to Address the Teacher Shortage

All students deserve access to high quality teachers to support their educational growth and aspirations. Yet, Colorado and the nation are facing a shortage of teachers who have the experience, skills, and requirements to meet the educational demand. This shortage is particularly pronounced in low income and rural communities, furthering educational inequities. To address this problem, Colorado created the Quality Teacher Recruitment (QTR) Grant Program, which authorizes the Colorado Department of Education (CDE) to fund programs to coordinate recruitment, preparation, and placement of highly qualified teachers in school districts that have difficulty attracting and retaining high-quality teachers.

From 2014 to 2020, OMNI provided the CDE with a report evaluating whether the grant recipients successfully recruited and placed highly qualified teachers who made a positive impact on student learning and remained in their placements. OMNI examined data from: (a) program-provided teacher recruitment, placement, and retention lists; (b) district-provided educator effectiveness ratings (via programs); (c) school leader and teacher surveys; and (d) key informant interviews with district and school partners and institutes of higher education. The report is shared with the Colorado State Board of Education and the Colorado Legislature, and includes information demonstrating how the grant has contributed to successfully filling hard-to-fill positions throughout Colorado.

+ Evaluating Outcomes Across Direct Service and Policy-Advocacy Organizations for The Women’s Foundation of Colorado’s WAGES Initiative

Too many Colorado women live in or on the edge of poverty or work in low-wage jobs with little opportunity for advancement. In 2017, The Women’s Foundation of Colorado (WFCO) implemented WAGES - Women Achieving Greater Economic Security - to advance and accelerate economic opportunities for Colorado women and their families. WFCO funds direct service (DS) and policy advocacy (PA) organizations to implement services and strategies that are designed to propel women into careers that enable them to meet their needs and the needs of their families, with a special focus on issues of equity and two-generational approaches. The OMNI Institute is the learning and evaluation partner of the initiative.

To support WFCO in understanding whether and how WAGES grantees are supporting women and their families’ progress toward a livable wage, OMNI implemented a mixed-methods evaluation that examines the progress of women served by DS grantees over time, tracks legislative and advocacy wins and learnings from the perspective of those working on the issues, and examines how the WAGES cohort learning model has accelerated support for women across Colorado. WFCO and the WAGES cohort have used the information to spark conversation and learning around what is working well and where there are systemic challenges in advancing women’s economic security.

+ Analyzing Parent Surveys and Teacher and Home Visitor Observational Data to Examine Outcomes for Statewide Parenting Programs

Especially in the early years, parents play a key role in supporting and nurturing their children’s development. However, parenting skills are not innate, and many parents lack the support they need to meet the complex and challenging job of raising children. Parent Possible equips parents with the tools and information to be their child's most valuable teacher, trainer, and mentor in life. As part of their work, they support organizations across Colorado to implement two home visiting programs, Parents as Teachers (PAT) and the Home Instruction for Parents of Preschool Youngsters (HIPPY).

From 2015 to 2018, OMNI conducted statewide evaluations examining changes in protective factors; parent practices, beliefs and confidence; the quality of parent-child interactions; and children’s school readiness. OMNI provided multiple levels of reporting – in aggregate, by program, by site, and by funding stream and conducted data analyses separately for new versus returning parents and by demographic characteristics. Over the course of our partnership, OMNI designed a parent survey for the PAT program integrating feedback from funded sites; consulted on best practices for reporting and disseminating program results to stakeholders; and advanced data visualizations, ultimately building the capacity of evaluation for Parent Possible to take the evaluation in-house in 2019.

+ Rigorous Research in a Rural Community Testing a Parent-Based Intervention on the Home Literacy Environment and Young Children’s Early Literacy Skills

Children’s early literacy skills are critical for school success and healthy development. Children who live in low-income, rural communities face challenges in accessing early literacy resources and supports, which can contribute to achievement gaps. Motheread/Fatheread is an early childhood intervention that aims to improve children’s literacy outcomes by increasing the quality and frequency of parent-child shared reading activities in the home. The curriculum uses award-winning children’s literature as texts, many from African-American, Latino, Native American and other ethnic cultures, to instill reading as a way to access personal and cultural narratives and to introduce parents and children to the humanities’ practice of conversing about shared texts.

OMNI was contracted to conduct an external evaluation of the Motheread/Fatheread Colorado (MFC) program. The challenge was implementing a rigorous research design in a rural community in which many families would be interested in participating in the program. OMNI successfully designed and implemented a randomized controlled trial study to test the impact of parental program participation on the home reading environment and child literacy outcomes. Assessments came from parent surveys and preschool teachers’ ratings of children’s literacy and language skills. Results from the study were disseminated to the practitioner and scientific communities, and used to support continued implementation and expansion of the program.

+ A Rigorous Study Testing an Afterschool Program’s Impact on Young Children’s Reading Skills in Very Low-Income Communities

Research continues to show the importance of early literacy and reading proficiency by third grade. Afterschool programs are a key strategy in low-income communities to enable at-risk youth to access to educational supports. The Bridge Project, a community outreach program of the University of Denver Graduate School of Social Work, sought a rigorous examination of its afterschool, three-pronged, early literacy intervention delivered to children residing in public housing communities. The challenge was to design a rigorous study knowing that random assignment, the gold standard for testing program impact, was not a possibility.

To meet this challenge, OMNI designed and implemented a quasi-experimental, between-group design with propensity score matching to assess whether kindergarten through third-grade students who received Bridge Project services showed greater gains in literacy skills than similar students who did not receive Bridge services. Study findings provide preliminary evidence that it is possible to impact reading proficiency for very high-risk students in the early grades of elementary school. Further, OMNI’s study design and execution successfully resulted in an increase in the program’s level of evidence from preliminary to moderate according to National Standards. Findings were disseminated to the research and practitioner community via a peer-reviewed journal publication.