By Kate Hagner, Program Quality Manager, Student Conservation Association
The Student Conservation Association (SCA) has been a partner of Search Institute since 2012, when we embarked on an ambitious, multi-year project to better understand young people’s experiences in our programs, build our capacity to deliver transformative youth development experiences, and measure the impact these experiences have on young people.
Founded in 1957, SCA aims to empower young people of all backgrounds to plan, act, and lead, while they protect and restore our natural and cultural resources. Among our programs, SCA annually fields scores of conservation crews comprised of two experienced leaders and eight crew members age 15-19. Crews serve for approximately one summer month at sites ranging from national parks to urban neighborhoods. Participants build hiking trails, restore habitats, and engage in a range of environmental education and trainings.
Prior to working with Search Institute, we’ve had external researchers study SCA—including a landmark analysis about a decade ago that showed how outdoor programs like those offered by SCA can have a considerable impact on participants, including increased self-confidence and a greater sense of personal meaning and direction. What we were missing, however, was a complete picture of the ways our program changes the lives of young people.
SCA approached Search Institute eager to dive right in. We hoped for a study that would show us quantitatively, once and for all, the impact of our program. In short, we wanted numbers. This is what our funders were asking for, this is what our Board of Directors was asking for, and to be truthful, it’s what we wanted too. We already knew that great things were happening in our program, but we wanted empirical evidence.
When Search Institute came to the table with a four-phase approach, we balked. It was going to take us years to get to the fourth phase—the quantitative research phase—as well as those beautiful, hard numbers we wanted so badly. But after much conversation, we dipped a toe into the first phase, and we’re glad we did.
In our first phase of work, Search Institute used a qualitative, grounded theory approach to identify the outcomes that SCA programs naturally produce and opportunities to build the capacity of programs to more consistently produce priority outcomes. The team at Search Institute visited our programs, talked with our participants around campfires and community tables, and even ventured into the wilderness to learn all they could about the experiences of our participants.
What surfaced from this qualitative investigation was an “emerging” Theory of Change; an early, fluid picture of the programmatic inputs that appeared to really matter in terms of positively impacting our participants’ readiness for school, work, and civic life. During data analysis, Search Institute paid particular attention to what was happening in our programs when participants found themselves transformed in meaningful ways, and looked for similarities in these stories. From this data, Search Institute produced a list of suggestions for places to focus program improvements to more intentionally and consistently produce these outcomes among participants. In other words, “We think that if you do more of this, you’ll likely get more of that.”
Building on these early findings, SCA and Search Institute staff worked together in Phase 2 to design and test improvements to our program based on what we’d learned from the data. We designed more intentional opportunities for our youth participants to engage in collaborative leadership, explore career opportunities in conservation fields, and reflect on their experience and what it meant to them. We also designed a new training for youth team leaders focused on the strategies that Search Institute research shows are key, including forming developmental relationships and harnessing the power of “sparks”.
In Phase 3, we started scaling up by taking the learning from the pilot and applying the most promising strategies to all of our youth programs. Only after testing and refining these improvements did we move into the fourth and most recent phase—the quantitative study—where we measured the ways in which SCA participants are transformed by their experience using a pre- and post-test survey designed by Search Institute. Because of the qualitatively rooted theory of change that had emerged from the earlier focus groups and interviews (and was validated by SCA staff and participants), researchers knew in very precise terms what to measure as they designed the instrument.
The survey found positive, statistically significant differences in pre- and post-program scores across 19 indicators. Nineteen! The numbers we’d been waiting for showed that as a result of their SCA experience, participating youth are more connected to nature and more committed to conservation values, and they’re also benefitting from significant gains in their thriving mindset, which we know through research is powerful for opening the door to a lifetime of new experiences through which they continue to build the knowledge, skills, and social connections necessary to succeed and to lead.
Even more importantly, we saw stronger results because of the earlier work to make our program more consistent and because we knew what achievement of our goals looked like in the field and had designed questions to get at those outcomes. The measures matched reality, and it paid off.
We’ve been thrilled to see that the hard work was worth it. Because of the phased approach, we ended up with stronger results, we improved our program, and we now understand more about the magic at work in SCA experiences (and how to be more intentional in leveraging that ‘magic’) than we ever knew was possible.
Our Search Institute partnership did more than measure what we do; it helped us get better at what we do.
Kate Hagner leads SCA’s national initiatives that use research and data to continuously improve programs. SCA is headquartered in Washington, DC and maintains regional offices across the country. For more, visit www.thesca.org.
Want to learn more? Download the key findings from the third phase of the partnership here.