News Nonprofits Measurement: Challenging nonprofits to define value, impact in community
Measurement: Challenging nonprofits to define value, impact in community
Order Reprints
Follow @MiBiz
By Nathan Peck | MiBiz
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
 |
|
Last year, Pfizer completed a $100 million investment to construct an aseptic processing area in the company’s Portage manufacturing facility to allow it to make injectable drugs. PHOTO COURTESY OF PFIZER |
WEST MICHIGAN — Paul Haan is 100-percent certain the Healthy Homes Coalition has lowered the levels of lead poisoning in Grand Rapids children.
The challenge is, he can’t always measure it 100 percent.
Haan, the group’s executive director, has solid data on the levels of lead poisoning in the region because Healthy Homes’ reporting is federally mandated. As the organization began its lead abatement activities under its previous guise as Get The Lead Out, it could monitor each initiative’s effect on regional data tracking lead poisoning.
“Health is an evidence-based science, so you have data-driven approaches. We had data from health care providers – really good surveillance data. … We could look at the childhood lead issue, not just anecdotally, but as a public health issue,” Haan said. “We have good evidence to make the decision on how to act on that (from a programming standpoint).”
As the organization evolved to include children’s environmental health under its mission when it became Healthy Homes, the data became much harder to come by. Information on rates of childhood asthma, accidents in the house and radon and carbon monoxide levels are collected in a much more haphazard manner. Hospitals may collect some data that is subsequently protected by HIPAA, and community health organizations may have differing data collection standards. The result is that the organization is left to extrapolate national data to the local level, complicating its efforts to evaluate the effectiveness of its initiatives.
“When we transferred to Healthy Homes, much to our frustration, (similar) data doesn’t exist for other children’s health issues,” Haan said. “We are working hard to pull together more robust data around asthma. We are trying to determine the source that is triggering asthma episodes.”
Healthy Homes’ struggle to develop more rigorous data collection and evaluation techniques is being mirrored at other nonprofits around the region. The trend isn’t just because organizations want to measure the good they’re doing: Donors and foundations are increasingly looking to gauge the impact their dollars are having in the community.
Charitable giving in the U.S. dipped more than 13 percent in 2008 and 2009 before rebounding slightly in 2010. Charitable giving was up $10.59 billion to $290.89 billion in 2010, according to data from the Center for Philanthropy at Indiana University. Donors are watching their investments more closely, which has them looking to nonprofit organizations to clearly articulate the impact of their programming.
Where exactly the money goes is at the center of the question, said James Edwards, executive director of the Dorothy Johnson Center for Philanthropy at Grand Valley State University. The problem in linking expenditures with community outcomes is that there is rarely a clear input-output relationship for nonprofits, particularly because of diverse funding sources.
“In a for-profit business, you are selling a product or service. There is a one-to-one relationship between service and measurement. If you sell a TV, you can count your sales,” Edwards said. “Nonprofit organizations in general, have embraced measurement and … the philosophy that you need to understand the benefit and impact. I think they have come to that because funders have asked for that. Smart nonprofits have realized that they need that information, they have put mechanisms in place to measure that and report it to funders.”
That’s all easier said than done. Measuring the impact and ROI of spending on nonprofits is more of an indirect art than a science at this point. In many cases, the central question is: What is the value of a service that is given away for free?
Ronald Visscher, assistant professor of business at Aquinas College, is working to give clarity to the valuation and evaluation issues. Whereas business has a commonly agreed upon set of standards to compare the performance of an auto supplier and a bakery – Generally Accepted Accounting Practices – there is not a parallel set of standards for nonprofits.
At this point, it is tough for anyone to measure the impact of a health care program versus one working to improve job skills. From simply asking a client what she or he would be willing to pay for a nonprofit organization’s services to conducting a complex regression analysis to determine the impact that programs have on community outcomes, there is little agreement on how to address the issue at this point.
“There is debate over the amount of evidence there needs to be to justify investment in social programs. More and more, you have to prove there was an effect. There is a trend to move from what works to what works better, to what works best. To compare across sectors – our methods need to be improved and the capacity of nonprofits to implement those methods” of evaluation must be improved, he said. “Whose values are you going to consider: funders, beneficiaries, administrators of the program, community leaders? You have to take a stakeholder approach – but this needs to be improved to incorporate multiple perspectives.”
The difficulty is in determining what to measure.
Rebekah Fennell, executive director of First Steps, a public-private partnership that works to strengthen and coordinate early childhood services in Kent County, said that a data-driven approach is a necessary part of evaluating nonprofits, but that it has its limitations.
“If (a program) is innovative, it may not be in the journals yet. It could take five years or more to appear (in academic journals) after the initial research,” Fennell explained. “Working with an evaluator was tremendously helpful. They helped us determine what is the best proxy for that (outcome) at the community level, as we cannot do primary data collection. We hope to drill down more deeply, to census tract, street. We want to find out where the hot spots are – to move the county measures.”
First Steps worked with the Community Research Institute at the Johnson Center to measure the effectiveness of one-on-one, home-based child education and play-and-learn groups on young children’s school readiness. An initial study in 2010 showed there was an effect when certified instructors visited childcare providers and when children took part in the group activities. Now the organization is interested in the minimum effective dose.
“We wanted to test how little a dosage can make an impact in programs with home-based childcare providers and play-and-learn groups. … We want to know what was due to the individual home visits, and what is attributed to what was done in the group,” Fennell said. The organization is interested in the outcomes of both interventions as “one is cheaper than the other.”
Data disruption
The Johnson Center’s Edwards said that as data becomes increasingly available, few organizations have the personnel available to wade through it due to the low overhead and lean operations many nonprofits operate under.
“It is actually about capacity in nonprofit organizations – we still lack capacity … to actually utilize the data,” Edwards said. “Too often, this falls on the plates of people who are already overwhelmed. If they can give it attention, they can’t give it their full attention. All over, Michigan nonprofits run very lean. They can see it and can ask questions about it, but they need more resources to incorporate it into their framework and business.”
He sees evaluation driving sea change in nonprofit organizations.
“We are at the beginning of a disruption,” Edwards said. “There is such granular data out there that systems and nonprofits are going to have to adjust quickly. Funders will adjust first, then nonprofits will adjust next. Data will start to show trends and themes and then the reactions to those things is going to have to be rapid. We will be in that cycle for a while, with pushes of data coming in, then the community reacting to it.
“It will probably be volatile for some time.”
SIDEBAR