My Public Comment on the Mental Health Innovation Grant Proposal

Shasta County Health and Human Services Agency (HHSA) has received a proposal for an Innovations project totaling ~$705,000. You may review Roughout Ranch’s Authentic Workshops and Horse Encounters plan proposal and program outline.

I submitted public comment as anyone can here and I’m sharing it here because I believe it is critical that we rigorously evaluate proposals for programs and services before committing limited county resources. This is one of the ways we hold the government accountable to deliver a benefit to the community.

 


 

Dear Shasta County’s Mental Health, Alcohol and Drug Advisory Board,

Thank you for the work you do to advance the mental and behavioral health of our community and for reviewing proposals for this Innovation Grant. 

I’m excited to hear that HHSA has received an innovation grant and is working to administer it. Improving the mental and behavioral health of our community must be a top priority! It is critical, in my opinion, that we use these funds to determine the efficacy of intervention(s) that address our most significant mental or behavioral health issues and fill an existing gap in our offering of services. 

While Roughout Ranch (RRFI) may be a worthy recipient of this grant, I believe its current proposal is far from sufficient to win this grant. Here are a list of my concerns regarding this proposal:

 

1. The intended outcomes of this program are too broad and too vague: Here are some selections of the program’s stated goals from the proposal:

The goal of both programs is to create communities of trust that work together for mutual benefit. In the short term, these sessions promote self-healing and empowerment and affect participants’ ability to enjoy interacting with others. The objective is to increase each participant’s sense of confidence.

The goal is to improve overall well being and functioning in multiple domains of an individual’s life. If completion of the Authentic Workshops and Horse Encounters creates any improvement for an individual, the program is successful.

The program’s goal is to improve wellness, recovery and resiliency of those who participate in the service.

While these are all important dimensions of mental/behavioral health, they are too broad to enable us to know exactly what they would hope to achieve and whether they would be a fit for the county’s strategy. In particular, the statement that if this program “creates any improvement for an individual, the program is successful,” demonstrates the lack of focus.

 

2. The current level of evidence for efficacy is very low: While it’s understandable for a program applying for an innovation grant to have a low level of evidence, the only cited evidence of efficacy is statements from participants. These statements have not been included in the proposal. Only one family presented during the last MHADAB meeting. 

Further, the program references the impact of “equine therapy,” but notes that the model proposed is not equine therapy. As a result, we cannot assume efficacy based on the use of evidence-based practices already widely established as effective.

 

3. There is no real plan for measuring effectiveness: There are 2 references to measurement strategy:

Tracking an individual’s functioning at referral then tracking it post program, the county will be able to determine the number of individuals reporting success.

The project included statements from individuals and families who have actively participated in Authentic Workshop and Horse Encounters sharing how this program has improved their lives. 

The first could hold potential in the sense that it involves pre-/post-evaluation, which would provide much stronger evidence than the anecdote(s) that exist today. However, this reference is too vague to determine what would be measured and if it could be measured successfully.

The second reference to measurement strategy is more of a reference to what has already been done – which as noted earlier, is limited to anecdotal evidence, the least robust form of evidence.

 

4. The target beneficiary population is far too broad: Social services tend to be effective for specific populations, requiring adaptation to work for other populations. While it’s not clearly articulated in the proposal, the in-person presentation at the last MHADAB meeting suggested that this program would serve men, women, children, those with special needs (e.g., physical and cognitive disabilities), and those with mental illnesses. Casting such a broad net is sure to reduce overall effectiveness and engagement because marketing/referral strategies would need to differ by target population, instructor skills/qualifications may need to differ, and the program itself may need to be adapted for specific populations. 

 

5. The focus should be on proving the efficacy of the program before scaling the workforce: The primary focus of this proposal (phases I-III) is training more people to deliver this intervention. However, we don’t know what outcomes it intends to achieve and if it currently achieves those outcomes. Phase I of this proposal should be testing the effectiveness of the program for the people it can already serve with its existing workforce. Only after proving it is effective, should we look to scale the intervention. Further, we should allow demand to drive scaling of the workforce – if RRFI can build a waitlist, then it should build the workforce to meet the demand. It is risky to build a workforce with the hope that demand will follow (as I discuss further below).

 

6. We don’t know the demand for this program: The proposal claims that scaling the workforce is an important first step because RRFI needs to “onboard adequate staff to meet the demand.” But we don’t have any quantitative evidence of the demand for this program. It would be hard to assess the demand given critique #4 – that is, we don’t really know who the program is for, so how can we know how many people want to participate? Further, even if we could demonstrate a significant need for the program, need doesn’t equal demand, as I have documented here.

 

It’s possible that these critiques can be addressed by information not in the proposals or information included that I overlooked, but I would encourage the MHADAB to withhold approving this proposal either entirely – or at a minimum, until these critiques are adequately addressed.

One of HHSA’s Mental Health Services Act program goals published last June is:

Identifying the best program outcome measures and improving monitoring and analysis through interdepartmental and community partner collaboration.

In my opinion, approving this grant without addressing these critiques would not be consistent with the intent of that goal.

Finally, I want to clarify that my intent is not to disparage RRFI. I appreciate their sacrifice to serve the community to date and their desire to meet what clearly is a real and growing need in our community. At the same time, it is critical that we spend our limited resources in the best possible way to improve the mental and behavioral health of our community. If they are interested, I’d be more than happy to talk with RRFI to help them address these critiques because I want them to be successful. 

 

With Gratitude for Your Service & Consideration of these Comments,

Matt