Module #2: The Screening Process and Quality Assurance

Contents

• 2.1 MH/AODA Practitioners Complete the Functional Screen with Consumers
• 2.2 The MH/AODA FS Requires an In-Person Interview
• 2.3 The MH/AODA FS is not an Interview Tool
• 2.4 The MH/AODA FS is a Needs Inventory
• 2.5 The Functional Screen is Voluntary for Consumers
• 2.6 Confidentiality
• 2.7 Requirements for Quality Assurance
• 2.8 Screener Qualifications
• 2.9 Screen Leaders' Role and Responsibility
• 2.10 Quality Assurance and Screening Limitations
• 2.11 Strategies to Reduce Screen Limitations
  • 2.12 Ensuring Inter-Rater Reliability
  • 2.13 Conflicting Information from Different People
  • 2.14 If the Consumer Seems to Overstate Their Needs
  • 2.15 If the Consumer Seems to Overstate Their Abilities
  • 2.16 If AODA is an Issue
  • 2.17 Frequency of Need for Assistance
  • 2.18 Ranking Fluctuating Needs
• 2.19 Impending Discharge 
  
  

2.1 MH/AODA Practitioners Complete the Functional Screen with Consumers

Screens may only be done with consumers by a mental health/AODA practitioner who has met DHFS qualifications and training requirements. Screeners must be direct or contracted employees of county agencies specified by DHFS as screening pilot agencies.

The MH/AODA FS needs to be completed by MH/AODA practitioners, not consumers themselves, in order to obtain federal approval of its use. We agree that this is not as consumer-focused as we'd like, and in fact adds a significant risk of practitioners' subjectivity to the screen. Instructions, trainings and quality assurance processes (including close communication between screeners and state staff) will reduce subjectivity enough to ensure reliable and valid results.

[Back to Top]

2.2 The MH/AODA FS Requires an In-Person Interview

The screening process requires face-to-face contact with the individual being screened. No screen should be completed without a meeting with the consumer, even if they have communication difficulties.

The face-to-face interview can take place in any setting (e.g., the applicant's residence, a group home, day services center, a hospital or a restaurant). It may take more than one contact with the consumer to complete the screen.

[Back to Top]

2.3 The MH/AODA FS is not an Interview Tool

The MH/AODA FS is NOT an interview tool; do not just go through it like a checklist with someone. The screen can be done in any order. More importantly, it should be done within a larger conversation about the whole person, not just their needs.

Screeners should use their best interviewing skills to gather information in a way that is appropriate for a given consumer. The screener should ask questions in a variety of ways, and use collateral informants as necessary. Collateral informants include family, significant others, formal or informal caregivers, health care providers, and agencies serving the consumer. A release of information must be signed by the consumer to access collateral information (either written or verbal). The screener must always have a face-to-face contact with the consumer, even if other informants are used.

[Back to Top]

2.4 The MH/AODA FS is a Needs Inventory

The MH/AODA FS is based on the person's diagnoses, symptoms, and need for help from others (i.e., functional impairments). Such a deficits-based approach to mental health service delivery and practice is fundamentally contrary to recovery-oriented practice. That said, program eligibilities (particularly under federal Medicaid guidelines) are based on the need for program services. A functional eligibility screen by definition must focus on functional impairments, i.e., needs for assistance. This means two things concerning the functional screen process and content:

• Screeners must complete the functional screen within a more holistic dialogue.
  Most people would find it insulting, upsetting, and discouraging if MH/AODA practitioners just read off the screen items like a checklist. Instead, screeners should use recovery-based best practices, including learning what the person needs help with within a larger, recovery-focused dialogue that includes the person's strengths, values, goals and perspectives.
  
  
• The screen content will be revised based on screeners' and consumers' feedback. We will work together to revise the screen (and instructions) based on your experience with it. Revisions will focus on making the screen process acceptable to consumers and screeners, and on making the screen work well, with accurate and reliable data.

[Back to Top]

2.5 The Functional Screen is Voluntary for Consumers

The person being screened should consent to completion of the functional screen and its submission to DHFS for screen development and aggregate data research. No screen should be completed without the person's signed informed consent. However, where the screen is the tool for determining need for services, the consumer needs to know that refusal to participate in the screening process could affect their eligibility for services. All information will be confidential within the Department and the screening agency.

The Department has chosen to make this screen the approved screen for CCS admissions for adults (the new children's screen will be used to determine CCS for children). It will also be the approved screen for COP eligibility for COP level 3 individuals with serious and persistent mental illness. For CCS, it is an annual requirement. It will remain voluntary for CSP at this time, however many counties may want to begin using the screen for CSP as it will give each program access to valuable data on the CSP population locally. For the first time, local agencies will have a mechanism to be able to sort caseloads by living situation, diagnosis, risk factors and ADL functional levels.

[Back to Top]

2.6 Confidentiality

Any information collected for the screen or during the screening process is confidential. It is to be treated with the same requirements for confidentiality as other long-standing screens and assessments. Screeners and screening agencies shall comply with confidentiality rules and requirements and shall obtain a signed release of information from the person or the person's guardian or power of attorney, where applicable, for the use of medical records, educational records and other records as appropriate before conducting the MH/AODA FS. Signed releases of information shall be included in the person's records as appropriate.

All aspects of the functional screen must comply with HIPAA (the Federal Health Insurance Portability and Accountability Act). The functional screen is a secure on-line computer application, with access limited to Department-assigned users verified by passwords.

Screeners should inform consumers of this, and of the following: A completed functional screen can be seen only by screeners of the screening agency. If MH/AODA screeners elsewhere in the state were to key in the individuals' name or Social Security Number, the program would inform them that a functional screen had been completed for that individual and would show the screening agency and date. That is all that can be seen; the content of the individual's functional screen cannot be seen by anyone except screeners of the screening agency (including the county agency if subcontracting).

Later, when the MH/AODA FS is fully implemented, it will form the basis of a comprehensive assessment by a MH/AODA provider. The provider will ask the consumer to sign a release of information, which allows the original screener to electronically "transfer" permission to see the completed functional screen content. If consumer chooses to sign the release, then the new case manager will read the functional screen to avoid asking all the same questions again. The new case manager will do a complete assessment building from the information shown in the MH/AODA FS.

[Back to Top]

2.7 Requirements for Quality Assurance

Because the MH/AODA FS will (after testing) determine program eligibility, special requirements for quality assurance and screener qualifications are necessary.

Each pilot county will assign a "Screen Leader" who will be responsible for facilitating screen testing, gathering screeners' questions and feedback, and passing them on to designated state staff as soon as possible.

Every screener should test the MH/AODA FS thoughtfully - do not simply fill screens out, but really critique it. Test it on people with high needs, low needs, different sorts of needs, people now eligible for CSP (or BRC target groups 1 or 2) and those who are not.

Never guess or extrapolate on anything. When you encounter something not covered in these instructions - and you will - leave it blank and discuss it with your Screen Leader. The Screen Leader will call or e-mail designated state staff, who will answer the question and the update instructions as necessary. No one should make interpretations; instead, pass your questions on to state staff (usually through your Screen Leader). Questions can be answered locally only if they are found in these instructions. This will be most helpful to this project as we test the screen and the instructions for clarity.

[Back to Top]

2.8 Screener Qualifications

All persons administering the functional screen must meet the following conditions:

1. Meet the following minimum criteria for education and experience:

• Nursing license or a BA or BS, preferably in a health or human services related field, and at least one year of experience working with at least one of the target populations, or
• Prior approval from the Department based on a combination of post-secondary education and experience or on a written plan for formal and on-the-job training to develop the required expertise; and

2. Meet all training requirements as specified by the Department. Currently that means:

• Completing the online course, or
• Attending an in-person training by Department staff (or watching video of same), and
• Reading and following screen instructions.

[Back to Top]

2.9 Screen Leaders' Role and Responsibility

Among the Screen Lead's roles is to oversee quality assurance. As the MH/AODA FS becomes more fully implemented, each agency will have on-going quality assurance methods, such as:

• Training, mentoring, and monitoring new screeners
• Providing on-going feedback and questions to state staff
• Random sampling for accuracy and consistency
• Inter-rate reliability testing.

[Back to Top]

2.10 Quality Assurance and Screening Limitations

It is recognized (and shown in research) that all functional assessments or screens are imperfect. In particular, screeners should be aware of the following limitations found in national studies to be characteristic of all functional screens:

• Health care and institutional providers tend to overrate the consumer's dependency on others.
• Guardians, spouses, and family members often tend to overrate the consumer's dependency on others.
• Consumers may be unable to provide accurate information or may exaggerate their abilities or their need for assistance.
• Consumers' status and abilities fluctuate, making it difficult to choose the most accurate frequency at which help is needed.
• Consumers can provide conflicting information at different times or to different screeners.
• Screen answers vary somewhat depending on whether the screener knows the consumer well or not.
• Screen answers vary somewhat depending on the profession of the screener.
• Particularly in the field of mental health, some subjectivity remains in some questions.

[Back to Top]

2.11 Strategies to Reduce Screen Limitations

The MH/AODA FS will be repeatedly revised based on feedback and testing until it has acceptable levels of validity and reliability. However, it is generally recognized that any objective rating of consumers' functioning, cognition, behavior and symptoms can be difficult. This challenge calls for extra vigilance to ensure the greatest possible accuracy in the functional screen. This is why screeners must be trained and experienced with the population, and why DHFS and counties using the screen must have ongoing quality assurance processes.

Screeners should adhere to the following guidelines:

• Read and follow screen definitions and instructions closely.
• Go slowly and carefully enough to be accurate even with someone you know well.
• Do not "inflate" any answers because you think a consumer has special costs not "visible" through the screen. Instead, you should always select the answer that most accurately describes the applicant's functional status

All screening agencies should have designated "Screen Leaders" to assist you with questions. Refer all questions to your Screen Leader, who will in turn refer questions to designated staff at the WI Bureau of Mental Health and Substance Abuse Services. (You can also contact designated State staff directly.) In this way, interpretations will be kept consistent and communicated to all screeners, and revisions can be made to the screen and instructions as necessary. It is absolutely critical that all screeners participate in this process.

[Back to Top]

2.12 Ensuring Inter-Rater Reliability

For many items in the functional screen, functional status is ranked by the frequency at which the person needs help from other people. If someone is marked as needing help from others, then it is expected that either the person is getting the help (i.e., it would be in a individualized recovery plan) OR they are suffering negative outcomes or risk thereof from not getting the help.

Screeners should always think about inter-rater reliability before selecting answers on the MH/AODA FS. Do not mark your impression without first considering whether you have objective information that indicates that answer such that most other screeners would select it. Follow these important steps before selecting answers on the functional screen:

• Follow the precise definitions on the MH/AODA FS.
  These were written precisely to reduce subjectivity and enhance inter-rater reliability.
  It is imperative that all screeners read the instructions and follow up with their Screen Leaders as needed.
  
  
• Ask yourself "What objective information do I have, and what would a different screener mark based on that info?" This means the following:

1. Gather objective information; do not merely mark your opinion.
2. Consider what other screeners-including those of a different gender, class, age, etc. from you-would mark based on the objective information.
3. Consider what is considered typical for members of the person's culture.
4. If you are unsure regarding A, B, or C, contact your Screen Leader or designated State staff to discuss the situation.

Screen Leaders should facilitate regular times for screeners to discuss screen questions, especially concerning cultural competence and inter-rater reliability. Screen Leaders can then share this feedback with Department staff for improving the screen and instructions.

2.13 Conflicting Information from Different People

Sometimes screeners will get different information from different sources. Consumers may function less independently in day care facilities or institutions than they do at home, and staff at such facilities may tend to perceive more dependency than family or peers in the community might perceive. Screeners are to use their best professional judgment to describe the person's functional abilities as accurately as possible given all the information from multiple sources.

[Back to Top]

2.14 If the Consumer Seems to Overstate Their Needs

Remember that the goal is to be as objective as possible, to have high "inter-rater reliability," meaning that other screeners would choose the same answer you did. That is why your professional judgment must be based on as much objective information as possible.

Objective information can be obtained by skilled interviewing and observation. If the proper answer is still not clear, discuss it with your Screen Leader, who can then, if necessary, ask DHFS for guidance. If the consumer appears to be overstating their needs, ask:

• Have they in fact been doing the task adequately and without significant hardship?
• Does someone in fact help them with it now?
• What has consumer done (or what would they do) if assistance were not available?
• What would the consequence be if assistance were not available?

If no hardship results from a lack of assistance, it implies that the person is in fact able to do the task himself or herself. This, of course, is the familiar "needs versus wants" distinction, but with the thought process clearly spelled out. Doing so shares the power with consumers, improves inter-rater reliability, and reduces at least some errors from assumptions or biases.

[Back to Top]

2.15 If the Consumer Seems to Overstate Their Abilities

Sometimes MH/AODA practitioners recognize a need for help that the consumer does not recognize. Recovery principles reject the old "professional knows best" model and require that the consumer's perspective be central. Yet MH/AODA practitioners, particularly professionals, are responsible for noticing things that consumers might not notice, and for following up as much as the consumer allows. Example:

• Joe has decided he doesn't need his Lithium anymore, and hasn't taken it for four days, and threw his remaining pills away. He says he'll be fine and you don't need to be coming around to see him. In the past, Joe's had big problems when he's gone off his medication. You negotiate with him to at least accept your visits. You'll be talking with his psychiatrist, getting the prescription refilled, negotiating every day with Joe to try to get him to take it and to see his psychiatrist. On the functional screen, you check "More than weekly" for both "Taking Meds" and "Managing Psychiatric Symptoms" for Joe.

[Back to Top]

2.16 If AODA is an Issue

Individuals' responsibility and choice-making capacities are less clear when alcohol or drug addiction is involved. If a person's purchase and/or use of drugs or alcohol interferes with their ability to meet basic needs (food, clothing, and shelter) the screener will probably indicate some need for assistance with some tasks. The things to remember are:

• Gather details from consumer and others (as permitted by consumer);
• Consider what other trained screeners would select given the same details; and
• Consider what frequency of assistance would seem appropriate in a MH service/treatment plan for this person.

[Back to Top]

2.17 Frequency of Need for Assistance

Check a given frequency on the functional screen if it reflects actual needs as closely as possible; not merely what the person says they need or don't need, and not merely what the screener thinks, but something in between. It is what most screeners would agree should be in a service/treatment plan for that individual. Check a given frequency:

• No matter who's providing it - family or providers.
  • Example: If the family provides med reminders every day, that's the frequency of help needed from others.
    
    
• Even if the individual verbally denies the need but accepts the help, including visits to monitor, cue, negotiate.
  • Example: Joe decided he doesn't need his Lithium anymore, and hasn't taken it for four days, and threw his remaining pills away. He says he'll be fine and you don't need to be coming around to see him. In the past, Joe's had big problems when he's gone off his med. You negotiate with him to at least accept your visits. You'll be talking with his psychiatrist, getting the prescription refilled, negotiating every day with Joe to try to get him to take it and to see his psychiatrist. On the functional screen, you check "More than weekly" for both "Taking Meds" and "Managing Psychiatric Symptoms" for Joe.
    
    
• Even if the individual currently refuses any help or discussion of the topic.
  • The functional screen is intended to indicate what the person needs, even if they refuse the help. An individualized service plan is negotiated with the consumer, but the functional screen looks only for needs.
    
    
• Even if it exceeds a provider's capacity to provide help at that frequency.
  • The functional screen seeks statewide consistency in indicating mental health service needs. Counties vary in how they distribute scarce services. Screeners need to think beyond their own agency to ask what other screeners in any county would indicate for an individual's needs. These instructions are intended to help you do that.

So, frequencies selected on the functional screen will usually match frequencies of MH/AODA case manager contacts with the individual. In this way, completing the functional screen is similar to developing an individualized service plan with someone. There are three exceptions:

• If the consumer declines the services, you still mark them on the functional screen.
• Your agency does not or cannot provide the high frequency needed.
• The functional screen looks for help needed from anyone, not just MH/AODA practitioners.

[Back to Top]

2.18 Ranking Fluctuating Needs

Mental illnesses are often cyclical, with varying levels of symptoms and functioning. The MH/AODA FS is a mix of a "snapshot" view - the person's current status now and over the past few weeks - and an historical (or "movie") view looking over the past few months up to the past year or two. Every day MH/AODA practitioners (and consumers) make judgments based on this mix of snapshot and historical views, to determine the frequency of contacts and of help needed now and for the next few weeks or months. Of course people's needs will change, and of course predictions are only approximate, but they reflect expert judgment (and sometimes research data) of the frequency of interventions needed to promote recovery and prevent crises.

The MH/AODA FS is similar. For some "Community Living Skills," you are asked to indicate the approximate frequency at which help is needed. To make it easier to select answers, the answer choices have been reduced to "Independent," "Less than monthly," "1 to 4 times a month," "More than one time a week."

Less than monthly
Check this for applicant who, for example:

• Sees their case manager only every few months and is otherwise independent.
• Has had two or three episodes over the past year, requiring interventions for 1 or 2 weeks each episode - such that it averages out to less than monthly; and the episodes are unpredictable such that regular and more frequent assistance would not prevent the episodes.
• Recently became independent with a skill, but still needs some follow up and back up.

1 to 4 times a month
Check this for applicant who, for example:

• Needs help with budgeting and finances just 1 to 4 times a month.
• Needs help every other week, for instance, with housekeeping or grocery shopping.
• Needs med boxes filled every two weeks (filling two weekly med boxes at once).
• Is still developing skills they learn during visits with their case manager every two weeks.
• Has crises if they don't get regular emotional support and reinforcement 2 or 3 times a month.
• Does pretty well most of the time, but sometimes calls their case manager for support; this happens irregularly, but on average over the past six months or so, it's about 1 to 4 times a month.
• Does not recognize when symptoms escalate, and they do so within 2 or 3 weeks; person needs someone every two weeks to monitor symptoms and prevent crises. For example, Stu has had crises from manic episodes 2 to 3 times a year. His mania progresses rapidly, within 2 or 3 weeks, and he doesn't see it starting. He needs someone to check in with him every two weeks to monitor for mania and help him prevent its escalation.

More than one time per week
Check this for applicant who, for example:

• Needs someone to give them their meds (psych and others) every day or more often.
• Forgets to take their meds unless the person's daughter calls to remind them every day to take them.
• Needs intensive case management and/or psych nursing visits 3 to 5 times a week.
• Comes in to the clinic every morning for meds and money.
• Does not recognize when symptoms escalate, and they do so within a day or two; person needs someone every two days to monitor symptoms and prevent crises. For example, Marilyn has schizophrenia and lives alone. Her MH case manager continues to see her three times a week to help her cope with her symptoms. With this support, Marilyn has only been hospitalized twice in the past three years.

[Back to Top]

2.19 Impending Discharge

When completing the functional screen with someone preparing for discharge from a mental health facility within the next few days, complete the screen based on how the person is expected to function at home when they go home. This looking ahead is a normal part of discharge planning. So, if, for example, although the facility administers medications now, mark on the screen if the person will need help with medication administration after they go home. It will take additional time and talking with the individual, facility staff, family, etc. to get the most accurate picture of the person's needs at home after discharge.

The screener must be able to envision the person at home. This is why screeners must have experience in community-based mental health/AODA services.

[Back to Top]