Appunti Public Management

CONS of evaluating client satisfaction

- do not answer questions about outcome in human service – that is, they don’t tell us whether clients actually improved as a result of the services provided to them;

- fail to provide objective information on the impact that receipt of services had on the reasons clients approached their agencies for help;

- preponderance of positive responses on consumer satisfaction surveys caused by gratitude or fear of alienating their caregivers or self involvement in the program otherwise it means to recognize the waste of out time. – the feeling of engagement could be a bias in receiving fair and true answer to the question that you make to the respondents, so you should have an high expectations of the positivity of the answers.

Why to evaluate client satisfaction:

- if consumer feedback studies are not conducted, then there is no organized or systematic means for learning about clients’ experiences with programs;

- even if the ratings do tend to come back relatively high,

that is a good thing—reassuring everyone (clients, staff, and management) that there are no hidden or obscure problems lurking just below the surface; - examine client satisfaction with different provision procedures, e.g. three therapies for alcohol dependence (cognitive-behavioural, motivational enhancement, and 12-step facilitation). If ratings come back with clients reporting below 75 percent satisfaction, then there is at least a suggestion that further investigation ought to probe for the source of dissatisfaction. The Client Satisfaction Questionnaire (CSQ-8) – easy, short questionnaire to have result of “are you satisfied”. Longer version for more detail information regarding the process of implementing the program you’re evaluating (CSQ-24) - The use of the same questionnaire gives the possibility to make comparisons; - Average evaluation is 24. – made by attendant of the program for 90%. But you should ask those questions also to the other. So you

have to compensate and to balance people that attend more with people that attend less.

In average, when a person is satisfied, it gives his good evaluation to 3 people. When he’s dissatisfied, he express his dissatisfaction to 20 people.

People who leave the program are more prone to be VERY negative, while people that are positive are usually prone to give and AVERAGE positive. So potentially, if you make a bad balance between the two, the final result could be negative for the preponderance of the negativity of people who left the program.

Advice: use already provided questionnaire in making client satisfaction, because already tested with the psychometric point of view, that is: the capacity of the questions to influence the answer of the people because of the way in which they are formulated.

Client satisfaction: some other comments

• Consumers’ evaluations of their service experiences can be influenced by the provider’s preestablished reputation for reliability;

Quantitative assessments tend to yield high levels of satisfaction, while qualitative reports tend to reveal lower satisfaction; - people much more prone to say something negative in speaking more than giving a number to evaluate the quality of the program.

Satisfaction with services has shown to be positively related to age; - older people more positive evaluation, you should consider and balance also the age of the people.

Those who are not happy with services tend to drop out. Satisfaction is correlated with clients' length of stay in treatment;

Client satisfaction has been shown to be associated with symptom relief; - important to consider when to make the evaluation, because usually if the program provide immediately some results, it's much better to delay the client satisfaction, because otherwise (because of the immediate relief of the problem) people could be incentivate to answer positively to questionnaire.

Open-ended questions can provide "illuminating"

Information about the client's perspective that may not be discovered through using objective instruments:

• Telephone surveys yield higher response rates than mailed surveys.
• Low response rates should be expected. The average response rate for mailed client satisfaction questionnaires is usually between 28 and 46 percent.
• Clients who return mailed surveys tend to have higher levels of educational attainment than those who do not respond.
• Families told to obtain services by the court or school were significantly less satisfied; volunteers were more satisfied than those court-ordered to attend divorce education.

Client satisfaction: how?

• Surveys: because of the convenience provided in either handing these out to consumers at the end of a treatment episode or mailing them at some point (e.g., a month) after terminating services.
• Focus groups are also an option: especially if the agency or evaluator wants to probe deeper than the typical questions asked on client satisfaction.

Forms:

When you use a survey and when a focus group?

Surveys:

More dissimilar the people receiving a program, much better it is to make a survey because you need to collect information related to as many people as possible.

Focus groups:

More similar the people receiving a program, much better it is to make a focus group because you can go much in detail.

- Use a scale that has good reliability and that has been used successfully in other studies. By all means, avoid the use of any hastily created questionnaires for which there is no psychometric information.

- Employ at least one and possibly two open-ended questions so that the consumers of your services can alert you to any problems that you did not suspect and could not anticipate.

- Use the same instruments on repeated occasions and develop a local baseline of data so that departures from the norm can be observed.

In terms of procedures, what you should do is to make something similar to the Single System Research.

Design: you should make an evaluation of the quality along time. This gives you the possibility to eliminate from the final evaluation of the satisfaction the answers that depends on the contingency of when you did the evaluation. Some stratagems that you could use along the process in order to make your final assessment more reliable: - to avoid the problem of surveying only the clients who remain in a program (and therefore would be more likely to have positive experiences than those who drop out), use a "ballot box" approach – in selecting people, use a procedures more "random". - Do not focus only on solving the problems of the dissatisfied consumers; - bias during evaluation: concentrate only to negative aspects and not on positive ones. - Reward respondents to increase the response rate; - Target specific dimensions of the quality that you want to evaluate. Instead of thinking about client satisfaction as an all-or-nothing phenomenon, try to identify an area.

• Identify areas where problems may exist
• Set realistic expectations
• Look for behavioral indicators (such as attendance) to supplement client satisfaction data

A quality model: what is important?

• The quality evaluation is considered useful only if the information you have can provide feedback to improve the service and revise the program
• Client satisfaction is just one dimension of overall quality; other factors to consider include staff results, societal impact, and agency results

European Foundation for Quality Management (EFQM) model

Lesson 15: Sampling

Especially for group-based design.

Introduction

Sampling is fundamental for quantitative evaluation.

In quantitative PE, it is normal to find answers for

questions like how many persons should we survey? What would be a good sample size? We have to learn how to distinguish among several typologies of samples without getting into many details. General Classification Nonprobability samples: - do not represent population - Used for Pre-experimental and the Quasi-experimental designs - Samples of convenience - Purposeful samples - Snowball samples - Quota samples - Maximum variation and deviant samples - so, no generally applicable results (because comes from a sample which is not representative of the entire population). So why we use it? - Standard (-> topic): For example to build a standard, so to have the definition/examples of some topics that you can consider relevant after the end of the evaluation, and that topic you start thinking that are the most important aspects that should be evaluated. - Build a standard: what is much more relevant and commonly perceived as a need for the population living in that area. - Controlgroup-> When population not well known-> For people reporting specific features, so you need a specific selection of people. Probability samples -> Entails a statistical representativeness of the population-> Used for experimental design -> possibility to extend the validity of your result to all the population. - Simple random samples - Systematic random samples - Stratified random samples-> Represent the population: the same results are also reach by all the member of all the population. The main criterion to respect for the arrangement of the selection is the RANDOMNESS. But also, PROBABILITY: Everybody, being part of the population, should have the same probability to be selected (likely occurrence). Features Nonprobability samples- Advantage: easier, cheaper, less time consuming and, sometimes, capable of representing the reality.- Disadvantage: the sample could represent very poorly what we are looking for in the population we have to study (conditions, thinking, feelings,

Probability samples

Advantage: the sample – if correctly chosen - should be a smaller mirror image of the larger population;

Disadvantage: more expensive, more time consuming.

NPS (Nonprobability samples)

Sample of convenience: a group of persons easily obtained since they are close at hand and accessible.

Examples:

• Approaching people in a mall;
• Handing out surveys to students sitting in a college course;
• The first 10 clients who walk in without appointment at an outpatient clinic on the first two days of the week

Typical case sampling: to obtain a collection of cases who represent the “average” client.

Example:

• We can ask clinicians to give us information that is to nominate some clients that, in their opinion, are the typical clients.

Purposeful samples: a group that by definition has certain characteristics in common (the ones the evaluator is looking for).

Examples:

• The oldest, the youngest, the most recent, etc.