Professional evaluation of career counseling programs is becoming increasingly important as funding becomes more limited and accountability more pervasive. The challenge is to make evaluation a positive experience for the counselors, administrators, and clients, while, at the same time, to satisfy the more quantitative needs of the policy planners and program funders. Greene (1994) has suggested that the more narrative, participatory approach of qualitative evaluation, can foster greater understanding, solidarity, and contextualization of a career counseling program. This paper describes the approach used by an outside team to evaluate seven small career counseling programs. The evaluations were funded mainly by the major sponsor of the counseling centers. Since the evaluation team wanted to involve and satisfy both the funder and those involved in service delivery, a quantitative/qualitative evaluation process and a strength challenge approach to counselor feedback was used.

Although the evaluation mandate varied somewhat among the seven centers, evaluators addressed two basic goals:

1. Appraise the effectiveness and efficiency of the administration and delivery of career counseling services, and

2. Suggest areas of improvement and identify areas where initiatives had been taken or where innovative procedures had been developed to improve counseling services. The following eight areas were identified for review: client inventories and record keeping; service-needs determination; leadership, planning and organization; client perception of employment coordination and counseling; liaison with referral agencies; individual and group counseling; employment success rate; and additional services and innovations.

DISCUSSION OF THE EVALUATION PROCESS

1. Examination of materials such as written mission statements, objectives, monthly and annual reports. The evaluation team wanted to learn as much as possible about the center before the on-site review.

2. Preparation of suitable review forms. Counselors and administrators had an opportunity to see various forms and select the ones they felt would be most helpful. The "Project Worker Issues" form, was used most frequently, since it helped to stimulate discussion. This open-ended form included five major areas:

a) Client Issues: Describe your typical clients. What do your clients expect of you? What changes would you like to see so that clients could be served better?

b) Operational Issues: What is done at your center to foster team building? What is done with client follow-up? What structural changes would help you offer better services to clients?

c) Accountability Issues: Comment on client feedback services provided by your center. What measures are taken to be accountable to funding agencies? Does management provide written materials on management style, standards of counselor conduct, etc.?

d) Self-Assessment Issues: Do you have a written set of standards for the services you offer (e.g., number of sessions, wait time, referrals, etc.)? Can you "deliver" the services that you offer?

e) Professional Development Issues: How are the competency levels of counselors assessed? Are professional development days provided for all counselors? What are your future plans for professional development?

3. Meeting of the review team members to plan the specifics of the 2-3 day on-site evaluation.

4. A 2-3 day on-site evaluation by the evaluators.

5. Evaluators meet to discuss findings and plan the recommendations.

6. Preliminary report of the evaluation team sent to the career counseling center for comments and to correct any inaccuracies. This report contained information on the eight areas for review identified earlier.

7. Final report sent to the career counseling center.

8. On-site visitation by the review team to discuss the recommendations and possible ways of implementing the recommendations.

The key part of the whole evaluation was the 2-3 day on-site visitation by the review team. The attitude the team attempted to foster was "what can we do in the next 3 days that might make this process useful to counselors and clients?" At an introductory meeting with all staff (sometimes lasting several hours) this question, other open-ended questions, and fun introductions helped to set a relaxed tone for the evaluation process.

The evaluation team found that helping clients, counselors and administrators become more comfortable talking about themselves and their career counseling was of great value in getting a better picture of what the counselors were doing and what clients were receiving or expecting. Savickas (1989) used the term "enhancing narrative ability" to describe this process. Several activities that worked well included the following:

The evaluation team shared their own experiences, talking about heroes or heroines who had influenced them. One counselor said that when she was seven years old she adored Florence Nightingale, and later, as an adolescent, she decided she wanted to be like Mother Theresa. One of her colleagues volunteered, "You are Mother Theresa to us and all the people you see." The evaluators learned a lot about this woman's attitudes toward people and counseling from her story.

"Tell us about significant people in your life who would be good to have as colleagues at your center."

List five of the most important things that are occurring at your center.

The remainder of the on-site evaluation time was spent talking with staff members, examining and discussing files, monthly and annual reports, and referral systems, and being observers in individual and group counseling sessions.

In response to the question, "What could make the evaluation worthwhile?", the majority of counselors asked for feedback on their counseling with clients. Most had arranged with a client to have an evaluator present during a counseling session. Formative evaluation for individual counselors was enhanced by using a strength-challenge style of feedback (Borgen, Amundson, & Westwood, 1988). In this approach the counselor receiving feedback collaborates beforehand about the focus or direction of the feedback. When the feedback session begins, the receiver is allowed some self-analysis, followed by specific comments from the observer. The feedback is very specific and focuses on what the observer saw. The emphasis is on the strengths or competencies that a counselor demonstrates and the possible need to make greater use of the talents or strengths. For example, an evaluator might say: "I thought your use of reflection of feeling was very effective. You said: '...and now you're really frustrated and angry that you can't find suitable employment...' Your client responded to this and vented some of her anger and frustration. This type of reflection of feeling works well for you and I encourage you to use it more" (Schulz, 1990.)

Several hours were used at the end of the on-site visit to again meet with all counselors and administrators. This final meeting gave the staff an opportunity to ask questions, to "dream a little" and list some things needed at their counseling center, and to talk about any special counseling services that their center had introduced for clients.

RECOMMENDATIONS FOR EVALUATORS

Take time to build rapport with the career counselors before beginning the actual review.

Inform all the stakeholders beforehand that they will be involved in reviewing the evaluation report before release.

Obtain staff and administration input into the review process and review instruments.

External reviewers should spend much time with the evaluation process and should be aware of the process interacting significantly with the outcome goals. Both formative and summative evaluation should be stressed.

Case studies and anecdotal information should be part of the career counseling evaluation report.

In-person follow-up of the review was highly valued and is strongly recommended as the final step of the evaluation process.

REFERENCES

Greene, J.C. (1994). Qualitative program evaluation: Promise and practice. In N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative research (pp. 530-544). Newbury Park, CA: Sage.

Savickas, M.L. (1989). Career-style assessment and counseling. In T. Sweeney (Ed.), Adlerian counseling: A practical approach for a new decade (3rd ed.) (pp. 289-320). Muncie, IN: Accelerated Development Press.

Schulz, W.E. (1990). Making feedback a positive experience. Unpublished manuscript. Winnipeg, MN: University of Manitoba.

William E. Schulz, Ph.D., is a professor in Counsellor Education at the University of Manitoba. He has published numerous books and articles on counseling, ethics, career counseling, interpersonal communication, and program evaluation.

ERIC Digests are in the public domain and may be freely reproduced and disseminated. This publication was funded by the U.S. Department of Education, Office of Educational Research and Improvement, Contract No. RR93002004. Opinions expressed in this report do not necessarily reflect the positions of the U.S. Department of Education, OERI, or ERIC/CASS.