September 4, 2008
Week 2 Reflections: Standards, principles & guidelines… Oh my!
If asked about evaluation a few weeks ago, I would have guessed that standards for it exist and that most of the evaluations I have been involved in would not “meet the test or measure” of those standards. Now I’m certain of it! Reading the Program Evaluation Standards was interesting and informative yet daunting and overwhelming. I certainly now understand why there are professional evaluators. While reading the standards, however, a list of words and phrases began playing in my head. First of all came “communication”, “respect” and “common sense”. Then I added “cover your bases”, “double check” and “document”. During the section on metaevaluation I was thinking of Tom Anderson’s course! This section seemed to be the meta of all metas. So I needed to add, “do it all over again” to my list.
Utility. If one is going to take the time to do something (especially those in a professional like ours where time is a valuable commodity), then certainly we want it to be useful and address its purpose. It makes sense to take into consideration those involved or affected, that the person doing the job should be competent and knowledgeable, that the task should be done in a timely manner and that the end product should be usable. The more time and resources spent on something, the more we expect to get in return. Probably the biggest task (which involves evaluating programs) I’m presented with every three years is writing the district tech plan. Too many of them have been met with indifference or roadblocks. I was even told once by a superintendent that “Nobody cares.” My biggest surprise in terms of the utility standards came in standard U4, values identification. Common error A states, “Assuming that evaluations can be objective in the sense of being devoid of value judgment.” A huge part of evaluating is trying to be impartial, yet in the end there is a value judgment.
Feasibility. I’ve always thought of that as whether something was possible or able to be done. The standards take it a few steps further. They speak of diplomacy and political viability, cost and disruption. How many times have school districts been guilty of “disrupting program activities in an attempt to collect information”? Our entire district staff took a time consuming, confusing and extremely long technology survey for our NextSteps program – twice. (They would have done so a third time but I pled repeatedly for mercy.) Although taken using a computer, the results could not be had electronically, but could only be printed. You could not get a comparison of the second against the first. Again, printouts were available. I spent untold hours trying to gather information from them and in the end they took up residence in a couple filing cabinet drawers (there’s that much paperwork). I had district data and data by schools – seven of them! Feasibility? Definitely disruptive. They took too much staff time and were so frustrating that some teachers did as they’ve complained students do – went through and just marked something, anything to be done. It didn’t do much for tech’s approval ratings either.
Propriety. I sure got that one wrong. I thought propriety had to do with “being proper” or following society’s rules (like when you’re allowed to wear white shoes or what fork to use at a formal dinner). In terms of evaluation, propriety has to do with being legal, respecting people’s rights and dignity, being ethical and not discriminating against any group or type of person. I like this propriety better than mine. It’s certainly more important to me as I don’t tend to care whether it’s the season for my color shoes or not. But I do care about ethics and how people are treated. All people. This is certainly important in the area of evaluation, but it is also important in education and our daily lives. Too many times I have seen non-certified staff treated as less than important. I’ve seen them ignored at meetings where their input could have been incredibly valuable. They play very important roles in our schools and in the daily school experience of our students. Just try to hold school for a few days without aides, secretaries, bus drivers and custodians! Our libraries and labs are staffed with aides. They are hard working, intelligent, caring and dedicated educators. I seek their input when looking at learning technologies. They’re on the front lines and have insights and knowledge that often proves invaluable. Yet some in “high” positions most likely couldn’t tell me their names. (Sorry. This one hit home.)
Accuracy. Well, that’s a no-brainer. Certainly evaluations need to be accurate. But wait. Once I read the accuracy standards I had to wonder, do they need to be THAT accurate? Be accurate, then check, then check again as in meta? Look at program, content, purposes, procedures, information, information sources, quantitative data, qualitative data, conclusions and reporting. Is it reliable? Valid? Defensible? Systematic? Justified? Impartial? But then again, as I read through the “illustrative cases” it became clear why the accuracy standards are necessary. Mistakes can lead to huge consequences. Little mistakes “hiding” in the midst of huge amounts of data. Divide by 6 instead of 4? Little mistake. Different (and incorrect) conclusion or outcome. As an educator, I have to wonder how many times in our busy-ness have we made such errors that were never uncovered as such, and what were the consequences to people and programs.
Then there are the AEA’s Guiding Principles. My suspicions were confirmed. Evaluation is a profession! Certainly I am not qualified to do this. And neither are my colleagues or our administrators, though hopefully they’ve had more training than we have. Evaluation is a complex task and I am grateful there are experts to be had. That said, the guiding principles go right along with the above standards. Inquiries should be data driven and systematic (accuracy Standard). Stakeholders should expect and get competence (utility standard) from evaluators. Evaluators should be honest, have integrity and respect other people (Propriety and Feasibility Standards). They should be articulate (communicate, communicate, communicate) and take into account diverse populations (Propriety again). Evaluations certainly need evaluators of high moral and ethical standards who include all types and groups of people while gathering information as well as when providing information. Evaluators need to not only be knowledgeable but also be effective communicators. In fact, at this point if I had to come up with one word to describe what I have read about evaluation it would be the word “communication”.
And finally, forms and procedures for use of human subjects in research. My first thoughts were in praise of the fact that I would not need to complete any of these forms as past students taking this course have. My second were in sympathy to those who have had to. My further thoughts, however, center around why they are necessary and how very awful and wrong evaluation can be if respect for human subjects in research is not present. Tuskegee and Nuremberg come to mind. Vulnerable populations. And us “regular folk”? A close friend of my husband’s has cancer and should have died long ago. He is, instead, doing pretty well and in a research program where he is being treated for free and basically has been a walking miracle as he continues to not only live but function with a relatively good quality of life. What if he had not been treated with dignity and respect? What if he had been in a control group where treatment was withheld in the name of science? Would doing nothing be acceptable as “do no harm”? Where is the line? “Doing nothing” can in actuality be “doing harm”. On the other hand, I think of one of my medical experiences and in that case, doing nothing was what I needed – and doing something would have most likely done me harm. How so? I had a rare form of cancer. They treated me. Many hours of surgery, weeks in intensive care, more weeks in the hospital, then two years when I could not lift my second child (who was 6 weeks old when I had the surgery). At the end of those two years, a small lump. A doctor visit then bad news. It was back. Nothing to be done. No hope. But there was this experimental program… My husband went with me to the fourth oncologist – “the” leading oncologist in the Chicago area. My husband (military trained in biological and nuclear type weapons) told me clearly that the treatment in the experimental program would kill me. I didn’t join the study and instead a trip to the Mayo Clinic proved all four Chicago oncologists wrong. (That was 23 years ago.) If I had participated in the study, I certainly would have been “done harm”. So, my point? How do we decide? How do we know? Often we just don’t know. So the best we can do is follow the Belmont Report’s guidelines. Carefully weigh options. Respect, justice and beneficence (do no harm). Information, comprehension and voluntariness. Weigh and assess risks and benefits. As humans without crystal balls, it’s the best we can do.
Standards, principles & guidelines… Oh my!
Big shoes to fill this semester.
– Pam
Addendum on weekly posts.
Case example responses are as varied as I expected along with a bit of confusion as to what direction to take for a response. Expected because the standards are many, interrelated and complex. Varied because varied people with their own perspectives on evaluation read the same information but processed it in their own way. Highlights for me included the addition of “planning” to my intro list above as well as a question. Can there be a true evaluation in the absence of stakeholders? I think not.
