September 10, 2008
Week 3 Reflections: AI and a big Q
Part One:
AI – Appreciative Inquiry
You become what you believe. Organizations become what their people believe. It’s amazing to think about but oh so true. The pictures of the future we hold in our minds, the thoughts that run through our heads, the labels were bear – they shape us and our future. So, appreciative inquiry, knowing this, turns us towards the positive. How can an evaluation do this? Through the questions it asks the people involved. What works, not what doesn’t. Let’s look at your best, not your worst. Then, let’s build on that. In education specifically we need to stop labeling people in negative ways, searching for problems to manage and “fix”. Instead we need to look for the strengths of people and programs, celebrate those and ask positive questions towards building on those strengths and positives towards a better, brighter future.
These are woven into AI’s eight principles. Questions are really interventions. Knowledge is shaped by experiences and conversations. Stakeholders co-author and direct the direction an organization or program takes. Collective imagination is key. Change requires large amounts of hope, joy and inspiration. Collectively we can bring out the best in individuals. We must model the change we envision. Freedom enables us to be better and perform better.
And then there are the 4-I’s of Inquire, Imagine, Innovate and Implement that are woven throughout the text. Taken from Discovery, Dream, Design and Destiny – Cooperrider and colleagues, 2003. Inquire about peak experiences, values and wishes. Positive stuff to engage people to share, open up and look at the past, present and future in a positive way. Next, imagine. You’ve already begun. Now take it another step. Let’s clarify that vision for the future. Innovate. What needs to be done to accomplish that vision. Pick your best vision and look at “what it might be” and how to get there. Choose wisely. Does it stretch and challenge while still being grounded? Is it” participative”, stimulating “intergenerational learning”? Is there a balance of “continuity, novelty, and transition”? Then Implement. Take action. But that’s not the end. As you implement, continue to inquire, imagine and innovate. Keep growing, expand your vision and shape the future.
The placebo effect. Is it real? Can it work in education? I believe so. You may have already heard this story, yet is speaks clearly to this topic. It was my first year in a new school district after moving away from my home town and eight years in a school system there. My past experiences had led me to believe we tend to live up to how people label and view us. So, I did not pour over student records and instead left everyone a clean slate in my mind and in my classroom. No preconceived notions, only expectations and hope. As the year began to develop I heard student conversations about reading groups. Which one was the low group? Can’t be this one because “Johnny” is in it. He never was before. Can’t be this one… “Sally” isn’t in it and she surely is a member of the low group. Could it be? Could it be that I’m not in the low group? And what if I am not? I could hear it in their voices and see it on their faces. What if? I thought it over and decided to run with it. The groups remained the same with no indication from me as to how to label them – which is the low group? Which “comes next”? Low group? Do we HAVE a low group? All of the reading groups were energized, “woken up” in a sense. There were no labels, no negative predictions for their reading future. What came from it was a collection of good, positive efforts that lead to amazing achievement gains. I am still stunned by the outcome when I remember it. These students were changed, not just academically, but in their very perceptions of themselves. Their parents saw it at home and wrote to me about it. It was the win win of all win wins! Huge reading gains – of which so much academically is built and huge gains in perceptions of self worth and capabilities – which can take you anywhere. All because they might not be in the low group. Our thoughts shape us.
Our thoughts shape us. My husband returned home after a year and a half away from home, a full year of it “boots on the ground” in Iraq. He came home a changed man. The families had been told about and given advice about how our loved ones would return to us. We were told, “The hard part’s over, they’re coming home. Right? Wrong! The hardest part is ahead.” How right they were. I came to see that a big piece of returning home and adjusting to living in such a different environment (from one environment to another where you adjust quickly or die, then back to another where people’s lives continue as if that other world doesn’t exist) plays out in a soldier’s mind. The thoughts that protected and saved me in this environment continue to play in the new, even when they are wrong or inappropriate or even dangerous. Simple things. Drive in the middle of the road. The IEDs are usually on the side. In East Peoria IL driving in the middle of the road means being pretty close to oncoming traffic, some of which are texting cell messages and not attending to much of anything else. Be alert. Always on guard. Don’t trust. No where is there a safe place to be. Don’t sleep deeply or well. The list could go on and on. After two and a half years with stalled progress I rang the bell. Loudly. Someone needed to do something. Enough is enough. So, three states later (there are no doctors in Illinois?) he was enrolled in a cognitive therapy program. What is it? What does it do? It retrains your mind and the thoughts that run through it. So, we are back to the beginning point. Our thoughts shape us. They do. I saw more progress in the course of a 12 session cognitive therapy plan than what we had seen in a several years. Think about the incident – daily incidents mishandled and misthought. What were you thinking? What thoughts were playing in your head? Were they right and true and appropriate here and now? What are right and true and appropriate thoughts? Think them. Break old thought habits and replace them with true appropriate ones. Because – our thoughts shape us. They truly do.
Part Two:
The big Q – Drum roll please…
And the project topic is… =?
I had more problems deciding on a topic for this class than any of the others. I have so many ideas to choose from! In the end, I decided on the one that I am most conflicted about right now, the one that holds the most impact for the future of our district in terms of tech right now and the one that I frankly am most interested in spending a lot of time with. It may not be one that “fits the mold” the easiest or the best - yet it is the one I personally and professionally will benefit from most by looking at closely. So, I feel settled in my choice and confident that what I learn from the experience is more important than “fitting a mold” to “doing it right”. And from the case examples we’ve been reading and discussing, I’m not sure there is a “perfect evaluation topic”. Just topics that are important and evaluations that evaluate them in the best ways possible.
Pam
September 4, 2008
Week 2 Reflections: Standards, principles & guidelines… Oh my!
If asked about evaluation a few weeks ago, I would have guessed that standards for it exist and that most of the evaluations I have been involved in would not “meet the test or measure” of those standards. Now I’m certain of it! Reading the Program Evaluation Standards was interesting and informative yet daunting and overwhelming. I certainly now understand why there are professional evaluators. While reading the standards, however, a list of words and phrases began playing in my head. First of all came “communication”, “respect” and “common sense”. Then I added “cover your bases”, “double check” and “document”. During the section on metaevaluation I was thinking of Tom Anderson’s course! This section seemed to be the meta of all metas. So I needed to add, “do it all over again” to my list.
Utility. If one is going to take the time to do something (especially those in a professional like ours where time is a valuable commodity), then certainly we want it to be useful and address its purpose. It makes sense to take into consideration those involved or affected, that the person doing the job should be competent and knowledgeable, that the task should be done in a timely manner and that the end product should be usable. The more time and resources spent on something, the more we expect to get in return. Probably the biggest task (which involves evaluating programs) I’m presented with every three years is writing the district tech plan. Too many of them have been met with indifference or roadblocks. I was even told once by a superintendent that “Nobody cares.” My biggest surprise in terms of the utility standards came in standard U4, values identification. Common error A states, “Assuming that evaluations can be objective in the sense of being devoid of value judgment.” A huge part of evaluating is trying to be impartial, yet in the end there is a value judgment.
Feasibility. I’ve always thought of that as whether something was possible or able to be done. The standards take it a few steps further. They speak of diplomacy and political viability, cost and disruption. How many times have school districts been guilty of “disrupting program activities in an attempt to collect information”? Our entire district staff took a time consuming, confusing and extremely long technology survey for our NextSteps program – twice. (They would have done so a third time but I pled repeatedly for mercy.) Although taken using a computer, the results could not be had electronically, but could only be printed. You could not get a comparison of the second against the first. Again, printouts were available. I spent untold hours trying to gather information from them and in the end they took up residence in a couple filing cabinet drawers (there’s that much paperwork). I had district data and data by schools – seven of them! Feasibility? Definitely disruptive. They took too much staff time and were so frustrating that some teachers did as they’ve complained students do – went through and just marked something, anything to be done. It didn’t do much for tech’s approval ratings either.
Propriety. I sure got that one wrong. I thought propriety had to do with “being proper” or following society’s rules (like when you’re allowed to wear white shoes or what fork to use at a formal dinner). In terms of evaluation, propriety has to do with being legal, respecting people’s rights and dignity, being ethical and not discriminating against any group or type of person. I like this propriety better than mine. It’s certainly more important to me as I don’t tend to care whether it’s the season for my color shoes or not. But I do care about ethics and how people are treated. All people. This is certainly important in the area of evaluation, but it is also important in education and our daily lives. Too many times I have seen non-certified staff treated as less than important. I’ve seen them ignored at meetings where their input could have been incredibly valuable. They play very important roles in our schools and in the daily school experience of our students. Just try to hold school for a few days without aides, secretaries, bus drivers and custodians! Our libraries and labs are staffed with aides. They are hard working, intelligent, caring and dedicated educators. I seek their input when looking at learning technologies. They’re on the front lines and have insights and knowledge that often proves invaluable. Yet some in “high” positions most likely couldn’t tell me their names. (Sorry. This one hit home.)
Accuracy. Well, that’s a no-brainer. Certainly evaluations need to be accurate. But wait. Once I read the accuracy standards I had to wonder, do they need to be THAT accurate? Be accurate, then check, then check again as in meta? Look at program, content, purposes, procedures, information, information sources, quantitative data, qualitative data, conclusions and reporting. Is it reliable? Valid? Defensible? Systematic? Justified? Impartial? But then again, as I read through the “illustrative cases” it became clear why the accuracy standards are necessary. Mistakes can lead to huge consequences. Little mistakes “hiding” in the midst of huge amounts of data. Divide by 6 instead of 4? Little mistake. Different (and incorrect) conclusion or outcome. As an educator, I have to wonder how many times in our busy-ness have we made such errors that were never uncovered as such, and what were the consequences to people and programs.
Then there are the AEA’s Guiding Principles. My suspicions were confirmed. Evaluation is a profession! Certainly I am not qualified to do this. And neither are my colleagues or our administrators, though hopefully they’ve had more training than we have. Evaluation is a complex task and I am grateful there are experts to be had. That said, the guiding principles go right along with the above standards. Inquiries should be data driven and systematic (accuracy Standard). Stakeholders should expect and get competence (utility standard) from evaluators. Evaluators should be honest, have integrity and respect other people (Propriety and Feasibility Standards). They should be articulate (communicate, communicate, communicate) and take into account diverse populations (Propriety again). Evaluations certainly need evaluators of high moral and ethical standards who include all types and groups of people while gathering information as well as when providing information. Evaluators need to not only be knowledgeable but also be effective communicators. In fact, at this point if I had to come up with one word to describe what I have read about evaluation it would be the word “communication”.
And finally, forms and procedures for use of human subjects in research. My first thoughts were in praise of the fact that I would not need to complete any of these forms as past students taking this course have. My second were in sympathy to those who have had to. My further thoughts, however, center around why they are necessary and how very awful and wrong evaluation can be if respect for human subjects in research is not present. Tuskegee and Nuremberg come to mind. Vulnerable populations. And us “regular folk”? A close friend of my husband’s has cancer and should have died long ago. He is, instead, doing pretty well and in a research program where he is being treated for free and basically has been a walking miracle as he continues to not only live but function with a relatively good quality of life. What if he had not been treated with dignity and respect? What if he had been in a control group where treatment was withheld in the name of science? Would doing nothing be acceptable as “do no harm”? Where is the line? “Doing nothing” can in actuality be “doing harm”. On the other hand, I think of one of my medical experiences and in that case, doing nothing was what I needed – and doing something would have most likely done me harm. How so? I had a rare form of cancer. They treated me. Many hours of surgery, weeks in intensive care, more weeks in the hospital, then two years when I could not lift my second child (who was 6 weeks old when I had the surgery). At the end of those two years, a small lump. A doctor visit then bad news. It was back. Nothing to be done. No hope. But there was this experimental program… My husband went with me to the fourth oncologist – “the” leading oncologist in the Chicago area. My husband (military trained in biological and nuclear type weapons) told me clearly that the treatment in the experimental program would kill me. I didn’t join the study and instead a trip to the Mayo Clinic proved all four Chicago oncologists wrong. (That was 23 years ago.) If I had participated in the study, I certainly would have been “done harm”. So, my point? How do we decide? How do we know? Often we just don’t know. So the best we can do is follow the Belmont Report’s guidelines. Carefully weigh options. Respect, justice and beneficence (do no harm). Information, comprehension and voluntariness. Weigh and assess risks and benefits. As humans without crystal balls, it’s the best we can do.
Standards, principles & guidelines… Oh my!
Big shoes to fill this semester.
– Pam
Addendum on weekly posts.
Case example responses are as varied as I expected along with a bit of confusion as to what direction to take for a response. Expected because the standards are many, interrelated and complex. Varied because varied people with their own perspectives on evaluation read the same information but processed it in their own way. Highlights for me included the addition of “planning” to my intro list above as well as a question. Can there be a true evaluation in the absence of stakeholders? I think not.
August 23, 2008
Week 1: Description and Hopes
This is my eleventh year as tech trainer for East Peoria School District 86 – plus… That “plus” part of my job includes bits of network and helpdesk responsibilities for our seven buildings, managing and maintaining the district web site, working with the district librarian to support and provide lessons for our library/computer lab aides, major technology plan responsibilities and “all other duties as assigned”. My ten years of public school teaching experience have been invaluable in my work with district staff and my eight years as a library/computer lab aide provide the same for my work with our library and lab aides. My (stay-at-home) mother status provided new perspectives on education and the classroom as well. I share tech responsibilities for our K-8 district with a young Linux geek and we are both incredibly busy with the start of a new school year. I wouldn’t trade my job for any other, even on our most disastrous days. I have the best job in the world!
My greatest hope for this course is to discover a way to truly evaluate the impact of technology on student learning. Tech use is so intertwined with personalities and teaching styles along with a myriad of changes and classroom influences. How do you pull the impact of tech out of such a bowl of spaghetti? Along side that, I/we attempt to evaluate training sessions and workshops, our district technology plan, pilot projects and initiatives, our helpdesk support, district infrastructure, hardware and software as well as our own job effectiveness. I have simply “picked up” what I know from experiences and discussions – and frankly I suspect much of it is not ideal. Often it is difficult to collect data which then remains basically unused. Problems and negativity can easily become the focus of evaluation, leaving a “bad taste” behind. And sometimes it seems fabricated to simply say it was done. After looking through our appreciative inquiry text, I am encouraged and hopeful! It offers a positive, affirmative approach. My past CTER course experiences leave me very hopeful as well. They’ve been incredibly useful, thought provoking and fun!
If so inclined, personal photos and info can be found at: Pam’s Mahara ePortfolio/Profile
Pam
