December 1, 2008
Week 14 & 15 Reflections: On polishing and finishing and final things…
The week of Thanksgiving I took another look at my presentation, cut it down a little and decided what specifically I wanted to say. I timed myself and literally let out a cheer – I’d done it! FIVE MINUTES!
What relief! I’m feeling ready to present and am hopeful to not be too boring as well. While my presentation isn’t spectacularly creative, I think it’s relatively “light”, “clean” and loaded with graphics.
After Thanksgiving I tackled my final paper again. I’d made revisions – quite a few revisions – and changed to APA style during week 13, but wanted to take another close look (after a bit of a well enjoyed break) before finally submitting it one last time. Final. Pretty much the end of the course but also the end of possibilities to change or improve. That makes it noteworthy, so I took that time and combed through my final paper one more time. After my final post, I got a great response from Luke and spent even more time with it. Making his suggested changes felt good and right. I’d spent a lot of time on my final evaluation paper, as all of us have, and wanted it to reflect that time and effort. Hopefully, it does that!
This course has completely changed the way I look at evaluation. I had a very limited view of what evaluation means and what an evaluation “looks like”. Unfortunately, I think most people think of evaluation the way I did pre-EPSY474 – as something you “do” every once in a while, particularly when you need to decide whether to keep something or get rid of it – or something you do because you “have to”, then file away and never really use or look at again. Evaluation, when looked at that way, is something that is negative or punitive and often to be feared. Through this course, we’ve all learned that evaluation should be an ongoing thing, a part of the basic fabric or structure of programs and processes. Evaluation done well can build up and energize, pull people in and make them feel as though what they say and what they do really counts for something. Evaluation done right adheres to high standards, standards which keep it focused and honest and true. Evaluation does not have to “dismiss” people or programs, it can instead find values and strengths that can lead to a better future.
In particularly, Appreciative Inquiry can add immensely to an evaluation, however it is used. While AI may not be “the answer” to every evaluation, its use (in varying forms and amounts) can transform that evaluation experience. Getting people to “stay the course” and remain in the positive is a challenge when using Appreciative Inquiry. People seem to be in a habit of looking at things in a negative light, focusing on what does not work, complaining and even predicting gloom. The heart of AI is to move away from that towards the positive, focusing on what does work, looking for core values and strengths and creating a positive vision of the future. I think the basic tenants behind Appreciative Inquiry are true. People and their interactions create their own reality. What we ask, how we ask it, what we think about and what we speak are so very important to how we perceive our world. We all have a story and it needs to be an important part of what we do and how we are treated. We have value, value as individuals but also as a part of a whole. That whole is shaped by us and others, our inspirations, hopes and joys. We need to be free, free to choose but also to contribute. We must be (right now) the future we want tomorrow. And we create that by our thoughts and images and actions. Thus, the questions we ask and the discussions we have are pivotal. They shape us and we shape the future.
I’ve enjoyed watching the wide variety of evaluations of my course cohorts develop and evolve. Reading and discussing what each other have written has been not only valuable but enjoyable. I have enjoyed blogging and found that it caused me to stop and step back, reflect a bit and even be proud of what had been accomplished that week. And while our evaluations were different, they were alike. So as we each worked through our own thoughts and processes, we learned from each other and hopefully supported each other as well. I have to admit, I was skeptical about the dyad work and the “pretend” portions of this course. They surprised me in their value and impact. It was a growth experience similar to the conflict resolution role playing for Tom Anderson’s course. It seemed suspect and unfamiliar, as well as awkward at the start. Yet it became much more than one would ever have imagined.
So, I have come to the end. The end of this blog and the end of this course. It is not, however, the end to my new found insights about evaluation. Those I carry with me.
The End!
Pam
November 22, 2008
Week 13 Reflection: On rough drafts, final papers and cutting presentations down to size…
I was glad to be able to post the rough draft of my final course project early. That meant I had some time to read what others had written, comment on them and think – through that activity – about what I might need to change or add. It also meant I could go back to my PowerPoint once again and try to whittle it down a bit. I would still have to rush through it to be done in 5 minutes. That was beginning to be a pretty hard number to hit! I wanted to have it finished this week pretty much so I could enjoy Thanksgiving week and still be able to post it early, per Cheryl’s instructions to have it up a couple days before we present.
Thanksgiving IS just around the corner…
Reading through Ashley’s paper, I got a good laugh at myself. I realized I hadn’t written about my focus group questions and protocols. Then Kona’s feedback was what I had suspected… I had to re-format my paper as APA. It wasn’t hard to do, but it took a good chunk of time! I did some double checking about APA and think I got everything pretty much the way it belongs. Only took me three tries!
Then I tackled my PowerPoint again. I only managed to get rid of two slides. I think it’s still too long. I’m hoping next week to look at it again. Maybe I can get rid of a couple more slides then!
So, it’s been busier this week than I had planned. Plus it was a busy week in other ways as well. Thursday and Friday I attended IETC (Illinois Education and Technology Conference) in Springfield. Among a zillion other things, I found out the cost of land in Second Life has gone WAY up and many can no longer afford it. They thought Google’s new virtual world Lively would be the answer, but evidently that is being shut down. Most of the conference, of course, was much more positive and up beat. The IETC is always great (I personally think it’s as good as the Chicago tech conference) – and this year was no exception – but you leave with a head full of thoughts and ideas and not enough time to do them all justice! I am resolved to make some time next week for that.
Happy Thanksgiving!
Pam
November 11, 2008
Weeks 11 & 12 Reflections: On Results, Discussions, Tweaking and Presentation Beginnings…
Week 11:
Before diving into my final section, the Discussion, I took another look at my conversation about data collection then added a conversational piece about my timeline. I think the text and the tables together provide a better picture than either could do alone. I hope it’s going to be OK to include those tables! While writing the timeline section, I was really glad that I looked at the actual school calendar and set out the evaluation schedule as if I were really conducting it. That made it easy to explain and discuss the timeline of the evaluation.
I’d been waiting as well for some input or discussion on my Results section that I posted. I began to wonder if “no news was good news” because that’s pretty much what I got – no takers on a response to my post. Then Kona who never disappoints took a look. She really helped me resolve an issue I’d not quite felt comfortable about – how to address myself in the paper. She said usually instead of I people refer to “the evaluator” or “primary investigator”. That made sense. I had a little trouble going back and changing what I’d written but that was OK… I needed to take another look and it kind of forced me to do that a bit more closely. So – that done, I intend to look back again after its finished and I’ve had some time for it to settle in… Just to make sure it’s just as it should be.
The Discussion section was the hardest to start. Should I just look at a variety of possible scenarios? Is there a way to make this more “real”? Then I thought – wait a minute, I have some actual, current data! We’ve been gathering survey data for the writing of our current tech plan and while not all of that data applies, some of it could be pertinent. Also, I do have lab stats collected that I could look at as well – a lot of them. So, I decided to approach it in a couple of ways and combine some possibilities and some probabilities (using actual data). One look would be at what current data, though incomplete, indicated in terms of a direction to go. Because that data is incomplete, there would be most likely a variety of possibilities and different directions to take. So the plan at that point was to do some analyzing of data in terms of my specific evaluation and see where that leads.
Well, that wasn’t as easy as I’d hoped! I started a couple different times. I couldn’t get focused. I wrote parts for the end, then parts for the middle, then stepped back and called a halt! Finally I decided to take a very careful look at what data I had and actually write about the data itself. That really helped me focus. Once that fell together, I started a list of ideas and reactions and possibilities. Then I grouped them into categories and tackled a few. As those sections began to flow, I’d have another thought and add it to my list. Gradually the paragraphs of ideas grew as my list shortened as I crossed out items I’d discussed. I’m not sure it’s what I was “supposed” to do or what it was “supposed” to look like. I sure hope so! Plus, isn’t this a ground breaking kind of semester with us being the “first” to write this way? (One can hope.) In any case, I feel solid about what I wrote. I know my evaluation is a bit different than many of my cohorts’ as theirs are more specific in nature than mine. Also, I have some actual data which most likely they do not. In any case, my plan is to post it and see what happens! If nothing else, I have learned a ton and have gotten a lot of great insights into technology in the district in which I work. That can only be a “good thing”!
Feelin' Good!
Week 12:
This week my goal was my presentation. I’d thought about it a lot and hoped to take a not-so-heavy approach – and keep the text minimal and to the point. That, of course, was not as easy as planned but I think definitely worth the trouble. After all, I have my finished paper with all the long drawn-out details. This presentation is to be relatively brief, and I remember past presentations going very quickly. My goal was to do this one a bit differently.
I began by looking for some pictures. After all, once you pick your PowerPoint background, what’s left but the clipart! Of course, after looking at pictures I had to go back and rethink my background or theme. It really was difficult not just trying to put everything in my report in my presentation. No way that’s going to work in the amount of time we have! So, in the end, I decided to paint a picture of the computer lab system I’m evaluating, look at the evaluation design as a whole, then discuss the “results”. We spent a lot of time posting questions – interview questions & survey questions – as well as commenting on them and rewriting them, so I think everyone has a feel for those. So, I left them out! I can certainly produce them if anyone asks or if they’re needed…
So now I’m whittling the presentation down and tweaking it. I’d really like to present it within five minutes and not feel rushed or like I didn’t get to portions I wanted to say. I have a feeling it will be a work in progress in any case… Once we post our final projects for people to discuss, most likely there’ll be more changes.
I for one REALLY appreciated the break last week. It kept us all from feeling like this poor guy who’s definitely in need of some R and R!
In need of a break...
Pam
October 30, 2008
Week 10 Reflections: On results and final things
After the week 9 blog postings and thoughts as well as concerns about the final product – How should it look? What should it include? – I was pleased to see some clarification with the posting of two example final projects from last year. Even though they are different than ours because they actually conducted the evaluation, they give good clues and ideas as to how it should all fit together. I had done some googling about how one might analyze data, what a Likert scale is and how to sift through open ended question answers. And I had taken a tentative start to writing the results section. After looking at the examples, I went back and added some narrative, then continued with more confidence.
Writing the results section made me take another look at each and every one of my questions in a very critical way. What key question(s) does this address? What information are you searching for by asking this question? What possible responses might it produce? How are you going to analyze the responses? Again, the process made me very glad to have those key questions to return to. And again, I was glad I’d focused on them when I initially wrote the interview, survey and focus group questions.
I did my writing this week in several sessions. After working on it for a while, I’d need to take a break and come back to it. Then I could take a fresh look at what I’d written, make some revisions, and continue writing. That speaks to its complexity and its importance. Luckily my husband is in South Dakota pheasant hunting (with a “gas saving” stop =? before and after in Wisconsin for a bit of deer hunting), so I’ve had some extra flexibility to my schedule. My Riley, however, is not feeling lucky. He has been very, very sad. He misses his dog-buddy Scout (who’s hunting with Frank) and has been spoiled since my husband retired. He’s gotten used to having someone home more often. It’s just me and Riley, and I go to work… So, I get barked at vigorously as he “tells me off” every time I walk in the door. Plus he is not sure where Scout went, but he is CERTAIN that wherever he and Frank are, they’re having FUN without him. And he’s right.
Riley Now Sad
I’m really grateful for the optional forum where we can get feedback from each other and toss around what we’ve done so far. After starting my results section, a few people posted and got responses indicating perhaps there should be more dialogue and less “listing”. Mine was kind of a mix of dialogue and lists when I began, as the instructions seemed to say to me “each question should be looked at”. So, after some thought, I went back and rewrote what I’d begun. It was good, I believe, to have looked at them in a list fashion first, as I got a feel for what they addressed and how they fit together – the interview, survey and focus group questions. That enabled me to do the rewrite more easily and hopefully better as well. I posted them in the optional forum for comments, so hopefully I’ll get some feedback. This is new territory for all of us and although I’m glad to have the examples from last year to look to, what we are doing is different in several respects. Plus, I know Kona is “out there” taking a look as well and she’ll tactfully guide and encourage us in the direction we should go! That’s “polite-speak” for, “She’ll tell us if we’re doing it wrong, without making us feel foolish.” Thanks, Kona!
Looking at those examples and some of the optional forum postings made me take another look at my timeline and a few other charts I created. I’d really like to include them in my final project, but I’m going back now and adding a little narrative to them as well. They seem to call for that and the examples are pure text… Once I get that done and I feel the rest of the final project is basically ready, I’ll start the final Discussion section.
Putting the sections we had already created, posted and revised together as a final product gave me a feeling of accomplishment and lessened thoughts of all that still needs to be done. At first, because portions had to be cut out – pre-interviews / surveys and discussions of changes made – I thought I’d “lose” so much there’d be very little left! But happily that wasn’t so, and looking again at the examples from last year, I felt secure that Cheryl had guided us well.
Fitting Some Pieces Together
October 22, 2008
Week 9 Reflections: Evaluation Capacity, SIPs and TIPs
Building Evaluation Capacity:
While reading chapters 5 and 6 of the Preskill text, I couldn’t help but think about our District Technology Integration Plan and our Building School Improvement Plans. Those are the two evaluation systems in our district that I know the most about and have the greatest contact with. As I’ve begun the process of writing another tech plan with yet another “team”, it brought to mind the differences between the two evaluation systems. It seems to me that the School Improvement Plan process has been embedded to a certain extent, many people are involved on some level, and there is a regular-ness and pattern to it. The District Technology Integration Plan always seems to take people by surprise and the process doesn’t “flow” as it should. So, bear with me while I look at them in terms of their structure/creation and whether they measure up to the building evaluation capacity descriptions in the text. I may ramble as well, as points or components come to mind…
Cheryl stated, “appreciative inquiry pulls in the leaders not just the drones, everybody is involved and everybody has something invested and hopefully that means there is leadership commitment to it.” Our Building SIPs definitely pull in the leaders. The building principals are responsible for them and those who have difficulty writing them find another principal willing to help. They definitely have an aura of importance. And we “drones” have numerous SIP days when progress is looked at and work on them is “done”. Certainly the leadership commitment is there, I’m often not sure about the drones, however! On the other hand, our District TIP writing has more often than not lacked leaders at the table. I don’t mean that the district librarian and I aren’t there. We certainly are. But often administrators are absent or make brief physical appearances with little actual commitment. Some are mildly interested in the look of the plan when done, but others have instantly given “disclaimers” such as “everything is tentative and has to be run by us” or “you know we really don’t get any money for this any more”. (Though in reality e-rate funds are tied to it now.) So, the TIP often gets drones who, if they become invested, are setting themselves up for a bit of a disappointment in the carry-through phase. This time there is potential for much more leadership involvement so I am hopeful.
So what about a vision and a philosophy? While the School Improvement Plans reflect a vision and a philosophy, I don’t believe they actually have “formal” ones. It certainly seems that the clear vision is better achievement scores with usually a community relationship type improvement piece. Ours are all data driven from start to finish – and most of the data is achievement scores. And the idea that schools need to be about improving and maximizing student achievement as we partner with the community is clearly universally embraced. It’s odd when you think about it because the SIPs, without a formal stated vision or philosophy, clearly have more common vision and focus than District Technology Integration Plans that include a formal, written vision. Each three year cycle we write, the formal “do’s and don’t’s” of writing the TIP include more and more specific student achievement ties. Yet the “feel” of writing them is always less focused and more torn. At times that’s because someone has an agenda, pre-determined, about some new technology they want to introduce to or expand within the district. Then there’s a tendency to want to “make” the rest of the plan fit their perceived outcome or vision of what it should look like. At other times, there is just a disconnect between people about how technology can best serve the needs of our students and staff. So, while there’s clearly a written vision in the TIP, it often isn’t really shared or agreed upon, even by those writing the plan.
Of strategic planning, the Preskill text states, “An evaluation strategic plan is an excellent tool for communicating the ongoing progress and outcome of various programs and processes within the organization.” Our SIPs and TIPs are chock full of goals and strategies and activities. On the surface, they look pretty equal in that regard. Yet our SIPs get the “communicating the ongoing process” piece on a regular, scheduled, routine basis. They’re collaborative, ongoing and useful. Our TIPs do not get that “communicating the ongoing process” piece regularly at all. There’s no real plan to “check in”, changes are often made with no formal acknowledgement and much of the time a good look doesn’t happen again for another three years. Well, in writing there’s a plan to “check in”, but in reality it doesn’t happen often. I wonder if ISBE figured that out, because I noticed a change in the required structure of the TIP this go around. There’s a new section titled “Monitoring”!
So that leaves the kind of personnel and financial resources available to take a look at. Personnel-wise all of the administrators are on board for the SIPs, especially the building principals. And because of that and the fact we’re talking AYP and NCLB (plus now RTI), the staff’s onboard as well. Not to mention that they’re a captive audience on SIP days! The only people on board for the TIP regularly have been the district librarian and me. This year the associate super is as well. The building principals seem to be reluctant participants and though staff will become excited if a piece of technology may become available that they are really interested in, there isn’t a high level of involvement in terms of the process. The staff members that are on board seem to see the value and potential that technology holds for their students and their curriculum. The “gripers” – well, I don’t believe they see the connection as yet. And financially? Financially, in a sense, the SIPs get the bulk of the school budget. Yet, technology is expensive and the monies spent are large. I do think, however, we tend to spend too much on “stuff” and not enough on professional development.
While certainly formal communication systems are in place, both seem to lack a bit in the flexible and responsive departments. Writing plans that are formally approved and regulated most likely puts a bit of a kink in prospects for flexibility, yet I’ve certainly seen formal plans ignored or abandoned before. And I believe for both our SIPs and TIPs, that’s what happens. I’m not saying we should lose sight of our goals, but sometimes our plans just don’t pan out. They beg for change and improvement. And rightfully so! On the surface, it seems as if our TIP has more difficulty in this department than our SIPs, but a lot of that is most likely due to the fact that SIPs are written yearly and TIPs are written every three years. Three years is a long time to clearly see and interpret that crystal ball into the future. Then when you add to the mix the fact that we’re talking technology here – technology that is changing so incredibly rapidly – it’s no wonder we look at the third year of our plan and scratch our heads. In that case, flexibility and responsiveness are exactly what we need.
With nine weeks of this course under my belt, I see the possibilities and opportunities that could be had here. Do we purposefully choose the system (or non-system) we have? I think, perchance, our administrators should take this class or one like it. Perhaps they have. But I cannot understand how someone would/could deliberately not choose something “ongoing and meaningful, and supportive and collaborative, and aimed at improvement.” …something that encourages success, is learning oriented and enhances creativity.
Final course project: Where I am and what I need to do…
I feel very solid about what I have completed so far on my final course project, which is pretty much what we have been asked to do on a weekly basis. I’ve gained a lot of insights into my evaluation topic even though I haven’t really “done” the evaluation. I certainly have been able to look at it, examine it and pick it apart in a variety of ways. And most certainly I understand the situation more clearly than before I began this course. It has helped me put some of my feelings about the situation aside and look at it in an impartial way. So – I haven’t looked ahead to see if more information is available about any other sections or finishing touches that will be necessary, but I really feel I understand my evaluation and will be able to handle any of those types of tasks.
My major concern is with the final presentation. I’ve presented twice in Elluminate and have been very anxious about both. I think it’s because I do not feel comfortable in that software, not having used it much. Although I know the basics and in reality did not come up against anything I didn’t understand how to do, I’m used to knowing all the ins and outs of the software I use – after all, I’m supposed to be the “expert” in our district! Hah!!! If they only knew… In any case, I think on some level that’s the issue. I don’t know everything there is to know about Elluminate and therefore it is anxiety producing. The first time I used it, I was so nervous I didn’t really watch the “chat” that was going on and wasn’t aware of the time I’d taken. During the Q and A I was able to “look around” and relax a bit, though I wasn’t certain I would be able to give intelligent responses to any and all questions or comments. I was surprised and taken a bit off balance, as I’m used to presenting and used to feeling comfortable and “in control”. (Control… It’s all about control!) The second time was a bit better but not quite “up to Pam’s standards”. So – here I am again looking at a presentation in Elluminate.
Both times I felt I tried to pack too much into the time I had. This time I have less time – 5 minutes! And then Cheryl presented the challenge – “but if you have a creative idea, I want to thoroughly encourage you to do that”… Uh oh! There are so many creative people in this course! I totally enjoy their creativity but feel a bit inadequate in that area. So, I need to pack everything into 5 minutes and do it in a creative way. THAT’s what concerns me at this point.
Pam
October 18, 2008
Week 8 Reflections: On surveys and waiting…
Creating the Survey:
Creating my survey questions was even harder than writing my interview questions! I spent quite a bit of time reading through the Preskill chapter, the interview tips and the additional readings that pertain to survey questions. There was a lot of helpful information.
I definitely would use an Internet survey for my evaluation. I’m surveying about 70 certified staff and I know that they have a good Internet connection available. It’s inexpensive and it doesn’t take me long to create one as I’ve done so before. I also think people will be more honest and forthcoming using a computer as opposed to speaking to a person. I also like the idea of beginning with a welcome message as well as a brief statement of the purpose of the survey or how it will be used. Finally, I want to reassure staff that their answers will be kept confidential.
It was interesting to think about the difference between 4 and 5 questions. I can certainly remember times when I took that “easy middle road”! It also makes good sense to begin with easier questions and end with harder ones. I plan to put my two open questions at the end. I don’t think we have enough questions to worry too much about “habituation” while taking the survey, but just in case I planned to vary my questions a bit. Finally, I like the idea of having an “Other Comments” section at the end. (I hope it doesn’t count as one of my open ended questions!)
After reading and thinking the above through, it was time to write! I looked for examples in the text as I did last week. They were harder to pinpoint. I liked “My best experiences…”, “If I had the opportunity, I would like to…”, “The most important…” and “Mark the statement that best represents your view.” But looking at my key questions, most of them just weren’t a good fit. So then I decided to concentrate on my key questions and how to best get the information I needed from the teachers I’d be surveying. Once I got the first question on paper, the second was a little easier. Then it finally started to flow!
1) With the current system, how easy is it for you to use the computer lab to integrate technology into your classroom curriculum?
Very easy
Relatively easy
Difficult
Impossible
2) When I integrate technology into my classroom curriculum, I prefer to:
Use educational games and activities.
Have students research and/or present on a topic.
Have students create projects and/or products.
Other: __________________________________
3) The current system of delivering core technology skills to students is:
Very adequate
Adequate
Somewhat adequate
Inadequate
4) Mark the statement that best represents your view.
I am very capable of integrating technology into my classroom curriculum.
I am capable of integrating technology into my classroom curriculum.
I am barely capable of integrating technology into my classroom curriculum.
I am not capable of integrating technology into my classroom curriculum.
5) Mark the statement that best represents your view.
I am very capable of teaching core technology skills to my students.
I am capable of teaching core technology skills to my students.
I am barely capable of teaching core technology skills to my students.
I am not capable of teaching core technology skills to my students.
6) Mark the statement that best represents your view.
Library computer lab aides should continue to deliver core technology skills to students.
Classroom teachers should deliver core technology skills to students as they integrate technology into their classroom curriculum.
A certified “computer teacher” should be hired to deliver core technology skills to students.
Other: _____________________________________________________
7) Based on your experiences so far with the computer lab system of delivering core technology skills and offering integration opportunities, what do you value most about it? What would you define as core characteristics of the system, characteristics without which the system would not be what it is?
8) (No matter what I do, #8 wants to be “cool”, so I’m just going to accept the fact and move on!) If you were in charge of the computer lab system and could have three wishes for the system granted, what would they be? How would the computer lab system be different if your three wishes were granted?
Finally, and a bit backwards, I wrote the welcome message, description of how the survey would be used and reassurance that it will be kept confidential.
“Thank you in advance for taking the time to complete this brief survey. Your responses will be anonymous and used only in combination with other survey results. The information from this survey will help the district as we write our technology plan, evaluate the current computer lab system and look towards the future. Your opinions and honesty are greatly appreciated!”
Self Discovery – Waiting for the “Think Aloud”:
Two things happened waiting for my Think Aloud with Kelly. Well, really one thing happened that is a two-part blog reflection… I don’t wait really well. Hah! Not exactly a NEWS FLASH here, but the large amount of difficulty I had was pretty significant to me! So, I got everything as ready as I could and waited. Well, no… I didn’t exactly wait. I field tested my survey at work. A couple times. Actually, that turned out to be a good thing because I asked more people for opinions than I did with my interview questions, and I got a lot of great feedback. Then I discovered something else. There was one person I had a little trouble accepting advice from. Hah! Another NEWS FLASH. She is an incredible person and an even more incredible teacher – and she is terrific with technology. (Gee – that’s a head scratcher why I had some difficulty taking advice from her…) She had a good suggestion that I (mentally) initially dismissed, although in person I was VERY POLITE. Her suggestion sat there in the back of my mind though, knowing it was true and deserving. So the next day I made sure to say something nice to her and brought it out to use.
NEWS FLASH for the week: Waiting is hard for Pam.
So – I’m going to post this and add to it later, after my Think Aloud date. Is that cheating? I sure hope not because I need a “quick fix”!
The “Think Aloud”:
It was worth the wait! I hadn’t been sure at first if Kelly would have the same thoughts as the staff members I’d taken the survey for a road test with – or if she’d have anything different to bring to the table. Any concerns were definitely unfounded! Kelly had many of the same thoughts as the staff members but she had a lot of new ones as well. Everybody thought I should clarify what “core technology skills” means and all agreed there was something not quite right with questions 4 and 5. Kelly, however, showed me that my first question wasn’t clear – not in terms of what I really wanted to know. And she pointed out that question 2 should be one where staff could pick multiple choices, not just one. Her suggestions really added to the revised package a great deal! After we went through the questions once, I talked to Kelly about what the staff members had said, then I ran a possible new question by her. She suggested a clarification for it as well.
Once we hung up, I got right to work. After all, I could FINALLY POST! I was feeling better already! By that point the re-write and changes flowed easily. I’m really pleased with my final product and I am also pleased with the process of the Think Aloud this week. Talking on the phone was better than the chat, as it was quicker and more “real”. I also preferred the idea of talking what you’re thinking. I really believe it produced a lot more ideas than “standing back” and discussing the interview questions – or even trying to put yourself in someone else’s shoes. Thinking aloud meant getting first impressions and responses. They turned out to be incredibly valuable!
Pam’s Post- and Final- Survey:
My final revised survey can be found here. (It was just too hard to get the survey into this blog, plus it looks much better in pdf format!)
Pam
October 9, 2008
Week 7 Evaluation Reflections: The Interview
There’s a lot to reflect on in terms of this week’s interview question development and test drive, so I’ve written it in sections. I know it’s a lot, but it has been a great learning experience and I felt the need to share with anyone willing to wade through it all! (Or parts, if they wish…)
Developing the interview guide:
Having read so many examples and so much information about AI, I felt pretty prepared to write my dyad interview questions. I decided, however, to double check what information this week’s lecture provided and re-read chapter 4 of the Preskill text with writing those questions in mind. From the lecture I discovered that I needed 4 to 7 questions, that I should test them out or take them for a bit of a test drive, make sure they’re linked to my key questions and focus on moments of excellence. So far so good. On to the Preskill text. There were a lot of examples of AI questions throughout the text, actually, but they were in different situations evaluating different types of programs. It became clear I needed to think about my key questions and evaluation situation while looking at examples in the text. It, of course, spoke of stories and wishes, values and visions, quality and worth, significance and excellence along side successful moments, pride in achievements, successful processes and outcomes, and moving towards more of these successes in the future.
I knew I wanted to document the strengths and best practices of technology integration in our district. Thus we could build on those strengths. So I began with a peak experience question. It wasn’t as quick and easy to write as I thought it would be. This had to be a peak experience aimed right at my key questions. I wanted them in the computer lab, integrating technology, but what else? I wanted them successful and proud and their students to be meaningfully engaged. With a mix of some examples in the text and these thoughts, my first interview question was born:
Think back to the times when you were in the computer lab with your students, integrating technology into the classroom curriculum. Describe a peak experience when you felt most successful, excited and proud – a time that you felt technology added real value to the lesson at hand – and students were engaged in meaningful ways.
The next two just flowed:
What circumstances or conditions made this exceptional experience possible?
If you had 3 wishes that would ensure that every technology integration experience would be as good as the one you just described, what would they be?
Then some failed attempts I immediately erased. That led to examining more examples, a look at my list of AI question qualities or often used words, and a look again at my key questions. What did the first three questions interview questions chosen not address? What was missing? Ah! At that point I knew it when I saw it.
Without being humble, what do you most value about yourself with regard to the technology integration opportunities you afford your students?
Good. Yet, I still wasn’t sure I’d get all the information I wanted about the current lab system and its delivery of core technology skills along with integration opportunities. Then I found the example that I could use as a model to perhaps get at this a different way.
Based on your experiences so far, what do you value most about the current computer lab system in terms of providing core technology skills and offering technology integration opportunities?
Four to seven and one more would make six. A good number. I wasn’t sure the 3 wishes question would get me everything I needed about the future direction(s) to take. I’d been drawn to the “pretend you were asleep for 5 years” question since I first saw it. It would offer that future info I want, so I decided to use it. Writing it wasn’t as easy as I’d thought, as well. I had to dream a vision as well to produce the scenario and Good Morning, America interview questions!
Imagine that you have been asleep for 5 years and it is now 2013. As you awake, you look around and see that District 43’s computer lab system is a model for effective technology integration and delivery of core technology skills. Not only are core skills mastered and integration opportunities provided, but students and staff alike are energized and engaged in meaningful activities that provide added value to the classroom curriculum. Lab scheduling runs like clockwork. Staff members use technology in innovative ways and are convinced technology brings an added dimension to their curriculum, something unique. Administrators are certain the monies spent on technology bring a much greater educational return than their actual monetary investment. They are extremely proud of the district staff’s integration of technology, student innovation and achievement, and the district’s computer lab system as a whole. As a matter of fact, the district has received a joint ISTE/ISBE award for excellence in technology integration and delivery of the NETS. Two weeks after receiving this award, you receive a phone call from the Good Morning, America show. They want you and five of your colleagues to appear on next Wednesday’s show. With the award in hand, you arrive in New York and are escorted onto the program set. Diane Sawyer starts the interview by asking you to describe what technology integration looks like in District 43 and how it works. She further asks: How are the core technology skills delivered? What key factors attribute to the fact the lab schedules are busy yet readily accessible? What advice would you give to a school district that wants what you have – energized students, staff and administration – innovative uses of technology that bring added value to the curriculum – unprecedented student achievement and creativity? What key features support this winning system? … What does your team tell Diane Sawyer?
I had my interview questions. The next step was to take them for a test drive… Before my dyad interview, I decided to give them a quick whirl at work.
Test Drive at Work:
I decided to try out my questions on our district librarian. She is always so willing to help and she has good insights and ideas. So, I slipped her the questions and told her “if and when she had time…” Within the hour she was standing at my desk. We talked about the similarity of some of the questions. How they almost asked the same question, but in a slightly different way. The “3 wishes” could be the same as the “what circumstances” one. Yet one might get a response that the other might not. Plus it might deepen the conversation, so we decided to leave those as is. Then she talked to me a little about how they applied to her when she was in the classroom. How when she had great lab experiences it was usually because there happened to be a good open lab time – which is a problem in our district – they’re rare. Then she talked about having a library lab aide that was really helpful and how valuable that extra pair of hands was. And finally she talked about how feeling comfortable in the software was important as well. So, I got a “bonus” mini-interview with my test drive! Plus she laughed and told me she loved my “asleep for 5 years” especially the award and Diane Sawyer. I didn’t know Diane was a personal favorite of hers…
Conducting the Interview:
Actually, Kelly and I did a little “groundwork” before the actual interview. We emailed a couple times about how to manage the interview, thought about Gizmo but decided on the Moodle chat. You can save the whole thing which is great to reflect with, it moves a bit slow but if you’re OK with that it gives some time to process and think and breathe. Plus it’s easy to use. Turns out I think it was a great choice. The other thing we did was toss around our question ideas ahead of time with each other. I think that helped in a couple ways. We got to think a bit and “get in character”. I had to go WAY back in my memory to when I first started teaching. I used my daughter too as she’s a second year teacher – to help me remember those days. We didn’t have an evaluation tool especially for non-tenured staff. My district does now where I work and I got to think about that tool as well kind of as a prep for the interview. Kelly was struggling a bit with her questions and she made some really great changes to them. I think they turned out terrific and she deserves to be proud of herself!
The interview itself went smoothly. I think both Kelly and I were surprised at how easily the conversation flowed from questions and comments. I think we both got involved with the other’s interview questions and ended up learning a lot and having a good time as well. I feel like I know Kelly now. Personally. And as I sat here thinking about that, it struck me… Duh!!! Isn’t that what AI is all about? Personal stories. And stories lead to connections. Plus I had to sit down and write a bit about it right away because I was feeling “cranked up”. Energized in AI terms. Kelly said she was excited about my school’s computer lab system and where it may head – and I really believe she meant it. Energized and connected. AI – the stuff of AI…
I don’t think I’d change my questions much, but I do think I realized a few things about the AI interview process. The text said people would want to slip back into the familiar negative. I had to catch myself a couple times. Stay positive. Stay AI. I also think interviewing is a real skill. Being able to take an answer and respond positively. And respond not just positively but in a way that perhaps can draw out more information, more stories, more wishes…
Summarizing the Interview Data:
I thought it would be difficult to find those patterns and get what you need from AI type questions, but I immediately knew that was not so – once I experienced a taste of it. I would imagine with a number of interviews you would get a lot of data, but definitely there would be patterns – with an added bonus of a lot of good ideas as well. From my mini-interview with Shanna and my full blown interview with Kelly I found out quite a bit about teachers using technology, integrating it well.
These are important ingredients of technology integration success. (Those with an asterisk * were common to both interviews.)
- Tech training
- Comfort level with software used*
- Lab availability*
- Extra pair of hands
- Tech help readily available*
- Timing (teachable moment, right time & place)*
- Engaging activity*
- Ability to get through filter when necessary/no over-blocking of sites
- Up to date, available hardware and software*
I also discovered:
- Success breeds more success (teachers copy and want what others have that works)
- We need to teach our students to be responsible, not block them entirely and not work with them.
- Education / professional development leads to a more positive attitude towards technology. (People are afraid of what they don’t know.)
- Doing something different and/or fun with students energizes them and brings even the shy student out of themselves and into classroom participation.
- Whether it’s a school building or a district, it takes time to get to that place of good technology use and integration.
- Delivering core technology skills to students in a structured way is important.
- Students need to see the value of the lesson(s) at hand.
- Writing AI evaluation questions isn’t as easy as one might think. It takes a lot of thought, revision and re-thinking to create a good list of AI questions.
Pam
Paths & Reflections
October 3, 2008
Week 6 Evaluation Reflections: Key Questions – How I got there…
Key Questions: How I got there…
Clarifying my purpose statement helped me focus in on what key questions to ask. This week’s PowerPoint guided that purpose statement. “What needs to be evaluated?” “What are the intended uses of the evaluation?” “Why evaluate now?” “Why evaluate this as opposed to something else?” I also looked at the example of the purpose statement in the text. I looked back at my purpose statement several times while writing my key questions. I also took a look back at my introduction.
Once I was focused in, I did a little research for instructions and some examples of key questions. I took a good look at the lecture for the week. Speaking to the key questions, the transcript reads, “They’re broad, over-arching, they’re not specific.” They “guide any evaluation efforts”. They’re not “questions on a survey”. Looked pretty important to me! I thought about the PowerPoint list of typical evaluation questions, which ones seem to “fit”, which do not? Then I re-read my introduction, especially the sections that spoke to the purpose of the evaluation.
I threw out “Is the program worth the cost?” That isn’t really the focus. As well, the district hadn’t provided a lot of “frills”. Many districts have computers in classrooms connected to the Internet. We have only one in each – the “teacher” workstation. The only way students get hands-on a computer is in the one lab in the building. Our 3-5 labs were reduced in a size a few years ago from 30 to 25. Now we have a classroom or two each year that can’t “seat the class”. The computers purchased are basic with no “add-ons”. We haven’t purchased monitors in years, so they don’t “match” the workstations. We didn’t purchase speakers, as we could get them cheaper elsewhere. As well, our labs are staffed with aides who run the library for the building as well as the computer lab. My point? We’re pretty “bare bones” now. As well, the focus of the evaluation is about teaching styles, lab schedules, opportunities, technology skills and integration – not cost.
Next I threw out “How sustainable is the program?” We already know it’s sustainable. We’ve had the same system or program in place since its inception fifteen years ago. The ear-to-the-ground “rumblings” are saying it needs to evolve, improve, adjust… Sustaining the program is definitely not “the question”.
So – back to the top of the list. “How well does the program…” just flowed into “provide integration opportunities?” That’s something the evaluation is definitely about. Central. Key. So then a bit of tweaking… We don’t want to forget those basic core skills, the foundation, ISTE’s NETs. “How well does the current computer lab system provide core technology skills and offer integration opportunities?” Over-arching. A guide. A key question for sure.
Moving on down – “What kinds of outcomes did the program produce?” Wow. Absolutely. This evaluation is all about an outcome – the qualitative and quantitative integration of technology. A vital outcome we want to increase. ISBE looks for it in the core of our technology plan. The NETS are based on it. It deserved “first place” on the list: “What kinds and quantities of technology integration does the current system produce?” Yes.
Two or three key questions, so that’s it – I have two – or do I need one more? Last example, “What was the impact of the program on…” Huh. Doesn’t fit. But yet, something seemed missing. I re-read the two key questions I’d decided on. Thought it over. Re-read my purpose and what I’d said about the goals of the evaluation in the introduction. Read my two questions again, and then it came to me. Both of my questions were about now. What we produce now. How well we do it now. What about the future? That’s really what this evaluation is about. Where do we head in the future? Dream. Wish. Envision. It was clear I needed a different sort of question. But just what question about the future? What’s the focus of the evaluation? It’s about integration opportunities. So I wrote, “In what ways might District 43 provide better integration opportunities?” Yes, but… What about Hannah and my major concern, maintaining the core technology skills students now receive? We need that. It’s key. So I added, “…while maintaining the delivery of core technology skills to students?” Balanced. Stable. “In what ways might District 43 provide better integration opportunities while maintaining the delivery of core technology skills to students?” Perfect.
Three. A good number. Complete. I set it aside and slept on it. Then I read them again, re-read my introduction and purpose, re-read the questions. Yep. They stand.
Pam
September 26, 2008
Week 5 Evaluation Reflections: Traditional vs. Appreciative Inquiry
As we pondered the similarities and differences of AI and traditional evaluation this week, I was drawn to thoughts of my limited experiences with evaluation of organizations or programs. We certainly are all familiar with evaluation of our professional performance (evaluating people), but what about that of programs and organizations? Were any of them appreciative in nature? I was involved with our district’s committee or LPDC (Local Professional Development Committee) to approve teacher’s recertification plans and activities. In a sense, we were evaluating and approving those plans and activities to be sent on to ISBE. I have been involved in writing our district technology plans and in a sense that process evaluates our past plan, our district’s progress or lack of it and provides a direction or plan for the future. And I have been on curriculum committees, evaluating for instance a variety of reading publishers and programs to decide which to purchase. Usually those committees don’t spend much time looking back at what has been used in the past, although perhaps they should. Instead they tend to look at their future options. The decision may be made on a list of pros and cons but often it has to do with financial considerations as well. Little data is collected, or at least it wasn’t on the committees I served on. Perhaps research studies are debated as to which curriculum approach is best, but it doesn’t seem to “fit” the overall evaluation scenario. The LPDC didn’t do much other than look at the official rules and decide if that person’s recertification plans and/or activities met them. So again, that was not a true evaluation experience. I’m left then with the district technology plans I have been involved with writing. Next week I begin that journey again as we are “writing” this year. While there will be rubrics and forms and rules and guidelines, there will be a look at where we are. Most of it is pretty traditional in nature. Where are your weaknesses. What isn’t working. How can we fix it. An evaluation of our current reality happens and then a look towards the future. The filling out of forms is pretty traditional evaluation data gathering. How many of this and how much of that… However, it also may include interviews and focus groups. At this point there is a possibility of a move from the traditional towards Appreciative Inquiry. The tone of these interviews and focus groups varies, depending on the administration at the time and how they approach gathering data and creating the plan. The most excitement I can recall at this point in the process occurred when staff members were asked about their successes with technology and what they would wish for if funds were available. Not everyone got everything they wished for, but many did. There was positive energy about it all as staff members formed those granted wishes into future reality. That was a piece, a small experience, of a component of Appreciative Inquiry. My hope is for that as we “write” this year. And perhaps with my new knowledge and awareness of AI, this “writing” can be energizing as we look to the next three years.
Reflections on others’ thoughts:
The postings this week have been more varied and interesting. We have three choices! That means less repetition and a greater variety of thoughts. Same topic, different perspective… Traditional evaluation vs. AI… How are the similar? How are they different? What about competencies? What makes AI fresh and positive? And how is all of this applied?
I loved what Julie Bryniczka said. “Making the similarities and differences table really drove home the point that AI focuses on the positives, strengths and desired outcomes – aka looking at a situation with a “glass half-full” mentality. On the opposite end of the spectrum, traditional evaluation focuses on the negatives, problems and weakness…and that’s never fun!” AI: positives, strengths, desired outcomes… Traditional: negatives, problems, weaknesses… Not only are the questions for each framed differently, but AI really looks to a fresh, new and positive future. By design, AI asks people to envision, to dream, in a “what if” perspective. Set aside the “can we” questions and barriers. Just dream. Wish. That makes all the difference. Traditional evaluations look to the future in a narrow way. What piece was wrong? How can we patch it? Stark contrasts for the future.
Another contrast? Traditional evaluation looks at problems, things to fix and patch. Pieces. Parts. A patchwork of sorts results. Appreciative Inquiry looks at and for a vision, a common goal, the betterment of the whole. AI speaks of teamwork, collaboration, constructive union, community.
And application? Applications of traditional vs. AI can be seen everywhere, if one just looks. Work to your strengths. I didn’t truly appreciate the meaning of that phrase until the past couple of weeks. David speaks of it in terms of the Marines and his experiences as a recruiter. Others speak to it in stories about coaching and sports. Use your strengths. Build from them. And then there is the half-empty, half-full scenario. In the same situation, some people will complain and see only the negatives. Others will find the good in it and build from that good. It doesn’t really matter if that situation is a day to day routine one, a crisis or a miracle. I’ve known people who find goodness in crisis and holes in miracles. AI has a lot to teach about life.
Pam
September 19, 2008
Week 4 Reflections: AI Annotated & Revisited – Reviewed and Re-thought
Part One -
Further thoughts from
my response to Rebecca:
“This week re-reading Chapter 2 in Preskill I was struck by the many uses of AI outside of evaluation specifically. I guess when I did an initial read-through when the text arrived at my doorstep, I was really thinking about and concentrating on what was said or how it applies to evaluation specifically. Your articles bring this home again. You speak to the beginnings of AI through Cooperrider who studied its use in terms of “models for organizational and professional development”. AI is useful for student affairs, working with 7-9 year olds (active listening) and working with Catholic high school students (school reform). What a variety of organizations and people! (I even thought about Tom Anderson’s course when you mentioned active listening…)”
As I have gone from re-reading Preskill’s Chapter 2 to exploring my cohorts annotated bibliographies, I am struck again by that thought. Large corporations, private and specialized schools, online communities, varying age groups and cultures and genders – everywhere the use of AI serves needs and serves them well. It’s use in evaluation is just a small piece of that. Cooperrider & Whitney (“A Positive Revolution in Change: Appreciative Inquiry” – Draft, no date – Recommended Reading) call it “a positive revolution”. While for the business world at the time of their writing that most certainly is true, I believe as a human race the underlying principles have held true back to our very beginnings. Someone tries something new or different. They are praised or encouraged. They then are energized in a positive way and the positive effect snowballs and grows. That is human nature. Or – also human nature… We have a problem. A dilemma. We grumble, complain and find fault or blame. Nothing good comes of it – instead hard feelings as well as negative thoughts and actions are produced. Yet – Someone has a dream. They think it and envision it, then begin and take steps toward it. They are committed and work hard at it. Good things happen and dreams may even be attained. Cooperrider & Whitney’s same article states that AI is a “search for the best in people” that “gives life to a living system”, “speeding imagination and innovation”. I think beyond that, AI gives a focus – a systematic way to take positive aim at a target. From Cooperrider’s perspective, you use the steps of Discover (“mobilizing a whole system inquiry”), Dream (clarifying your vision), Design (look at exactly how to attain that vision) and Destiny (solidifying it all with “deep purpose”). In a sense, you take aim and head for that target – and along the way you refine or readjust that aim and clarify the target. But it’s more than that – because a group dynamic is involved as well… Group process and wholeness… Together, not alone… With purpose and some group creativity thrown in.
So I’m left with thoughts of “random” affirmation or positive effects versus “targeted” affirmation as in Appreciative Inquiry (a systematic thing)…. When my youngest daughter now twenty-five was born, I was thrust into a three year medical nightmare. Within six weeks of her birth I’d been swept from suspicions to doctor visits to tests to a hospitalization complete with “haven’t seen this or done this before”, “gather all the help and support you can find”, and “we don’t really know”. A lot of people rose to the occasion. Some sent food, some supported us with their presence and prayers and conversation, some took my three year old on excursions (and away from the fray) and one even threw her a birthday party (as Mom was in intensive care). All were appreciated and wonderful, in and of themselves. Yet they were random. Individual. Unorganized. Sporadic. In contrast there was the church we attended. Someone there was organized. Took careful aim. Designed a plan. Refined that plan and carried it through. Though the numbers of people and kindness of thought and action – taken individually – equaled no more than the randomness of others’ kindnesses and actions, the impact of this organized group was incredible. I received a steady stream of visitors to chat and pray and meet my every need, not too many and not too few. My family received a steady supply of meals – varied and balanced. My newborn daughter was delivered to the hospital three times a day for five weeks once I was out of intensive care and given the OK to nurse her. My three year old was not forgotten, receiving systematic attention as well. Cards and gifts and prayers and calls and organized-acts-of-kindness were heaped upon myself, my children, my husband and my family (who flew in to help as well). Our every need and more was met accompanied by smiling faces and prayerful intentions. Mobilized and organized. With a shared vision and shared values. Dreamed and designed and carried through. With deep purpose.
Part Two –
Diversity Confirmed
Which brings me back full circle to the annotated bibliographies’ articles… From Girl Scouts Behind Bars, to a small alternative U.S. high school, to a Family Rehabilitation Center in Sri Lanka. So varied and diverse yet all benefit from AI…
Girl Scouts Behind Bars
Smart, Dawn Hanson and Mann, Mariann (2003). Incorporating appreciative inquiry methods to evaluate a youth development program. New Directions for Evaluation. 100, 63-74.
In this case, the Girl Scouts Beyond Bars program choose the piece or part of AI that would best suit their purposes and needs. They used the inquiry portion of AI, recording and using interviews to identify and confirm their ideas for change and the future. They also used the AI data to identify and confirm their long term outcomes. As Scott C. mentions in his annotated bibliography, this is the perfect program to accentuate the positive using Appreciative Inquiry. GSBB in state after state after state gives hope and promise to children and families, “support girls” and “connecting families”.
A Small Alternative U.S. High School
De La Ossa, P. (2005). “Hear my voice:” alternative high school students’ perceptions and implications for change. American Secondary Education. 34, 24-39.
In an attempt to understand how new students adjust to and make sense of their alternative school experience, a quantitative study was designed to find out why traditional education does not “work” for all students. AI focus groups were used and evaluators found that indeed high school students can provide valuable information on the topic. I found it interesting that they used video to document these focus groups. I had to wonder if doing so would cause some to be more hesitant to speak. After all, one of the benefits of AI is that it can free people to speak honestly and without reservations – as opposed to more problem-based, negative interview situations.
Family Rehabilitation Center in Sri Lanka
Jacobsgaard, Mette (2003). Using appreciative inquiry to evaluate project activities of a nongovernmental organization supporting victims of trauma in Sri Lanka. New Directions for Evaluation. 100, 53-62.
In this case, the Family Rehabilitation Center, a program that serves victims of torture and trauma used AI to evaluate their program. Actually, they speak of Cooperrider’s four D’s, yet still AI… Absolutely a group in need of positive, affirmative thoughts! And two-fold in need as Scott C. comments. Not only are the victims obviously in need of the affirmative, the evaluators themselves could easily be drawn into much worse than the term “negative” connotes. And leaving the last two stages, Design and Delivery, to the clients to put into action must have added a certain empowerment to their healing-towards-wholeness process.
Children of prisoners, “ordinary” U.S. high school students and victims of torture…
Pam
