WEBVTT 1 00:00:02.680 --> 00:00:15.750 Amy Pinkerton, BSPH CTL Teaching Toolkit: All right, hello, and welcome to Analyzing Student Course Evaluations. My name is Amy Pinkerton, and I'm a Senior Instructional Designer at the Center for Teaching and Learning, or CTL, and I'm joined by my colleague… 2 00:00:16.070 --> 00:00:20.740 Lauren Dana: Hi, I'm Lauren Dana. I'm also an instructional designer for the Center of Teaching and Learning. 3 00:00:21.240 --> 00:00:31.379 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and today, we are going to start out by identifying strategies that can help alleviate stress surrounding course evaluation. 4 00:00:31.380 --> 00:00:49.260 Amy Pinkerton, BSPH CTL Teaching Toolkit: Then we'll overview the course revision process, identify sources of course data to consider in that process, and then we'll finally conclude where we'll actually get to practice analyzing some qualitative student course evaluation data to identify specific course revisions and actionable next steps. 5 00:00:49.850 --> 00:01:09.479 Amy Pinkerton, BSPH CTL Teaching Toolkit: But first, let's start out with why is this important? Why should you take the time to analyze your student course evaluation and your course data? Well, student course evaluations, they're an important factor in determining the perceived quality of your teaching and learning, and the learning experiences of your students for your course. 6 00:01:09.910 --> 00:01:23.219 Amy Pinkerton, BSPH CTL Teaching Toolkit: And this data is used in a number of ways. First, as a faculty member, it allows you to critically reflect on your teaching and help you identify possible updates to your course and its organization and delivery. 7 00:01:23.670 --> 00:01:30.900 Amy Pinkerton, BSPH CTL Teaching Toolkit: It can offer insights into your teaching effectiveness, and maybe even inspire some professional development. 8 00:01:31.320 --> 00:01:53.609 Amy Pinkerton, BSPH CTL Teaching Toolkit: And finally, course evaluations are used even beyond the scope of a course. You as the faculty would want to review your course evaluations, but they can also be used by school administrators, academic departments, and programs to make data-informed decisions about the larger programs and series of courses that your course fits within. 9 00:01:53.880 --> 00:01:56.650 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, it's important data to consider. 10 00:01:56.880 --> 00:02:11.660 Amy Pinkerton, BSPH CTL Teaching Toolkit: But there's an elephant in this room that's a little bit hard, and we wanted to address it before we go any further. And that is that sometimes feedback can be difficult, especially subjective, qualitative feedback. 11 00:02:11.660 --> 00:02:23.230 Amy Pinkerton, BSPH CTL Teaching Toolkit: It can be tough. When you've poured a lot of time and energy into your course, it's natural to feel a little defensive, or maybe even discouraged as you're reading through your student comments. 12 00:02:23.230 --> 00:02:40.320 Amy Pinkerton, BSPH CTL Teaching Toolkit: So you might have thoughts like, I worked hard on this, why are my students not satisfied, or maybe even something has worked in the past, why is it not working now? So, when revision is tied to these negative emotions, the whole… it can sour the whole revision process. 13 00:02:40.940 --> 00:02:52.550 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, to address this, I wanted to reframe how we think about revision. So revision isn't a bad thing. It's not… it's not a bad thing to go back and review your course and 14 00:02:52.550 --> 00:03:02.919 Amy Pinkerton, BSPH CTL Teaching Toolkit: make changes and updates to it. It's actually a natural… it's a good sign if you're reviewing your course. It's a natural part of the course revision cycle of development. 15 00:03:02.920 --> 00:03:14.839 Amy Pinkerton, BSPH CTL Teaching Toolkit: implementation, evaluation, and revision. So instead of thinking of revision as, like, punitive or negative, think of it as refinement, like perfecting a recipe. 16 00:03:15.440 --> 00:03:26.760 Amy Pinkerton, BSPH CTL Teaching Toolkit: And, welcome to those who've joined us, and thank you, Lauren, for posting in the chat. If you have questions, you can post those in the chat. Lauren and I will be monitoring, and there will also be opportunities for Q&A throughout. 17 00:03:27.830 --> 00:03:28.760 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright. 18 00:03:28.760 --> 00:03:48.049 Amy Pinkerton, BSPH CTL Teaching Toolkit: And just to really hone in on why… how feedback can be a little difficult, I wanted to share a quick story about how I've dealt with, course… student course evaluations, from the TA training course that I co-teach with some of my colleagues at the Center for Teaching and Learning. 19 00:03:48.520 --> 00:04:06.620 Amy Pinkerton, BSPH CTL Teaching Toolkit: So back in 2020, our course evaluations were clear that students wanted two things. They wanted more instructor presence, and they wanted clearer guidance for each of the modules throughout the course. There were some frequently asked questions and some things that needed clarity. 20 00:04:07.160 --> 00:04:26.819 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, my co-instructor and I, we created these short module introduction videos. We thought, oh, this is great. We'll be on screen, we'll increase our presence, and we'll also be able to introduce each module and proactively address frequently asked questions. We were like two birds, one stone, this is great. 21 00:04:26.990 --> 00:04:40.650 Amy Pinkerton, BSPH CTL Teaching Toolkit: Fast forward to 2023, and the feedback was clear. The students hated the videos. They… they disliked the videos, from an information standpoint, because the frequently asked 22 00:04:40.650 --> 00:04:57.079 Amy Pinkerton, BSPH CTL Teaching Toolkit: the answers to the frequently asked questions, they would have to scrub through the video to get to that information, or go through the transcript. Also, there was, like, extra clicks because they had to go to the page to watch the video, and then click off of it and go to the next page to get to the next lecture. 23 00:04:57.080 --> 00:05:03.530 Amy Pinkerton, BSPH CTL Teaching Toolkit: And I also had the feedback that my voice sounded like a cartoon character, which that one… that one stung a little bit. 24 00:05:03.530 --> 00:05:25.039 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, so we regrouped, and we kept the core content from the video, but we converted it to concise, text on overview pages so that students could quickly go back and look at that information, and we decided to increase instructor presence in other ways, like office hours and more meaningful feedback, for the assignments. 25 00:05:25.430 --> 00:05:35.629 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then fast forward again, we went… we reviewed the course another second time, and the evaluations improved after the second round of revisions. 26 00:05:35.710 --> 00:05:40.480 Amy Pinkerton, BSPH CTL Teaching Toolkit: And the key takeaways are, from this experience, is that feedback 27 00:05:40.480 --> 00:06:00.110 Amy Pinkerton, BSPH CTL Teaching Toolkit: can be hard, and sometimes if you respond to feedback, it might not work the first time. So when you're doing your course revisions, there is some trial and error. Even we as instructional designers don't always get it right the first time, so you might have to try something, see how it goes, and then make more changes, and respond and pivot as necessary. 28 00:06:00.290 --> 00:06:13.090 Amy Pinkerton, BSPH CTL Teaching Toolkit: So just give yourself a moment to feel what you feel about your revisions or your feedback, and then when you're ready, use those insights to make changes that, will really benefit your learners. 29 00:06:13.480 --> 00:06:23.919 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and now I will… oh, I just realized I wasn't clicking through the animation, sorry about that. Now I'll pass to Lauren to tell us about the course revision process. 30 00:06:24.420 --> 00:06:44.299 Lauren Dana: Thank you, Amy. So, after reflecting upon your most recent course offerings, or offering, many of you are probably going to want to make some course revisions. And these revisions may be major, they may be minor, or somewhere in between, but regardless, we are really hoping that you follow these steps as you begin to think about course revision. 31 00:06:45.560 --> 00:07:03.820 Lauren Dana: So our first step is determine the why. So Amy spent some time talking about why evaluations are important, but you need to determine your own why. Maybe you noticed that the course felt stale, or that you received negative student feedback. Maybe students aren't picking up on concepts as quickly as they had in the past. 32 00:07:03.900 --> 00:07:09.370 Lauren Dana: Or maybe you're more worried than ever about academic integrity in this world of generative AI. 33 00:07:09.480 --> 00:07:18.520 Lauren Dana: And even though you may add more reasons to your why as you dig more deeply into course data, it's really important to approach revisions with your own why. 34 00:07:20.100 --> 00:07:35.820 Lauren Dana: Okay, so the next step, you're going to gather any applicable qualitative and quantitative data from your courses. We are going to review the quantitative and qualitative data available to you in this workshop, but our main focus is going to be on student evaluations as a data point. 35 00:07:36.830 --> 00:07:48.560 Lauren Dana: Next, you need to spend some time analyzing and reflecting on what this data tells you. You will create an action plan for changes that you hope to implement in your course based on this analysis and reflection. 36 00:07:50.270 --> 00:07:59.279 Lauren Dana: And then, over the next few months, the goal is to revise this action plan and make sure that you have a concrete set of changes. 37 00:08:00.930 --> 00:08:18.170 Lauren Dana: Next, you're going to implement this action plan. This can take a couple months, it could take a year, multiple years, but you want to make sure you have implemented what you would like to implement for the next time your course or courses run. And unfortunately, the revision process doesn't stop here, as Amy spoke about. 38 00:08:18.660 --> 00:08:32.590 Lauren Dana: The course revision and reflection process is cyclical, and you'll repeat these steps every time after your course runs. However, we did want to remind you that you're human, which we can all forget sometimes, with many competing priorities. 39 00:08:32.840 --> 00:08:46.970 Lauren Dana: And the quality of the course reflection and revision process is really going to vary from year to year. There may be some years you can only make one or two small changes, and other years where you feel like you have the time and the energy to revamp most of your course. 40 00:08:47.120 --> 00:08:59.539 Lauren Dana: We do promise today, however, that you're going to complete steps 2, 3, and 4, or at least get started on this with us, and we hope this makes the revision and reflection process a little bit more accessible. 41 00:09:01.930 --> 00:09:12.869 Lauren Dana: So, as promised, Amy and I wanted to first give you an overview of the type of course data that's available to you, and we're first going to start with quantitative analytics that are available in Course Plus. 42 00:09:14.200 --> 00:09:27.060 Lauren Dana: So to access these quantitative analytics, you'll first go to Faculty Tools, and on the right column, you're going to see a section labeled Administrative Tools. And the quantitative data is going to be under Student Activity Reports. 43 00:09:27.200 --> 00:09:39.240 Lauren Dana: So you're gonna see that CoursePlus offers a variety of reports. You can look at lecture content access, live talk attendance if your course is fully online, discussion forum activity, and site access. 44 00:09:39.600 --> 00:09:48.330 Lauren Dana: So, the data and learning analytics from this Course Plus Analytics is very important, and it is great to use in course revisions. 45 00:09:48.440 --> 00:10:06.500 Lauren Dana: So, I do recommend, I'm gonna plug Amy in my former workshop. So, in the worksheet I put in the chat, you're gonna see a link to one of our former workshops, diving deep into course data. And so this is going to go a lot more deeply into how you can use this quantitative data in course reflection and revision. 46 00:10:08.890 --> 00:10:12.989 Lauren Dana: Okay, I'm now going to go over quickly the qualitative data available in Course Plus. 47 00:10:15.960 --> 00:10:26.190 Lauren Dana: Okay, so the communication that you have with students are a qualitative data source. So you can review emails that you can send… that you sent to students in the email archive feature in Course Plus. 48 00:10:26.200 --> 00:10:41.170 Lauren Dana: I did want to note, you can only view the emails that you sent if you use the Course Plus email tool and selected the Save to Email Archive setting. Amy and I do recommend that you do this for every class, it just makes life a lot easier to have access to these emails. 49 00:10:41.240 --> 00:10:55.870 Lauren Dana: But we do recommend that you consider gathering information from any emails that you receive from students as well, such as frequently asked questions or reported issues. So, for example, maybe Lecture 3 needs a frequently asked questions associated with it. 50 00:10:56.010 --> 00:11:09.210 Lauren Dana: And then you can post that on the lecture page. Or students might repeatedly identify an assignment with a broken link, or an online library resource that doesn't work, and you can make the necessary fixes for next year. 51 00:11:09.760 --> 00:11:16.439 Lauren Dana: So just make sure that you're, as you're going through the emails, that you're looking for trends, frequently asked questions, and any reported issues. 52 00:11:18.660 --> 00:11:32.859 Lauren Dana: Okay, and last but not least, which is the main point of this workshop, is student evaluations. So, as many of you know, each term, students provide anonymous feedback on their courses during a predetermined evaluation period. 53 00:11:32.880 --> 00:11:41.829 Lauren Dana: There are standard questions that the students receive that consist of the students rating the course on a numerical scale, and there's also open-ended questions. 54 00:11:41.960 --> 00:11:49.290 Lauren Dana: However, for today's purpose, we're gonna place the four open-ended questions in the chat, so I'm gonna put that in right now. 55 00:11:50.190 --> 00:12:07.450 Lauren Dana: Just so you can take a look at them. And you do actually have the option to add 3 open-ended questions to your own course evaluations, which many faculty don't take advantage of. So if you're looking for something specifically from students about your course, I highly recommend that you consider adding an open-ended question. 56 00:12:07.560 --> 00:12:21.349 Lauren Dana: And once students have submitted the evaluations, the faculty can go into the course evaluation system and take a look, and you can compare course evaluations from up to 5 courses, and the courses can be from any of the last 3 academic years. 57 00:12:21.660 --> 00:12:30.380 Lauren Dana: So these evaluations should help provide you with student perspectives, as well as guide future course modifications and redesigns if necessary. 58 00:12:35.000 --> 00:12:45.470 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so when you're identifying changes for your course, it's also a good moment to reflect on your own experience. So, so far, we've focused on data that shows the student experience. 59 00:12:45.470 --> 00:12:55.059 Amy Pinkerton, BSPH CTL Teaching Toolkit: But it's… it is important to think about you as the faculty, and then also, ask about the perspectives of any co-instructors or teaching assistants that you work with from the course. 60 00:12:55.060 --> 00:12:59.550 Amy Pinkerton, BSPH CTL Teaching Toolkit: So if you were leaving qualitative feedback for your course, what would you say? 61 00:12:59.910 --> 00:13:11.409 Amy Pinkerton, BSPH CTL Teaching Toolkit: Taking a moment to reflect critically about your course will help you contextualize and analyze the available course data, not just from your students' perspective, but from your perspective as… your perspective as well. 62 00:13:11.770 --> 00:13:26.969 Amy Pinkerton, BSPH CTL Teaching Toolkit: It can deepen your understanding of your teaching practice, reveal how your course actually functions day-to-day in the behind the scenes, and can clarify that why certain, the why reason, why certain changes might be necessary. 63 00:13:27.170 --> 00:13:29.310 Amy Pinkerton, BSPH CTL Teaching Toolkit: So do take a moment to reflect. 64 00:13:30.650 --> 00:13:47.409 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so, so far, we've identified how reframing the revision process as refinement can help alleviate some stress surrounding course evaluation, and then Lauren overviewed the course revision process and identified sources of course data to consider in the revision process. 65 00:13:47.410 --> 00:13:54.909 Amy Pinkerton, BSPH CTL Teaching Toolkit: Before we go any further, are there any questions? And you can post those in the chat, or, raise your hand in Zoom. 66 00:14:04.160 --> 00:14:13.229 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, I don't see any hands raised. We'll continue monitoring the chat, so if you do have a question, you can post it in there, or there'll be more opportunities for Q&A. 67 00:14:14.870 --> 00:14:24.929 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so, now we're going to take some time to model… to go over our model for analyzing qualitative data. 68 00:14:24.940 --> 00:14:45.110 Amy Pinkerton, BSPH CTL Teaching Toolkit: Again, evaluations are vital to the learning process for both students and faculty, but it can be difficult to analyze those qualitative, subjective student comments that come in through the course evaluations. So we're going to model a method for analyzing student responses that's based on the type of feedback that you receive. 69 00:14:45.130 --> 00:15:02.340 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, first we'll categorize comments, then sort them into themes, and then identify actionable next steps based on the themes. And thank you, Lauren. In the, in the chat, Lauren just shared a worksheet where you can follow along, and we'll also use this worksheet for our activity. 70 00:15:02.970 --> 00:15:05.270 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so let's dive in. 71 00:15:05.270 --> 00:15:27.659 Amy Pinkerton, BSPH CTL Teaching Toolkit: onto step number one, which is categorize student comments. So here, you're going to go through, your student comments, and start categorizing. And for this, what I like to do is I like to copy and paste them into a Word document, and then use, like, a highlighter, or maybe a comment tool, and then also the strikeout, so that I can quickly sort through and, 72 00:15:27.810 --> 00:15:45.119 Amy Pinkerton, BSPH CTL Teaching Toolkit: go through my comments. So let's see, how do we categorize them? We're going to categorize them by type of comment. So, if your comments are unrelated to teaching and learning, or are non-specific enough to be helpful, then those can be discarded, because they're not going to contribute to your course review efforts. 73 00:15:46.030 --> 00:15:48.449 Amy Pinkerton, BSPH CTL Teaching Toolkit: Then, start with your positive comments. 74 00:15:48.450 --> 00:16:13.370 Amy Pinkerton, BSPH CTL Teaching Toolkit: These show what's working well in your course, and you can enjoy these comments, you can go back to them if you're feeling, sour about the experience, you can go back and enjoy what's working well. You also want to take note, if something's working well, you don't want to break what's working well in your future changes, so make sure you take note of what not to change. And you can look for themes, between what's 75 00:16:13.370 --> 00:16:21.109 Amy Pinkerton, BSPH CTL Teaching Toolkit: working well and what's not working well with your less positive feedback, and see if students might agree or disagree on what's working well. 76 00:16:22.320 --> 00:16:28.329 Amy Pinkerton, BSPH CTL Teaching Toolkit: Then, we're going to distinguish between actionable suggestions and unactionable suggestions. 77 00:16:28.480 --> 00:16:38.419 Amy Pinkerton, BSPH CTL Teaching Toolkit: Starting with actionable. These are comments or suggestions that you, as your faculty, have the ability to address and control within the context of your course. 78 00:16:38.420 --> 00:16:51.049 Amy Pinkerton, BSPH CTL Teaching Toolkit: So these are things like clarifying instructions, or changing your assignments around, or changing the way your course is organized. These are, actionable suggestions are suggestions that you can make within your course. 79 00:16:51.660 --> 00:16:55.029 Amy Pinkerton, BSPH CTL Teaching Toolkit: Non-actionable or unactionable suggestions. 80 00:16:55.030 --> 00:17:11.730 Amy Pinkerton, BSPH CTL Teaching Toolkit: are, issues that you can't address or fix within the scope of your course. These are things like where the course falls within a program, or, the format that the course takes, whether it's a… maybe it's an in-person class and a student has asked for it to be online. 81 00:17:11.730 --> 00:17:20.419 Amy Pinkerton, BSPH CTL Teaching Toolkit: So these are things that you need to have a higher conversation with, with someone else, or maybe it's something out of your control, like a Course Plus tool. 82 00:17:20.420 --> 00:17:33.270 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, for these, we recommend passing or sharing this information to the people who can make use of that information, like your program coordinator, classroom facilities, or the Center for Teaching and Learning. 83 00:17:33.790 --> 00:17:45.539 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, using this approach helps you to quickly sort through your student comments in a clear, objective way. And so now, we're going to actually practice this step, so I will pass to Lauren. 84 00:17:46.180 --> 00:18:01.210 Lauren Dana: Thank you, Amy. So, as Amy mentioned, we're actually going to look at a sample course's student evaluations. So, I want you to first take a look at the student responses to the open-ended question. Please identify what you consider to be strengths of the course. 85 00:18:01.620 --> 00:18:17.229 Lauren Dana: So I'm going to give you just about a minute to read the responses on either your worksheet or the slide, and I want you to think about Amy's suggestions on how to categorize these responses, which would you discard, which would you consider as actionable, and which would you possibly refer to a higher up? 86 00:18:17.610 --> 00:18:22.629 Lauren Dana: Alright, so we have some music for you while you review this, and we will give you some time. 87 00:19:25.800 --> 00:19:38.080 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, that was our one-minute timer. I know that went by pretty quickly. So if you were able to categorize some of these, let us know in the chat. The comment number 88 00:19:38.080 --> 00:19:46.489 Amy Pinkerton, BSPH CTL Teaching Toolkit: Which comment would you discard? So I'll put a discard in the chat, so let us know which one you discarded, by sharing the number. 89 00:19:54.680 --> 00:19:57.779 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, we see we discarded number 4. 90 00:19:58.990 --> 00:20:14.549 Amy Pinkerton, BSPH CTL Teaching Toolkit: All right, and then, oh, nope, yep, number 4. And number 4 was, everything about the course was awesome. So that's a nice one to hear, but it's not specific. Alright, and then which one did you… were any of these actionable? 91 00:20:19.150 --> 00:20:22.000 Amy Pinkerton, BSPH CTL Teaching Toolkit: Oh, we saw number 1 is actionable. 92 00:20:22.570 --> 00:20:25.079 Amy Pinkerton, BSPH CTL Teaching Toolkit: Got a number… another number 1. 93 00:20:26.620 --> 00:20:31.199 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and then… which were unactionable. 94 00:20:34.890 --> 00:20:37.830 Amy Pinkerton, BSPH CTL Teaching Toolkit: Oh, got a number 1. Number 5… 95 00:20:39.570 --> 00:20:42.149 Amy Pinkerton, BSPH CTL Teaching Toolkit: One to two may lead to more interviews. 96 00:20:43.140 --> 00:20:44.690 Amy Pinkerton, BSPH CTL Teaching Toolkit: 4 and 5… 97 00:20:44.970 --> 00:20:52.689 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so, now I'll share how I categorized these, and I'll give a little bit of, 98 00:20:53.290 --> 00:20:55.469 Amy Pinkerton, BSPH CTL Teaching Toolkit: Explanation for each one. 99 00:20:56.110 --> 00:21:00.060 Amy Pinkerton, BSPH CTL Teaching Toolkit: And this is how I would organize them, but you might… you might have other insights. 100 00:21:00.510 --> 00:21:14.969 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, number one, I thought this was actionable because, number one says, the group assignments were practical and could be applied to real life. I do wish directions had been a little bit clearer, so that's actionable. I can look at my instructions and make them clearer. 101 00:21:15.400 --> 00:21:32.079 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 2 says, I loved the videos with the interviews, the teaching team was amazing. So those are positive, specific trends, so I can… I can say, oh, okay, students like the videos, students like how the teaching team is working, those are two trends to… to think about. 102 00:21:32.160 --> 00:21:47.909 Amy Pinkerton, BSPH CTL Teaching Toolkit: Again, with number 3, the weekly email that the TA sent really helped me stay organized, so that's more positive, trend going towards the teaching team. And the video interviews were great. So again, students are… seem to be liking the videos. 103 00:21:48.860 --> 00:22:08.010 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 4, just like you said in the chat, you can discard number 4, which says, everything about the course was awesome. That's good to know, but it's not specific enough to let me know why it was awesome. So, it's a good comment to come back to, but, it's non-specific for our review efforts. 104 00:22:08.510 --> 00:22:14.009 Amy Pinkerton, BSPH CTL Teaching Toolkit: And number 5 says, I loved the course and wish it would be offered online as well. 105 00:22:14.010 --> 00:22:29.429 Amy Pinkerton, BSPH CTL Teaching Toolkit: And I said, non-actionable share, I might share that with my, program coordinator to see, would it be feasible to offer this online? What would that mean? And have a larger conversation about, that information, if that's a route you would want to pursue. 106 00:22:30.120 --> 00:22:34.369 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so now I'll pass back to Lauren, and we'll do another round. 107 00:22:34.770 --> 00:22:45.500 Lauren Dana: Great, so now we're going to go in the little ones that could be a little bit harder to digest, so these are going to be the responses to the questions. Please identify areas where you think the course could be improved. 108 00:22:45.530 --> 00:22:54.919 Lauren Dana: So again, you're going to take a minute to skim through the comments, and mark the comments you would discard, refer to a higher-up, or ones you might actually use for actionable suggestions. 109 00:22:54.940 --> 00:23:07.469 Lauren Dana: I do want to give you a quick reminder before you start that you can discard parts of a comment and keep other parts, so that's just something to consider. And at this point, you also want to start to be looking for common themes. 110 00:23:07.700 --> 00:23:18.929 Lauren Dana: Again, another reminder, one comment from one student doesn't necessarily mean you should act on it. You want to make sure that other students have a shared experience before you change your whole course on one opinion. 111 00:23:19.490 --> 00:23:25.379 Lauren Dana: So we're gonna give you another minute with our lovely free beat audio, and you can take a look. 112 00:24:29.700 --> 00:24:32.580 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, that's our one-minute timer. 113 00:24:32.720 --> 00:24:43.659 Amy Pinkerton, BSPH CTL Teaching Toolkit: And we'll do this again, where we'll start with, did you discard any? So let us know the number in the chat, which ones, or which one or ones you might have discarded. 114 00:24:45.040 --> 00:24:47.150 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, I see number 2. 115 00:24:47.770 --> 00:24:50.170 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 2 says, this course was boring. 116 00:24:50.720 --> 00:24:54.460 Amy Pinkerton, BSPH CTL Teaching Toolkit: So yeah, you can discard that, because it doesn't say why the course is boring. 117 00:24:55.810 --> 00:25:04.519 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and then… How about actionable? Were any of these actionable? 118 00:25:10.000 --> 00:25:11.739 Amy Pinkerton, BSPH CTL Teaching Toolkit: 1, 3, and 4. 119 00:25:11.900 --> 00:25:14.640 Amy Pinkerton, BSPH CTL Teaching Toolkit: 1… I'm seeing more 1s. 120 00:25:16.000 --> 00:25:21.800 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and then how about unactionable? I know I'm going fast in the chat, so… 121 00:25:31.100 --> 00:25:35.349 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so now… oh, and Celine, I just saw your hand was raised. Yes? 122 00:25:36.980 --> 00:25:54.549 Celine Greene: Thanks, Amy. So the… I put that 5 was actionable, because it might not go toward course design, but it goes toward a greater conversation that something might need to be evaluated at the department level, for instance. So there is action, it's just not necessarily related to 123 00:25:54.550 --> 00:25:58.470 Celine Greene: direct, immediate course design for the next offering. 124 00:25:58.470 --> 00:26:12.030 Amy Pinkerton, BSPH CTL Teaching Toolkit: Right, right. That's a good, good distinction there, where it's like, there is an action to be taken, which is share it with the, with, your program, or the, the program that your course fits within. 125 00:26:12.220 --> 00:26:21.280 Amy Pinkerton, BSPH CTL Teaching Toolkit: And that's a great segue to, here's how I would, categorize these comments. I'll go through each one with a little bit of a rationale. 126 00:26:21.560 --> 00:26:30.749 Amy Pinkerton, BSPH CTL Teaching Toolkit: So number, 1 says, the assignment directions were really confusing, and I wasn't sure what the professor was looking for in the submitted assignment. 127 00:26:30.750 --> 00:26:42.040 Amy Pinkerton, BSPH CTL Teaching Toolkit: And so that's actionable, and that's very similar to another comment that we received, about the instructions being unclear. So that's a trend now that we're… we've seen it a few times. 128 00:26:42.450 --> 00:26:57.330 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number two, you are correct, we can discard this one. Number two says the course… this course was boring. They don't… unfortunately, they don't tell us why the course was boring, so it's so nonspecific that we can't, act on it, so we can discard that comment. 129 00:26:57.760 --> 00:27:12.950 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 3, I see the point of the group assignments, but my group really struggled. We weren't sure where to submit our assignment, and if we each needed to submit the assignment. That's actionable, that again has to do with clarity for the instructions, and how the assignments are submitted. 130 00:27:13.670 --> 00:27:23.469 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 4. Now, number 4, I think, the way that I categorize this is I pulled out some parts that were actionable and some parts that were unactionable. 131 00:27:23.910 --> 00:27:25.550 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, let's go through it. 132 00:27:25.560 --> 00:27:48.689 Amy Pinkerton, BSPH CTL Teaching Toolkit: It says, the teaching team never responded to me on time, so that's actionable. I can talk to my TAs and say, hey, did something happen? Was there an issue? Was there some kind of delay that had happened? And also knowing that we had had other comments that said that the teaching team was great, so I'd want to know, like, was there a specific issue that we would want to proactively avoid in the future? 133 00:27:48.690 --> 00:27:52.130 Amy Pinkerton, BSPH CTL Teaching Toolkit: It would require some conversation with my teaching team. 134 00:27:52.670 --> 00:28:06.979 Amy Pinkerton, BSPH CTL Teaching Toolkit: The next part of the comment, number 4, says, also, way too much work for a two-credit course. So this one, I think this could go either way, actionable or unactionable. If I know that my workload is… 135 00:28:06.980 --> 00:28:19.009 Amy Pinkerton, BSPH CTL Teaching Toolkit: appropriate for the course credit. So if I know that the workload is calculated for a two-credit course, then I could discard that, knowing, like, oh, no, the student, you know, it's not… 136 00:28:19.010 --> 00:28:22.090 Amy Pinkerton, BSPH CTL Teaching Toolkit: I know that that's incorrect. 137 00:28:22.630 --> 00:28:27.920 Amy Pinkerton, BSPH CTL Teaching Toolkit: Or, if they are an incongruent, then I have two routes. 138 00:28:27.920 --> 00:28:52.850 Amy Pinkerton, BSPH CTL Teaching Toolkit: I could either, take this and say, hey, my credit… my course is listed as a two-credit course, but actually, it's a three-credit course, and can we reassign it and get it bumped up to a three credit? Or I could look at that and say, oh, like, oh, my course has way too much in it, I need to pare it down and do a course revision, to make it fit within that 2 credit. So there's two routes that you can take if it doesn't, in fact, align with the credit 139 00:28:52.850 --> 00:28:58.629 Amy Pinkerton, BSPH CTL Teaching Toolkit: So that could go either way. And then the third part of this number 4 comment says. 140 00:28:58.850 --> 00:29:12.079 Amy Pinkerton, BSPH CTL Teaching Toolkit: My group also did nothing, yet they got the same grade as me, so that's something to do with the, group assignment, and that's something that I, as the faculty, can look at how the group assignment is managed and make some changes, so that part's actionable. 141 00:29:13.070 --> 00:29:17.499 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then number 5, and this is the one that, Celine had brought up, earlier. 142 00:29:17.500 --> 00:29:41.000 Amy Pinkerton, BSPH CTL Teaching Toolkit: Where, it says, the course had similar learning objectives to another required course in my program. I'm not sure if both should be required, and Celine was correct. There's an action here, the action is to share it, but it's listed as unactionable because it's not something that we as the faculty control within the scope of the course. So you are correct that there's an action to be taken, which is share that with your program. 143 00:29:41.000 --> 00:29:51.960 Amy Pinkerton, BSPH CTL Teaching Toolkit: But for the labeling system, I labeled it as unactionable, because it's not specifically actionable within the context of the course. So I hope that's clear. 144 00:29:52.780 --> 00:30:11.239 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so that was… we just practiced Step 1, now we're moving on to step two, and so for step two, we're only taking the comments that we pulled that we categorized as actionable, and now we're going to further organize them by thematic category. 145 00:30:11.250 --> 00:30:25.659 Amy Pinkerton, BSPH CTL Teaching Toolkit: So from here, you can… remember, you don't have to look at your discarded comments, you've already passed along the things that needed shared with other people, and now we're focusing on actionable comments to look for themes. 146 00:30:25.780 --> 00:30:50.530 Amy Pinkerton, BSPH CTL Teaching Toolkit: So these categories that we have listed, like syllabus, course organization, content, assessments, feedback, resources, other… these are categories that we at CTL see a lot, but if you're creating categories for your course, it would depend on what themes you are picking up within your student comments. So your themes might be different from these… your categories might be different from these categories. 147 00:30:50.530 --> 00:30:53.179 Amy Pinkerton, BSPH CTL Teaching Toolkit: Categories as you're going through your own course. 148 00:30:53.670 --> 00:31:06.160 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, using the sample comments we just reviewed, let's focus on the category assessments, because we had quite a number… quite a few comments that had to do with the instructions and the group assignment. 149 00:31:07.880 --> 00:31:15.419 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, looking at the category assessments, we want to synthesize and, 150 00:31:15.660 --> 00:31:26.079 Amy Pinkerton, BSPH CTL Teaching Toolkit: Synthesize those comments into bullet points, that can quickly communicate what's positive and what's constructive criticism about this category. 151 00:31:26.210 --> 00:31:43.549 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, our positive feedback, said that the assessments were practical, applicable to real life. The constructive criticisms had to do with unclear instructions, unclear submission guidelines, and then also some perceived unfairness, especially around the group assignment. 152 00:31:43.860 --> 00:31:49.019 Amy Pinkerton, BSPH CTL Teaching Toolkit: And again, as you work through your own student course evaluations, you'll… you'll… 153 00:31:49.260 --> 00:31:56.380 Amy Pinkerton, BSPH CTL Teaching Toolkit: Go through the same process for each of your categories, according to your comments that you receive. 154 00:31:56.600 --> 00:32:08.059 Amy Pinkerton, BSPH CTL Teaching Toolkit: As you go through, you might end up having to re-read comments, and you might find that some comments might fit across categories, so you might have the same comment in two different spots, and that's okay. 155 00:32:08.350 --> 00:32:18.840 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, you might find that this, this step usually takes a while, because, especially if you have a large course with a lot of comments, going through and synthesizing, those themes can take a while. 156 00:32:19.220 --> 00:32:42.220 Amy Pinkerton, BSPH CTL Teaching Toolkit: You also might find that you might discard more comments at this stage, because you might have had a comment that was actionable, but if only one student brought it up, it might be an outlier and not, reflect the broader student experience. So, as you're looking for themes, you might say, oh, maybe one student had a bad experience, maybe that 157 00:32:42.220 --> 00:32:50.810 Amy Pinkerton, BSPH CTL Teaching Toolkit: the one that had the miscommunication with the TAs, but that one was an outlier, so you can discard it as an outlier. 158 00:32:50.810 --> 00:32:52.060 Amy Pinkerton, BSPH CTL Teaching Toolkit: As an example. 159 00:32:54.000 --> 00:33:11.830 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so, once you've identified those criticisms… the, the critiques of your categories, you're then going to identify actionable steps based on, how you would want to respond to the suggestions. 160 00:33:12.000 --> 00:33:23.650 Amy Pinkerton, BSPH CTL Teaching Toolkit: So here I have pulled out those three, I'm gonna switch back, these three constructive criticism critiques, the unclear instructions, the unclear submission, and perceived unfairness. 161 00:33:23.650 --> 00:33:32.009 Amy Pinkerton, BSPH CTL Teaching Toolkit: And that is what informed these three, rows. The unclear instructions, unclear submission, and then perceived unfairness. 162 00:33:32.490 --> 00:33:33.410 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright. 163 00:33:34.670 --> 00:33:53.749 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, considering this, my actionable steps for unclear instructions and unclear submission guidelines is to review the instructions for clarity, and then also I added, ask another person to review. And for this course, the other person could be my TA, or if I have a co-instructor or colleague. 164 00:33:53.750 --> 00:34:13.689 Amy Pinkerton, BSPH CTL Teaching Toolkit: But I would also… I could ask a CTL instructional designer, an ID. We at CTL love to review, instructions and course sites, so if you need feedback, we're happy to review your, your instructions. We love doing that, so, that's another person that I could ask is, 165 00:34:13.690 --> 00:34:15.190 Amy Pinkerton, BSPH CTL Teaching Toolkit: I could ask for review. 166 00:34:15.860 --> 00:34:30.559 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then for the perceived unfairness between group members, I have a couple of options available. I could create some group accountability contracts, I could consider incorporating some peer evaluations where peer… where group members evaluate each other. 167 00:34:30.560 --> 00:34:36.790 Amy Pinkerton, BSPH CTL Teaching Toolkit: And I can create a communication route specifically to deal with group 168 00:34:36.790 --> 00:34:40.890 Amy Pinkerton, BSPH CTL Teaching Toolkit: issues, like in-group member issues. 169 00:34:41.120 --> 00:34:50.259 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, considering these possible actions, what benefits, drawbacks do you see in incorporating any of these actions? 170 00:34:50.530 --> 00:34:58.799 Amy Pinkerton, BSPH CTL Teaching Toolkit: You can either let me know in the chat if you can think of any benefits or drawbacks, or, you can raise your hand in Zoom. 171 00:35:13.960 --> 00:35:16.769 Amy Pinkerton, BSPH CTL Teaching Toolkit: Oh, time! Celine said time. 172 00:35:17.280 --> 00:35:19.229 Amy Pinkerton, BSPH CTL Teaching Toolkit: Is that you save time? 173 00:35:19.620 --> 00:35:21.299 Amy Pinkerton, BSPH CTL Teaching Toolkit: Feedback on changes… 174 00:35:23.460 --> 00:35:24.400 Celine Greene: Well, the… 175 00:35:24.400 --> 00:35:24.939 Amy Pinkerton, BSPH CTL Teaching Toolkit: Or is those… 176 00:35:24.940 --> 00:35:25.600 Celine Greene: Drawback. 177 00:35:25.620 --> 00:35:28.739 Amy Pinkerton, BSPH CTL Teaching Toolkit: Drawbacks, drawbacks, oh, these take time. Okay. 178 00:35:29.010 --> 00:35:35.250 Amy Pinkerton, BSPH CTL Teaching Toolkit: I was thinking… They take time, but they could also save time. But I'll get into that. 179 00:35:35.900 --> 00:35:41.830 Celine Greene: But then also, it's like, how do you get the feedback? So you thought you had it right the first time, so how do you get the feedback? 180 00:35:42.180 --> 00:35:44.879 Celine Greene: That's a drawback, also, I'm sorry. 181 00:35:44.880 --> 00:35:47.119 Amy Pinkerton, BSPH CTL Teaching Toolkit: You're right, no, you're right! 182 00:35:51.650 --> 00:36:00.499 Amy Pinkerton, BSPH CTL Teaching Toolkit: Yeah, those are all good… all good things. All right, so now I'll share, what I highlighted as benefits and drawbacks. 183 00:36:00.500 --> 00:36:13.479 Amy Pinkerton, BSPH CTL Teaching Toolkit: So for the unclear instructions, I said I focus on the benefits for the students, which would be increased clarity, but, not on the slide. I thought of this as we were chatting. 184 00:36:14.510 --> 00:36:24.250 Amy Pinkerton, BSPH CTL Teaching Toolkit: There's also a benefit for you as the faculty. If you have clear instructions or clear submission guidelines, that means you're spending less time answering questions 185 00:36:24.250 --> 00:36:41.990 Amy Pinkerton, BSPH CTL Teaching Toolkit: or clarifying for students, and maybe, like, emails or discussion forums, about the assignments. So, even though it might take some upfront time to review the instructions and edit them, it might save you time down the line, because you're not answering questions, to clarify later. 186 00:36:42.750 --> 00:36:56.299 Amy Pinkerton, BSPH CTL Teaching Toolkit: For drawbacks, I said not applicable, but Celina's correct, it does take some time to actually do this, especially if you're asking another person to review with you. So there is a time… there is a time, 187 00:36:56.520 --> 00:36:59.789 Amy Pinkerton, BSPH CTL Teaching Toolkit: That is cost, in this 188 00:37:00.040 --> 00:37:18.979 Amy Pinkerton, BSPH CTL Teaching Toolkit: Oh, and I just saw another benefit. Students get to pay more attention to what they're learning and don't get caught up in the distraction of the activities. Yes, that's huge. That's a huge benefit. We want students to spend time actually learning and doing, not figuring out. So, that's a good benefit to highlight. 189 00:37:18.980 --> 00:37:34.669 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then for level of effort, we just had this scale, level of effort, 1 being the lowest level of effort, and 4 being the highest. I put, the first two as 1s, only because, I had a really clear idea of who I would ask. 190 00:37:34.670 --> 00:37:47.129 Amy Pinkerton, BSPH CTL Teaching Toolkit: To review, but if you're… if it… if you have to, like, track someone down to help review your instructions, or if you're not sure where to start, then you might bump that up to a 2, depending on your situation. 191 00:37:47.980 --> 00:38:05.829 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then for number 3, the perceived unfairness between group members, this one takes a little bit more. So the benefits are increased accountability for group members, especially if you have the group members evaluating each other, and improved clarity around the group, the group 192 00:38:06.040 --> 00:38:30.859 Amy Pinkerton, BSPH CTL Teaching Toolkit: assignment, and how it works, and also communication. But some drawbacks are the administrative time that it takes to actually implement, those changes. That's going to take a lot of time, especially time during the term when the group project is running. So that's, so that requires more effort, and because of that, I labeled it as a 3 in our level of effort scale, so not quite a major over 193 00:38:30.860 --> 00:38:34.930 Amy Pinkerton, BSPH CTL Teaching Toolkit: Overhaul, but it does take some significant time and effort in the administration. 194 00:38:35.660 --> 00:38:45.590 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so at this point, if you were doing your own evaluations, you'd create actionable steps, for each of your, each of your 195 00:38:45.800 --> 00:38:55.670 Amy Pinkerton, BSPH CTL Teaching Toolkit: Pain points, and then, at this stage, you might find that you can identify the pain point, but you're not sure what the actionable steps are, or the solutions. 196 00:38:55.670 --> 00:39:14.920 Amy Pinkerton, BSPH CTL Teaching Toolkit: And, that's why I want to remind you that, the CTL instructional designers are here to help you through this whole process. So if you find, like, I can identify the problem, I'm not sure what the solution would be, that's a great time to bring in your instructional designer, or ask for an ID consult from CTL. 197 00:39:15.110 --> 00:39:24.089 Amy Pinkerton, BSPH CTL Teaching Toolkit: Another thing is, as you're going through your actions and considering your level of effort, you might realize you have a couple of ones. 198 00:39:24.140 --> 00:39:31.889 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then, like, a couple… three… twos and threes, and then maybe 1 or 2 4s in your level of effort. 199 00:39:31.940 --> 00:39:43.810 Amy Pinkerton, BSPH CTL Teaching Toolkit: You don't have to do all of your changes all at once by your next course offering. You can focus on your 1s, think through, and maybe another term passes before you work on your twos and threes. 200 00:39:43.810 --> 00:39:54.059 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then maybe a whole other term passes before you address the fours. And that's okay. You can make small, incremental changes over time, and that's also a great time to, 201 00:39:54.060 --> 00:40:01.700 Amy Pinkerton, BSPH CTL Teaching Toolkit: get more feedback from students as you're pivoting and adjusting, your class. So, that's okay if you don't make every change 202 00:40:01.840 --> 00:40:03.090 Amy Pinkerton, BSPH CTL Teaching Toolkit: Every time. 203 00:40:05.060 --> 00:40:22.739 Amy Pinkerton, BSPH CTL Teaching Toolkit: All right, so, so far, we've identified how reframing the revision process as refinement can help alleviate stress. We overviewed the course revision process and identified sources of data, and then we actually spent some time practicing analyzing student course evaluation together. 204 00:40:22.740 --> 00:40:28.200 Amy Pinkerton, BSPH CTL Teaching Toolkit: Are there any questions before we actually dive into your student course evaluation data? 205 00:40:32.860 --> 00:40:42.610 Amy Pinkerton, BSPH CTL Teaching Toolkit: All right, I don't see any hands raised. If you do have questions, you can post those in the chat. Oh, and Lauren, I just realized that was you, so I'm gonna pass to Lauren. 206 00:40:42.610 --> 00:40:52.249 Lauren Dana: So, as Amy mentioned, now is the time where you actually get to start with your work with your own evaluation and apply some of the skills that Amy and I walked through with you. 207 00:40:52.400 --> 00:40:59.920 Lauren Dana: So, the first thing that we're gonna have you do, is download your student course evaluation results, if you haven't already. 208 00:40:59.980 --> 00:41:18.079 Lauren Dana: So your other option is you can… alternatively, you can use the worksheet that we have in the chat. We did create a sample course evaluation that you can use if you want to practice, but you're not quite ready to… to do your own course, or maybe you haven't taught a course yet and are just preparing. So that's going to be our step one. 209 00:41:18.780 --> 00:41:20.480 Lauren Dana: And then step 2, 210 00:41:20.700 --> 00:41:34.119 Lauren Dana: you're gonna open that workshop worksheet that we had, so hopefully it's already open and you've been using it, but if not, you want to go back in the chat, I think it's in there, I posted it a little bit earlier, and you can open up that again, so that's gonna be where you take some of your notes. 211 00:41:35.210 --> 00:41:42.519 Lauren Dana: And then step 3, we are basically going to walk you through each step of the qualitative data analysis process. 212 00:41:42.680 --> 00:41:48.859 Lauren Dana: So we will walk you through. It looks like our slides are… are, like, there's, like, little… 213 00:41:49.080 --> 00:42:02.260 Lauren Dana: Oh, weird, there's, like, little squares. That's okay, they're… it's too hot, they're tired. So, before you get started, I do just want to give you some realistic expectations. There's no way that you're going to be able to get through all of these steps in the next. 214 00:42:02.260 --> 00:42:12.290 Lauren Dana: 16 minutes, but Amy and I did want to have you get started, so that way, if you have any questions for us, you can ask, or it just is going to get you the first step 215 00:42:12.350 --> 00:42:16.850 Lauren Dana: In this process, and you can return to this when you have a little bit more time. 216 00:42:17.790 --> 00:42:19.589 Lauren Dana: Okay, so we're gonna head to our first… 217 00:42:19.700 --> 00:42:26.420 Lauren Dana: Step 1. So what we're gonna do is we're going to ask you to collate and organize your responses. 218 00:42:26.600 --> 00:42:35.989 Lauren Dana: So, go through your own evaluation and mark it up however it helps you. So, due to the sake of time, I'd pick one of those open-ended questions to work with. 219 00:42:36.010 --> 00:42:50.130 Lauren Dana: And then decide what you're gonna keep, what you're gonna discard, what you're gonna enjoy. And I just want to give you, before you get started, a couple pro tips that have always helped me. So you could highlight each type of comment in a specific color. 220 00:42:50.350 --> 00:43:06.749 Lauren Dana: you could strike through the comments that you plan to discard, that way you're not even looking at them. And sometimes it can be helpful to actually copy and paste the comments onto a labeled Word document. So you might want to not be working on that evaluation, but copy and paste the comments you plan to keep. 221 00:43:07.080 --> 00:43:10.599 Lauren Dana: Okay, so we're gonna give you, I believe it's 2 minutes to work on this step. 222 00:45:15.660 --> 00:45:22.919 Lauren Dana: Okay, great. So hopefully you got started a little bit with our Step 1. So we are gonna head to Step 2. 223 00:45:23.690 --> 00:45:35.350 Lauren Dana: So, for step two, as Amy had mentioned when she did her sample, you're gonna just completely ignore that discard pile, you're not gonna use those comments, and we're gonna try to synthesize your comments into themes. 224 00:45:35.410 --> 00:45:52.289 Lauren Dana: As Amy again mentioned, maybe these themes don't work for you. Feel free to pick your own categories, but for the sake of time, again, I would just recommend picking one. You could pick assessments, but that's probably going to be a big one. Maybe go something a little bit smaller, like course organization or syllabus. 225 00:45:52.290 --> 00:46:01.210 Lauren Dana: And see if you can pull out some positive feedback and constructive criticism that encompass all of the comments, so just a couple points. 226 00:46:01.730 --> 00:46:07.500 Lauren Dana: Okay, so we're gonna give you again another 2 minutes to see if you can synthesize some of these comments into themes. 227 00:48:10.210 --> 00:48:11.190 Audio shared by Amy Pinkerton, BSPH CTL Teaching Toolkit: Great, thank you. 228 00:48:11.660 --> 00:48:15.770 Lauren Dana: Thank you, Amy. Alright, so we're gonna head now into our final step. 229 00:48:15.800 --> 00:48:28.910 Lauren Dana: So this again is the hardest part. So you may, as Amy said, know the pain point, but not what to do next. But we're gonna ask you just to take 2 minutes to think about any next steps that you could do to address the comments. 230 00:48:28.910 --> 00:48:35.810 Lauren Dana: something to think about, like, are there changes that need to be made? And again, you're going to consider the benefits and the drawbacks, like we did before. 231 00:48:36.710 --> 00:48:38.459 Lauren Dana: So we'll take 2 minutes here. 232 00:50:45.450 --> 00:50:59.949 Lauren Dana: Great, so we've now gone through all of our three steps. Hopefully that's given you a little bit of taste of what this would look like and feel like. And Amy and I did just quickly want to mention, usually we're asked about generative AI and what role that can play in the evaluation process. 233 00:51:00.000 --> 00:51:19.029 Lauren Dana: And Amy and I encourage you, first of all, you want to make sure you're not submitting any identifying information to generative AI, so any course name, student name, anything like that can ever be submitted. But we also encourage that GenAI would be more like a colleague. It can't replace the knowledge of your course, the tone of the evaluations, 234 00:51:19.030 --> 00:51:37.619 Lauren Dana: And this revision process is really valuable for you as a faculty and for your students, so we do recommend that you don't outsource this, to generative AI. It can be a helpful support or help you categorize, but again, we definitely encourage you to go through this process with you and your teaching team. 235 00:51:37.700 --> 00:51:48.399 Lauren Dana: We also encourage you to meet with an instructional designer, so as Amy has mentioned a couple times, you are not alone. We are happy to help you go through quantitative and qualitative data. 236 00:51:48.400 --> 00:52:02.119 Lauren Dana: So if you already have an instructional designer, we encourage you to reach out to them, they're happy to walk you through this, and if you do not, on that initial resource sheet that we shared, there'll be a link to schedule a consultation, so we encourage you to do that. 237 00:52:02.240 --> 00:52:13.100 Lauren Dana: And before we head to our final Q&A, it would be remiss if Amy and I didn't get your feedback, so we are going to have you quickly fill out this evaluation. 238 00:52:15.880 --> 00:52:26.469 Lauren Dana: And I placed it in the chat. So your feedback is important, and it is also very valuable to us, and it benefits our instructional efforts. So we really appreciate you taking the time to do this. 239 00:52:28.540 --> 00:52:30.119 Lauren Dana: Okay, and I'm going to pass it back to Amy. 240 00:52:31.730 --> 00:52:35.620 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and I'm actually… I'm gonna stop the recording. 241 00:52:35.890 --> 00:52:39.269 Amy Pinkerton, BSPH CTL Teaching Toolkit: So that we can… have open Q&A.