WEBVTT 1 00:00:02.750 --> 00:00:17.099 BSPH CTL Teaching Toolkit: Alright. Hello, everyone, and welcome to today's teaching toolkit workshop on analyzing student course evaluations. My name is Amy Pickerton, and I'm a senior instructional designer at the center for teaching and learning, and I'm joined by my colleague. 2 00:00:17.460 --> 00:00:23.429 Lauren Dana: Hi! I'm Lauren, Dana. I am an instructional designer, also at the center for teaching and learning, and I work with Amy. 3 00:00:24.220 --> 00:00:44.649 BSPH CTL Teaching Toolkit: And today's session is being recorded, and the recording, along with all of the resources that we're sharing today will be sent as a follow up email to all of the Zoom registrants. Also we'll be monitoring the chat. So if you have any questions, there'll be chances for Q&A throughout today's session. But you can also post questions in the chat. 4 00:00:45.220 --> 00:00:47.940 BSPH CTL Teaching Toolkit: Alright. So we can go ahead and begin. 5 00:00:48.620 --> 00:00:51.290 BSPH CTL Teaching Toolkit: So in today's workshop, we're going to first.st 6 00:00:51.290 --> 00:00:54.610 Lauren Dana: Just give me a minute. I'll pause you just a moment. The screen didn't change. So yeah. 7 00:00:54.610 --> 00:00:58.179 BSPH CTL Teaching Toolkit: Oh, okay, let me see, that's so strange. 8 00:01:02.240 --> 00:01:05.530 BSPH CTL Teaching Toolkit: Are you still seeing the slides or no? 9 00:01:05.530 --> 00:01:08.259 Lauren Dana: Now we see the slides, but not in presenter view. 10 00:01:08.260 --> 00:01:15.540 BSPH CTL Teaching Toolkit: Oh, okay, actually, let me stop sharing and just restart my share. 11 00:01:16.250 --> 00:01:20.870 BSPH CTL Teaching Toolkit: Sorry about this. We're having some technical difficulties with zoom share today. 12 00:01:21.380 --> 00:01:22.125 Lauren Dana: Temperamental. 13 00:01:22.870 --> 00:01:23.550 BSPH CTL Teaching Toolkit: Yes. 14 00:01:23.550 --> 00:01:24.840 Lauren Dana: It's cold. 15 00:01:28.380 --> 00:01:39.000 BSPH CTL Teaching Toolkit: Oh, goodness, I lost zoom for a second. There we go share, and then 16 00:01:42.960 --> 00:01:49.070 BSPH CTL Teaching Toolkit: okay, so you should be able to see the slides, but not my presenter notes. Is that correct? 17 00:01:50.360 --> 00:01:54.579 Lauren Dana: It looks just like a gray. I'm just seeing a gray box. I don't know if anyone else has that. 18 00:01:56.050 --> 00:02:00.280 BSPH CTL Teaching Toolkit: Let me move this out of the way. So sorry. 19 00:02:02.500 --> 00:02:07.469 Lauren Dana: Oh, yes, okay, now. So the gray box is still there, but I can see parts of the slide. 20 00:02:09.050 --> 00:02:16.360 Celine Greene: Okay, we're seeing your your zoom boxes are hiding part of it. 21 00:02:16.360 --> 00:02:24.846 BSPH CTL Teaching Toolkit: Yes, that's so weird. It's never happened like that before. Let me open up the slides in a different on a different device. 22 00:02:26.050 --> 00:02:31.530 BSPH CTL Teaching Toolkit: and just share my screen from a different device. I think that will be better. 23 00:02:31.530 --> 00:02:37.579 Mia Lamm: Those boxes sometimes if you're playing it from not downloaded to your desktop. 24 00:02:38.300 --> 00:02:40.659 Mia Lamm: if you're playing it like a Yeah. 25 00:02:41.230 --> 00:02:47.909 BSPH CTL Teaching Toolkit: I have it in the desktop app, so I'm sharing it like I always share. But it's really strange that it's blocking it 26 00:02:48.040 --> 00:02:49.005 BSPH CTL Teaching Toolkit: weird. 27 00:02:49.970 --> 00:02:52.860 Lauren Dana: Naomi. I can work on getting that one pulled up as well. 28 00:02:52.860 --> 00:02:54.830 BSPH CTL Teaching Toolkit: Okay, thank you. 29 00:02:59.390 --> 00:03:21.069 BSPH CTL Teaching Toolkit: While we're figuring that out, I'll go ahead and keep going over the agenda while we figure out the visuals. Just so we don't lose too much time on the technical difficulties. So today we are going to go over feedback and course revision. And then types, of course, data that are available to you today's session is going to focus on qualitative data. But we will also 30 00:03:21.480 --> 00:03:45.790 BSPH CTL Teaching Toolkit: go over some quantitative data that's available in courseplus, and then we will end the session with a hands-on activity, with some recommended next steps and part of that hands-on activity. We ask that if you have a course that you're working on. You bring your course evaluation data. But if you don't have your course evaluation data on hand, we also have some mock data that you can use for the hands-on activity. 31 00:03:47.280 --> 00:03:48.330 BSPH CTL Teaching Toolkit: Okay? 32 00:03:50.650 --> 00:04:10.529 BSPH CTL Teaching Toolkit: And then our course, our objectives for today is overview. The course revision process identify sources. Of course, data to consider in the revision process identify strategies that can help alleviate stress surrounding course evaluation and finally analyze student course evaluation data to identify some specific course. Revisions. 33 00:04:11.410 --> 00:04:17.360 BSPH CTL Teaching Toolkit: Then I am still pulling up. Sorry. Sorry for the delay again. I'm still pulling up my 34 00:04:19.860 --> 00:04:22.570 BSPH CTL Teaching Toolkit: My visuals 35 00:04:23.030 --> 00:04:32.630 BSPH CTL Teaching Toolkit: so far. Are there any questions about what we're going over today, or any questions about course data in general that you'd like us to cover. You can add that in the chat. 36 00:04:35.950 --> 00:04:38.540 Lauren Dana: I should almost be set up as well. 37 00:04:39.110 --> 00:04:42.600 BSPH CTL Teaching Toolkit: Okay. Do you want to try sharing your screen, Lauren, to see if. 38 00:04:45.780 --> 00:04:47.359 Lauren Dana: Yes, give me just a moment. 39 00:04:47.360 --> 00:04:48.690 BSPH CTL Teaching Toolkit: Okay, thank you. 40 00:04:49.400 --> 00:04:54.950 BSPH CTL Teaching Toolkit: And if not, I just pulled it up on my other, my secondary device. 41 00:05:06.970 --> 00:05:09.449 BSPH CTL Teaching Toolkit: and again apologize for the delay. 42 00:05:16.160 --> 00:05:22.630 BSPH CTL Teaching Toolkit: Okay, thank you. And then we can proceed to slide. 4. 43 00:05:24.460 --> 00:05:25.770 BSPH CTL Teaching Toolkit: Thank you, Lauren. 44 00:05:27.510 --> 00:05:37.319 BSPH CTL Teaching Toolkit: Alright. So when we think about course evaluation data, we should stop and think about the why. Why are we doing this? And 45 00:05:37.950 --> 00:05:51.169 BSPH CTL Teaching Toolkit: when we think about the Why course evaluations. The biggest thing on our mind is that course. Evaluations are an important factor in determining the perceived quality of teaching and learning in a course 46 00:05:51.930 --> 00:05:53.300 BSPH CTL Teaching Toolkit: next slide. 47 00:05:53.610 --> 00:06:17.429 BSPH CTL Teaching Toolkit: They allow faculty to critically reflect on your teaching and also help faculty and teaching assistants identify possible updates to the course and its delivery student. Evaluations also offer insight into faculty teaching effectiveness and inspire some professional development and your professional development efforts. So you can sort of hone in on what might need to be improved 48 00:06:17.430 --> 00:06:25.029 BSPH CTL Teaching Toolkit: beyond the course faculty student evaluations can also be be used by school administrators to make data informed decisions. 49 00:06:26.160 --> 00:06:47.910 BSPH CTL Teaching Toolkit: Sometimes revisions can be associated with negative thoughts or feelings, especially if you've put a lot of time and effort into your course. So a lot of times you might think I put a lot of thought or effort into this. Why do my students not like it? Or why has this worked in the past? And it's not working anymore. And when revision is 50 00:06:47.910 --> 00:07:01.540 BSPH CTL Teaching Toolkit: sort of associated with these negative types of these negative thoughts. It can sour the entire process for you. So I'd like to switch the script around a little bit about how we 51 00:07:01.540 --> 00:07:05.939 BSPH CTL Teaching Toolkit: view revision and engage with their revision process as 52 00:07:05.950 --> 00:07:18.079 BSPH CTL Teaching Toolkit: revision, as refinement. So think of revision as a good sign, because it's good. If you're looking at your course, you're reviewing it and you're updating it. That's a good sign. 53 00:07:18.110 --> 00:07:43.420 BSPH CTL Teaching Toolkit: That means that you're taking into account your experiences and the needs of you and your students. Courses have this natural cycle of development, implementation, evaluation, and revision. And that's a natural thing for a course to undergo. So think of it as refinement, like perfecting a recipe of something that is already good, but could be really delicious. 54 00:07:43.420 --> 00:07:49.089 BSPH CTL Teaching Toolkit: So now I'm going to pass things off to Lauren to tell us about course the course revision process in more detail. 55 00:07:49.950 --> 00:08:08.950 Lauren Dana: Thank you, Amy. So, after reflecting upon your most recent course, offering or offerings, many of you will probably want to make revisions. As Amy just discussed. The revisions may be minor major, or somewhere in between, but regardless, we hope that you will follow these steps as you begin to think about. Course. Revision. 56 00:08:09.070 --> 00:08:20.990 Lauren Dana: First, st as Amy discussed, you need to determine why the course needs revisions. Maybe you notice the material felt stale or you receive negative student feedback in your evaluations. 57 00:08:21.170 --> 00:08:28.639 Lauren Dana: Maybe your students aren't picking up concepts as quickly in the past. Or maybe you're more worried about academic integrity. 58 00:08:28.770 --> 00:08:38.489 Lauren Dana: even though you may add more reasons why you need to revise your course as you dig more deeply into course data, it's really important to approach the revision process with your own. Why. 59 00:08:38.799 --> 00:08:44.480 Lauren Dana: next you will gather any applicable and available data from your courses. 60 00:08:44.640 --> 00:08:55.420 Lauren Dana: We will review much of the quantitative and qualitative data available to you in this workshop. But I do want to emphasize. Our main focus will be on student evaluations as a data point. 61 00:08:56.030 --> 00:08:57.090 Lauren Dana: Next. 62 00:08:58.600 --> 00:09:10.269 Lauren Dana: you need to spend time analyzing and reflecting on what this data tells you, you will create an action plan for changes. You hope to implement in your course, based on your data analysis and reflection 63 00:09:10.470 --> 00:09:33.430 Lauren Dana: over the next few months. You will then implement your action plan for the next time. Your course or course is run. However, the course process doesn't stop here. As we discussed a few minutes ago, the course, reflection and revision process is cyclical, and you will create different action plans. After each iteration of your course. 64 00:09:33.890 --> 00:09:52.739 Lauren Dana: However, we do want to remind you that you are just a human with many competing priorities and the quality of the course, reflection and revision process may vary from year to year, you may only be able to make one or 2 small changes some years, while other years you can have time to revamp the entire course. 65 00:09:52.850 --> 00:10:01.870 Lauren Dana: We promise, however, that you will leave this workshop completing the bolded steps. 2, 3, 4 for your student evaluations from at least one course. 66 00:10:06.070 --> 00:10:19.130 Lauren Dana: Okay, so as promised, we're going to give a very brief overview of some of the types of available course data that may help you in the course revision process before we focus on student evaluations as a source of data. 67 00:10:21.900 --> 00:10:27.059 Lauren Dana: we are going to 1st discuss the quantitative analytics available to you within courseplus 68 00:10:28.960 --> 00:10:55.230 Lauren Dana: to access these quantitative analytics. You will 1st go to faculty tools on the right column, you will see a section labeled Administrative Tools, and the quantitative data is under student activity reports. You will see that courseplus offers data on lecture, content, access, live talk attendance. If your course is fully online, discussion forum access and site access. All of the data points are available to download in either a Pdf or Excel format. 69 00:10:55.380 --> 00:11:17.299 Lauren Dana: This data and the learning analytics from this data are incredibly important in course revisions, and to learn more on how to analyze this data, I recommend watching Amy and my presentation from last June that focused specifically on quantitative data learning, analytics and course revision. We promise to send you that link in a follow-up email after this workshop. 70 00:11:19.540 --> 00:11:24.730 Lauren Dana: We are now going to shift our data to focus, to qualitative data available in courseplus. 71 00:11:26.190 --> 00:11:37.370 Lauren Dana: The communications that you have with students are a qualitative data source. You can review emails that you send to students by using the email archive feature in the course plus course email tool. 72 00:11:37.490 --> 00:11:50.649 Lauren Dana: just as a reminder. This only tracks emails that you send to students through courseplus while selecting the save to email archive setting. We do recommend that you do this for this very reason, so that you can use this as a data source. 73 00:11:50.650 --> 00:12:08.610 Lauren Dana: You may also consider gathering information from any emails that you receive from students like frequently asked questions or reported issues. For example, if Lecture 3 has a lot of questions with it, you may want to consider posting an FAQ. Or Q&A on the lecture page. 74 00:12:08.610 --> 00:12:19.210 Lauren Dana: If students constantly are reporting an issue with an assignment like unclear instructions or an external link that doesn't work. Then you can make any necessary fixes for the next offering. 75 00:12:22.210 --> 00:12:28.779 Lauren Dana: Last, but not least, course, evaluations are a great source of quantitative and qualitative data 76 00:12:28.910 --> 00:12:51.510 Lauren Dana: course. Evaluations are vital to the learning process at Bsph, for both students and faculty. So each term students will provide anonymous feedback on their courses during a predetermined evaluation period there are standardized evaluation questions that students receive that consist of students rating the course on a numerical scale and open-ended questions. 77 00:12:51.850 --> 00:12:56.260 Lauren Dana: We will send the complete list of these questions in a follow-up email. 78 00:12:56.956 --> 00:13:14.029 Lauren Dana: I also did want to let you know that you do have the opportunity to add 3 open-ended questions of your own to your course evaluations, and we'll give you these instructions in our follow-up email as well. Once the evaluation period has closed, you will receive an alert to view the evaluation process. 79 00:13:14.310 --> 00:13:18.549 Lauren Dana: The evaluation reports by logging into the course evaluation system. 80 00:13:18.910 --> 00:13:26.880 Lauren Dana: Faculty can compare course evaluations from up to 5 courses, and the courses can be from any of the last 3 academic years. 81 00:13:27.100 --> 00:13:37.360 Lauren Dana: These evaluations should provide faculties with the students, perspectives as well as guide future course modifications and redesigns as necessary. 82 00:13:38.370 --> 00:13:44.010 Lauren Dana: Okay, I'm going to pass it over to Amy to talk about how we can reflect on these student evaluations. 83 00:13:48.240 --> 00:14:07.160 BSPH CTL Teaching Toolkit: Thank you, Lauren. I was muted. So this is also a really great time, as you're reviewing your course, and you're looking at your student evaluations to reflect on not only your students perspective, but also your own experience and your own perspectives around the course. 84 00:14:07.160 --> 00:14:33.659 BSPH CTL Teaching Toolkit: So like, I said, when you're identifying changes for your course. Think about you and your teaching team. If you're also, if you're a ta, or if you're working with tas, or maybe co-instructors, if you were leaving qualitative feedback about the course, what would you say? What would you say? Worked well, what would you say? Would need improving? So take a moment to reflect critically about your course, and this will help you contextualize and analyze 85 00:14:33.660 --> 00:14:52.400 BSPH CTL Teaching Toolkit: the available course data and identify what to do next. So, while reflection is often overlooked in the course evaluation process. This can really help you understand your teaching practice and elevate your understanding of how your course is working and why. 86 00:14:52.570 --> 00:15:05.029 BSPH CTL Teaching Toolkit: and also, perhaps, why changes might need to be made so again, this is a this is a great starting point, like forming a hypothesis that you're going to test against the data. So do take time to reflect about your course. 87 00:15:06.050 --> 00:15:21.510 BSPH CTL Teaching Toolkit: Alright. So we've gone over quite a bit so far. Are there any questions before we move on. And I did receive a a direct question in the chat, asking if the slides will be available after and yes, the slides will be shared with the recording. 88 00:15:22.810 --> 00:15:25.889 BSPH CTL Teaching Toolkit: so please let us know if you have any other questions. 89 00:15:27.247 --> 00:15:32.800 BSPH CTL Teaching Toolkit: I don't see any hands raised, and I don't see any of the more any movement in the chat. So 90 00:15:33.340 --> 00:15:35.059 BSPH CTL Teaching Toolkit: all right, we're gonna go. 91 00:15:35.180 --> 00:15:39.530 BSPH CTL Teaching Toolkit: We're going to keep going on. But if you do have questions. Please post them in the chat. 92 00:15:40.030 --> 00:16:06.469 BSPH CTL Teaching Toolkit: So now we're going to switch gears and think about a model for analyzing qualitative data. So like, we said, student evaluations are vital to the learning process for both students and faculty. But it can be difficult to analyze that qualitative, open-ended, subjective responses from students. So we're going to go through a step by step method that you can use to analyze this type of feedback. 93 00:16:08.580 --> 00:16:26.430 BSPH CTL Teaching Toolkit: So we shared a worksheet with you, and we're going to share a document with you in the chat. So please, when you see that file in the chat, please download your own copy to follow along. But again, we're going to go step by step through this process and step one is to categorize your student comments. 94 00:16:26.430 --> 00:16:53.070 BSPH CTL Teaching Toolkit: So comments that are either unrelated to teaching and learning, or are so nonspecific or vague that they're not helpful. You can just disregard those types of comments. You can just x those out or clear them out. So if you're going through your comments, I like to copy and paste them into a word document, and then just strike through the ones that I am discarding. So again, if they're unrelated to teaching and learning, or if they're nonspecific or vague, then you can discard those. 95 00:16:53.380 --> 00:17:20.279 BSPH CTL Teaching Toolkit: Now, if they're positive comments. You can enjoy those comments, because that tells you what students think is working well in your course. So enjoy those comments, and then maybe look for themes about what's working really? Well, because you don't want to change what's working? Well, you want to be able to identify what what students enjoy about your course. So look for areas of agreement and disagreement among your students when you compare your positive comments. 96 00:17:20.950 --> 00:17:34.090 BSPH CTL Teaching Toolkit: Next, you want to look at your comments as actionable or unactionable suggestions. So these are specific changes that your students are recommending to you. And if they're actionable suggestions. 97 00:17:34.090 --> 00:17:54.289 BSPH CTL Teaching Toolkit: these are comments that offer suggestions or shed light on pain points in your class that you, as your as a faculty, or perhaps a member of the teaching teams of your ta, you can actually do something about. So look for themes, and then consider the trade-offs associated with making each suggested change. So, for example, the level effort required to make that change 98 00:17:54.400 --> 00:18:01.059 BSPH CTL Teaching Toolkit: now, unactionable suggestions. These are comments that are suggestions or shed light on pain points. 99 00:18:01.060 --> 00:18:25.969 BSPH CTL Teaching Toolkit: but are beyond the scope of your course or beyond your control within your role of the course. So this might be something that you could consider passing to individuals who can make use of that information. So, for example, your academic or program coordinator, classroom facilities management or the center for teaching and learning. So, for example, if a student comes up with a really great course, plus feature or tool, you can always pass that on to us. 100 00:18:26.985 --> 00:18:27.590 BSPH CTL Teaching Toolkit: So 101 00:18:27.710 --> 00:18:38.330 BSPH CTL Teaching Toolkit: by using this method, the step, one categorization, you can quickly, systematically organize, and your your student responses in an objective way. 102 00:18:44.820 --> 00:19:05.750 Lauren Dana: Great. Thank you, Amy. We are going to start by looking at a sample courses, student evaluations. So let's take a look at student responses to the question, and this is on your worksheet as well. Please identify what you consider to be the strengths of the course. I will give you about 1 min to read your responses, either on this worksheet or the slide. 103 00:19:05.750 --> 00:19:18.839 Lauren Dana: and I want you to think about Amy's suggestions on how to categorize these responses. Which would you discard? Which would you use to consider as actionable suggestions? And which would you possibly refer to a higher up? 104 00:19:20.520 --> 00:19:22.510 Lauren Dana: So I'll give you about a minute. 105 00:19:32.320 --> 00:19:34.850 Lauren Dana: I'm not sure you can hear the audio right, Amy, can you hear the audio. 106 00:19:34.850 --> 00:19:37.310 BSPH CTL Teaching Toolkit: No, I think you'll have to reshare with your sound. 107 00:19:37.310 --> 00:19:38.560 Lauren Dana: I'll I'll pause. Yeah. 108 00:19:38.560 --> 00:19:39.170 BSPH CTL Teaching Toolkit: Okay. 109 00:20:15.070 --> 00:20:17.129 Lauren Dana: Okay. Slides are shared. Shown. Right, Amy. 110 00:20:17.580 --> 00:20:18.220 Lauren Dana: Okay. Great. 111 00:20:18.220 --> 00:20:18.740 BSPH CTL Teaching Toolkit: Yes. 112 00:20:20.560 --> 00:20:27.550 Lauren Dana: Okay. So we didn't get to listen to our beautiful music that we had to time the minute. So I'll give you about another 10 seconds. 113 00:20:47.980 --> 00:20:52.785 BSPH CTL Teaching Toolkit: They don't. I just heard the music play all right. So 114 00:20:53.690 --> 00:21:16.109 BSPH CTL Teaching Toolkit: So now I'd like to ask you if anyone would like to share, either raise your hand and share verbally, or share in the chat any comments that stood out to you as actionable, or something that you might pass on to another, another person higher up or in a different role. So would anyone like to share what they did with these comments. 115 00:21:23.100 --> 00:21:40.090 BSPH CTL Teaching Toolkit: And if not, that's okay, I will share. I don't see any hands raised, so I'll go ahead. Oh, I just saw in the chat everything about the course was awesome could be discarded. Yes, that's correct. So yeah, that one. It's a it's a nice comment, but it's not it's nonspecific. So it's nice, but 116 00:21:40.310 --> 00:21:47.129 BSPH CTL Teaching Toolkit: it won't help with your review. Carrie said. The last comment about being offered online would be sent higher up. Yes. 117 00:21:48.990 --> 00:21:49.770 BSPH CTL Teaching Toolkit: yep. 118 00:21:49.850 --> 00:22:19.249 BSPH CTL Teaching Toolkit: and yes, I agree with that. So when I look at the last comment. I love the course and wish it could be offered online as well. That's something that if I were a faculty member I would share that with perhaps my program or academic coordinator to talk about, maybe offering the course in multiple formats. And then another comment that stood out to me, was the 1st comment. The group assignments were practical and could be applied to real life. I do wish directions had been a little bit clearer, so that to me, if I see more than one 119 00:22:19.650 --> 00:22:31.800 BSPH CTL Teaching Toolkit: students saying that the directions are unclear, then that's something that I need to go back and read redo for clarity. So that's a potential suggestion. Yes, definitely actionable, I agree. Mia 120 00:22:32.540 --> 00:22:39.949 BSPH CTL Teaching Toolkit: alright so any other comments about these particular comments before we go on to our next batch. 121 00:22:41.230 --> 00:22:42.120 BSPH CTL Teaching Toolkit: Okay. 122 00:22:42.340 --> 00:22:43.950 Lauren Dana: We look good, Amy. Great. 123 00:22:46.900 --> 00:23:11.189 Lauren Dana: Okay. So now we're going to do the same activity. But with the sample student responses to what can sometimes be the dreaded. Please identify areas where you think the course could be improved. So again, we're going to give you another minute, this time with fantastic music to skim through the comments and mark the comments that you would discard, you would refer to a higher up, and the ones you would possibly use as actionable suggestions 124 00:23:11.240 --> 00:23:33.430 Lauren Dana: as a reminder. You may choose to discard parts of a comment and keep other parts. You also want to make sure you are looking for common themes. Just as a reminder. One comment from one student does not mean that you shouldn't necessarily act on this, whereas 10 comments from multiple students is something that you might want to consider. So let me get us started here. 125 00:24:35.370 --> 00:24:38.339 Lauren Dana: Okay, it sounds like our time is up. 126 00:24:39.910 --> 00:24:53.680 BSPH CTL Teaching Toolkit: Alright. So now, looking at these, what comments stood out to you, or what would you do with these comments? How would you categorize them. And again, you can share in the chat or raise your hand and share verbally. 127 00:25:07.358 --> 00:25:12.770 BSPH CTL Teaching Toolkit: Okay. So option to raise. I'm not sure if both should be required to program. Director. Yes. 128 00:25:12.960 --> 00:25:22.320 BSPH CTL Teaching Toolkit: yeah. So that's a good one, too. That would be non actionable for you, but would be something that you would pass on to someone. Who can use that information 129 00:25:24.420 --> 00:25:45.889 BSPH CTL Teaching Toolkit: all right, and then I'm not seeing anything else in the chat, so I'll go ahead and say the point out the ones that I thought that I highlighted. So again, the last comment about this course is just like another required course in my program. Not sure if both should be required. That is definitely something. I would pass on to my 130 00:25:45.890 --> 00:26:10.029 BSPH CTL Teaching Toolkit: my program coordinator or academic coordinator. And then the other comments that stood out to me, starting with the 1st comment. The assignment directions were really confusing, and I wasn't sure what the professor was looking for in the submitted assignment. So that's another indication that the instructions or the directions are not clear. So that's another point for go back and review instructions for clarity. 131 00:26:10.340 --> 00:26:40.029 BSPH CTL Teaching Toolkit: And then our next comments about the group assignments. There's a little bit, it says. I see the point of the group assignments, but my group really struggled. We weren't sure where to submit our assignment, and if we each needed to submit the assignment. So my group really struggled. That might be a little bit big, but it indicates to me that I might need to look into the group group work dynamic of the course. And then again, if the they weren't sure how to submit the assignment. That means that the directions were unclear. 132 00:26:40.750 --> 00:27:03.459 BSPH CTL Teaching Toolkit: And then, finally, the teaching team never responded to me on time. I also are also way. Too much work for a 2 credit course. So here I would perhaps meet with my teaching team to see you know what happened. Was there a delay. Was there like one delay, or was there like 5 delays like? What were the expectations for communication in the course and amongst the teaching team to see if there was an issue. 133 00:27:03.460 --> 00:27:25.879 BSPH CTL Teaching Toolkit: or if there was just like maybe a 1 off. And then for the 2 credit course, comment the center for teaching and learning instructional design team can actually help you calculate your course workload. So if you're concerned about workload and versus or workload within your course credit amount, we can help you determine if if the workload is appropriate for your course. Credit number. 134 00:27:28.470 --> 00:27:36.670 BSPH CTL Teaching Toolkit: so yes, so those are the the things that stood out to me, and then also I see more in the chat. So yes, I agree with what's going on in the chat. 135 00:27:37.380 --> 00:27:38.430 BSPH CTL Teaching Toolkit: All right. 136 00:27:39.390 --> 00:27:40.629 BSPH CTL Teaching Toolkit: So now 137 00:27:42.210 --> 00:27:50.149 BSPH CTL Teaching Toolkit: excuse me. So now, after that was our 1st sort of our 1st step was sort of filtering the the cat the the 138 00:27:50.180 --> 00:27:57.999 BSPH CTL Teaching Toolkit: comments by, if they're actionable or unactionable. And now we're going to take all of the comments that we can act on 139 00:27:58.000 --> 00:28:22.829 BSPH CTL Teaching Toolkit: and sort of organize them by thematic category. So here you're looking at all of those comments, and you're sort of putting them in these different category buckets. Now, these categories that we came up with syllabus course, organization, course, content, etc. These are categories that we frequently see, but your categories might look a little different. So if you see something in your course, data that doesn't fit in one of these buckets. You can make your own buckets 140 00:28:22.830 --> 00:28:27.650 BSPH CTL Teaching Toolkit: based on whatever themes come up for your course evaluation data. 141 00:28:28.260 --> 00:28:46.189 BSPH CTL Teaching Toolkit: But here, at this step. So you want to categorize by theme, and you want to put all of the positive feedback related to that theme and gather all of the constructive criticism or those actionable suggestions. And we're going to compare and contrast those to decide if it's actionable on our part. 142 00:28:46.860 --> 00:28:48.499 BSPH CTL Teaching Toolkit: So that is 143 00:28:49.350 --> 00:28:58.459 BSPH CTL Teaching Toolkit: so again, using those sample comments. We just reviews. Let's pick a category to work on together. We have a couple comments on 144 00:28:58.800 --> 00:29:05.699 BSPH CTL Teaching Toolkit: that had to do with the assignment instructions. So I think a good theme to go over would be using the assessments category. 145 00:29:06.300 --> 00:29:07.160 Lauren Dana: I agree. 146 00:29:07.160 --> 00:29:07.800 Lauren Dana: Amy, yeah. 147 00:29:07.800 --> 00:29:31.960 BSPH CTL Teaching Toolkit: Yes. So let's look at the assessments. So if we go to the next slide, thank you. So here we'll summarize with a few bullet points, some of the positive feedback given so for our assessments, some highlights were that they were practical, and that they were applicable to real life. So those are really good things that we don't want to change if we're reviewing and revising this assignment. 148 00:29:32.120 --> 00:29:48.080 BSPH CTL Teaching Toolkit: But some of the constructive criticism was that the instructions were unclear. The submission guidelines were unclear, and there was also some perceived unfairness, especially around the group assignment or the group work dynamics. And is this actionable on our part. Yes. 149 00:29:48.670 --> 00:30:13.260 BSPH CTL Teaching Toolkit: So as you're working with your own evaluations, you would want to follow the same process for your remaining categories. So you're going to. You may need to reread your comments multiple times before you feel that you're ready to synthesize and clearly summarize all of your comments like this, and this can take some time, especially if you have a large course with a lot of student comments to go through, but 150 00:30:13.350 --> 00:30:28.620 BSPH CTL Teaching Toolkit: this time will pay off later, as you're going through, because once we finish this step, we'll have a clear picture of what? How? The how, the students perceive the course, and then also what we might want to do as a list of revisions for the course. 151 00:30:28.630 --> 00:30:53.000 BSPH CTL Teaching Toolkit: So again, as you complete this step, you may find you discard more comments, and you also, things might emerge as outliers versus general student experiences. So that's another thing to look out for at this step is, maybe you have one comment, that is, that is criticism. But you might have 10 comments that were positive, so that can help you weigh whether or not this is a an action that you should take, or if this is just an outlier. 152 00:30:56.510 --> 00:31:04.000 BSPH CTL Teaching Toolkit: all right, and then our next step. So after you've completed, that is to consolidate actionable steps. 153 00:31:04.000 --> 00:31:27.119 BSPH CTL Teaching Toolkit: So if we go back to our category, we discussed in Step 2, which was assessment. Using our feedback from both positive and constructive criticism, we're going to look at some actionable items that we could change, based on the student experiences and then weigh the benefits and drawbacks against a level of effort scale, and our scale is one being the lowest level of effort, and 4 being a high level of effort. 154 00:31:27.580 --> 00:31:44.179 BSPH CTL Teaching Toolkit: So off the bat we want, we know that the instructions were unclear, and we would want to. Our actionable item is to review the instructions for clarity, and then ask another person to also review. So either a teaching assistant, an instructional designer, or perhaps a colleague. 155 00:31:44.710 --> 00:31:56.089 BSPH CTL Teaching Toolkit: and the benefit of this would be increased clarity, and there are no drawbacks to improving the clarity of your instructions, and the level of effort is fairly low for this. So I put it as a number one. 156 00:31:56.440 --> 00:32:19.279 BSPH CTL Teaching Toolkit: Similarly, unclear submission review the instructions for clarity. So this was the actual, like submission guidelines for the group assignment. And again I would review the instructions for clarity, and then ask another person to review so same benefits as before. You have increased clarity, not very many drawbacks, and it's a low effort action to take. 157 00:32:20.090 --> 00:32:44.169 BSPH CTL Teaching Toolkit: And then our final actionable item within this assessments category is perceived unfairness between group members. So some of the actionable items that I could take for this would be to create a group accountability contract for group members to sign. We might consider incorporating some pure evaluation and then maybe create some communication routes for issues specific to group work. 158 00:32:44.360 --> 00:33:03.869 BSPH CTL Teaching Toolkit: So for this the benefits is increased accountability for group work and increased clarity. But a drawback is, if we're going to incorporate all of these changes, then it's going to increase the administrative work for running the group work assignments. So I bumped the level of effort up for this particular change up to a 3. 159 00:33:07.240 --> 00:33:25.339 BSPH CTL Teaching Toolkit: So as you also, if you, if you're unsure, how you would work towards an actionable goal, you can always reach out to an instructional designer. So say you have some issues, but you're not sure what the actionable item is. You can always reach out to us to help for that. So now let's talk about the elephant in the room. So 160 00:33:25.770 --> 00:33:46.249 BSPH CTL Teaching Toolkit: when it comes to qualitative data, receiving feedback, especially subjective feedback, can be difficult, especially when you've put a lot of time and effort into your course. So I'm actually going to share a personal experience of feedback that I received that hurt a little bit, but ultimately helped how I facilitate the Ctl teaching assistantship training course 161 00:33:47.170 --> 00:34:14.810 BSPH CTL Teaching Toolkit: because feedback isn't always easy to receive, especially subjective feedback. So the Ta training course that Ctl offers has multiple facilitators, and I'm 1 of the facilitators, and it has run for many years. But back in 2020, which feels forever ago my co-instructor and I noticed from the evaluation data that students wanted more presence from the instructors and clear information about each of the modules. 162 00:34:14.909 --> 00:34:41.020 BSPH CTL Teaching Toolkit: So, based on this feedback, we decided to produce one to 2 min introductory videos for each module that sort of introduced the module and answered some frequently asked questions that were associated with that module. Then in 2023, me and the co-facilitators, we decided to update the course. And we looked at the feedback about these videos, and the feedback was clear that the students hated them. 163 00:34:41.130 --> 00:35:09.600 BSPH CTL Teaching Toolkit: So students wrote in the course evaluations that they thought the videos were pointless, that they wasted time. I was told that my voice sounded like Minnie Mouse. It was not. They were not good reviews for the videos, and that kind of hurt a little bit, because we were very excited about these videos, and we thought it would solve all of our evaluation problems. And they did not. So based on this new evidence, we ended up scrapping the videos. 164 00:35:09.600 --> 00:35:20.169 BSPH CTL Teaching Toolkit: And we ended up in. So the students, what they really wanted was increased instructor presence. So we we increased that presence through the discussion forum and some live sessions. 165 00:35:20.180 --> 00:35:42.959 BSPH CTL Teaching Toolkit: and for those frequently asked questions and clarity we just provided more information in the syllabus and on the course overview page, not in the form of a video, but in the form just like a bullet point list and some other ways. So we clarified the instructions, and we. We delivered what the students wanted in the 1st place, in a different format. 166 00:35:42.960 --> 00:35:50.759 BSPH CTL Teaching Toolkit: And that happened, we just re we just launched the new version of the Ta training course this past 167 00:35:50.760 --> 00:36:07.200 BSPH CTL Teaching Toolkit: July, and we're still collecting data. But so far in the interim, since then the data, the anecdotally it seems to be working much better than the videos that we originally did. And so the the lesson learned here is that even if you make a change 168 00:36:07.200 --> 00:36:30.790 BSPH CTL Teaching Toolkit: well, one, the 1st part is feedback can be hard to hard to receive, so make sure you're in a good head space to receive feedback and 2, even if you make a change, go back and look at how that change impacted the course, because sometimes if we make a change, sometimes they work and sometimes they don't work. And that's okay. As long as you keep going back to the data and seeing what the students are saying. 169 00:36:30.790 --> 00:36:37.699 BSPH CTL Teaching Toolkit: and just be flexible and give yourself and your students a little bit of grace in this process in this refinement process. 170 00:36:37.700 --> 00:36:43.269 BSPH CTL Teaching Toolkit: So again, allow yourself time to take in the feedback, positive and negative before just diving in. 171 00:36:43.270 --> 00:36:45.600 BSPH CTL Teaching Toolkit: and it's a cycle. 172 00:36:45.710 --> 00:36:52.189 BSPH CTL Teaching Toolkit: So all right. So and now I'm going to pass to Lauren to do our next round of Q&A. 173 00:36:53.060 --> 00:36:59.560 Lauren Dana: Great. Thank you for sharing that, Amy. I'm sure we all have had similar and equally painful experiences. 174 00:36:59.810 --> 00:37:04.119 Lauren Dana: So we're going to take a pause, just to see if there's any additional questions. 175 00:37:05.710 --> 00:37:09.019 Lauren Dana: Feel free to raise your hand, and I'll keep an eye on the chat. 176 00:37:11.680 --> 00:37:15.189 Lauren Dana: Looks like we're good. Do you see anyone, Amy, I think we're good. 177 00:37:15.460 --> 00:37:16.679 BSPH CTL Teaching Toolkit: Nothing on my end. 178 00:37:18.180 --> 00:37:42.700 Lauren Dana: Okay. So now you guessed it. We are going to have you start to complete these steps for one of your own courses. Evaluations. We know that you most likely actually not even most likely you definitely will not complete these steps by the end of the workshop. We only have 20 min. But our goal was to get you started so that we can provide you with any needed support so that you can continue this process outside of the workshop. 179 00:37:42.950 --> 00:37:45.740 Lauren Dana: All right. So we're going to now, start with step one. 180 00:37:47.820 --> 00:38:14.289 Lauren Dana: okay, we're going to ask you to download your student course. Evaluation results. We did mention in an email. If you have it, that's great. We can give you a moment to do that as well. You can alternatively use our mock data, which is on that worksheet I shared in the chat earlier and via the email yesterday. So I'm just going to give you a moment or 2 to either pull up your student evaluation results or just make sure you have our worksheet open. 181 00:38:32.400 --> 00:38:35.339 Lauren Dana: Okay? So hopefully, you are ready with that. 182 00:38:37.420 --> 00:38:48.370 Lauren Dana: Okay? So the next is, we're going to make sure. And this should hopefully, is already open the workshop worksheet that we were working with, because we have some some charts that you can use to complete the activity. 183 00:38:52.040 --> 00:39:17.810 Lauren Dana: Okay? And then the next step is when prompted, you're going to apply each step of the qualitative data analysis process. As I mentioned before, we're only giving a set amount of minutes for each of the steps. So we know that you're going to need more time on this later. We will also be here after this session. So we're going to try to save some time for Q&A. Or, if you need any, have any personal questions for us, we are happy to help. 184 00:39:17.810 --> 00:39:23.810 Lauren Dana: And again, if you do not have student data, you can use the data that we have on the on, the worksheet. 185 00:39:26.890 --> 00:39:51.299 BSPH CTL Teaching Toolkit: All right. So again, you're going to use your own data or the mock data in the worksheet to, we're going to start with. Step. One categorize, the student comments. So as you're going through your comments, think about what you can discard, what you can enjoy and compare and consider. And then what comments you might consider sharing with others outside of your role. So I think we have a 2 min timer starting now. 186 00:41:53.640 --> 00:42:02.169 BSPH CTL Teaching Toolkit: Alright, I think that is the end of our timer. So again, we don't expect you to go through all of your comments in this short amount of time, but this is a good starting point. 187 00:42:02.330 --> 00:42:22.469 BSPH CTL Teaching Toolkit: So now the comments that you were able to categorize in Step one, we'd like to you to start on Step 2, which is organized by thematic category. So if you're using your own data, you can always come up with your own categories again, these categories are what we frequently see, but your categories might look a little different, based on the comments that you 188 00:42:22.470 --> 00:42:36.269 BSPH CTL Teaching Toolkit: that you've received. But you're going to organize your your comments by category, and then put them in these 2 buckets, positive feedback, and then constructive criticism, and then decide if they're actionable or not. 189 00:42:37.750 --> 00:42:39.750 Lauren Dana: Sorry. I jumped the gun on the music. 190 00:42:40.267 --> 00:42:41.819 BSPH CTL Teaching Toolkit: No, we're good. 191 00:44:35.810 --> 00:44:38.680 BSPH CTL Teaching Toolkit: all right, and that is our 2 min timer. 192 00:44:38.760 --> 00:44:49.979 BSPH CTL Teaching Toolkit: So now, finally, for step 3. So by now you should pick a bucket. So, for example, in our examples, earlier, we picked the assessments bucket. 193 00:44:49.980 --> 00:45:13.050 BSPH CTL Teaching Toolkit: But now you're going to want to look at that specific bucket or that specific category and identify some actionable items and outweigh the benefits and drawbacks of making those changes. And again, you can use our scale, our level of effort scale, one being the lowest level of effort, and 4 being the highest level of effort to help you determine if this is a change that you would actually 194 00:45:13.050 --> 00:45:14.790 BSPH CTL Teaching Toolkit: want to make or can make. 195 00:45:14.890 --> 00:45:17.720 BSPH CTL Teaching Toolkit: So 2 min timers starting now. 196 00:47:19.700 --> 00:47:35.960 BSPH CTL Teaching Toolkit: all right, and that is our timer. So as you've gone through those steps again, we know that you probably didn't get through all of your comments. But again, it's a good starting point to start getting you to think about what actions you can take, and their benefits and drawbacks. Now I'm going to pass to Lauren. 197 00:47:36.190 --> 00:47:53.959 Lauren Dana: Thank you, Amy, and our final recommendation is to meet with a Ctl instructional designer who can accompany you through this, your courses, quantitative and qualitative data. So with that in mind, we did want to give some time for Q&A to see if there's anything else that you would like to discuss. 198 00:47:54.200 --> 00:47:56.130 Lauren Dana: or any questions that you have. 199 00:47:59.530 --> 00:48:11.350 BSPH CTL Teaching Toolkit: Yep. So open floor for questions. And then also, if you ran into anything in your qualitative data that you'd like to bring up, we can also answer questions there, like perhaps an actionable item. 200 00:48:20.800 --> 00:48:35.400 Lauren Dana: Great. I don't see any questions. If anything comes up later we will be sending you a follow up email later today. So please feel free to respond to us. We are happy to to speak with you further about this, and make this process as smooth and productive as possible. 201 00:48:37.830 --> 00:48:49.160 BSPH CTL Teaching Toolkit: All right. So we, speaking of feedback and evaluation, we would love to hear your thoughts about this workshop and how it went, so please do 202 00:48:49.340 --> 00:49:04.150 BSPH CTL Teaching Toolkit: take 2 min to fill out this workshop evaluation form. It's anonymous. And we really do look at this feedback when we plan this and future workshops. So we appreciate your time. So please complete this 2 min survey for us. 203 00:49:09.320 --> 00:49:11.600 BSPH CTL Teaching Toolkit: and then after this, we will. 204 00:49:13.120 --> 00:49:17.010 BSPH CTL Teaching Toolkit: We'll share some other resources as well, so don't log off quite yet. 205 00:49:18.420 --> 00:49:20.720 Lauren Dana: Amy, do you mind sharing the link in the chat? I'm just having. 206 00:49:20.720 --> 00:49:21.265 BSPH CTL Teaching Toolkit: Yes. 207 00:49:21.810 --> 00:49:22.930 Lauren Dana: Yeah, thank, you. 208 00:49:22.930 --> 00:49:23.840 BSPH CTL Teaching Toolkit: Of course. 209 00:49:33.180 --> 00:49:37.570 BSPH CTL Teaching Toolkit: all right. The link to the form, the workshop evaluation form is in the chat. 210 00:49:38.180 --> 00:50:02.969 Lauren Dana: Great. So while you're filling that out, we just want to share a couple additional resources. Ctl has a self-paced course for faculty who want to learn more about course management, redesign, and other pedagogical topics. I specifically want to point out that in this course there's a module called Reflecting on a course that provides more information about course, reflection, review, and revision. That I think, would be a really great accompaniment to today's workshop. 211 00:50:03.100 --> 00:50:22.400 Lauren Dana: If you are a ta or work with tas. Then we highly recommend Ctl's Ta training course, and then, finally, you can always stay up to date on the latest events and recommendations from Ctl. By checking out our events, page and subscribing to our blog. So, Amy, again, if you don't mind placing those links in the chat that would be great. 212 00:50:23.030 --> 00:50:24.850 Lauren Dana: and thank you so much. 213 00:50:26.150 --> 00:50:29.620 BSPH CTL Teaching Toolkit: Alright! Links are in the chat. 214 00:50:32.010 --> 00:50:40.280 BSPH CTL Teaching Toolkit: There you go. Oh, I should have hyperlinked those apologies you have to copy and paste them. But yeah, we really do recommend these resources for you. 215 00:50:40.950 --> 00:51:05.569 BSPH CTL Teaching Toolkit: And then we also have our references for this presentation. So again, we will share the slides with you along with all of the resources, the worksheet, and everything that we shared today, and thank you so much for your time, and we hope that you found this a helpful session, and we look forward to hearing about your course, revisions, and refinement. But thank you so much, and if you'd like to stay on. We have a couple minutes, we can. We can talk things. 216 00:51:05.950 --> 00:51:15.720 BSPH CTL Teaching Toolkit: course, review. Or if you have some specific comments that you'd like to look at, or actionable items you'd like some Id input in. We're happy to help. 217 00:51:16.040 --> 00:51:17.890 BSPH CTL Teaching Toolkit: but otherwise, thank you so much.