WEBVTT 1 00:00:02.840 --> 00:00:16.969 Amy Pinkerton, BSPH CTL Teaching Toolkit: Hello, and welcome to today's Teaching Toolkit workshop on analyzing student course evaluations. My name is Amy Pinkerton, and I'm a Senior Instructional Designer at the Center for Teaching and Learning, and I'm joined by my colleague. 2 00:00:17.440 --> 00:00:31.149 Lauren Dana: Hi, my name is Lauren Dana. I'm also an instructional designer at the Center for Teaching and Learning, and before Amy begins, I am going to place in a chat a worksheet with some links that we'll be discussing throughout, so you have it throughout the entire presentation. 3 00:00:31.760 --> 00:00:48.310 Amy Pinkerton, BSPH CTL Teaching Toolkit: All right, thank you, Lauren. So today, we will start by identifying strategies that can help alleviate stress around course evaluation, and then we'll overview the course revision process and identify sources of course data to consider during the revision process. 4 00:00:48.490 --> 00:00:57.289 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then we'll actually get a chance to practice analyzing some qualitative student course evaluation data to identify specific course revisions. 5 00:00:58.030 --> 00:01:11.459 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, why is it important? Why should we take the time to analyze student course evaluation data? Well, student course evaluations are an important factor in determining the perceived quality of teaching and learning of a course. 6 00:01:11.730 --> 00:01:13.960 Amy Pinkerton, BSPH CTL Teaching Toolkit: And it can be used in different ways. 7 00:01:14.450 --> 00:01:25.569 Amy Pinkerton, BSPH CTL Teaching Toolkit: As a faculty, student comments allow you to critically reflect on your teaching and learning, and help you identify possible updates to your course and its delivery and organization. 8 00:01:25.790 --> 00:01:33.730 Amy Pinkerton, BSPH CTL Teaching Toolkit: Student data can also offer insights into your teaching effectiveness and inspire some professional development efforts. 9 00:01:34.270 --> 00:01:50.069 Amy Pinkerton, BSPH CTL Teaching Toolkit: Beyond the course faculty, student course evaluation data can also be used by school administrators, academic departments, and programs to make data-informed decisions about your course, and also the greater, wider program that your course lives within. 10 00:01:50.070 --> 00:01:55.739 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, this course data is, is helpful for data-informed decisions. 11 00:01:56.650 --> 00:02:11.719 Amy Pinkerton, BSPH CTL Teaching Toolkit: But, there's an elephant in the room, and that is, is that receiving feedback, especially feedback from students that can be subjective, can be difficult, especially if you put a lot of time and effort into your course. Sometimes. 12 00:02:11.720 --> 00:02:28.090 Amy Pinkerton, BSPH CTL Teaching Toolkit: course review and revision can be associated with negative thoughts, like, I put a lot of work into this, why isn't it working? Or, why don't my students like this or think it's good enough? Or even, this has worked in the past, why is it not working anymore? 13 00:02:28.160 --> 00:02:35.310 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, when revision has these types of negative associations with it, it can sour that entire process for you. 14 00:02:35.450 --> 00:02:48.899 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, I'm gonna challenge you, if you fall into that negative pattern of thought, to rethink how you approach revision. And instead of thinking of revision as correction, I want you to think of it as refinement. 15 00:02:48.920 --> 00:03:06.569 Amy Pinkerton, BSPH CTL Teaching Toolkit: It's a good sign if you are going back and reviewing and revising your course. It means that your course is going through a healthy course life cycle of development, implementation, evaluation, revision, and then back to development. That's a good sign. 16 00:03:06.600 --> 00:03:15.789 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, so again, I want you to rethink of this, of revision from correction to refinement, like perfecting a recipe. 17 00:03:16.250 --> 00:03:22.610 Amy Pinkerton, BSPH CTL Teaching Toolkit: And that can help with some of those negative associations with… Course evaluation and revision. 18 00:03:22.660 --> 00:03:33.090 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, to really hone this in, I'm going to share a quick personal story of whenever I did some course revisions, review and revision, that, was tough, but worthwhile. 19 00:03:33.090 --> 00:03:43.109 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, I'm one of the co-facilitators for the teaching assistantship Training Course that CTL offers to students, to prepare them for teaching assistantship at BSPH. 20 00:03:43.110 --> 00:03:58.259 Amy Pinkerton, BSPH CTL Teaching Toolkit: And back in 2020, my co-facilitator and I noticed from the course evaluations that students wanted more instructor presence and clearer module info. There were some frequently asked questions and points that needed to be clarified for students. 21 00:03:58.500 --> 00:04:17.190 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, we created some short introductory videos for each module. These were 1-2 minute intro videos, that played at the beginning of every… of each module, and it, it showed, me on camera talking about each module, going over the goals. 22 00:04:17.190 --> 00:04:21.360 Amy Pinkerton, BSPH CTL Teaching Toolkit: And also clarifying those points of, confusion. 23 00:04:21.680 --> 00:04:27.730 Amy Pinkerton, BSPH CTL Teaching Toolkit: And, then we looked back at the data again after some time, so back in 2023, 24 00:04:27.730 --> 00:04:42.269 Amy Pinkerton, BSPH CTL Teaching Toolkit: And the feedback was clear that the students hated the videos. They were not a successful change to the course. They said that the videos were hard to… you had to go back and rewatch the videos multiple times for that important information. 25 00:04:43.200 --> 00:04:57.480 Amy Pinkerton, BSPH CTL Teaching Toolkit: the… I was told that the videos were pointless, and I sound like a cartoon character, so the students really hated the videos, and that's okay. So after taking a moment to, say, oh, okay, that didn't work. 26 00:04:57.480 --> 00:05:20.209 Amy Pinkerton, BSPH CTL Teaching Toolkit: We regrouped, and we reworked the course. So we went back to what did the students really want? They wanted clarity on frequently asked questions, and they wanted more instructor presence. So we took the information that we wanted to give in the videos, and we just put that as text on an overview page in really clear language, so that the students could always go back to that page and find 27 00:05:20.230 --> 00:05:23.280 Amy Pinkerton, BSPH CTL Teaching Toolkit: Really quickly what the information they were looking for. 28 00:05:23.360 --> 00:05:29.269 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then we put in instructor presence in other ways through the discussion forum, live talks, and 29 00:05:29.270 --> 00:05:47.280 Amy Pinkerton, BSPH CTL Teaching Toolkit: through the, the way that we were giving feedback to students on assessments. And now, fast forward, now it's 2025, that's worked much better. We have much better results from that. But that just goes to show that, feedback can be hard, and sometimes your changes don't always work the first time. 30 00:05:47.280 --> 00:05:51.290 Amy Pinkerton, BSPH CTL Teaching Toolkit: But, it's a process, and it's a process of refinement. 31 00:05:51.360 --> 00:05:57.170 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so enough about that, now I'm going to pass back to Lauren to talk about the course revision process. 32 00:05:57.170 --> 00:06:08.670 Lauren Dana: Thank you, Amy, for being so candid about that. I think we've all had similar experiences, so I appreciate you sharing. As Amy mentioned, I'm going to talk a little bit about course review and revision and the steps. 33 00:06:08.910 --> 00:06:15.770 Lauren Dana: So after reflecting upon your most recent course offering, my guess is that many of you are going to want to make some changes. 34 00:06:16.020 --> 00:06:24.310 Lauren Dana: And these revisions may be minor, major, or somewhere in between, but I hope that you follow these steps as you begin to think about course revisions. 35 00:06:24.630 --> 00:06:28.220 Lauren Dana: So first, we're asking you to determine the why. 36 00:06:28.360 --> 00:06:44.780 Lauren Dana: Why does the course need revisions? Maybe you noticed the material felt stale, or you received negative student feedback in your evaluations. Maybe your students aren't picking up on concepts as quickly as they have in the past, or maybe you're more worried than ever about academic integrity. 37 00:06:45.170 --> 00:06:55.709 Lauren Dana: Even though you may add more reasons why you want to revise your course as you dig more deeply into the course data, it's really important to approach the revision process with your own why. 38 00:06:56.590 --> 00:07:01.420 Lauren Dana: Next, you'll want to gather any applicable and available data from your courses. 39 00:07:01.580 --> 00:07:10.099 Lauren Dana: Today, we're going to review much of the quantitative and qualitative data available to you, but our main focus will be on student evaluations as a data point. 40 00:07:11.830 --> 00:07:16.349 Lauren Dana: Next, you're going to spend time analyzing and reflecting on this available data. 41 00:07:17.240 --> 00:07:24.000 Lauren Dana: You will create an action plan for changes you hope to implement in your course based on your data analysis and reflection. 42 00:07:26.130 --> 00:07:29.750 Lauren Dana: And you'll create this action plan and work towards changes. 43 00:07:31.850 --> 00:07:47.810 Lauren Dana: And over the next few months, you're going to implement the action plan. And you can do this the next time your course or courses run, or you could think about doing it the next year. So you want to make sure that this implementation process is not something that has to happen immediately, but can happen over time. 44 00:07:48.460 --> 00:07:58.509 Lauren Dana: And finally, as we discussed earlier, the course reflection process is cyclical. Okay, you'll repeat this process each time that you review your course. 45 00:07:58.660 --> 00:08:15.450 Lauren Dana: But we do just want to remind you that you are human, and you have many competing priorities, and the quality of the course reflection and revision process may vary from year to year. Maybe this year you're only able to make one or two small changes, and other years, you have time to revamp your entire course. 46 00:08:15.680 --> 00:08:25.249 Lauren Dana: We do promise you, however, that you will leave this workshop today completing the bolded steps 2, 3, and 4 for your student evaluations for at least one course. 47 00:08:27.880 --> 00:08:38.579 Lauren Dana: Okay, so as promised, we're going to give a brief overview of some of the types of available course data that may help you in the course revision process before we shift our focus to student evaluation. 48 00:08:40.010 --> 00:08:44.140 Lauren Dana: Okay, first we'll discuss some quantitative data available within Course Plus. 49 00:08:45.740 --> 00:08:57.760 Lauren Dana: To access these quantitative analytics, you'll first go to Faculty Tools, and on the right column, you'll see a section labeled Administrative Tools. And the quantitative data is under Student Activity Reports. 50 00:08:57.970 --> 00:09:06.890 Lauren Dana: You'll see that Course Plus offers data on lecture content access, live talk attendance if your course is fully online, discussion forums, and site access. 51 00:09:07.140 --> 00:09:11.959 Lauren Dana: All of these data points are available to download in either a PDF or Excel format. 52 00:09:12.130 --> 00:09:27.489 Lauren Dana: The data and learning analytics from this data are incredibly important in course revisions, and to learn more about analyzing this data, I'm going to self-promote. I recommend watching, Amy and my presentation. It'll be in that worksheet that I placed in the chat. 53 00:09:27.650 --> 00:09:34.419 Lauren Dana: And we do have a workshop that focused specifically on this quantitative data, learning analytics, and course revision. 54 00:09:38.820 --> 00:09:44.029 Lauren Dana: And now I'm just going to go a little bit quickly over other qualitative data available on Course Plus. 55 00:09:46.130 --> 00:09:57.309 Lauren Dana: The communications that you have with students are considered a qualitative data source. You can review emails that you send to students by using the email archive feature in the Course Plus Setting course email tool. 56 00:09:57.340 --> 00:10:11.710 Lauren Dana: I do want to mention that this only tracks emails sent to students within the Course Plus email tool, and if you've selected Save to Email Archive setting. And we do recommend that you do this, so that you can use your emails in the future to help course revisions. 57 00:10:12.540 --> 00:10:19.040 Lauren Dana: So you may want to consider gathering information from these emails. So, for example, maybe you look at frequently asked questions. 58 00:10:19.220 --> 00:10:21.050 Lauren Dana: or reported issues. 59 00:10:21.140 --> 00:10:39.900 Lauren Dana: For example, maybe one of your lectures, is really confusing to students, and you need to add an FAQ. Or maybe students have repeatedly put information on how something's not working, like an external link. Or maybe they've even provided evaluative feedback, like instructions are confusing, and you've had to send an email to explain it. 60 00:10:40.080 --> 00:10:45.320 Lauren Dana: So these are all things that you might want to consider putting on your list to rework on the next offering. 61 00:10:45.540 --> 00:10:51.930 Lauren Dana: But as you go through emails, I encourage you to look for trends, frequently asked questions, and reported issues. 62 00:10:54.760 --> 00:11:04.019 Lauren Dana: Last but not least, which is why we're here today, is course evaluations. So they are actually an excellent quantitative and qualitative data source for your course. 63 00:11:04.290 --> 00:11:10.059 Lauren Dana: Course evaluations are vital to the learning process for both BSPH students and faculty. 64 00:11:10.520 --> 00:11:17.519 Lauren Dana: Each term, a student will submit an anonymous feedback on their courses during a predetermined evaluation period. 65 00:11:17.670 --> 00:11:26.630 Lauren Dana: There are standardized evaluation questions that students receive that consist of students rating the course on a numerical scale, as well as open-ended questions. 66 00:11:26.880 --> 00:11:32.099 Lauren Dana: The complete list of questions are, again, in that worksheet I pasted in the chat. 67 00:11:32.200 --> 00:11:42.639 Lauren Dana: You also have the opportunity to add 3 open-ended questions of your own to course evaluations. I find many faculty don't know that this is there, or they don't utilize this feature. 68 00:11:42.800 --> 00:11:45.939 Lauren Dana: Again, these instructions will be in our follow-up email. 69 00:11:46.420 --> 00:11:55.230 Lauren Dana: So once the evaluation period has closed, you receive an alert to view the evaluation reports by logging into the course evaluation system. 70 00:11:55.560 --> 00:12:03.049 Lauren Dana: Faculty can compare course evaluations from up to 5 courses, and the courses can be from any of the last 3 academic years. 71 00:12:03.300 --> 00:12:11.049 Lauren Dana: These evaluations should provide you with student perspectives, as well as guide future modifications and redesigns if necessary. 72 00:12:12.310 --> 00:12:16.879 Lauren Dana: I'm now going to pass it to Amy, who's going to guide us through how to reflect on these course evaluations. 73 00:12:16.880 --> 00:12:34.730 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, thanks, Lauren. So, when you're identifying changes for your course, this is also a great time to reflect on your perspective of how the course ran and how it's going, and also the perspectives of your teaching team if you're working with a co-instructor or with teaching assistants. 74 00:12:34.940 --> 00:12:39.559 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, if you were leaving qualitative feedback about the course, what would you say about it? 75 00:12:39.690 --> 00:12:56.719 Amy Pinkerton, BSPH CTL Teaching Toolkit: Taking a moment to reflect critically about your course will help you contextualize and analyze the available course data, and help you identify, not only the rationale for why the course is running a certain way, but also can help you determine the why any changes need to be made. 76 00:12:56.930 --> 00:13:01.649 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, do take a moment to think about your perspective, and the perspective of your teaching team. 77 00:13:03.700 --> 00:13:23.370 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so, so far, we've identified how reframing the revision process as refinement can help alleviate stress surrounding course evaluation, and then we overviewed the course revision process and identified some sources of course data to consider in the revision process. Before we go any further, are there any questions? 78 00:13:23.830 --> 00:13:27.439 Amy Pinkerton, BSPH CTL Teaching Toolkit: And you can either raise your hand or post questions in the chat. 79 00:13:36.840 --> 00:13:49.009 Amy Pinkerton, BSPH CTL Teaching Toolkit: All right, I don't see any hands raised or anything in the chat, so we'll go ahead, but if you do have questions, you can post those in the chat, and there'll be more Q&A breaks throughout today's session. 80 00:13:50.210 --> 00:14:15.200 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so, now let's pivot gears and think specifically about our qualitative data and how to analyze that subjective, open-ended data that we receive from students. So, as Lauren said, evaluations are vital to the learning process, but it can be difficult to analyze that qualitative data. So, we've come up with a method for analyzing student responses based on the type of feedback that you receive. 81 00:14:15.890 --> 00:14:25.409 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, we'll categorize the comments, sort them by theme, and then identify actionable next steps, for… based on that feedback that we get. 82 00:14:25.410 --> 00:14:43.410 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, we'll post a worksheet in the chat, and this is the same worksheet that we shared in our reminder email, and this will also be shared in our follow-up email along with our resources. But you can follow along by opening the worksheet that Lauren just shared in the chat. Thank you, Lauren. 83 00:14:44.510 --> 00:14:50.420 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so our Step 1, and if you're in the worksheet, this is part one, step one, 84 00:14:50.680 --> 00:15:11.889 Amy Pinkerton, BSPH CTL Teaching Toolkit: is to categorize your student comments, based on the type of comment that you receive. So as you're reading through your student comments, you can annotate them or copy and paste them into a new Word document or something, and this will help you organize those comments. So right away, if you have a comment that's either unrelated to teaching and learning. 85 00:15:11.890 --> 00:15:22.820 Amy Pinkerton, BSPH CTL Teaching Toolkit: or is so non-specific that it's unhelpful for your review efforts, you can just discard those. So if you're working in a Word doc, you can just strike those out, or even delete them from your list. 86 00:15:22.950 --> 00:15:29.479 Amy Pinkerton, BSPH CTL Teaching Toolkit: So discard those, because they're not going to be helpful for your review and revision process. 87 00:15:29.910 --> 00:15:47.230 Amy Pinkerton, BSPH CTL Teaching Toolkit: Next, consider your positive comments. These are nice. Enjoy these. And you can also look for trends in what students are enjoying about the course, and if you notice that, for example, something is working really well, then you might make a note. Don't change that, it's working well. 88 00:15:47.230 --> 00:15:51.730 Amy Pinkerton, BSPH CTL Teaching Toolkit: And you can also compare your positive comments To, 89 00:15:51.730 --> 00:15:58.139 Amy Pinkerton, BSPH CTL Teaching Toolkit: Areas of disagreement among your students, where things might not have worked as well, to compare and contrast. 90 00:15:59.060 --> 00:16:23.150 Amy Pinkerton, BSPH CTL Teaching Toolkit: Next, consider the actionable suggestions. These are comments or suggestions that shed light on pain points in your class that you, as the faculty or a teaching team member, can actually change or act on. So these are things that, things like how the course is organized and delivered, the content, stuff that's within your control within the course. 91 00:16:23.680 --> 00:16:31.360 Amy Pinkerton, BSPH CTL Teaching Toolkit: So look for themes, and then consider, we'll then, in our next steps, consider trade-offs for making those suggested changes. 92 00:16:31.910 --> 00:16:51.869 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then finally, you're going to come… you're going to run into non-actionable suggestions. And these are comments or suggestions that shed light on pain points, but are items that you as the faculty or teaching team cannot address within the context or confines of your course. So for these actions, consider sharing them with 93 00:16:51.870 --> 00:17:06.310 Amy Pinkerton, BSPH CTL Teaching Toolkit: people who can make use of that data point. So, like your program coordinator, classroom facilities, IT, or the Center for Teaching and Learning. We'd love to hear, for example, if there's, if students have a suggestion about Course Plus. 94 00:17:06.990 --> 00:17:13.109 Amy Pinkerton, BSPH CTL Teaching Toolkit: By using this method, you can systematically sift through your comments, in an objective way. 95 00:17:13.369 --> 00:17:16.839 Amy Pinkerton, BSPH CTL Teaching Toolkit: And now, I'm gonna pass to Lauren to practice this step. 96 00:17:17.680 --> 00:17:22.869 Lauren Dana: Great, so on the worksheet, we have a sample course's student evaluations. 97 00:17:22.990 --> 00:17:32.759 Lauren Dana: So, we're first going to take a look at student responses to question 1 on the evaluation, which is, please identify what you consider to be strengths of the course. 98 00:17:33.030 --> 00:17:48.370 Lauren Dana: I'm going to give you about 1 minute to read the responses, either on your personal worksheet or on this slide. And as you do this, I want you to really think about Amy's suggestions. Which would you discard? Which would you consider as actionable? Which would you possibly refer to a higher-up? 99 00:17:48.660 --> 00:17:58.440 Lauren Dana: So what we're gonna do is we have some music, to help time this out, so just take a minute and look at these, and again, determine where you would place these, using Amy's. 100 00:17:58.440 --> 00:18:00.120 Audio shared by Amy Pinkerton, BSPH CTL Teaching Toolkit: Decision-making chart. 101 00:18:55.290 --> 00:18:58.790 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, that was our one-minute timer. Thank you, Lauren. 102 00:18:58.790 --> 00:19:15.619 Amy Pinkerton, BSPH CTL Teaching Toolkit: So before we… before we go… before I reveal my… how I would categorize these comments, I'd like to hear from you. So, in the chat, let me know what number comments, so we have comments 1, 2, 3, 4, and 5, which ones would you discard? 103 00:19:26.020 --> 00:19:27.790 Amy Pinkerton, BSPH CTL Teaching Toolkit: And if you're not sure, that's okay. 104 00:19:27.990 --> 00:19:29.840 Amy Pinkerton, BSPH CTL Teaching Toolkit: Or maybe you don't discard any. 105 00:19:30.000 --> 00:19:38.159 Amy Pinkerton, BSPH CTL Teaching Toolkit: Okay, so Celine said 4, Lou said 4. Okay, so we were seeing some 4s. Alright, so which, which comments do you think are actionable? 106 00:19:48.850 --> 00:19:51.340 Amy Pinkerton, BSPH CTL Teaching Toolkit: 1… Alright. 107 00:19:52.040 --> 00:19:54.520 Amy Pinkerton, BSPH CTL Teaching Toolkit: I'll give it another couple seconds. 108 00:19:56.400 --> 00:20:00.020 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and now, which ones do you think are unactionable? 109 00:20:08.060 --> 00:20:10.509 Amy Pinkerton, BSPH CTL Teaching Toolkit: It's okay if you don't think any are unactionable. 110 00:20:10.870 --> 00:20:18.869 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so I'll go ahead and reveal what… how I categorize these. Okay, so Celine put 1 and 2, actionable, unactionable, 4. 111 00:20:19.190 --> 00:20:33.059 Amy Pinkerton, BSPH CTL Teaching Toolkit: So here's how I would categorize these. So, one I said is actionable, because, you as the teaching team member can, can look at the group assignment instructions to make them clearer. 112 00:20:33.150 --> 00:20:53.080 Amy Pinkerton, BSPH CTL Teaching Toolkit: For 2 and 3, I said those were positive trends, so I didn't necessarily discard them, but, just things to keep in mind. And then number 4, I said that was non-specific, so I discarded that. Everything about the course was awesome, that's a great comment to hear, but it's not helpful to tell me what exactly about the course was awesome. 113 00:20:53.240 --> 00:20:57.090 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then number 5, I put that as non-actionable, 114 00:20:57.090 --> 00:21:16.009 Amy Pinkerton, BSPH CTL Teaching Toolkit: because… so that I love the course and wish it would be offered online, that wish it would be offered online, that's something I might talk to my academic coordinator about to see if there's a lot of interest for an online offering, then maybe that's something I would pursue. But I'd need… I would talk to, my… my academic or 115 00:21:16.010 --> 00:21:17.510 Amy Pinkerton, BSPH CTL Teaching Toolkit: Program coordinator for that. 116 00:21:17.790 --> 00:21:23.120 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, that's how I categorize these, and now we're gonna do another practice run. 117 00:21:23.500 --> 00:21:33.379 Lauren Dana: Great. So as you can guess, we are now going to look at student responses for the second question, so please identify areas where you think the course could be improved. 118 00:21:33.570 --> 00:21:46.099 Lauren Dana: Okay, so again, we're going to take a minute to skim through the comments, and you're going to use the same process as you did for the first set. So think about what you would do with these comments. Discard, share with a higher-up. 119 00:21:46.320 --> 00:22:03.889 Lauren Dana: Or if they're actionable, so all things to consider. Just a quick comment is you can choose to discard parts of a comment and keep the other parts, so not all comments are discarded the same. There might be something helpful at the end of the comment, so that's something to consider. 120 00:22:03.890 --> 00:22:19.570 Lauren Dana: And you also do want to make sure that you're looking for common themes. So one comment from one student doesn't necessarily mean that you should act on it. We are limited to only 5 comments for this activity, but you do want to see if you can start pulling those common themes. 121 00:22:19.670 --> 00:22:22.929 Lauren Dana: So again, we have our awesome music while you take a look at this. 122 00:23:19.260 --> 00:23:21.669 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and that is our timer. 123 00:23:21.670 --> 00:23:39.829 Amy Pinkerton, BSPH CTL Teaching Toolkit: So we'll do, same as before, let me know which ones you would discard, and while you're typing, I'll also read the comments, because I forgot to do that on the previous slide. So I will read, so one says, the assignment directions were really confusing, and I wasn't sure what the professor was looking for in the submitted assignment. 124 00:23:39.830 --> 00:23:42.710 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 2 is, this course was boring. 125 00:23:43.120 --> 00:23:54.859 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 3 is, I see the point of the group assignments, but my group really struggled. We weren't sure where to submit our assignment, and if we needed to submit the assignment. Or if we each needed to submit the assignment. 126 00:23:55.250 --> 00:23:56.489 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 4? 127 00:23:56.630 --> 00:24:13.159 Amy Pinkerton, BSPH CTL Teaching Toolkit: The teaching team never responded to me on time. Also, way too much work for a two-credit course. My group also did nothing, yet they got the same grade as me. And comment number 5 is, the course had similar learning objectives to another required course in my program. I'm not sure if both should be required. 128 00:24:13.320 --> 00:24:17.130 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, let me know in the chat, what numbers would you discard? 129 00:24:18.760 --> 00:24:25.719 Amy Pinkerton, BSPH CTL Teaching Toolkit: Okay, so we have, for Lou, she said, discard 2, actionable 1, 3, and 4, non-actionable 5. 130 00:24:25.940 --> 00:24:30.010 Amy Pinkerton, BSPH CTL Teaching Toolkit: And Emily said, discard two, unless there were a lot of them. A trend. 131 00:24:30.210 --> 00:24:34.769 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and since, here, I'll go ahead and… 132 00:24:35.980 --> 00:24:44.780 Amy Pinkerton, BSPH CTL Teaching Toolkit: ask if you have, you can also share which ones you thought were actionable, and Lou, thanks for preemptively telling me actionable 1, 3, and 4. 133 00:24:58.640 --> 00:25:07.779 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then, if you thought any were unactionable. Oh, okay, I think that 1, 3, and 4 from Emily is under actionable, and then also let me know what you thought was unactionable. 134 00:25:10.140 --> 00:25:12.460 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, I see some thumbs up. 135 00:25:13.330 --> 00:25:31.329 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so I'll go ahead and reveal, how I categorize these, along with a little rationale. So, number one, I said that was actionable, again, because, that's another, another comment about the assignment directions not being clear, so that's something that you can take action on. 136 00:25:31.400 --> 00:25:47.950 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number two, the course was boring. I discarded that, because they didn't tell… like, it would be interesting to see if a lot of students think the course is boring, but, as just a single comment, unless it's a trend, it would, it's not specific enough to be helpful. 137 00:25:48.510 --> 00:25:57.660 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 3, actionable. Again, the, there was some confusion about the group assignment, in this case, the submission guidelines. 138 00:25:58.270 --> 00:26:16.659 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 4 has some actionable and unactionable aspects to it. The teaching team never responded to me on time. I think that's actionable. If I were the faculty, I would meet with my teaching team and say, hey, did something happen with communication? Was there a drop or a… 139 00:26:18.880 --> 00:26:27.749 Amy Pinkerton, BSPH CTL Teaching Toolkit: a, what is it? I just lost my train of thought. A, disconnect between who was doing what when communicating with students, so that's something I could address. 140 00:26:27.750 --> 00:26:49.589 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then the unactionable part was the way too much work for a two-credit course. So for that, I, as a faculty or teaching team, I could go in and… or ask an instructional designer from the Center for Teaching and Learning to help me calculate what is the actual workload for this course, and if I find that it doesn't align with a two-credit course, 141 00:26:49.670 --> 00:27:00.999 Amy Pinkerton, BSPH CTL Teaching Toolkit: then that's something I might have to talk to my program about, either… either adding content if it's not enough, or, making it align with the credit hour that it is. 142 00:27:01.090 --> 00:27:09.040 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then my group did nothing, yet they got the same grade as me, that's another, actionable item where I could look at the group… how the group project is set up. 143 00:27:09.590 --> 00:27:18.289 Amy Pinkerton, BSPH CTL Teaching Toolkit: Number 5, I said this was unactionable. The course had similar learning objectives to another required course in my program. I'm not sure if both should be required. 144 00:27:18.360 --> 00:27:39.850 Amy Pinkerton, BSPH CTL Teaching Toolkit: That's something I would pass on to my academic department or program, to look at, because that has to probably do with a competency requirement, or some kind of requirement for your program. So, if there is redundancy, then they should know that, or if there's a rationale behind why both courses are required, then that might be something you'd want to communicate to students. 145 00:27:40.530 --> 00:27:52.779 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so that's how I categorized those comments. So now, thinking about, so after that step one, categorize the comments, by now we've discarded some comments, we have some actionable. 146 00:27:52.780 --> 00:28:05.629 Amy Pinkerton, BSPH CTL Teaching Toolkit: comments, so at this stage, you would pass on your unactionable comments and focus on trends and actionable comments. And from here, we're going to organize them by a thematic category. 147 00:28:06.090 --> 00:28:27.709 Amy Pinkerton, BSPH CTL Teaching Toolkit: Now, on our list, we have categories syllabus, course organization, Course Content, Assessments, feedback, course resources, and other. But your list might look different, depending on the feedback that you receive, so create categories that make sense for the feedback that you received. These are just the categories that we as instructional designers see often when we're looking at course reviews. 148 00:28:28.810 --> 00:28:47.570 Amy Pinkerton, BSPH CTL Teaching Toolkit: So from here, we're going to organize and synthesize the comments thematically. So here, and for our example, we're going to go through assessments, because there were a lot of comments about, the instructions, submission guidelines, and the perceived fairness around the group project. So we're gonna focus on our category assessments. 149 00:28:49.540 --> 00:28:59.839 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, looking at, or considering the comments that we received, some positive feedback were that they, the course, or the assessments were practical and applicable to real life. 150 00:28:59.840 --> 00:29:10.979 Amy Pinkerton, BSPH CTL Teaching Toolkit: However, the constructive criticisms were really around unclear instructions, unclear submission guidelines, and a perceived unfairness, especially regarding the group work activity. 151 00:29:11.240 --> 00:29:17.009 Amy Pinkerton, BSPH CTL Teaching Toolkit: And are these actionable? Yes, we can do something about these, constructive criticisms. 152 00:29:17.290 --> 00:29:19.470 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, when you're working with your own 153 00:29:19.470 --> 00:29:43.939 Amy Pinkerton, BSPH CTL Teaching Toolkit: evaluations, you would then follow the same process for each remaining category. You may need to reread comments multiple times, and like before, you might have comments that have, multiple categories or themes in one single student comment, so you might want to pull out, different pieces here and there, and you might end up breaking student comments into different parts, and that's okay. But once you, 154 00:29:44.540 --> 00:30:02.789 Amy Pinkerton, BSPH CTL Teaching Toolkit: again, you might need to reread comments multiple times, and then to synthesize and clearly summarize the positive and critical feedback that you receive. So this step might take a little bit of time, especially if you have a large course and a lot of student comments to read through, but once you get through this step. 155 00:30:02.790 --> 00:30:05.279 Amy Pinkerton, BSPH CTL Teaching Toolkit: The next steps will be much easier. 156 00:30:06.900 --> 00:30:26.159 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then also, Lauren mentioned this earlier, but I want to reiterate it here as well. As you're going through your student comments, you also might find that you discard more comments at this stage, especially if you have, like, outliers, where a lot of students enjoyed something or didn't have a comment about something, and only maybe one student 157 00:30:26.160 --> 00:30:34.150 Amy Pinkerton, BSPH CTL Teaching Toolkit: had, a really negative reaction to something. So it's up to your own discernment if you want to, disregard 158 00:30:34.150 --> 00:30:35.310 Amy Pinkerton, BSPH CTL Teaching Toolkit: an outlier. 159 00:30:37.260 --> 00:30:43.839 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, so, from our… from those, the critical, 160 00:30:43.840 --> 00:30:57.560 Amy Pinkerton, BSPH CTL Teaching Toolkit: feedback, I've now identified… I've broken those into specific, actionable steps. Now, from step 2 to step 3, step 3 is… are, steps that I can take to address that criticism. 161 00:30:57.560 --> 00:31:06.140 Amy Pinkerton, BSPH CTL Teaching Toolkit: And if you're not sure how to go from step 2 to step 3, maybe you've identified the issue, but you're not sure about the actual action you can take. 162 00:31:06.140 --> 00:31:31.129 Amy Pinkerton, BSPH CTL Teaching Toolkit: the steps you can take, or the solution, or possible solutions, that's when I encourage you, you can contact a CTL instructional designer, and we can help you come up with different options for actions you can take to address those criticisms. So if you're having a hard time consolidating actionable steps, that's where we can… we shine in that. So we'd be happy to help you, with different… come up with different options for how to respond to your students 163 00:31:31.130 --> 00:31:31.880 Amy Pinkerton, BSPH CTL Teaching Toolkit: data. 164 00:31:32.430 --> 00:31:36.360 Amy Pinkerton, BSPH CTL Teaching Toolkit: So for our examples, I have our actionable steps. 165 00:31:36.430 --> 00:31:51.330 Amy Pinkerton, BSPH CTL Teaching Toolkit: For the unclear instructions, review the instructions for clarity, and then ask another person to review. And in this case, I have a teaching assistant, or maybe an instructional designer at CTL. We love reading instructions for clarity, so we're happy to do that. 166 00:31:51.380 --> 00:32:00.100 Amy Pinkerton, BSPH CTL Teaching Toolkit: For the second item, which were unclear submission guidelines, again, review the instructions for clarity and ask another person to review. 167 00:32:00.650 --> 00:32:19.679 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then, to address the perceived unfairness between group members, some actionable steps are create a group accountability contract, consider peer evaluation, maybe group members get to evaluate each other, and create a communication route for issues, especially pertaining to the group work. 168 00:32:19.970 --> 00:32:26.190 Amy Pinkerton, BSPH CTL Teaching Toolkit: So those are the actionable steps from our, critical feedback. 169 00:32:26.790 --> 00:32:31.469 Amy Pinkerton, BSPH CTL Teaching Toolkit: And now, let's consider the benefits and drawbacks for each of these steps. 170 00:32:32.710 --> 00:32:38.960 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, for unclear instructions and the unclear submission guidelines, the benefit is clear. 171 00:32:38.960 --> 00:33:03.939 Amy Pinkerton, BSPH CTL Teaching Toolkit: It's clarity. Students will have a better understanding of what the assignment is and how to submit their assignment. So that's a huge benefit for students. There's no drawback to, reviewing and rewriting the instructions and submission guidelines for clarity. So zero drawback. And the level of effort, I put this as a 1. If you don't have someone to review, again, you can reach out to the Center for Teaching 172 00:33:03.940 --> 00:33:17.219 Amy Pinkerton, BSPH CTL Teaching Toolkit: and learning instructional design team. We would be happy to help you review your instructions or submission guidelines. So that is a level of effort one, one being the lowest level of effort, and four being the highest level of effort. 173 00:33:17.740 --> 00:33:36.909 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then let's look at the third item, perceived unfairness between group members. For that, that takes a little bit more effort. The benefits, though, are there's more accountability among group members and greater clarity in the group assignment. But the drawbacks are, with that greater, 174 00:33:37.370 --> 00:33:54.120 Amy Pinkerton, BSPH CTL Teaching Toolkit: It requires more course administrative time, so there might be more behind-the-scenes work working with the teaching team, or more items for the teaching team to have to sort through from a course administrative perspective. 175 00:33:54.120 --> 00:34:05.109 Amy Pinkerton, BSPH CTL Teaching Toolkit: So there's a little bit of a drawback there, but I, as an instructional designer looking at this, I think the drawbacks probably outweigh the benefits, so I would recommend to continue on with this actionable item. 176 00:34:05.160 --> 00:34:14.360 Amy Pinkerton, BSPH CTL Teaching Toolkit: But the level of effort is a 3, so not quite the highest level of effort, it's not, like, a complete overhaul of the course, but it will take some effort. 177 00:34:14.960 --> 00:34:25.289 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, if you were completing this using your own evaluations, you would create these actionable steps for each category. Again, you can consult with an instructional designer if you're not sure what steps to take. 178 00:34:25.560 --> 00:34:35.940 Amy Pinkerton, BSPH CTL Teaching Toolkit: And, and then consider the benefits and drawbacks of taking each step. And you might find that you might have a really great suggestion, but it's just not, 179 00:34:35.969 --> 00:34:47.770 Amy Pinkerton, BSPH CTL Teaching Toolkit: realistic within your course. So you might find the drawback outweighs the benefit, and that's okay. And you can always, consider that maybe for a future revision, so you don't have to do that right away. 180 00:34:47.820 --> 00:34:56.789 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then also, considering the level of effort scale, having a scale like this, can help you identify and prioritize 181 00:34:56.790 --> 00:35:14.670 Amy Pinkerton, BSPH CTL Teaching Toolkit: your revisions. So say, you only have a little bit of time before your next offering, you might focus on the items that are only a one level of effort, and then for the next offering, do some twos and threes, and then, like, two offerings down the line, plan to do a major overhaul, which would be a 4. 182 00:35:14.710 --> 00:35:21.370 Amy Pinkerton, BSPH CTL Teaching Toolkit: So it's okay to do some incremental changes over time. You don't have to do all of your, 183 00:35:21.450 --> 00:35:26.659 Amy Pinkerton, BSPH CTL Teaching Toolkit: your revisions all at once. You can prioritize based on effort or importance. 184 00:35:27.700 --> 00:35:34.179 Amy Pinkerton, BSPH CTL Teaching Toolkit: So, yes. So, now I'll pass to Lauren for our next Q&A break. 185 00:35:34.630 --> 00:35:40.960 Lauren Dana: Great, so before we move forward, I just want to cover what we've gone over. So we've overviewed the course revision process. 186 00:35:41.370 --> 00:35:46.189 Lauren Dana: We've identified how to reframe the revision process and alleviate some stress. 187 00:35:46.470 --> 00:35:57.879 Lauren Dana: We've identified course data to consider in the revision process, and we've actually practiced how to approach an evaluation, and make it meaningful for you. 188 00:35:58.490 --> 00:36:03.709 Lauren Dana: So before we actually start analyzing our own course evaluations, does anyone have any questions? 189 00:36:07.410 --> 00:36:08.320 Lauren Dana: Celine. 190 00:36:09.800 --> 00:36:19.139 Celine Greene: So, I just tried something, because I was thinking about how you were, you know, we were summarizing comments and everything, and I just tried something using 191 00:36:19.280 --> 00:36:35.390 Celine Greene: Adobe Reader, which is available to everybody for free. There's an AI assistant in Adobe Reader, and I asked it, can you summarize the open-ended comments in questions 13 through whatever, starting on page 9? I think it's 13 through 16. 192 00:36:35.450 --> 00:36:44.600 Celine Greene: it did a really decent job of summarizing those comments and picking up the trends, and I was just wondering, 193 00:36:44.980 --> 00:36:53.100 Celine Greene: If you guys had any concerns about using AI for summarizing those trends. 194 00:36:53.100 --> 00:37:10.060 Amy Pinkerton, BSPH CTL Teaching Toolkit: Right, yes. Yeah, I'm so glad you asked that. So AI can really… it can help summarize and synthesize and identify trends. However, you want to be careful with your course evaluation data that if you're going to use an AI, to help you with this step. 195 00:37:10.060 --> 00:37:25.960 Amy Pinkerton, BSPH CTL Teaching Toolkit: Then you want to anonymize the data first. So you want to take out any names, you want to take out any course numbers, or even, like, faculty names or TA names, anything that could be, linked back to your course. You want to make sure you anonymize the data. 196 00:37:25.960 --> 00:37:39.119 Celine Greene: Yes, but this is actually built into Acrobat Reader and Acrobat, excuse me, Adobe Reader and Adobe Acrobat. It's just when you open the document, it asks you, do you want this? Do you want to use the assistant? So you're not uploading it anywhere? 197 00:37:39.830 --> 00:37:51.690 Celine Greene: So, I take your point, if we were to upload 100%, but beyond the anonymization of it, again, because it's within a controlled environment of our machine or our cloud. 198 00:37:52.050 --> 00:37:54.680 Celine Greene: What do you think we might be missing? 199 00:37:54.980 --> 00:37:58.719 Amy Pinkerton, BSPH CTL Teaching Toolkit: Right. I think also you'd want to… 200 00:37:58.880 --> 00:38:15.560 Amy Pinkerton, BSPH CTL Teaching Toolkit: You would probably at least want to manually read through your comments as a human at least once before just solely relying on the, on the generative summary, just in case the summary misses something, because it is fallible. 201 00:38:15.560 --> 00:38:33.219 Amy Pinkerton, BSPH CTL Teaching Toolkit: So that would be the other concern, is that the AI might miss something, that might be key. So, you still… you can use that tool, granted it's private and, within that… that secure space, but you want to still go through and with human eyes and look through your data at least once or twice. 202 00:38:35.660 --> 00:38:58.669 Lauren Dana: Matt, I do think it's valuable to take the time to process your evaluations. Like, AI can summarize and synthesize that, but I do think there is something about actually taking the time to read each comment, think about it, reflect on it, and I think having AI completely summarize it takes a little bit of that reflection process out. It's absolutely a wonderful tool if you're time-crunched, 203 00:38:58.670 --> 00:39:04.649 Lauren Dana: Or if you just want to see what it comes up with compared to what you came up with as trends, I think that could be really helpful, but… 204 00:39:04.650 --> 00:39:10.609 Lauren Dana: I do encourage that I think people should look through the evaluations, and take the time to reflect on it. 205 00:39:10.610 --> 00:39:14.900 Lauren Dana: And then use AI, or the Adobe app, as a supporting tool. 206 00:39:18.520 --> 00:39:20.379 Lauren Dana: Any other questions? 207 00:39:23.570 --> 00:39:30.219 Amy Pinkerton, BSPH CTL Teaching Toolkit: Well, and Celine just posted in the chat, I like the comparison of your thoughts versus the AI summary. Yeah, that was a good, good suggestion. 208 00:39:30.950 --> 00:39:31.850 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright. 209 00:39:33.740 --> 00:39:37.269 Lauren Dana: Okay, I think we're ready, Amy, for what everyone's been waiting for. 210 00:39:38.010 --> 00:39:47.310 Lauren Dana: Okay, so it's now your turn, to use this method that we've taught you to complete and complete some of these steps with your own course evaluations. 211 00:39:47.440 --> 00:40:05.520 Lauren Dana: As you can see at the time, it's only… it's 1140, so there is no way that you're going to finish every step of this workshop and completely go through a course evaluation. But the hope is that you can get started, and we can provide any needed support, and that you can continue this process outside of the workshop. 212 00:40:05.670 --> 00:40:17.980 Lauren Dana: So for step one, hopefully you've already come with your student course evaluations downloaded, so we will give you a moment if you want to access these or go into the evaluation database to pull them up. 213 00:40:17.980 --> 00:40:28.229 Lauren Dana: If you don't have any course data, or you're just not really in the mood to work with it right now, we have added, some sample data from the TA training course. 214 00:40:28.410 --> 00:40:33.749 Lauren Dana: If you go to your worksheet under Part 2, you can take a look and scroll down. 215 00:40:34.140 --> 00:40:39.139 Lauren Dana: To page 4, and you'll see some sample comments. So feel free to use these if you prefer. 216 00:40:41.110 --> 00:40:54.390 Lauren Dana: Okay, and then step two, hopefully this is already open, but we are going to have you open up the workshop worksheet. Hopefully you've been interacting with it throughout, but as a reminder, we're now on Part 2, so we have completed part one. 217 00:40:56.310 --> 00:40:57.780 Lauren Dana: And then step 3? 218 00:40:57.910 --> 00:41:13.689 Lauren Dana: When prompted, you're going to apply each step of the qualitative analysis process. But Amy and I are going to prompt you through each step, you don't have to do it all at once, and we're going to give you some concrete times to work with, just so that we can get through all of the steps. 219 00:41:16.370 --> 00:41:36.319 Lauren Dana: Okay, so we're going to start with step one, categorize student comments. So this is what Amy had walked us through about 20 minutes earlier. So, you're going to go through your evaluation and mark it up however it helps you. So just some pro tips that I use for discard. Sometimes I'll just cross out a comment or use the strikethrough. 220 00:41:36.320 --> 00:41:48.950 Lauren Dana: You could highlight, different actions, so pink could be consider, yellow could be enjoy and compare. So feel free to organize it or categorize it however works for you. 221 00:41:48.960 --> 00:41:55.649 Lauren Dana: And I do recommend copying and pasting the comments onto a labeled Word document. You can use the chart we give you. 222 00:41:55.700 --> 00:42:00.269 Lauren Dana: If you want to get off that PDF from the, evaluation database. 223 00:42:00.490 --> 00:42:06.159 Lauren Dana: So, I'm gonna give you, I believe it's 2 minutes, Amy, right? To, go through the evaluation, mark it up? 224 00:42:06.370 --> 00:42:15.330 Lauren Dana: decide what comments you should discard, which you should enjoy and compare, which you could consider are actionable, and which you might share to some higher-ups at BSPH. 225 00:44:09.000 --> 00:44:10.400 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, that is our timer. 226 00:44:10.400 --> 00:44:17.059 Lauren Dana: Okay, hope we were able to get through some of the comments. So we are going to move to step two. 227 00:44:17.520 --> 00:44:31.070 Lauren Dana: So for step two, what you're first going to do is you're going to disregard those discard comments, okay? So these are not going to be used in the next set of our organization or analysis, so you can remove those and focus just on the ones that you decided to keep. 228 00:44:31.450 --> 00:44:42.950 Lauren Dana: And we're going to ask you to now put your comments into themes. Again, for this activity, we organized it by categories that we see a lot as instructional designers. You may change this in the future. 229 00:44:43.020 --> 00:44:52.100 Lauren Dana: And we're going to encourage you for today, because you only have 2 minutes, just pick one theme. So maybe it's assessments, or maybe it's… there's a lot of comments about your syllabus. 230 00:44:52.240 --> 00:45:02.840 Lauren Dana: So focus on one theme, and start to place the comments. You can either place them, copy and paste, or you could summarize them yourself as you place them into the chart. 231 00:45:03.060 --> 00:45:06.770 Lauren Dana: So, we're gonna give you 2 minutes to start working on this. 232 00:47:06.040 --> 00:47:14.830 Lauren Dana: Great, thank you, Amy. Okay, we're gonna head into our final step, which is the most challenging one, and you're now going to consolidate actionable steps. 233 00:47:15.020 --> 00:47:15.920 Lauren Dana: So… 234 00:47:16.040 --> 00:47:24.290 Lauren Dana: Again, using the same theme that you picked, you're going to identify some actionable steps that you could take to address the comments. 235 00:47:24.340 --> 00:47:38.269 Lauren Dana: So think about changes that you could make, anything that you might need to update based on the comments, and then also consider the benefits and the drawbacks, your time being one of them, right? So you have to consider how much time you have to make these. 236 00:47:38.390 --> 00:47:42.979 Lauren Dana: And then try to consider the level of effort, how much time it would take you to complete these. 237 00:47:43.110 --> 00:47:49.629 Lauren Dana: So, again, we're going to give you 2 minutes, not a lot of time, but try to get one or two actionable steps down. 238 00:49:45.450 --> 00:49:56.740 Lauren Dana: Okay, thank you, Amy. So again, today was just a start, but we do want to talk to you a little bit about the next step, which is to assure you that you are not alone in this process. 239 00:49:56.760 --> 00:50:11.100 Lauren Dana: And our final recommendation is to meet with a CTL instructional designer who can accompany you through your course's quantitative and qualitative data. If you have an instructional designer, feel free to reach out to them, they'd be happy to help you with this. 240 00:50:11.150 --> 00:50:29.140 Lauren Dana: And if you don't have an instructional designer that you're working with, there's a link on, again, that resource sheet we provided you in the beginning, and you can click on this to schedule a consultation. We are happy to help, and we really do hope that you found today helpful and productive to start your course review process. 241 00:50:29.260 --> 00:50:37.389 Lauren Dana: And before we get to our final Q&A session, I am going to hand it to Amy to do some, I guess, end of workshop housekeeping items. 242 00:50:37.390 --> 00:50:39.970 Amy Pinkerton, BSPH CTL Teaching Toolkit: Yes. Thank you so much. 243 00:50:39.970 --> 00:51:04.940 Amy Pinkerton, BSPH CTL Teaching Toolkit: Just like, you benefit from hearing your students' feedback, we benefit from hearing your feedback. So please take a moment to complete our 2-minute workshop evaluation survey. I'm putting the link in the chat as we speak. There we go. And this is an anonymous survey, that will help us know how today's workshop went. It will also… we use this data to plan future workshops 244 00:51:04.940 --> 00:51:22.970 Amy Pinkerton, BSPH CTL Teaching Toolkit: Just like this one. So your feedback really helps us. So you can go to the link I posted in the chat, or you can scan the QR code on the screen, and again, your responses are anonymous, and we very much value your feedback. So, thank you for taking a moment to fill that out. 245 00:51:23.650 --> 00:51:25.729 Amy Pinkerton, BSPH CTL Teaching Toolkit: I'll just give you a few seconds. 246 00:51:27.160 --> 00:51:30.789 Amy Pinkerton, BSPH CTL Teaching Toolkit: Alright, and again, the link for that is in the chat. 247 00:51:30.940 --> 00:51:53.539 Amy Pinkerton, BSPH CTL Teaching Toolkit: I also wanted to highlight some resources from CTL, starting with CTL Help and the Course Plus Faculty Guide. So we mentioned a few times that you can contact CTL to ask for an instructional design consultation for your course, to work through your course reviews, or to have an ID look at your course. So if you don't have an ID assigned, you can contact CTL Help, and they'll connect you with an instructional designer at CTL. 248 00:51:53.670 --> 00:52:06.950 Amy Pinkerton, BSPH CTL Teaching Toolkit: If you find that as you're going through your course revisions, you might have questions about Course Plus tools or features, you can check out the Course Plus Faculty Guide for step-by-step instructions on how to use Course Plus tools and features. 249 00:52:07.180 --> 00:52:28.040 Amy Pinkerton, BSPH CTL Teaching Toolkit: I also wanted to highlight the CTL Teaching Toolkit website. This is a resource site that the instructional design team has created on all things course design, development, and teaching and learning. So, for example, if you want to create more instructor presence, or put in some more active learning activities, then the Toolkit site is a great place to start. 250 00:52:28.040 --> 00:52:30.540 Amy Pinkerton, BSPH CTL Teaching Toolkit: We have an entire page on active learning. 251 00:52:31.420 --> 00:52:44.159 Amy Pinkerton, BSPH CTL Teaching Toolkit: Other things, if you want to stay up-to-date on professional development at CTL, the most up-to-date and just-in-time information can be found on our CTL blog. 252 00:52:44.170 --> 00:52:53.070 Amy Pinkerton, BSPH CTL Teaching Toolkit: Our teaching toolkit website that I mentioned earlier houses our events calendar and our on-demand workshop videos page, and 253 00:52:53.190 --> 00:52:57.669 Amy Pinkerton, BSPH CTL Teaching Toolkit: That way, if you want, you can watch workshops just like this one. 254 00:52:57.740 --> 00:53:09.290 Amy Pinkerton, BSPH CTL Teaching Toolkit: And then finally, earlier today, I mentioned our teaching assistantship training course. That is an online, self-paced course to prepare students for the many roles and responsibilities of teaching assistantship. 255 00:53:09.290 --> 00:53:18.230 Amy Pinkerton, BSPH CTL Teaching Toolkit: So the links to all of these are in our, our resources document that we shared, and we'll also share all of this in our follow-up email. 256 00:53:18.920 --> 00:53:21.139 Amy Pinkerton, BSPH CTL Teaching Toolkit: There's our references. 257 00:53:22.490 --> 00:53:27.049 Amy Pinkerton, BSPH CTL Teaching Toolkit: And, I'm going to stop the recording now. We still have about 5 minutes left. 258 00:53:27.050 --> 00:53:44.169 Amy Pinkerton, BSPH CTL Teaching Toolkit: On our time, so we'll open that for Q&A, but I just wanted to say that we hope that you found this workshop to be helpful as you analyze your student course evaluations, and please reach out to us at CTL if you'd like a one-on-one consultation, or help with your evaluation results. 259 00:53:44.170 --> 00:53:45.090 Amy Pinkerton, BSPH CTL Teaching Toolkit: Thank you. 260 00:53:45.410 --> 00:53:47.640 Amy Pinkerton, BSPH CTL Teaching Toolkit: I'll go ahead and stop the recording.