WEBVTT 1 00:00:02.750 --> 00:00:18.950 Amy Pinkerton (BSPH CTL Teaching Toolkit): Alright hello and welcome to today's picnic. Pd workshop. The large enrollment course learner how to create a clear, consistent student experience in complex, large enrollment courses. And now I'll pass to our speaker to introduce himself. 2 00:00:19.710 --> 00:00:36.129 Boris Buia: Thank you, Amy. My name is Boris Boya. I'm an instructional designer at the center for teaching and learning. And as Amy mentioned, we're going to focus today on large enrollment courses and specifically on how to create an engaging experience for students. 3 00:00:36.410 --> 00:00:40.950 Boris Buia: Let's start by defining what we mean by larger, unknown course. 4 00:00:41.593 --> 00:00:50.200 Boris Buia: The definition can vary according to contest, background and qualification of students. But for stem courses, health, profession, courses 5 00:00:50.450 --> 00:00:53.769 Boris Buia: in public health classes, we are anywhere from 6 00:00:54.100 --> 00:00:57.440 Boris Buia: 100 to all the way to 500 students 7 00:00:58.130 --> 00:01:04.920 Boris Buia: at the School of Public Health we have courses that are offered concurrently, both as 0 1 and 8, 1 8 00:01:05.030 --> 00:01:08.759 Boris Buia: on site and online, in Biostat Epi. 9 00:01:09.080 --> 00:01:15.340 Boris Buia: And in terms of enrollment. We had in one instance enrollment all the way up to 800 students. 10 00:01:16.470 --> 00:01:21.910 Boris Buia: Now that we have defined these courses, let's focus on one question. 11 00:01:22.600 --> 00:01:27.000 Boris Buia: how comfortable would you be teaching a course with 100 or more students? 12 00:01:27.460 --> 00:01:31.128 Boris Buia: We're gonna have a poll everywhere, 13 00:01:32.400 --> 00:01:40.829 Boris Buia: where we can post our quest. Our answers. We keep the poll open for approximately 25 seconds. 14 00:01:41.090 --> 00:01:47.619 Boris Buia: and we would like to see whether you are somewhat comfortable, not very comfortable. 15 00:01:48.160 --> 00:01:51.680 Boris Buia: or perhaps you would rather have 16 00:01:51.890 --> 00:01:55.650 Boris Buia: frozen broccoli instead of thinking about a larger almond court. 17 00:02:00.320 --> 00:02:03.139 Boris Buia: It looks like we are somewhat 18 00:02:04.090 --> 00:02:12.830 Boris Buia: comfortable and not very comfortable. So we are right in between. And the follow up question is, why is that? 19 00:02:13.130 --> 00:02:19.360 Boris Buia: What is it that is most challenging in terms of offering a large enrollment course. 20 00:02:19.510 --> 00:02:32.449 Boris Buia: Again, we're gonna have a quick poll everywhere question to verify and see what are the aspects that makes offering a large enrollment course, challenging. 21 00:02:32.750 --> 00:02:44.169 Boris Buia: designing, and outfitting a large enrollment course, connecting with students, engaging students, creating assessments, grading and offering timely feedback. 22 00:02:55.630 --> 00:03:02.179 Boris Buia: It looks like connecting and engaging with students is perceived as a challenge. 23 00:03:02.440 --> 00:03:06.059 Boris Buia: grading and offering timely feedback is perceived as a challenge. 24 00:03:07.050 --> 00:03:13.219 Boris Buia: So what to do do we run away in despair. 25 00:03:13.640 --> 00:03:21.989 Boris Buia: or we raise to the challenge and focus on some of the key aspects that will make offering a large enrollment course successful 26 00:03:22.940 --> 00:03:35.040 Boris Buia: in this presentation, we'll touch upon designing and offering larger enrollment course, connecting with students, engaging students, creating assessments and grading and offering timely feedback. 27 00:03:35.350 --> 00:03:38.939 Boris Buia: Let's start with designing and offering a larger loan. Of course 28 00:03:42.220 --> 00:03:51.139 Boris Buia: you want to think of your large enrollment course as an assessor would do so. You want to start by identifying your course. Learning objectives. 29 00:03:51.240 --> 00:04:03.279 Boris Buia: Once you have a clear sense of what you are, your desired results, you want to determine what are what is acceptable evidence. What summative assessments are you going to ask students 30 00:04:03.460 --> 00:04:04.280 Boris Buia: to take? 31 00:04:04.850 --> 00:04:07.170 Boris Buia: Once you have these 2 pillars for your course. 32 00:04:07.720 --> 00:04:22.740 Boris Buia: course outcomes and summative assessments? You can think about planning learning experiences and instructions for your courses, so you can think about lecture sections, live talks, reading materials, labs, and formative assessments. 33 00:04:23.100 --> 00:04:30.370 Boris Buia: thinking about your course as an assessor, and focusing on these elements, will help you drafting 34 00:04:30.660 --> 00:04:36.359 Boris Buia: 2 key pieces of information, 2 key pages of your course, the syllabus and the schedule page 35 00:04:36.760 --> 00:04:42.140 Boris Buia: we're using as a reference here the learning management system that we are adopting at the school of Public Health. 36 00:04:42.380 --> 00:04:52.270 Boris Buia: And so in relation to the course, plus syllables. Page, you want to think of your this page as both an operational roadmap and a source of cultural capital. 37 00:04:52.640 --> 00:04:57.270 Boris Buia: You want to provide students in your syllabus page with all the information necessary. 38 00:04:58.070 --> 00:04:58.950 Boris Buia: 4. 39 00:04:59.660 --> 00:05:02.009 Boris Buia: Knowing who to reach out to 40 00:05:02.700 --> 00:05:14.800 Boris Buia: in terms of members of the teaching team, what is their email, how they can reach out to them. And in relation to what aspect of the larger enrollment course you want to list, for instance, labs, and how students are 41 00:05:14.980 --> 00:05:18.349 Boris Buia: supposed to attend these labs, whether online or on site. 42 00:05:18.560 --> 00:05:28.000 Boris Buia: what kind of assessments will be included in your course, formative and summative. Any relation to which you want to detail what you're expecting students to accomplish. 43 00:05:29.710 --> 00:05:38.750 Boris Buia: The syllabus as a source of cultural capital should serve to share with students all the relevant resources the students will need. 44 00:05:38.970 --> 00:05:49.570 Boris Buia: at the course level in terms of office hours at tutoring, at the school level, in terms, for instance, of students, health and well-being and accommodation and accessibility. 45 00:05:50.680 --> 00:05:55.709 Boris Buia: As a best practice, we recommend for a teaching team or large enrollment courses 46 00:05:55.810 --> 00:06:02.170 Boris Buia: to set up a lost day quiz, or a survey to encourage students to get familiar with key aspects of the syllabus 47 00:06:03.070 --> 00:06:11.679 Boris Buia: in relation to both the syllabus page and the schedule page of the course you want to make sure to update these pages as early as possible. 48 00:06:12.140 --> 00:06:36.589 Boris Buia: and in relation to the schedule page, you need to keep in mind that these courses can run with anywhere from 15 all the way to 25 lectures are 2 dead labs on a weekly basis, quizzes, perhaps live talks, and it's easy to see how this page can become quickly complicated and complex. So we recommend to break down this page by week or module, to use consistent titles. 49 00:06:36.810 --> 00:06:51.870 Boris Buia: both for lecture sections and all the different course events that you're linking to the page. And equally as important, you want to link to the schedule page or the I stake assessments and labs, and in relation to each you want to list the due dates. 50 00:06:52.720 --> 00:06:56.229 Boris Buia: Once you have updated your syllabus and schedule page. 51 00:06:56.720 --> 00:07:03.640 Boris Buia: you want to pay attention to how you're setting up lecture section pages and assessment pages. 52 00:07:04.130 --> 00:07:15.069 Boris Buia: We recommend, in relation to all these pages, to use a consistent layout, so the students can see a lecture or a session page once, and be familiar with the layout of all the other pages. 53 00:07:15.170 --> 00:07:18.590 Boris Buia: You want to map each lecturing section 54 00:07:19.090 --> 00:07:25.620 Boris Buia: in light of the course, learning objectives developing for each lecturing session, learning objectives. 55 00:07:26.240 --> 00:07:37.670 Boris Buia: You want to share resources, of course, both for the lecture and the assessment and in relation to the assessments, make sure to update guideline resources and logistics. We'll focus on assessments more 56 00:07:37.790 --> 00:07:39.139 Boris Buia: at a later stage. 57 00:07:39.660 --> 00:07:42.869 Boris Buia: Now that you have updated your course. 58 00:07:43.290 --> 00:07:47.579 Boris Buia: You want to pay attention to another vital aspect to offer 59 00:07:47.930 --> 00:07:52.509 Boris Buia: a large enrollment course. It's teaching team communication and coordination 60 00:07:53.820 --> 00:08:02.939 Boris Buia: in large enrollment courses. The teaching team can be comprised of anywhere from 15 to 20 people of more, depending on the number of students and number of labs 61 00:08:03.520 --> 00:08:19.689 Boris Buia: and the number of Tas. So you want to have a dedicated channel of communication that can be used by all the members of the teaching team to pick quickly, get up to speed in relation to everything that is relevant for the course. And this can be something as a chat in teams. 62 00:08:20.070 --> 00:08:40.000 Boris Buia: You want to have outstanding recording meetings where the teaching team that the teaching team can use to address all the most important aspects of planning for the course and offering the course. And you want to have on top of outstanding meetings, touch points for a quick get together in relation to high stake time. Sensitive questions. They may arise 63 00:08:41.230 --> 00:08:43.340 Boris Buia: in relation to large 64 00:08:43.440 --> 00:09:09.649 Boris Buia: enrollment courses, you may experience a significant turnaround in terms turnover in terms of tas and lab instructors, so it is beneficial to have onboarding guidelines for new members, and this onboarding guidance can be something as simple as a word document with a set of frequently asked questions that the new members of the teaching team can use to quickly get up to speed 65 00:09:12.010 --> 00:09:13.879 Boris Buia: other aspects to consider. 66 00:09:14.010 --> 00:09:24.680 Boris Buia: You want to make sure that everybody on the teaching team knows how to address high stakes, situations, whether it's a matter of students asking for an essential, or students asking for special accommodations 67 00:09:26.200 --> 00:09:37.789 Boris Buia: in relation to setting up a course, especially a large enrollment course. You want to make sure that everybody's on the same page in terms of who does what? When in relation to updated course plus 68 00:09:37.950 --> 00:09:44.340 Boris Buia: in relation to offering, live to us in relation to being familiar with the technology of on-site sessions. 69 00:09:44.470 --> 00:09:45.560 Boris Buia: and so on. 70 00:09:48.310 --> 00:09:55.879 Boris Buia: Of particular importance is consistency in the use of technical terminology and explanation of complex topics. 71 00:09:56.320 --> 00:10:10.270 Boris Buia: It is not uncommon for large enrollment courses to have quite a few students who are taking the course and then getting exposed to these complex topics for the 1st time, and it may be confusing for them 72 00:10:10.420 --> 00:10:24.059 Boris Buia: to being exposed to a way of explaining complex topics, of using technical terminology during the main session in a different way of explaining these complex topics during the live session. So insofar as possible. 73 00:10:24.160 --> 00:10:31.709 Boris Buia: Ask all the members of the teaching team to be consistent in the use of technical terminology and the use of an explanation of complex topics. 74 00:10:32.800 --> 00:10:41.509 Boris Buia: Other aspects to consider in relation to the members of the teaching team. You want to make sure that everybody is on the same page in terms of guidelines. 75 00:10:41.760 --> 00:10:52.129 Boris Buia: for I stake assessments in terms of which questions to answer, and which questions not to answer in relation to these assessments. And if you're setting up a deadline by which students 76 00:10:52.450 --> 00:10:58.790 Boris Buia: are allowed to ask questions in relation to estate assessments. Make sure that everybody in your team is aware of it. 77 00:10:59.430 --> 00:11:04.800 Boris Buia: We also want to touch upon grading and renorming, and we'll do so in a later session. 78 00:11:05.180 --> 00:11:18.459 Boris Buia: Now that we focus on how to design and think about your large enrollment course and on how to communicate among the members of the teaching team. Let's focus on how to connect with students in a large enrollment course. 79 00:11:21.620 --> 00:11:27.469 Boris Buia: Often larger online courses are associated with the in-personal atmosphere. After all. 80 00:11:27.897 --> 00:11:34.559 Boris Buia: The standard image of a large enrollment course is a large lecture hall with hundreds of students instructors behind the podium 81 00:11:34.870 --> 00:11:39.949 Boris Buia: on stage explaining slides for an hour, an hour and a half. 82 00:11:40.170 --> 00:11:55.540 Boris Buia: In such a setting you have increasing noise and distractions, and often research has shown that in such a setting there is lack of interaction with faculty members at Eas, both in and out of class, and there is lack or insufficient student to student interaction. 83 00:11:55.700 --> 00:11:57.369 Boris Buia: Given these premises. 84 00:11:57.650 --> 00:12:09.680 Boris Buia: it's reasonable as a consequence, to experience less in individual accountability, irregular attendance, and in relation to assessments, lack of frequent testing and perhaps slow feedback. 85 00:12:09.850 --> 00:12:16.750 Boris Buia: What can we do to turn the personal atmosphere in personal atmosphere or a larger courts into a 86 00:12:17.040 --> 00:12:18.940 Boris Buia: personal atmosphere? 87 00:12:19.050 --> 00:12:21.829 Boris Buia: Let's start by getting to know your students. 88 00:12:22.610 --> 00:12:26.959 Boris Buia: You can start by letting students know the teaching team 89 00:12:27.070 --> 00:12:34.980 Boris Buia: with a welcoming introductory section in which introduce each and every member of the teaching team and explain what everybody is doing. 90 00:12:35.290 --> 00:12:46.310 Boris Buia: And in during your welcome session you offer a course tour going over the syllabus and all the relevant pieces of information that we mentioned earlier and going over the course schedule 91 00:12:47.900 --> 00:12:51.769 Boris Buia: research has shown that you can also deploy a survey. 92 00:12:52.040 --> 00:12:54.140 Boris Buia: and you can introduce the survey 93 00:12:54.430 --> 00:13:01.500 Boris Buia: is something that is not mandatory, mandatory, and in relation to which are going to ask questions that are not personal. 94 00:13:02.100 --> 00:13:08.559 Boris Buia: You want less students. Now that the answers will not be shared outside of the teaching team. 95 00:13:08.800 --> 00:13:19.389 Boris Buia: and some of the questions you may ask in this survey are about how students feel what are the most excited about the course? What are the most anxious, what they expect to get out of the course. 96 00:13:20.290 --> 00:13:31.790 Boris Buia: a few elements that you can consider to get to know your students in relation to course, plus use the courseplus roster page to get familiar with their names. Programs prefer pronouns and location. 97 00:13:31.990 --> 00:13:39.280 Boris Buia: which is especially important when you're going to set up groups for labs. If you're working on online, course. 98 00:13:39.600 --> 00:13:42.860 Boris Buia: you want to invite students to introduce themselves 99 00:13:43.180 --> 00:13:49.340 Boris Buia: whenever they have a question or interact with the main instructors or lab instructors. 100 00:13:50.000 --> 00:13:57.409 Boris Buia: and you want to use their names as you address their questions. Knowing your students names and using them consistently. 101 00:13:57.960 --> 00:14:09.780 Boris Buia: has proven to be valuable in increasing a sense of belonging on the part of the students. Research has shown the students who are taking large enrollment courses where their names are consistently used. 102 00:14:09.980 --> 00:14:28.930 Boris Buia: perceive these courses to be more engaging, perceive the teaching team to be more committed to their success, and students are more likely to think of the instructors as instructors that can be their mentors and instructors. They can reach out for letters of recommendation after the course 103 00:14:29.540 --> 00:14:34.500 Boris Buia: we know, yet a decreased sense of belonging at the university level 104 00:14:34.810 --> 00:14:48.029 Boris Buia: is correlated, positive, correlated with a better academic audience. So there are plenty of reasons to know your students use their names consistently, and make them feel welcome in your course. 105 00:14:48.260 --> 00:15:00.050 Boris Buia: Another aspect to consider in terms of communicating and connecting with your students. You want to communicate with empathy early and often, and you can use a few tools that make this very easy. 106 00:15:00.230 --> 00:15:12.659 Boris Buia: You can use the course plus email tool and send weekly emails to the entire class. And this will work very well if you set up the course schedule page on a weekly basis. By breaking down the content by week 107 00:15:13.280 --> 00:15:14.530 Boris Buia: you can 108 00:15:14.730 --> 00:15:27.839 Boris Buia: set, send emails out to specific subgroups of your large enrollment course. For instance, you may notice at the beginning that you have some students, some students who are not taking live live talks were not attending live talks 109 00:15:28.040 --> 00:15:39.260 Boris Buia: who are not taking formative quizzes who are not watching lectures. And so you can reach out to these subgroups by using the course plus email filters 110 00:15:39.430 --> 00:15:46.010 Boris Buia: to definitely make them feel that they're part of the course, and they need to engage with the course from the get go. 111 00:15:47.240 --> 00:15:48.750 Boris Buia: You want to plan 112 00:15:49.060 --> 00:16:00.789 Boris Buia: all your synchronous sessions later, so that you make time for students questions, and both in on-site and online courses, you can use the discussion forum in such a way. The students 113 00:16:01.160 --> 00:16:18.749 Boris Buia: can ask questions in relation to sessions, labs, and in relation to discussion. Forum page setup. We recommend to use a setup that is similar to what you have then, to what you have on the course schedule page, because they will make it easier for students to orient themselves and know where to post their questions. 114 00:16:19.220 --> 00:16:41.279 Boris Buia: Another tool that you can use very effectively in larger learning courses is a. Podcast this could be something as simple as a few slides that you're self recording to address, for instance, a complex topic that there was not enough time to explain. During asynchronous sessions or by interacting with your lab instructors. You notice the students are. 115 00:16:41.490 --> 00:16:47.129 Boris Buia: are expressing challenges and difficult, having some difficulties in understanding 116 00:16:47.740 --> 00:17:02.869 Boris Buia: topics that they were expressed in the module of your course, the self-recorded podcast can help clarify this, and by doing sharing a self-recorded, podcast you are again proving that the teaching team is committed to the student success. 117 00:17:04.440 --> 00:17:05.290 Boris Buia: We have 118 00:17:05.470 --> 00:17:13.699 Boris Buia: have analyzed a few ways to connect and communicate with students. Let's now focus on how to engage with students. 119 00:17:15.359 --> 00:17:31.869 Boris Buia: Traditionally, in lecture development courses, the content was delivered to some extent. Maybe it is still delivered as a traditional lecturing, where you have the main instructor spending an hour, an hour and a half going over complex topics with using complex terminology. 120 00:17:32.010 --> 00:17:43.360 Boris Buia: And this traditional approach to explaining content in a large enrollment course is based on some assumptions. All students possess the prerequisite knowledge to benefit from the lecture. 121 00:17:43.640 --> 00:17:47.469 Boris Buia: All students need the same information, delivered at the same time 122 00:17:47.830 --> 00:17:50.390 Boris Buia: presented orally at the same pace. 123 00:17:50.570 --> 00:17:56.030 Boris Buia: All students have high working memory capacity. All students have good note-taking skills. 124 00:17:57.060 --> 00:18:13.449 Boris Buia: Unfortunately, research has shown that when exposed to passive learning, students have a difficult time paying attention are inconsistent. And note taking. And when quiz after a passive learning session. 125 00:18:13.870 --> 00:18:19.470 Boris Buia: they show low retention, of course, content. So how do we address this aspect? 126 00:18:19.720 --> 00:18:25.300 Boris Buia: We want to transform what is a passive learning experience into an active learning, one 127 00:18:25.980 --> 00:18:32.369 Boris Buia: because active learning engages students in the process of learning through activities, in discussions, in class. 128 00:18:32.510 --> 00:18:37.420 Boris Buia: it emphasizes higher order thinking, and often it involves group work. 129 00:18:38.600 --> 00:18:49.159 Boris Buia: Freeman Scott et Al. In 2,014, conducted a meta-analysis of 225 studies on examination scores and failure rates 130 00:18:49.730 --> 00:18:57.150 Boris Buia: when comparing student performance in undergraduate stem courses with traditional lecturing versus active learning sections. 131 00:18:57.540 --> 00:18:59.330 Boris Buia: The results are clear. 132 00:18:59.680 --> 00:19:04.880 Boris Buia: Average examination scores improved by about 6% in active learning sections. 133 00:19:05.340 --> 00:19:14.690 Boris Buia: Students in classes with traditional lecturing were 1.5 times more likely to fail than were students in classes with active learning. 134 00:19:15.320 --> 00:19:21.159 Boris Buia: The evidence is clear. So the question is, how do we implement active learning in a large enrollment course? 135 00:19:23.080 --> 00:19:25.890 Boris Buia: We start by engaging students 136 00:19:26.250 --> 00:19:38.669 Boris Buia: by chunking. We want to chunk our sessions and our live talk. And we need to. We need to think of chunking as a way to maximize learning efficiency, not as a way to gather 137 00:19:39.020 --> 00:19:48.359 Boris Buia: to weak, generational attention span you want also in your chunk session or live talk model problem solving. 138 00:19:48.470 --> 00:19:59.880 Boris Buia: So that when you're actually learning. You're when you're actually lecturing in your chunk section, you're showing students how to solve concrete problems 139 00:20:00.500 --> 00:20:08.539 Boris Buia: after the chance section, you want to switch the focus from the podium and the slides to the classroom. And you want to elicit students questions. 140 00:20:08.940 --> 00:20:12.569 Boris Buia: You can, for instance, ask students to think about a problem. 141 00:20:12.790 --> 00:20:16.719 Boris Buia: Students should be able to solve in light of what you already shared with them. 142 00:20:16.950 --> 00:20:24.540 Boris Buia: You can ask students to pair with their neighbor and share to the class after they've 3 to 5 min team pair shared activity. 143 00:20:24.810 --> 00:20:29.599 Boris Buia: You may use technology like poll everywhere to pull students on the mightiest point. 144 00:20:29.760 --> 00:20:36.069 Boris Buia: Edu May deploy structured group, learning in a form of a lab which is very common in large enrollment courses. 145 00:20:36.620 --> 00:20:40.279 Boris Buia: This is something you can do during the synchronous sessions or a live talk. 146 00:20:40.490 --> 00:20:52.910 Boris Buia: you can do all of the same, all of the above. In an asynchronous online lecture you can chunk the lecture, and as a Cta, we're going to encourage you to chunk your lecture and help you. As you record the lecture with us. 147 00:20:53.250 --> 00:20:57.720 Boris Buia: you can model problem solving in your pre-recorded sections. 148 00:20:58.030 --> 00:21:07.419 Boris Buia: You can use structured group learning. And you can use the discussion forum to elicit students, questions in relation to specific lectures or the modest point of a lecture. 149 00:21:08.630 --> 00:21:16.380 Boris Buia: You can deploy formative quizzes both in your on-site online courses. And we'll focus more on this later. 150 00:21:17.920 --> 00:21:24.700 Boris Buia: We mentioned a couple of times the use of small groups or labs in large enrollment courses. 151 00:21:25.470 --> 00:21:29.519 Boris Buia: It is a very valuable tool to promote cognitive collaboration. 152 00:21:30.810 --> 00:21:33.429 Boris Buia: It is valuable to enhance critical thinking. 153 00:21:33.660 --> 00:21:43.069 Boris Buia: It does foster the formation of a community of learners, which is all the more important in a larger enrollment courses, because that will increase a sense of belonging 154 00:21:45.080 --> 00:21:48.059 Boris Buia: to promote the quality of small group work. 155 00:21:48.410 --> 00:21:53.100 Boris Buia: You want to provide explicit instruction and timeframe before the lab. 156 00:21:53.500 --> 00:21:57.630 Boris Buia: You want to ask students to assume roles if applicable during the lab. 157 00:21:57.770 --> 00:22:02.069 Boris Buia: And you want to ask groups to share learning outcomes after the lab. 158 00:22:03.270 --> 00:22:13.149 Boris Buia: Now that we have seen a few key strategies to create engaging lecturing sections in how to deploy labs in your course. Let's focus on how to 159 00:22:13.280 --> 00:22:14.850 Boris Buia: create assessments. 160 00:22:15.460 --> 00:22:21.100 Boris Buia: Let's start by making a clear distinction between formative assessment and summative assessments 161 00:22:22.465 --> 00:22:26.160 Boris Buia: assessments for learning, affirmative assessments are 162 00:22:26.500 --> 00:22:34.670 Boris Buia: assessments for learning. They're meant to improve learning. They take place while the learning process is in progress. 163 00:22:35.610 --> 00:22:50.860 Boris Buia: They can be collaborative or student regulated. They provide feedback to both students who are going through the formative assessments and to faculty who can take a snapshot. Our students are doing in relation to formative assessment and take corrective measures if needed. 164 00:22:51.320 --> 00:22:58.139 Boris Buia: Summative assessments on the other end are assessments of learning. They measure learning outcomes. 165 00:22:58.390 --> 00:23:05.579 Boris Buia: they take place more often than not at the end of the learning process, and they are focused on learning products. 166 00:23:07.060 --> 00:23:11.999 Boris Buia: They are more often than not instructor, director, and they are part of the final evaluation. 167 00:23:12.630 --> 00:23:17.370 Boris Buia: A few examples of formative assessment. We already seen team per share 168 00:23:17.770 --> 00:23:24.610 Boris Buia: polling that can be used for the magist point discussion, forum polls we mentioned on work 169 00:23:25.160 --> 00:23:31.860 Boris Buia: quizzes, empathics, exams are very common format of formative assessments in large enrollment courses. 170 00:23:32.160 --> 00:23:58.989 Boris Buia: Formative assessments are typically low, state graded or ungraded assessments an important point to consider, even though they are ungraded. You still want to use score quizzes and practice exams, so the students may take these practice, quizzes and exams receive a score, even though the score is not part of their final grade, and this will help students become self-regulated learners as they go through the process of taking. These practice quizzes and exams 171 00:24:00.443 --> 00:24:04.930 Boris Buia: formative assessment can be used to create connection between faculty and students. 172 00:24:05.715 --> 00:24:11.890 Boris Buia: If the students are allowed to take them in group. They can be a student to student community 173 00:24:12.420 --> 00:24:21.149 Boris Buia: they reinforce learning. Formative assessments should be aligned with modular electoral learning objectives to help students navigate your course. 174 00:24:21.920 --> 00:24:29.470 Boris Buia: and there is evidence that there is a positive correlation between onboard and exam performance. 175 00:24:30.410 --> 00:24:33.250 Boris Buia: Research has shown that 176 00:24:34.410 --> 00:24:43.959 Boris Buia: what we can define as a testing effect. And so that repeated testing leads to greater memory retention of material than repeated study alone. 177 00:24:44.820 --> 00:24:56.200 Boris Buia: Also, in relation to formative assessment. We have plenty of evidence that practice exam should mirror graded exams to increase students, confidence and familiarity with the exam. 178 00:24:57.090 --> 00:25:02.619 Boris Buia: and in relation to online multiple choice quizzes and practice exams with automatic grading. 179 00:25:02.770 --> 00:25:11.520 Boris Buia: There is evidence because these formative assessments, I'm providing students with instantaneous feedback regarding their performance. 180 00:25:11.670 --> 00:25:16.490 Boris Buia: They are a great tool to help students become self-regulated learners. 181 00:25:17.160 --> 00:25:21.279 Boris Buia: A self-regulated learner is a student who can generate internal feedback 182 00:25:21.640 --> 00:25:33.990 Boris Buia: as they go through the process of engaging with informative assassins. They will understand what they are doing well where they need to improve, and they will know where to find their resources to improve. 183 00:25:34.260 --> 00:25:48.669 Boris Buia: And to this effect. Research has recommended, for instance, to provide students with a table with quizzes and practice text that direct students to the relevant learning material for a given question or a given set of quizzes 184 00:25:49.810 --> 00:26:04.750 Boris Buia: in relation to questions, research has shown that it is important and pedagogically sound to create multiple questions addressing similar topics, and to align these questions with lectures and course, learning objectives 185 00:26:05.380 --> 00:26:30.730 Boris Buia: in relation to the sum of the most complex questions that quizzes may have multiple choice questions. Researchers share best practices. And at the end of the presentation we're going to share with you a great webinar on how to draft multiple choice questions. And Ctl. Created Pdf. With step-by-step instructions on how to create multiple choice questions 186 00:26:31.230 --> 00:26:39.240 Boris Buia: by creating for your large enrollment course a significant bank of questions 187 00:26:39.580 --> 00:26:49.339 Boris Buia: that are aligned with your learning objectives. You're gonna have plenty of questions to use in your formative assessments that could be a team paid or share during a session. 188 00:26:49.450 --> 00:26:56.050 Boris Buia: You're going to have questions for your practice exam and your summative exams. Deploying these 189 00:26:56.780 --> 00:27:04.319 Boris Buia: complex set of formative and summative assessments will help students become self-regulated learners. 190 00:27:05.370 --> 00:27:10.239 Boris Buia: Let's focus now on summative assessments 191 00:27:10.780 --> 00:27:21.249 Boris Buia: in larger German classes we already mentioned exams a few times. You also want to consider that other forms of summative assessments can be research 192 00:27:21.830 --> 00:27:27.100 Boris Buia: projects such as a policy, brief presentations, either synchronous 193 00:27:27.760 --> 00:27:40.450 Boris Buia: presentations in offer during a live talk or a synchronous presentations. Often during a voice trade via voice trade portfolios. Typically summative assessments are high stakes 194 00:27:41.580 --> 00:27:42.540 Boris Buia: in 195 00:27:42.720 --> 00:27:51.699 Boris Buia: in relation to large enrollment classes. Research has shown that exams are a very common form of summative assessions. 196 00:27:51.870 --> 00:27:59.730 Boris Buia: Given the ease of automated scoring, the minimization of grading bias and the strong assessment validity that exams are offering. 197 00:28:01.580 --> 00:28:07.790 Boris Buia: you can use exams to quiz and test students at a lower level, learning 198 00:28:08.010 --> 00:28:18.210 Boris Buia: where you can use questions to test students on remembering and understanding, you can craft questions to test 199 00:28:18.400 --> 00:28:21.270 Boris Buia: and quiz students at higher level learning. 200 00:28:21.520 --> 00:28:28.470 Boris Buia: such as applying, analyzing, and evaluating researchers, shared that stocking 201 00:28:28.870 --> 00:28:38.589 Boris Buia: whereby stacking, we mean inside a single question. You're requiring students to apply more than one concept, more than one set of skills. More than one decision 202 00:28:38.980 --> 00:28:48.180 Boris Buia: is a very valuable solution to engage students at the highest level, evaluating and analyzing in our courses. It's common to have 203 00:28:48.560 --> 00:28:59.080 Boris Buia: one or more set of data sets and a question blocks. And in relation to each question of the question, block students are asked to engage with the data sets 204 00:28:59.500 --> 00:29:03.180 Boris Buia: at different levels of learning. 205 00:29:04.990 --> 00:29:22.060 Boris Buia: A few additional consideration in relation to summative assessments, and especially in relation to ice, take exams, communicate clearly dates and time, so the students can plan accordingly, let students know whether the exams will be time box or not, whether they're going to be open book or closed book. 206 00:29:22.260 --> 00:29:27.549 Boris Buia: whether students are allowed to use resources. And if so, which resources are they allowed to use 207 00:29:27.790 --> 00:29:36.859 Boris Buia: whether you're going to break down the exam in question blocks everything. So what are the main topic or topics? What is the main topic, or the main topics of each question block. 208 00:29:37.080 --> 00:29:39.929 Boris Buia: are you going to use randomization in your exam? 209 00:29:40.870 --> 00:30:01.500 Boris Buia: Importantly, you want to avoid scheduling conflicts because it's already stressful for a student to take a summative exam in a large enrollment course, it will be even more so if they have to take more than one exam. So avail yourself of the school, large enrollment class exam calendar that we have created. To avoid this conflict. 210 00:30:01.780 --> 00:30:03.300 Boris Buia: Let's switch now 211 00:30:03.810 --> 00:30:16.289 Boris Buia: projects or open ended assessments, assummative assessments. This is where the teaching team can test with students at the highest level, creating and evaluating 212 00:30:16.430 --> 00:30:23.809 Boris Buia: and research has shown that there are a few important principles that you want to consider to help students succeed 213 00:30:24.030 --> 00:30:29.340 Boris Buia: while they are taking this assessment. Scaffolding, so break down your assessment 214 00:30:29.580 --> 00:30:35.679 Boris Buia: in steps, in relation to each step, provide clear guidance, timelines. 215 00:30:36.140 --> 00:30:39.800 Boris Buia: rubrics, and then examples of good assessments 216 00:30:39.990 --> 00:30:50.349 Boris Buia: also, in preparation for each step of the open-ended assessments, offer a live talk or a session to explain to students the guidelines 217 00:30:50.670 --> 00:31:04.619 Boris Buia: to go over the rubrics and illustrate how the rubrics have been used and deployed in relation to the examples of good assessments, so that everybody in your course knows how you're going to evaluate this assessment. 218 00:31:05.980 --> 00:31:20.860 Boris Buia: also very valued in the evaluations of larger enrollment courses is a mini session of explaining the pitfalls, so the students know what to avoid, to succeed when they are taking open-ended assessments. 219 00:31:22.020 --> 00:31:29.170 Boris Buia: Let's focus now on grading and offering a timely feedback in relation to large enrollment courses. 220 00:31:29.430 --> 00:31:34.579 Boris Buia: 1st and foremost, how do we define good feedback pedagogically sound feedback? 221 00:31:34.760 --> 00:31:41.439 Boris Buia: You need to have 3 key pieces of information. You need to let students know what good performance is. 222 00:31:41.630 --> 00:31:47.600 Boris Buia: You need to let students know how their current performance relates to good performance. 223 00:31:47.940 --> 00:31:51.509 Boris Buia: And lastly, you want to let students know 224 00:31:52.030 --> 00:31:57.239 Boris Buia: what they need to do to close the gap between their current performance and good performance. 225 00:31:58.060 --> 00:32:00.760 Boris Buia: Key tool that you want to use 226 00:32:00.880 --> 00:32:07.789 Boris Buia: in this process is grading rubrics. You want to share the rubrics prior to the assessment. As we have mentioned. 227 00:32:08.160 --> 00:32:19.989 Boris Buia: you want to share the rubrics after the assessments, so the students can identify, and their performance in relation to good performance. And as self-regulated learners 228 00:32:20.340 --> 00:32:23.010 Boris Buia: can understand what they need to improve. 229 00:32:24.520 --> 00:32:31.849 Boris Buia: Also, rubrics will help the teaching team to streamline the grading process and render the grading more consistent 230 00:32:33.750 --> 00:32:49.420 Boris Buia: in relation to large enrollment courses. We receive a common questions, what are the best practices for grade norming to make sure that all the members who are engaged in grading possibly hundreds of open-ended assessments 231 00:32:50.350 --> 00:32:54.340 Boris Buia: in a matter of days can do so consistently 232 00:32:55.010 --> 00:32:59.640 Boris Buia: so for lack of better definition, we can call this a 2 steps approach. 233 00:32:59.760 --> 00:33:05.130 Boris Buia: During your 1st meeting you want to have the most experienced member of the grading team 234 00:33:05.350 --> 00:33:12.359 Boris Buia: share with the new members of the grading team old anonymous assessments, a set of rubrics. 235 00:33:12.570 --> 00:33:22.590 Boris Buia: the grade and the feedback that was shared in relation to this assessment the 1st meeting should focus on sharing with them the grading process, and how they took place. 236 00:33:24.200 --> 00:33:28.590 Boris Buia: The second meeting should be a meeting that is focused on 237 00:33:28.930 --> 00:33:31.370 Boris Buia: the new members of the grading team 238 00:33:32.160 --> 00:33:44.320 Boris Buia: who are going to apply the same rubrics to a different batch of old anonymous assessments that are not graded in relation to which there is no feedback, and the second meeting should give an opportunity 239 00:33:44.640 --> 00:33:51.739 Boris Buia: for the most experienced member of the grading team to review and share feedback with the new members of the teaching team 240 00:33:52.160 --> 00:33:55.930 Boris Buia: a few other aspects to consider after the assessment 241 00:33:56.600 --> 00:34:04.729 Boris Buia: for once the grading is completed, you want to turn around the graded assessments in a timely manner. 242 00:34:05.160 --> 00:34:09.720 Boris Buia: This will be important to provide students with a timely feedback. 243 00:34:10.000 --> 00:34:21.410 Boris Buia: so the students have time to go over the feedback, process it and adopt the feedback for their next assessment. And doing this will help students become self regulator 244 00:34:22.010 --> 00:34:23.050 Boris Buia: learners. 245 00:34:23.340 --> 00:34:29.629 Boris Buia: You also want to consider offering a live talk or a synchronous sessions to review the assessment. 246 00:34:29.780 --> 00:34:35.229 Boris Buia: For instance, in relation to summative exams, you can turn these sessions into 247 00:34:35.750 --> 00:34:42.130 Boris Buia: open discussion where the teaching team goes over all the questions of the final exam one by one. 248 00:34:42.489 --> 00:34:48.090 Boris Buia: and elicits from students questions, doubts, and addresses these questions and doubts. 249 00:34:48.440 --> 00:34:59.270 Boris Buia: This is an engaging way of reviewing the exam, and it is very valued by students. You also want to consider recording this live talk or session, so that it is available for students 250 00:34:59.590 --> 00:35:00.780 Boris Buia: where maybe 251 00:35:00.990 --> 00:35:08.529 Boris Buia: taking the final exam so they can review how the feedback they received during the review, the meter, exam. For instance. 252 00:35:08.840 --> 00:35:13.160 Boris Buia: another aspect to consider. You want to share data on grade distribution. 253 00:35:13.900 --> 00:35:21.689 Boris Buia: and you want to offer office hours, especially after having shared with students. If somebody graded assessment. 254 00:35:22.660 --> 00:35:28.569 Boris Buia: And with this we have seen how to design in a 255 00:35:29.320 --> 00:35:35.690 Boris Buia: offer a larger learning course. We have seen them to connect with students, how to transform. 256 00:35:35.980 --> 00:35:41.059 Boris Buia: possibly passive learning experience to an active learning experience while delivering lectures. 257 00:35:41.510 --> 00:35:56.400 Boris Buia: and how to create assessments and how to grade. So I hope that when you're going to think about a larger online course, you will be able to avail yourself of these best practices and research that we're going to share with you after the meeting. 258 00:35:57.061 --> 00:36:02.938 Boris Buia: Let me conclude by just pointing out that we have at Ctl. 3 grade 259 00:36:03.640 --> 00:36:26.950 Boris Buia: webinars on formative assessment what was offered recently by Emily and Lauren. We have a webinar on multiple choice questions that was offered by Mia and Lou, and on top of that we're gonna share a Pdf, and we also have a webinar on rubrics that was offered by Amy. And we're going to share links to all of them with resources in relation to each and everyone. 260 00:36:27.440 --> 00:36:35.179 Boris Buia: And, Amy, I'm gonna pass it to you now and let me go to the relevant slide. 261 00:36:36.410 --> 00:36:38.880 Amy Pinkerton (BSPH CTL Teaching Toolkit): Thank you. Thank you, Boris. 262 00:36:38.950 --> 00:37:07.910 Amy Pinkerton (BSPH CTL Teaching Toolkit): I 1st wanted to highlight some other continuing professional development opportunities. As Boris mentioned, we have a lot of past workshops that you can check out on our on demand videos page, we'll also share links to the resources mentioned in today's workshop in our follow-up email. So keep an eye out for that in your email inbox. I also wanted to highlight. We have some training and professional opportunities for faculty and teaching assistants. 263 00:37:07.910 --> 00:37:33.390 Amy Pinkerton (BSPH CTL Teaching Toolkit): Especially, I wanted to highlight the teaching Assistantship training course. It's an online, self-paced modular course that prepares students to become to take on the roles and responsibilities of becoming a ta at Bsph. And completion of the Ta training course results in a certificate and our newest, our current cohort just started a few weeks ago. So it's a great time to join that course. 264 00:37:33.590 --> 00:37:36.270 Amy Pinkerton (BSPH CTL Teaching Toolkit): and then I think it's the next slide. 265 00:37:38.340 --> 00:37:59.049 Amy Pinkerton (BSPH CTL Teaching Toolkit): Yes, we also. We would love to hear your feedback. So please take a moment to complete this anonymous workshop evaluation survey. It's a really quick survey, but it gives us the. It gives us feedback about how this workshop went, and then we also use this feedback to plan future workshops. So we appreciate your feedback. 266 00:37:59.400 --> 00:38:05.740 Amy Pinkerton (BSPH CTL Teaching Toolkit): but otherwise I think I'll pass it back to Boris to for open Q. And a Thank you. 267 00:38:09.457 --> 00:38:11.069 Boris Buia: Thank you, Amy. 268 00:38:12.200 --> 00:38:26.930 Boris Buia: So if you have any questions in relation to your large enrollment course, or you're just curious about some aspects of the presentation. Please let me know. We'll be more than happy to address these questions. If you would like to have access to more resources, we can definitely 269 00:38:27.260 --> 00:38:29.240 Boris Buia: look for more resources. 270 00:38:38.380 --> 00:38:48.199 Boris Buia: I'm going over the chat now to make sure that did. I didn't questions that we didn't address. Okay 271 00:38:51.722 --> 00:38:58.600 Boris Buia: one question that you may have is about the use of generative AI in a large enrollment course. 272 00:38:59.845 --> 00:39:04.720 Boris Buia: So in course, plus, we have a listed tool that can be useful. 273 00:39:04.930 --> 00:39:09.630 Boris Buia: and it is a tool that you can use in relation to pre-recorded lectures. 274 00:39:11.100 --> 00:39:13.439 Boris Buia: It is inside the lecture material box. 275 00:39:13.950 --> 00:39:21.060 Boris Buia: It is a tool that will generate for you a set of questions based on the transcript of election. 276 00:39:21.670 --> 00:39:26.800 Boris Buia: and sometimes I receive questions in relation to how to deploy this tool. 277 00:39:26.950 --> 00:39:37.699 Boris Buia: So a couple of considerations, because larger enrollment courses may run, especially with pre-recorded lectures, may run anywhere from 15 to 25 lectures. 278 00:39:38.420 --> 00:39:44.059 Boris Buia: so make sure that the lectures have been recorded in a relatively recent period of time. 279 00:39:44.200 --> 00:39:48.710 Boris Buia: That's a key factor in relation to the presenters. 280 00:39:48.890 --> 00:39:59.269 Boris Buia: If you have multiple presenters who recorded these lectures you want to verify to some degree that they've used technical complex terminology consistently. 281 00:39:59.730 --> 00:40:10.809 Boris Buia: if not using the I tool, may create some inconsistencies the way terms are defined in the questions that are generated by the generative AI tool. 282 00:40:11.640 --> 00:40:14.959 Boris Buia: Another aspect to consider. 283 00:40:15.980 --> 00:40:25.829 Boris Buia: Are you using a textbook with your course? Sometimes there is a recommended textbook, and new edition may change the way things are explained. 284 00:40:26.430 --> 00:40:31.779 Boris Buia: and these students in larger learning courses may use the test book as a reference. 285 00:40:31.900 --> 00:40:35.120 Boris Buia: and if you are using lectures. 286 00:40:35.550 --> 00:40:43.419 Boris Buia: it perhaps may reference to an older version of the textbook, and on top of that you created generative AI generated 287 00:40:43.840 --> 00:40:48.699 Boris Buia: quiz that might increase confusion. So these are some aspects to consider 288 00:40:49.524 --> 00:40:57.170 Boris Buia: another question that we receive unless there are other question posted to the chat, it doesn't seem that is the case. 289 00:40:58.840 --> 00:41:01.659 Boris Buia: Do we use in lecture quizzes? 290 00:41:02.290 --> 00:41:05.299 Boris Buia: This is something important, because 291 00:41:05.570 --> 00:41:13.459 Boris Buia: again, you may have 15 to 25 lectures in a course. Each lecture may have 4 to 5 sections. 292 00:41:13.680 --> 00:41:25.140 Boris Buia: and each lecture may pack quite a lot of complex content. So if you're going to use in lecture quizzes, make sure to have a comprehensive plan for the entire course. 293 00:41:25.410 --> 00:41:39.530 Boris Buia: because it can be easily be a situation where you end up with 40, 50, 16 lecture quizzes. So if you want to experiment with that, I would say, start small and consider pros and cons of having. 294 00:41:39.800 --> 00:41:50.030 Boris Buia: on top of your practice quizzes on top of your practice exams another set of 40 to 50 quizzes, because it might take a little bit to get familiar 295 00:41:50.150 --> 00:41:51.879 Boris Buia: with all these quizzes 296 00:41:52.720 --> 00:42:03.789 Boris Buia: in relation again. If there are no questions in relation to in lecture quizzes, there is another aspect to consider. We have courses where, the turnaround 297 00:42:03.920 --> 00:42:07.969 Boris Buia: of the main instructor is significant, and as a consequence. 298 00:42:08.611 --> 00:42:24.160 Boris Buia: often new instructors re-record the lectures, sometimes changing the content, and at that point your existing in lecture quizzes will need to be recreated and realigned with the newly recorded lectures. 299 00:42:34.840 --> 00:42:36.069 Boris Buia: go ahead, I mean. 300 00:42:36.190 --> 00:43:01.140 Amy Pinkerton (BSPH CTL Teaching Toolkit): Of course I have a question. So you are often the instructional designer for large enrollment courses. So if let's say, we have a faculty member who's really busy, and they can only make one or 2 changes, each offering. What would you prioritize as the top thing to do to engage students if they were worried about engaging their students, but could only make one or 2 301 00:43:01.520 --> 00:43:06.770 Amy Pinkerton (BSPH CTL Teaching Toolkit): changes, or add one or 2 more things to their to-do list, each offering. 302 00:43:07.490 --> 00:43:10.559 Boris Buia: In. In the case of online courses. 303 00:43:10.850 --> 00:43:15.390 Boris Buia: I would say, offer more opportunities for students to engage with the teaching team. 304 00:43:15.770 --> 00:43:20.590 Boris Buia: and that could be more live talks, or if you have labs, model apps. 305 00:43:20.980 --> 00:43:24.189 Boris Buia: making sure that all the lab instructors 306 00:43:24.500 --> 00:43:28.209 Boris Buia: are on the same page in terms of 307 00:43:28.910 --> 00:43:34.210 Boris Buia: what to pay attention to to engage students that would be a low hanging fruit 308 00:43:34.450 --> 00:43:36.700 Boris Buia: that can be taken care of 309 00:43:37.120 --> 00:43:42.740 Boris Buia: in relation to on-site courses. I would recommend to 310 00:43:43.090 --> 00:43:49.229 Boris Buia: chunk the session as much as possible, and open the session for an 311 00:43:49.640 --> 00:43:56.360 Boris Buia: Q&A a list of 3 to 5 min every 15 to 20 min and 312 00:43:56.700 --> 00:44:06.549 Boris Buia: in relation to labs. There are a few things that can be done you want to make sure at the beginning of the term that there is a clear 313 00:44:06.730 --> 00:44:10.499 Boris Buia: plan in terms of what resources will be shared for each lab. 314 00:44:10.870 --> 00:44:25.160 Boris Buia: It may happen that you have lab instructors who are sharing different sets of slides, and that may create confusion and disengage students. So make sure that you have a common set of slides for all the labs. 315 00:44:26.410 --> 00:44:38.499 Boris Buia: Also, as we are mentioning labs and engagement, we have courses where you have labs that are offered on the same date and time, consistently through the term, and that is the ideal scenario. 316 00:44:38.650 --> 00:44:41.140 Boris Buia: We have courses where labs are offered 317 00:44:41.380 --> 00:44:44.440 Boris Buia: at different dates and times during the week 318 00:44:44.550 --> 00:44:51.789 Boris Buia: for different reasons, and in that case you want to consider as you design your court schedule 2 aspects. 319 00:44:51.950 --> 00:44:53.270 Boris Buia: the learning flow. 320 00:44:53.660 --> 00:45:06.760 Boris Buia: So you want to make sure the students have enough time to study the lecture the section, the reading material, so that they are prepared when they join the lab, and they can be actively engaging during the lab. 321 00:45:07.400 --> 00:45:16.979 Boris Buia: Another aspect to consider in relation to large enrollment course. When you are designing the course schedule page is paying attention to what we can consider degrading flow. 322 00:45:16.990 --> 00:45:40.400 Boris Buia: After the assessment, the teaching team will need time to grade hundreds of assessments, especially open-ended assessment. So as you design the syllables. Sorry the schedule page pay attention to this factor, because it will make for a better, more engaging experience and hopefully more reasonable experience for the teaching team as well 323 00:45:40.840 --> 00:45:44.439 Boris Buia: and in terms of learning flow 324 00:45:46.830 --> 00:45:49.049 Boris Buia: a few other things to consider. 325 00:45:49.290 --> 00:45:55.190 Boris Buia: In case you have students who need to take labs at different dates and times. 326 00:45:55.850 --> 00:46:01.860 Boris Buia: it will help to make available the lab material that they need as early as possible. 327 00:46:02.570 --> 00:46:04.660 Boris Buia: If we'll have to ensure 328 00:46:04.820 --> 00:46:22.789 Boris Buia: that all the lab instructors are aware of this difference in scheduling and times, and they are aware of what students need to know by the end of the lab, so they can take the assessments. So this is possibly more than 2 things, but if they have to choose, at least they have a few options to choose. 329 00:46:23.010 --> 00:46:34.969 Boris Buia: and it should matter. It should be a matter of redesigning. Perhaps the course schedule, offering a few more live tours, coordinating with lab instructors. In the case. You have a complex set of labs in your course. 330 00:46:36.400 --> 00:46:37.959 Amy Pinkerton (BSPH CTL Teaching Toolkit): Alright! Thank you, Forrest. 331 00:46:38.040 --> 00:46:39.169 Boris Buia: Thank you. Amy. 332 00:46:41.480 --> 00:46:43.670 Boris Buia: Okay, perfect. Go ahead, Emily. 333 00:46:44.180 --> 00:47:02.389 Emily Haagenson: I have a clarifying question on on one of your last slides, you said. It's important to share data on grade distribution with the summative assessments. Did you? Is that important to share with students or across the teaching team? Or can you just say more about why that matters. 334 00:47:02.390 --> 00:47:06.410 Boris Buia: Yes, thank you. So you want to share with students. 335 00:47:06.790 --> 00:47:12.490 Boris Buia: And you want to share with the key members of the teaching team. 336 00:47:12.650 --> 00:47:19.789 Boris Buia: so that everybody's on the same page in terms of students are doing and performing in relation to this course. This is 337 00:47:20.120 --> 00:47:27.279 Boris Buia: all the more important if you're offering a large enrollment course, if it is a part of a series of courses. 338 00:47:27.840 --> 00:47:30.830 Boris Buia: because you may have your larger learning course during turn one. 339 00:47:30.940 --> 00:47:42.629 Boris Buia: then you're going to have the same court of students who are taking in term 2, the follow-up course. And so tracking this across terms will help you understand how students are doing 340 00:47:42.910 --> 00:47:47.930 Boris Buia: so it's a matter of communicating both in relation to the students and sharing with the students and 341 00:47:48.120 --> 00:47:59.260 Boris Buia: of keeping track among the members of the teaching team. Sometimes you have a series of courses that are offered by different teaching teams from one time to the next. So 342 00:47:59.390 --> 00:48:15.260 Boris Buia: it's important to have some key pieces of information to let the next teaching team who is going to take over the second term, for instance, if you offer term one how students are doing and where they are experiencing some pain points. 343 00:48:17.610 --> 00:48:18.370 Boris Buia: Thank you. 344 00:48:27.740 --> 00:48:30.712 Lauren Dana: I have a I have a question as well, Boris. 345 00:48:31.180 --> 00:48:38.440 Lauren Dana: So as the instructional designer. When the students have or submit their evaluations. 346 00:48:38.730 --> 00:48:53.659 Lauren Dana: it is a lot to go through. Do you have any recommendations on how to approach? I mean, it's a lot different to look through evaluations for a 20 person course than like a 350 person course. Quantitative data is easy. But those qualitative. 347 00:48:53.810 --> 00:48:58.480 Lauren Dana: Do you have any recommend? How do you approach it when you're trying to think about ways to improve the course. 348 00:48:58.860 --> 00:49:01.680 Boris Buia: Yes, double your daily intake of chocolate. 349 00:49:02.924 --> 00:49:06.560 Boris Buia: That's it. The 1st step, the second step, is 350 00:49:07.390 --> 00:49:13.640 Boris Buia: right now, through our pins we have access to some generative AI tools that are recommended. 351 00:49:13.800 --> 00:49:20.099 Boris Buia: And so we can start considering using these tools to perhaps generate a summary. 352 00:49:20.470 --> 00:49:32.560 Boris Buia: but for large enrollment courses that ran in their offer, and they value 3 to 5 credits. I actually go through all the evaluations 353 00:49:32.790 --> 00:49:36.759 Boris Buia: minorly. There is no summary they can substitute for that. 354 00:49:37.020 --> 00:49:47.549 Boris Buia: And then I kind of digest a process at this evaluation and share key points with the main instructors of the teaching teams. 355 00:49:48.870 --> 00:49:53.949 Boris Buia: Hopefully, we will develop tools that make it possible 356 00:49:54.552 --> 00:50:03.009 Boris Buia: to consistently and reliably use generative AI tools to create a summary. But I will still use the main resource. 357 00:50:03.980 --> 00:50:14.229 Boris Buia: It requires a little bit of timing, I would say, if you're working with a series of courses, try to go over these evaluations and think of them as a series that will help you. 358 00:50:15.100 --> 00:50:22.120 Boris Buia: As you become more familiar with analyzing these evaluations, and as you are familiar with the course, you will know. 359 00:50:22.634 --> 00:50:30.569 Boris Buia: what to focus on in relation to the evaluations in light of how the course is set up, what are the main assessments. 360 00:50:31.030 --> 00:50:36.500 Boris Buia: and you will know what to consider as a pain point that can be improved 361 00:50:37.500 --> 00:50:42.270 Boris Buia: based on the capacity of the teaching team. As Amy mentioned. That's a big constraint. 362 00:50:43.670 --> 00:50:44.490 Lauren Dana: Thank you. 363 00:50:44.660 --> 00:50:45.860 Boris Buia: Thank you, Lauren. 364 00:51:03.550 --> 00:51:07.790 Amy Pinkerton (BSPH CTL Teaching Toolkit): All right. Well, I'm not seeing any more questions or hands up. 365 00:51:08.480 --> 00:51:24.890 Amy Pinkerton (BSPH CTL Teaching Toolkit): so if there are no more questions, we can go ahead and close out the workshop. But, Boris, thank you so much. This was really enlightening about how? How all of the the elements and the thought that goes into designing and running a large enrollment course. So thank you so much. 366 00:51:25.604 --> 00:51:31.079 Boris Buia: Thank you, everybody, and we shared slides and resources at a later time. Thank you.