WEBVTT

1
00:00:02.890 --> 00:00:19.070
Amy Pinkerton (BSPH CTL Teaching Toolkit): All right. Hello, and welcome to today's picnic Pd workshop. This workshop actually marks the midpoint of our summer picnic. Pd series that happens on Tuesdays, in June and July. If you miss any of our previous workshops, you can catch those, the recordings of those

2
00:00:19.430 --> 00:00:33.469
Amy Pinkerton (BSPH CTL Teaching Toolkit): on our on demand workshop videos page. And today's workshop is the developing learner using formative assessment to facilitate learning and connection. And now I'll pass to our speakers to introduce themselves.

3
00:00:34.310 --> 00:00:55.339
Emily Haagenson: Hello! All thanks for joining us today for this session of picnic. Pd, I'm Emily Hagenson, and this is my colleague, Lauren Dana. We're both instructional designers with the Bsph Center for teaching and learning. And we're eager to get into the value and practicalities of formative assessment with you today. Specifically, we're going to talk about knowledge checks.

4
00:00:56.680 --> 00:01:18.520
Emily Haagenson: So my hope is that by the end of this session we will have convinced you of the usefulness of using knowledge checks both as a formative assessment for developing, but also connecting with your learners, and that you'll feel confident you can create a knowledge check for your courses and use the data from the knowledge checks to improve your courses.

5
00:01:19.360 --> 00:01:36.769
Emily Haagenson: big goals, but we can do it. We're going to start off with an example. Maybe some of this will feel relatable. So imagine, as a summative assessment or a final project. You ask students to play the role of a health minister in a chosen country.

6
00:01:37.180 --> 00:02:03.499
Emily Haagenson: They need to make a case for a specific adolescent health program amid competing funds. So you expect your learners to present a problem statement with clear goals and objectives. Apply Kirby's Bdi logic model as they describe an epidemiological situation and a demographic dividend. You want them to do all of this while providing clear scope and approach for a proposed intervention.

7
00:02:04.220 --> 00:02:11.899
Emily Haagenson: This is a robust, summative assessment. Lots of competing components calling for your feedback and support

8
00:02:12.400 --> 00:02:30.159
Emily Haagenson: as you begin grading, you get frustrated. Several of your learners are misapplying the logic model or skipping steps of it altogether. A lot of the feedback you're giving is about helping them restructure their argument to fit the model. And you don't have the time or the bandwidth to provide feedback

9
00:02:30.410 --> 00:02:40.930
Emily Haagenson: on their ideas or the development of their suggested intervention. When you reach out to your students to clarify, your students are angry and defensive.

10
00:02:41.520 --> 00:02:46.429
Emily Haagenson: This is our. This is our hypothetical but possibly relatable scenario. So

11
00:02:46.690 --> 00:02:57.630
Emily Haagenson: I'm going to explain to you guys today why using knowledge checks could help prevent this situation from happening and improve your students. Summative assessments overall.

12
00:02:58.190 --> 00:02:59.040
Emily Haagenson: Okay.

13
00:03:00.600 --> 00:03:23.280
Emily Haagenson: specifically, today, we're going to look closely at knowledge checks which are just one variety of formative assessment. But generally formative assessment checks learners, necessary base level knowledge. It's an opportunity for students to share what they know and receive feedback or support before they apply the knowledge in a summative assessment like a test or a major project.

14
00:03:23.860 --> 00:03:34.930
Emily Haagenson: Therefore formative assessments don't need to be and arguably shouldn't be. We can talk about that later. If you want graded based on getting a correct answer the 1st time

15
00:03:35.120 --> 00:03:48.119
Emily Haagenson: they're supposed to be developmental supports. Both learners and instructors can use the results of a knowledge check to inform individual or collective progress.

16
00:03:49.100 --> 00:04:07.460
Emily Haagenson: So knowledge checks are very often simple quizzes with an immediate and automatic grading and feedback. They're meant to scaffold deeper learning and overall. The purpose is to provide our learners with personal and academic check-ins that they're craving.

17
00:04:10.040 --> 00:04:20.490
Emily Haagenson: There are many benefits to using knowledge checks in your course, and in particular, when you have an online course or a course that relies on asynchronous knowledge, production.

18
00:04:21.380 --> 00:04:49.410
Emily Haagenson: knowledge checks are an excellent form of self assessment. Self-assessment is especially useful for adult learners who want an adaptive, self-regulatory process. They want a process responsive to their specific context, and they want their learning to be iterative but self-generated. All that to say, we can provide structures that scaffold this learning, this individualized learning journey for them

19
00:04:50.060 --> 00:04:59.359
Emily Haagenson: as our learners review basic content within a logical timeline that works for them. We can give them structure and reinforcement through knowledge checks.

20
00:04:59.470 --> 00:05:08.219
Emily Haagenson: Their memories are activated through this retrieval process. This helps improve their retention and connection to our material

21
00:05:09.180 --> 00:05:38.859
Emily Haagenson: research also shows that frequency of engaging with self-assessment resources has a significant impact on learning, regularly offering knowledge checks, provides supportive pacing of learning by offering continual touch points throughout a learning experience, and the automated grading and feedback provide learners with an opportunity to receive, albeit passive validation from you that they're on the right track.

22
00:05:38.930 --> 00:05:46.030
Emily Haagenson: When the automated feedback is done well, it can feel like a small moment of connection between the learner and the teaching team

23
00:05:46.710 --> 00:05:48.670
Emily Haagenson: as a teaching team.

24
00:05:49.460 --> 00:06:00.520
Emily Haagenson: Our learner self-assessment is a good opportunity for us to think about using our time. Well, not all student learning requires active engagement from us.

25
00:06:01.020 --> 00:06:11.589
Emily Haagenson: So we should allocate our energy toward a course. Intentionally, knowledge checks allow us to monitor academic understanding. While students are getting the check-in and feedback they need.

26
00:06:12.180 --> 00:06:15.400
Emily Haagenson: So I want you to take a moment and think back to our example.

27
00:06:15.510 --> 00:06:22.150
Emily Haagenson: What might have gone differently if the students in the example, had had a knowledge check

28
00:06:22.330 --> 00:06:31.579
Emily Haagenson: on the Bdi logic model early in the term. How might how might that have helped the situation? And we'll get more into that in just a second. But keep that in mind.

29
00:06:33.380 --> 00:06:47.530
Emily Haagenson: We can also use knowledge checks to communicate with our learners directly. So knowledge checks are a good opportunity to communicate clearly what the most salient takeaways are from a lecture a week, or a module

30
00:06:47.530 --> 00:07:05.710
Emily Haagenson: similar to flagging information in a learning objective. If a topic is covered in a knowledge check. It says to the student, Hey, this is important. Pay attention to this. This is super helpful. As our students experience our courses and organize the material for their own learning.

31
00:07:05.800 --> 00:07:11.239
Emily Haagenson: Also, when one of these key topics is brought to their attention

32
00:07:11.470 --> 00:07:22.869
Emily Haagenson: and they don't understand it, it flags for them that they need to make sure they do understand it, pivot their learning, seek resources, ask questions, or review content

33
00:07:23.390 --> 00:07:45.520
Emily Haagenson: for the teaching team again, we don't need to review every student's knowledge check. So reviewing the data will indicate what just in time adjustments we can make to address any trends or provide individual support. And this truly provides formative feedback throughout their learning journey. Our data will show

34
00:07:45.680 --> 00:07:49.680
Emily Haagenson: where we can make adjustments to make our course more clear as well.

35
00:07:52.750 --> 00:08:15.569
Emily Haagenson: There are many benefits to using knowledge checks in our courses. So, thinking back to our example, if we had used a knowledge check early on in the term, attention to the model would have communicated to students, hey, you need to flag the Bdi logic model. This is important. This would have activated their attention and energy resulting in more expert application.

36
00:08:15.740 --> 00:08:26.049
Emily Haagenson: The collective foundational competency would have been established early by the teaching team. We would have ensured with greater confidence that all the students know the model and how to apply it.

37
00:08:26.640 --> 00:08:44.690
Emily Haagenson: Specifically, we would have been able to skim data and identify the trend in misunderstanding around the logic model and address it in a timely fashion. Right? Maybe via a course email or synchronously in class or during a live talk.

38
00:08:45.000 --> 00:08:53.439
Emily Haagenson: Some individuals. More significant misunderstandings would also have been repaired before they were misapplied in our summative assessment.

39
00:08:54.110 --> 00:09:02.040
Emily Haagenson: using the knowledge check would also create opportunities not just to repair these misunderstandings, but also enrich our course.

40
00:09:02.150 --> 00:09:16.910
Emily Haagenson: Using knowledge checks would have helped learners feel like the teaching team is aware of how they're doing. And we've laid a clear path for them to the summative assessment. This improves their sense of belonging and connection to our course as well.

41
00:09:17.340 --> 00:09:38.489
Emily Haagenson: The collective competence in a solid foundation opens up greater confidence in initiating high level discussions of application, either synchronously or in the discussion Forum. This could be initiated by students or by you. But if we have a collective understanding that we all know the basic information that's easier for us to do

42
00:09:38.890 --> 00:09:46.089
Emily Haagenson: also, because we, as the teaching team, weren't exhausting our time and energy with lower level grading.

43
00:09:46.240 --> 00:10:03.920
Emily Haagenson: we may have reallocated our valuable expertise toward higher level engagement with students making time to engage students with a particular passion for our subject, or maybe facilitate general probative discussions to further knowledge on the topic of our course.

44
00:10:04.780 --> 00:10:30.570
Emily Haagenson: So in our scenario, by providing a knowledge check on the foundational concept of the Bdi logic model. We could have prevented this cascading misunderstanding. Of major course, topics and structures. This small shift would have helped our learners feel more seen and more supported, and prevented frustration and disconnection from the content overall. A knowledge check would help us build a positive learning, experience and environment.

45
00:10:32.120 --> 00:10:49.449
Emily Haagenson: Okay, so I want to know what you all think. What's a foundational concept central to one of your courses that you could make a knowledge check to cover. You can go ahead and add your ideas to the chat or raise your zoom hand. If you want

46
00:10:50.900 --> 00:10:55.819
Emily Haagenson: a topic that you might be able to cover in one of your courses with a knowledge check.

47
00:10:58.210 --> 00:11:09.950
Emily Haagenson: At this point I also welcome any questions or skepticism, so how might a knowledge check, help you and your learners, or maybe what are your reservations about using them

48
00:11:21.120 --> 00:11:29.830
Emily Haagenson: again when you're when you're thinking about what topics would be good for a knowledge check. These are going to be foundational topics, right?

49
00:11:30.760 --> 00:11:34.720
Emily Haagenson: The the real basics of your course or your content area.

50
00:11:40.460 --> 00:12:07.330
Emily Haagenson: Thanks, Dorothy. Thanks, Lauren, yeah, Dorothy says, maybe an intro to case control study designs. Yeah, introductory level content is excellent to cover with a knowledge check. Right? You make sure everybody has that introductory level information, and then we can all build on it together throughout the course. Lauren says she teaches an advanced instructional strategies course, and

51
00:12:09.210 --> 00:12:16.099
Emily Haagenson: the students learn about writing objectives. How to assess these objectives are met would be a foundational concept. So

52
00:12:16.510 --> 00:12:23.070
Emily Haagenson: how how do we do objectives? What are objectives? How do they work is an important foundational concept for Lauren's class.

53
00:12:23.430 --> 00:12:30.419
Emily Haagenson: Another great topic to cover in in a knowledge check. Thank you.

54
00:12:31.260 --> 00:12:44.980
Emily Haagenson: Well, if no one is skeptical. I have convinced you all that the concept of knowledge checks have sold everyone. Everyone's convinced. Let's discuss what we need to consider as we're making our knowledge checks.

55
00:12:45.150 --> 00:13:08.100
Emily Haagenson: Remember that knowledge checks are best when they check on understanding of foundational concepts like we just discussed. And they can be graded automatically and immediately. So we're looking for learners to use lower level verbs in Bloom's taxonomy. We're asking them to recall, define, identify, list.

56
00:13:08.100 --> 00:13:16.659
Emily Haagenson: describe, maybe summarize information. So this means using question types that have clear, correct answers.

57
00:13:16.790 --> 00:13:24.990
Emily Haagenson: multiple choice, true, false matching. Fill in the blank. Okay, these ones that will tell them right away. They they're on the right track.

58
00:13:25.920 --> 00:13:54.340
Emily Haagenson: We also want to make sure the questions we're asking are important. We're not trying to catch students out for missing minor details in our lecture. We're trying to help them confirm. They've registered the most important takeaways. We can do this by looking at our questions alongside course objectives. How does this basic understanding lead them toward achieving one or maybe multiple course, level assignment or lecture objectives?

59
00:13:57.170 --> 00:13:58.340
Emily Haagenson: I hope

60
00:13:58.770 --> 00:14:08.330
Emily Haagenson: I hope you've noted the ways in which a knowledge check can save you time once you have them established in your course. But I acknowledge that creating knowledge checks

61
00:14:08.630 --> 00:14:14.200
Emily Haagenson: will take you time as you're preparing your course time, which you don't have a lot of

62
00:14:14.510 --> 00:14:28.119
Emily Haagenson: creating. A short knowledge check can be a nice task to assign to a teaching assistant. It's an opportunity for them to practice a real lower stakes. Teaching skill right? But also Ctl can help.

63
00:14:29.640 --> 00:14:43.720
Emily Haagenson: If you teach with us. At Bsph we have a tool in courseplus called the AI Review quiz maker. This tool uses AI to review your recorded lectures and suggest multiple choice questions for you to use in a knowledge check.

64
00:14:43.870 --> 00:15:08.350
Emily Haagenson: This button is available on all the lecture landing pages, and we'll send you an email with recommended questions that you can review and use as you see fit, so you can see a screenshot there on the slide for more information on the AI Review, Quizmaker and Courseplus. Lauren's going to put a link in the chat, but we're also happy to talk more about it during the

65
00:15:09.070 --> 00:15:11.719
Emily Haagenson: office hours after this session.

66
00:15:15.230 --> 00:15:21.079
Emily Haagenson: As you're creating your knowledge checks. I think the feedback is as important as the questions that we ask.

67
00:15:21.260 --> 00:15:51.090
Emily Haagenson: So every learning management system will have options to provide additional feedback to students within the quiz tool, even if you're outside. Bsph, look for these features within your tool. But within courseplus specifically for each question. You have the choice to provide students with a hint before they answer a question. So use the hint, feature to refer students back to specific content or resources. That'll help them answer the question correctly the 1st time.

68
00:15:51.120 --> 00:16:03.550
Emily Haagenson: The explanation feature is similar, but it will pop up after the learners have answered a question, and I like to use this feature to solidify or extend their learning

69
00:16:04.420 --> 00:16:15.159
Emily Haagenson: currently courseplus and canvas as well only provide one explanation for all answers. But some tools like Microsoft forms will allow you to create different feedback

70
00:16:15.420 --> 00:16:20.419
Emily Haagenson: depending on. If students answer correctly or incorrectly, or even by answer.

71
00:16:20.860 --> 00:16:43.259
Emily Haagenson: So I've talked a lot about knowledge checks. We're going to take an opportunity to experience a knowledge. Check ourselves. I'm trying a new tool here. You should be able to see the 1st question on the screen, but Lauren will put the link to the knowledge. Check in the chat for you as well, and I'll give you a couple minutes to complete this knowledge check.

72
00:16:46.380 --> 00:16:49.479
Emily Haagenson: and we'll look at some of these answers together.

73
00:17:11.050 --> 00:17:13.599
Emily Haagenson: There are 4 questions total.

74
00:17:13.730 --> 00:17:26.250
Emily Haagenson: Once you click, submit, you'll be able to see examples of question feedback like I was describing. They'll either invite you to expand or redirect you.

75
00:17:46.760 --> 00:17:54.429
Emily Haagenson: Okay, getting 3 responses. That's great. Maybe I'll wait. Oh, 4, all right.

76
00:17:55.040 --> 00:18:00.709
Emily Haagenson: I'll start looking at some of our answers. So notice in this, in this knowledge check.

77
00:18:01.120 --> 00:18:24.840
Emily Haagenson: I'm trying to ask you basic questions about items. I hope you'll take away from this workshop. So student experience, retention grading, I'm asking you to recall or define. And then I'm asking you to begin to apply to your own context. So I can use these results to confirm. We're all on the same page. That's what I'm doing with question one and 2 knowledge checks help student feel seen and validated. Great.

78
00:18:24.840 --> 00:18:39.479
Emily Haagenson: We all know that. And and that's something I feel like is important for you to take away. Same here with question. 2 retrieval practice helps improve retention. Great. Another thing that I feel like is important for you to take away.

79
00:18:39.480 --> 00:18:59.989
Emily Haagenson: Additionally, I can use the results to start a conversation. So here knowledge checks should be graded and scored. We're completely divided. It's amazing. That's awesome. So this would be a great conversation for us to have in my course or in office hours after after this session.

80
00:19:00.000 --> 00:19:23.670
Emily Haagenson: And here this again is within your context. Automated feedback is a good way to clarify what might you try? So it looks like most of us are willing to try to prompt students to think about applying the answer with an extension. All of us are into it. Yes, let's extend learning, but it sounds like none of us want to scold our students. So

81
00:19:24.040 --> 00:19:29.979
Emily Haagenson: that's okay, right? But this again will help us have a meaningful conversation about the topic.

82
00:19:30.200 --> 00:19:53.570
Emily Haagenson: So thank you all for trying out a knowledge check with me. I hope you learned something by stepping through a knowledge check experience together. If you want to talk more about any of these questions or ideas, please stick around for office hours at the end. Lauren and I and Amy are happy to chat with you about these things, or you're more than welcome to reach out to someone on our Id team

83
00:19:54.090 --> 00:19:55.020
Emily Haagenson: and.

84
00:19:57.020 --> 00:20:09.239
Lauren Dana: Thank you, Emily. So now that you've learned the why you create knowledge knowledge checks and how to create effective knowledge checks, I'm going to go over to some more practical side of knowledge checks and focus on some key logistics.

85
00:20:10.880 --> 00:20:18.510
Lauren Dana: As Emily mentioned, research has consistently shown that the frequency of self-assessments is a key factor in the success of any course.

86
00:20:18.640 --> 00:20:25.869
Lauren Dana: In fact, frequent engagement and self-assessment has been strongly linked to improved comprehension and retention. Of course, material.

87
00:20:26.120 --> 00:20:44.689
Lauren Dana: However, as a subject matter expert for your course, you have to decide and define what frequent means to you. So fortunately, you have a lot of options. So something to think about is if your course is organized by thematic module you could consider, including a knowledge check after each module to assess specific objectives.

88
00:20:44.920 --> 00:20:55.559
Lauren Dana: If your course is structured by weeks, a weekly knowledge check might be more appropriate. You can even create lecture texts for each lecture to reinforce learning in real time.

89
00:20:56.064 --> 00:21:19.789
Lauren Dana: And if you're using a Ctl recorded lecture, we do offer a feature that lets you embed these quizzes directly within the video. These are actually called in lecture quizzes, and they can be a very powerful way to keep learners engaged throughout the presentation as well as check retention. So I'll share details about this feature in the chat, and you can also reach out to us or stay after Q. And A, if you have questions.

90
00:21:19.970 --> 00:21:26.560
Lauren Dana: But I wanted to ask, just looking at your class. I'm gonna have you take a moment, pull up your courses.

91
00:21:27.280 --> 00:21:29.090
Lauren Dana: So we go next slide, Emily.

92
00:21:33.100 --> 00:21:55.259
Lauren Dana: and so, pulling up your courses and looking at your current content organization. Where do you think knowledge checks would be most effective? Module weekly by lecture, class, session or other. Maybe none of that works for you. So you just want to quickly type in the chat. We're just curious to see how you've structured your classes and what you think might realistically work for your course.

93
00:21:59.490 --> 00:22:03.409
Lauren Dana: Great class session. Yep, I feel the same for that one from the class. I teach.

94
00:22:03.760 --> 00:22:05.170
Lauren Dana: Modular

95
00:22:05.880 --> 00:22:12.909
Lauren Dana: module. Yeah, I see a lot of faculty that do it by module. I think that can be a great way to meet the thematic objectives.

96
00:22:18.260 --> 00:22:34.830
Lauren Dana: Great. So these are all possibilities that we can use. But I do just want to highlight that you can be flexible. There is no one. Size fits all for knowledge checks. And as the subject matter expert, you're the one that decides what frequent is what's realistic and what's helpful. And I did just share the in lecture quizzes information in the chat.

97
00:22:36.820 --> 00:22:58.910
Lauren Dana: Okay? Great. But as you create knowledge checks for your course, it's important to make full use of the tools available to you in your learning management system, particularly how to create a quiz. So I believe most of us are using canvas or course plus, and both have similar features, and these features are designed to support both the organization and pedagogical goals of your course.

98
00:22:59.520 --> 00:23:03.419
Lauren Dana: So one of the 1st recommendations that I have is tagging.

99
00:23:03.590 --> 00:23:17.940
Lauren Dana: So what you can do is you can actually tag each quiz question, and I highly encourage doing this for a couple of reasons. I would encourage tagging them by the Associated Content. So, for example, if it's associated with a lecture or a reading

100
00:23:18.410 --> 00:23:21.029
Lauren Dana: or even an activity that you've done in class

101
00:23:21.929 --> 00:23:42.220
Lauren Dana: and you could also tag each quiz question for the learning objective that it supports. This is super helpful, because this makes it easy for you to find the question and edit and change it if the content changes, and it also forces you to actually look deeply into the question like, What is it doing? Is it asking something that matters? Or is it more of a filler question?

102
00:23:42.980 --> 00:23:47.179
Lauren Dana: I also recommend for the setting to take a look at attempts and retake.

103
00:23:47.660 --> 00:23:57.140
Lauren Dana: So you can set the number of attempts that students have to take a quiz. Multiple attempts encourage students to revisit material and improve their understanding.

104
00:23:57.650 --> 00:24:04.229
Lauren Dana: So we do recommend that you make this a multiple attempt quiz. So students can go back and review as needed.

105
00:24:05.660 --> 00:24:31.710
Lauren Dana: Next, I'm going to talk about multiple access sessions. So this is different than multiple attempts. So what we do is we recommend enabling the option so that students can pause the quiz, leave the course site and then come back to it later. This flexibility supports deeper engagement with the material, and it allows students to go back and review the content. If they don't know the answer. It allows them to scaffold their own learning, and tackle difficult questions with greater confidence.

106
00:24:32.370 --> 00:24:58.369
Lauren Dana: Finally, a big question that I always get is, How long should the quiz be open, and again, this differs. But for formative assessments, we would recommend keeping it open for a longer window to give students time to review and revisit content. However, we do recommend setting a due date, because this can be helpful as a clear milestone, for when students should demonstrate mastery before progressing to more advanced material.

107
00:24:58.920 --> 00:25:05.829
Lauren Dana: These are just some of the recommendations we have when setting up formative assessments in your quiz, generator or tool.

108
00:25:08.160 --> 00:25:30.039
Lauren Dana: And then Emily discussed this briefly, but we do want to highlight grading. Keep it low, stakes and purpose driven, and we'd be remiss not to talk about this, because I feel like that's 1 of the most 1st questions you get asked right? How? How are these knowledge checks graded? How am I going to get an a in this course, and it's important to keep your grading strategy aligned with the pedagogical purpose.

109
00:25:30.340 --> 00:25:42.319
Lauren Dana: The knowledge checks are meant to be formative. They're meant to help students assess their own understanding, receive feedback and guide their learning journey. So with this in mind, we recommend keeping these assessment low stake.

110
00:25:42.560 --> 00:25:47.809
Lauren Dana: Here are just 2 options that you could consider one is just participation based grading.

111
00:25:48.030 --> 00:26:06.189
Lauren Dana: So this is the simplest and most supportive approach, and you can grade it solely based on participation. So in this model students receive full credit just for completing the quiz regardless of their score. This reinforces a safe stress, free space for self-assessment, and encourages continued engagement.

112
00:26:06.540 --> 00:26:10.470
Lauren Dana: Another option to consider is pass fail with multiple attempts.

113
00:26:11.050 --> 00:26:25.559
Lauren Dana: Students can take the knowledge, check multiple times, and the system will then record their highest score, and you can decide what percentage qualifies as a passing score. So how much information do they need to know for you to consider that they're ready to move on.

114
00:26:25.820 --> 00:26:34.740
Lauren Dana: And what this does is allow students to review and retry until they've mastered the material again. This supports retention and understanding over performance

115
00:26:35.340 --> 00:26:48.299
Lauren Dana: again. These options might not fit your course, goals or structure, and we are here to help so you can reach out to us or stay after for our Q&A, and we can help find a grading method that aligns with your content and learning objectives.

116
00:26:50.710 --> 00:26:55.259
Lauren Dana: Okay? So at this point, we've now covered the why, the how, and some of the logistics.

117
00:26:55.430 --> 00:26:58.930
Lauren Dana: and the best practices for creating and implementing them.

118
00:26:59.120 --> 00:27:08.299
Lauren Dana: But we're going to explore the next step, and this may be one of the most important steps, and this is actively applying the results to enhance both teaching and learning.

119
00:27:08.420 --> 00:27:16.980
Lauren Dana: This step is super important, as it helps you identify patterns, adjust instruction and provide targeted support when and where it's needed most.

120
00:27:18.900 --> 00:27:27.900
Lauren Dana: Well, one of the great advantage of using an online quiz tool. Is the wealth of data that it provides at both the class level and individual level.

121
00:27:28.010 --> 00:27:33.050
Lauren Dana: And this data can offer insights into student learning and help guide your teaching decision

122
00:27:33.230 --> 00:28:00.820
Lauren Dana: so really quickly, I'm just going to go over some key metrics available within most quiz tools again. It might differ a little bit, depending on what learning management system you're using. But in general the tool is going to offer class level data. So this is going to give you a snapshot of how your class performed overall. It's going to offer standard deviation. And this shows variability. So a low standard deviation would mean the scores are closely clustered, a high one would suggest a wider range of performance.

123
00:28:01.200 --> 00:28:12.820
Lauren Dana: We have the mean and median which can basically just highlight any outliers that you may find and minimum and maximum scores. And this just helps you get a quick glance of the performance range.

124
00:28:13.080 --> 00:28:21.720
Lauren Dana: What might be even more helpful, though, is the question level data. So this is really helpful when identifying areas where students may be struggling.

125
00:28:21.850 --> 00:28:46.809
Lauren Dana: the discrimination index is, it measures how well a question distinguishes between higher and lower performing students. So in general, a di of 0 point 3 or higher is considered good. But for formative assessments our goal is actually for all the students to demonstrate understanding. So in this case, consistently, high performance would actually indicate effective instruction.

126
00:28:47.580 --> 00:29:04.619
Lauren Dana: The question data can also show full and partial credit. So you can see what parts of a question a student might not be understanding. And finally, I think this one is particularly helpful. Answer distribution. So you can see how many students chose each answer and the percentage who are in full or partial credit, if applicable.

127
00:29:05.160 --> 00:29:11.989
Lauren Dana: can you also have access to individual data. So you can see a student's grade. You can see the number of attempts.

128
00:29:12.120 --> 00:29:23.739
Lauren Dana: And this actually might be helpful for identifying a persistence of a student or repeated challenges. If a student is taking a quiz 20 times, that means they were probably struggling with material a little bit

129
00:29:23.940 --> 00:29:31.070
Lauren Dana: and also time to complete. But this just gives you insights on how confidently or thoughtfully a student approached the quiz.

130
00:29:31.260 --> 00:29:39.319
Lauren Dana: So all of this data can help you identify learning trends spot content that may need clarification and provide targeted feedback or follow up.

131
00:29:41.630 --> 00:30:03.909
Lauren Dana: I just wanted to quickly show you an example of how a formative assessment can help the teaching team. So here, for example, on the course, you can see a quiz and looking through it, if you're the teaching team, you'll see. Wow! A lot of students got question 8 wrong, in fact, 18 out of 26 missed question 8. So this would be an invitation for the teaching team to take a look at it?

132
00:30:04.020 --> 00:30:16.549
Lauren Dana: Is the question. Important is the topic of the question not covered enough? Is the question worded unclearly? So this just inspires some discussion to see what you can do about this question and improve student understanding.

133
00:30:19.830 --> 00:30:27.329
Lauren Dana: As Emily mentioned, we do formative assessments, not just for content, right that matters, but we also do it to help connect with our students.

134
00:30:27.600 --> 00:30:33.549
Lauren Dana: So what you do with this data and how you use it to connect with your students really makes the difference.

135
00:30:33.800 --> 00:30:49.760
Lauren Dana: Frequent knowledge checks. They should serve as touch points. So opportunities to engage with your students in meaningful ways when used effectively. Data shows that this contact, these touch points can allow you to connect, motivate, and support your learners through the course

136
00:30:50.220 --> 00:31:02.589
Lauren Dana: they boost student motivation. They provide clear, actionable feedback. Even when not individualized. You don't have to give each individual student feedback, but that general feedback that Emily described can be as motivating.

137
00:31:02.930 --> 00:31:12.349
Lauren Dana: It offers guidance for success in the course. And you can build stronger relationships with students. They feel like they know you more, making them more likely to reach out with questions or concerns.

138
00:31:12.730 --> 00:31:19.629
Lauren Dana: I'm now going to quickly walk through a few practical scenarios where quiz data can serve as a bridge between you and your student.

139
00:31:22.820 --> 00:31:47.969
Lauren Dana: Okay, so scenario one. Your students just completed their knowledge check for your 1st course session. All of your students did. Well, yay, but there were 2 questions that about 30% of the class had the incorrect answer. So, keeping in mind, we're not talking about how to change the course or how to change the quiz. But how might you use this as a touch point? How might you use this data to create a student connection?

140
00:31:48.320 --> 00:31:57.989
Lauren Dana: So give just a minute to think about it, and feel free to raise your hand or put in the chat some of the actions you might take to use this data again as a touch point for students.

141
00:32:12.290 --> 00:32:13.420
Lauren Dana: Any thoughts?

142
00:32:14.630 --> 00:32:15.699
Lauren Dana: Oh, great!

143
00:32:16.640 --> 00:32:29.210
Lauren Dana: Oh, excellent! Amy! Yep, class email explaining the correct answer for the 2 questions missed right? It might need a little bit more than that feedback session and quiz generator. You may need to actually write a paragraph or guide them. Excellent!

144
00:32:31.890 --> 00:32:53.229
Lauren Dana: That's a great question, Dorothy. Do you address the class as a whole? Or maybe you take those 30% and you email them. I wouldn't put them in a group where they could see each other, but you could BCC them and address just the students who struggled with the question. That's an excellent point, Dorothy. You don't necessarily have to email the whole class. You could just reach out to those students who struggled with that? Those 2 questions.

145
00:32:53.540 --> 00:32:54.800
Lauren Dana: Great question.

146
00:32:58.750 --> 00:33:22.839
Lauren Dana: good. Yeah. Some of the thoughts I had were just, you could recommend material to review, so you could send them readings, or partly even timestamps of lectures where they could go back and review. You could write the detailed explanations like Amy asked. You could even open an office hour for students who need one on one guidance to support this. So all of these are just ways that you could address

147
00:33:23.390 --> 00:33:30.400
Lauren Dana: some of this 30% that was struggling and help create a touch point. So they feel comfortable with the material.

148
00:33:31.380 --> 00:33:36.189
Lauren Dana: Since we're a little short on time, Emily, I'm gonna skip scenario 2, and we'll jump to

149
00:33:37.290 --> 00:34:04.980
Lauren Dana: course revisions. Okay? So again, as we were saying. Formative assessments can help create relationships with students. So that's 1 benefit. The other benefit is actually can help you revise your course so you could either adjust instruction before students fail behind, or you could look at this after the running of the course and improve it for next year, since it's also pretty valuable. So again, I'm going to go through 2 scenarios on how we could use data to support course revision.

150
00:34:07.400 --> 00:34:29.519
Lauren Dana: Okay? So you created a knowledge check in your module, environmental health and recent legislation. The data shows that students are consistently missing questions that focus on global legislation which is more heavily focused on in your readings than lectures. The discrimination index is also negative on these questions, what are some next steps you might take when revising the course for next year.

151
00:34:30.620 --> 00:34:39.749
Lauren Dana: Okay, so students are really missing this one topic. So what are some things you might change or look into? Changing doesn't necessarily mean you have to, but you might look into it.

152
00:34:51.000 --> 00:34:52.789
Lauren Dana: What do we think any thoughts.

153
00:35:06.750 --> 00:35:16.915
Lauren Dana: Justin? Give a couple of ideas of what I thought of? I did some of the the easy ones, the low hanging fruit. So I said, maybe just review and revise the questions right? The questions could be the issue.

154
00:35:17.360 --> 00:35:28.890
Lauren Dana: I would also always check to see that the course objectives are aligned. Excellent! Faye. Yes, that was a great suggestion. Have a focus lecture. If they're doing better, absorbing the material with the lecture than the readings

155
00:35:29.030 --> 00:35:49.070
Lauren Dana: create a lecture. It can be short 1015 min. But I think that's a great idea, Dorothy. Excellent! Make the readings more understandable. Right? Maybe the readings just are either above a student level or not engaging, so you could find different ones, or you could do excerpts of it. Excellent. Yeah, add other media types to material lecture, video, love that podcast

156
00:35:49.560 --> 00:36:03.829
Lauren Dana: we love other media, material, other media types. So thank you. Dorothy, Amy, yeah. Record a new lecture. Excellent or reinforce the topic in an activity. So maybe they need to actually apply versus just just read or listen.

157
00:36:04.000 --> 00:36:22.950
Lauren Dana: Excellent, Carrie. Yep, have a more focused lecture or activity. You and Amy were on the same wavelength. Excellent! That is everything that I came up with, and you could also go back and do that targeted feedback, or hints that Emily suggested. But I think these ideas are much, much better and more robust, so excellent.

158
00:36:23.970 --> 00:36:42.389
Lauren Dana: And then, my full time. I just want to go through this one quickly. You're in the middle of your teaching your course on public policy, and you have a good problem. Students are getting all your knowledge questions correct and taking very little time to complete the knowledge check. You are worried. Your students may think the course is too easy.

159
00:36:42.510 --> 00:37:08.579
Lauren Dana: So here, what are some mid course adjustments you could make? I see this sometimes I'll see quizzes where students are finished in like 10 seconds. So that's not exactly quite the vibe we want with these checks. Right? We do want to make them think so. What are some things that you could do mid course to try to to make it a little more challenging, or address this issue where the course the students might think it's too easy, and I'll give everyone a couple of minutes to chat. I kind of jump the

160
00:37:09.150 --> 00:37:10.539
Lauren Dana: something to head there.

161
00:37:21.810 --> 00:37:36.279
Lauren Dana: Excellent. Yeah. Beef up. The knowledge checks. Maybe your questions are just kind of surface level. They're not meeting any objectives. If they're meeting an objective, it should take them a while. Right should make them think a little bit excellent. Yeah, go back and check. See what those questions are.

162
00:37:40.050 --> 00:37:52.099
Lauren Dana: Good. Okay? So add, no. I like, I like the beef up. That's perfect, Amy, check your course level objectives. So if they're complex, you can bump the level of complexity in your lecture level objectives and knowledge checks

163
00:37:52.190 --> 00:38:18.210
Lauren Dana: to match up excellent right? So we can use those lower level blooms. But you want to make sure that they're scaffolding to the higher objectives. Right? You can't just only do meaningless retention. Right? You want stuff that's going to help them later. Good at a multitude of true and false items. Right. You could increase the quiz questions. If you feel like that would be. I like the word, beep it up, make it a little bit more challenging. Make them think a little bit more

164
00:38:18.460 --> 00:38:30.369
Lauren Dana: good. Yes, I agree with all these. I thought I ever increased the depth of the question. You could add optional challenge questions. I sometimes have faculty that have, like a very challenging extra credit quiz that lets students who feel ready to jump in

165
00:38:30.480 --> 00:38:33.470
Lauren Dana: and always check alignment with objectives.

166
00:38:33.750 --> 00:39:02.100
Lauren Dana: That's a good question. What if they are using AI to answer the items? That, again, is always a good point. So if you've gone through the questions. And you're like, these are our solid, effective questions. We'll never know for sure. Right? If they're using AI. It does mean they're missing out on the self assessment part, and they're the ones that aren't learning this. But I guess the nice thing for this, too, is low stakes, formative. If they're choosing to not go down that path.

167
00:39:02.370 --> 00:39:05.339
Lauren Dana: That's going to be a struggle for them later.

168
00:39:05.841 --> 00:39:20.478
Lauren Dana: But it's not necessarily affecting like a summative grade where that would be. I mean, it'd be cheating. But that's a great question, Dorothy. So yeah, so I would. Just if you feel confident that those questions are solid and good. Then there may be an issue.

169
00:39:20.990 --> 00:39:23.110
Lauren Dana: that you wanna take a look at

170
00:39:23.380 --> 00:39:26.649
Lauren Dana: with the students using AI, not your questions. Great.

171
00:39:28.050 --> 00:39:54.670
Lauren Dana: Okay. So before we head to Q. And A. We don't have a ton of time. Sorry, Emily and I just got so excited about this topic, so we wanted to just finish up with some key takeaways. So if you only leave here with a couple a couple things, we want you to remember that knowledge checks both help the students and the teaching team establish course foundations. And these are both academic and relational right. We know that the academics matter. But so does your relationship with students, and these can really help with that.

172
00:39:55.412 --> 00:39:58.959
Lauren Dana: You want to incorporate knowledge checks to fit the needs of your course

173
00:39:59.170 --> 00:40:06.800
Lauren Dana: and use your Lms tools to redirect and guide students. You, said Lms, as much as you can, and we are happy to help if you have any questions.

174
00:40:07.452 --> 00:40:14.499
Lauren Dana: And then, finally, that data from knowledge checks can support and guide course revisions both during and after the course.

175
00:40:14.620 --> 00:40:37.999
Lauren Dana: So these are all things that you want to take away. And great, Amy, that's a great idea. You can remind students of AI policy and, like Emily, talked about it. I think it does matter to let students know why you're doing these checks they're supposed to help. Are you going to catch every student with that? Not necessarily. But it's, I think, the why is really important. But yeah, that's a really, that was a great question, Dorothy.

176
00:40:38.820 --> 00:40:47.200
Lauren Dana: Okay, so here are references. If you want to explore a little bit further. And I'm gonna just quickly pass it over to Amy for some additional information.

177
00:40:47.200 --> 00:40:58.709
Amy Pinkerton (BSPH CTL Teaching Toolkit): Yes, thank you. Thank you, Emily and Lauren, that was great. So 1st is a little bit of housekeeping. So we would love to hear your feedback. As you know, feedback is important to us.

178
00:40:58.710 --> 00:41:21.279
Amy Pinkerton (BSPH CTL Teaching Toolkit): so please take a moment to fill out our anonymous workshop evaluation form, either by going to the link which is on screen. I think it'll be added to the chat or by scanning the QR code. Those results are anonymous, and we. But we do use your feedback to plan future workshops just like this one. So we appreciate you taking the time to give us that feedback.

179
00:41:21.840 --> 00:41:46.670
Amy Pinkerton (BSPH CTL Teaching Toolkit): And then also, we have some other continuing professional development opportunities from the center for teaching and learning. First, st I want to highlight our teaching assistantship training course. This is an online self paced course for students who are interested in becoming tas, and it prepares them for all of the roles and responsibilities of becoming a ta at Bsph. And our new cohort just started July first.st So it's a great time to join the course.

180
00:41:46.970 --> 00:42:04.900
Amy Pinkerton (BSPH CTL Teaching Toolkit): and then we also have more workshops like this one, both upcoming workshops that are posted on our events, calendar and past workshop recordings on our on demand videos page. And we also, if you want to stay up to date on all things. Ctl, I recommend subscribing to our Ctl. Blog.

181
00:42:07.610 --> 00:42:11.669
Amy Pinkerton (BSPH CTL Teaching Toolkit): All right. And now I'm going to pass it back to Emily and Lauren. Thank you.

182
00:42:17.750 --> 00:42:28.692
Lauren Dana: I was. Gonna say, we're going to choose this time. If anyone has questions or we'd love to hear some of your own experiences. If you've implemented formative assessments or are thinking about it, or any pitfalls.

183
00:42:29.090 --> 00:42:33.649
Lauren Dana: I think this would be a great time. As a community of practice, we can talk about anything you would like.

184
00:42:38.060 --> 00:42:45.170
Emily Haagenson: Any follow up, as well from the questions or discussions we started in the presentation.

185
00:42:46.640 --> 00:42:48.980
Dorothy Faulkner: Ladies, can you hear me? This is Dorothy.

186
00:42:49.330 --> 00:42:50.100
Lauren Dana: Yes.

187
00:42:50.660 --> 00:43:14.970
Dorothy Faulkner: Thanks for the great session. I'm not at Hopkins. I grew up in Baltimore, but I'm actually at Old Dominion University in Virginia, Norfolk, Virginia. So I have a question. I have one student. She's a practicum student in public health. There's no learning management system that we're using. But I do want to do. Knowledge checks on her. What do you recommend in terms of tools?

188
00:43:16.460 --> 00:43:35.422
Emily Haagenson: I would say, Dorothy, we were just talking about this. I try to use the tools that I already have access to. So at Hopkins. We have Microsoft suite. So I would use Microsoft forms right? Because I already have access to that. And I know my students have access to it?

189
00:43:36.100 --> 00:43:45.769
Emily Haagenson: so it depends that that would be my advice. Instead of asking your students to sign up for something new, or try a new tool, start with what they already have.

190
00:43:46.890 --> 00:43:48.159
Dorothy Faulkner: Great idea. Thanks.

191
00:43:48.330 --> 00:43:51.649
Emily Haagenson: Yeah. Do you have my? Do you use Microsoft or.

192
00:43:51.650 --> 00:43:52.430
Dorothy Faulkner: Yes.

193
00:43:52.820 --> 00:44:09.260
Emily Haagenson: Okay, I like forms, and it'll let you. I don't know if you noticed in the in the knowledge check I gave you all I could give individualized and like feedback answers depending on any answer they gave, which is nicer than some learning management system. So.

194
00:44:10.900 --> 00:44:12.400
Dorothy Faulkner: Great thanks.

195
00:44:12.570 --> 00:44:13.350
Emily Haagenson: Yeah.

196
00:44:13.350 --> 00:44:30.330
Lauren Dana: Yeah, this is deviating a little off your question, Dorothy, because you only have the one practicum student. But I know I've used Kahoot on onsite classes and Kahoot can actually give you. It doesn't necessarily give you individualized question data because most of the time the students are in anonymously. But I can definitely tell if they've missed a complete

197
00:44:30.490 --> 00:44:44.789
Lauren Dana: concept. So I definitely use things outside of the Lms depending on on what you need. And who's just kind of fun. So get some data without making it seem like you're getting data. So.

198
00:44:44.790 --> 00:44:46.940
Dorothy Faulkner: Okay. Great. Thanks.

199
00:44:46.940 --> 00:45:04.030
Emily Haagenson: And for those of you at Hopkins. We also have a poll everywhere license that you can use to do a lot of a lot of this this same kind of check-ins with your students, too, if you're if you're if the Lms isn't working for you.

200
00:45:04.730 --> 00:45:06.569
Dorothy Faulkner: Great appreciate it.

201
00:45:12.360 --> 00:45:21.839
Emily Haagenson: We were kind of split on grading knowledge checks. Does anyone have any questions or experiences to share around?

202
00:45:22.650 --> 00:45:27.379
Emily Haagenson: Yeah, how do we grade this, how do we get students to

203
00:45:28.440 --> 00:45:32.220
Emily Haagenson: care about something? Maybe we don't grade or, yeah, Amy.

204
00:45:34.080 --> 00:45:53.020
Amy Pinkerton (BSPH CTL Teaching Toolkit): at least, for in my experience I've not given like a numerical grade for knowledge checks, but I've counted it towards like a participation grade so like the knowledge check itself isn't like, Oh, you got that wrong. So you're missing like 2 point, you know, minus 2 points. But it's more of a did you complete the knowledge checks.

205
00:45:53.060 --> 00:46:06.369
Amy Pinkerton (BSPH CTL Teaching Toolkit): and if you completed them all, then you that was towards participation. But otherwise I lean towards not grading either it's participation or not graded. But I'm curious to see if anyone else has graded them, or how that how that

206
00:46:06.470 --> 00:46:10.000
Amy Pinkerton (BSPH CTL Teaching Toolkit): turned out, because that's interesting.

207
00:46:10.000 --> 00:46:16.049
Emily Haagenson: I think so many of our students are so motivated by grades that for me, you have to explain

208
00:46:16.140 --> 00:46:43.290
Emily Haagenson: what they're for, right? Because I think we have a lot of students whose 1st reaction is you're not grading it. Then I'm not doing it right, like you have to explain. If great. If the grade isn't the purpose, then what is the purpose? And so but I do think, you know, especially at Bsph. These are graduate students. They should have some motivation to be there and to learn the content, and so explaining how this will support them.

209
00:46:43.520 --> 00:46:46.159
Emily Haagenson: should hopefully be enough. But.

210
00:46:53.040 --> 00:47:08.850
Dorothy Faulkner: This is Dorothy. I'd like to grade them, but they only get extra credit for passing. So it's pass fail, and if they pass they get extra credit. If they fail, there's no deduction or anything. There are no losses. They seem to like that.

211
00:47:11.740 --> 00:47:12.170
Lauren Dana: Yes.

212
00:47:12.170 --> 00:47:12.510
Emily Haagenson: Yeah.

213
00:47:12.510 --> 00:47:12.895
Lauren Dana: Great.

214
00:47:13.280 --> 00:47:14.659
Amy Pinkerton (BSPH CTL Teaching Toolkit): Like that. That's cool.

215
00:47:16.120 --> 00:47:23.741
Lauren Dana: Yeah, I have a I have a faculty member who does. He'll create. Actually, it's a little bit more work. But like a quiz one and a quiz, 2. So quiz one

216
00:47:24.120 --> 00:47:34.729
Lauren Dana: If you don't do well, you get like a second quiz to do check the same type of knowledge and multiple attempts so they can do it a couple of different times. But it's an interesting concept like you get.

217
00:47:35.050 --> 00:47:41.409
Lauren Dana: If you really aren't getting it, then you get a second quiz after you've reviewed the materials so.

218
00:47:43.760 --> 00:48:05.089
Emily Haagenson: I think one thing to consider as well again, especially with our graduate students, is that they? They come to us with lots of different experiences and lots of different backgrounds, so some of them might need the check in more than others, right need the establishment of foundational concepts more than others, so

219
00:48:05.210 --> 00:48:08.450
Emily Haagenson: their engagement with the knowledge check

220
00:48:09.200 --> 00:48:14.333
Emily Haagenson: will be variable as well. I have a faculty member who?

221
00:48:16.790 --> 00:48:45.620
Emily Haagenson: I think it's just participation, right? Or or she might not even do them for participation. But then, if I come to her and I'm really struggling, or my have questions about my grade, I know she's gone back through and said, Emily, you didn't do any of the knowledge like she can use. She can still use the data to say like you didn't watch any of the lectures or do any of the knowledge checks. Of course you're confused. So there's lots of ways to use the data, no matter how you're grading, too.

222
00:48:50.330 --> 00:48:51.610
Lauren Dana: You need hear me.

223
00:48:52.010 --> 00:49:11.660
Amy Pinkerton (BSPH CTL Teaching Toolkit): All right. I have a question. So many like knowledge check questions are usually like multiple choice questions, and sometimes multiple choice questions can be difficult to actually write. Do you have any recommendations for actually writing multiple choice questions that you could share with us?

224
00:49:17.040 --> 00:49:25.060
Lauren Dana: Well, I can start, Emily, and then I'll pass to you. So they are very challenging. And again, I do think there is you want to make sure that you start with a clear

225
00:49:25.330 --> 00:49:47.470
Lauren Dana: set that you're looking for. You don't need 50 right like 10 could suffice. What I always look for is again the alignment of the objectives. So like, what objective is this meeting? And, as I said earlier, I do like to think, what will this question help later on in the course? When I'm looking at questions I. Some of these are more in the weeds, but, like I never use double negatives so like

226
00:49:47.710 --> 00:50:13.935
Lauren Dana: this would not do what? Like things like that. You don't want to have answers that are trick answers, but you also don't. I don't necessarily love the question where there's 1 that's like, clearly not right? Because I'm like, well, then, why did you put that in at all per se. So I think you want to try to anticipate what students might be misunderstanding, and those would be the answers that you could provide feedback on, but I do think that's how I approach it.

227
00:50:14.460 --> 00:50:40.279
Lauren Dana: And it is it takes a long time. I do think AI can be helpful. I don't want to say that that's what everyone should do. But I do think you know things like I. We now have access to the Jhu. It's like the the Chat gpt of Jhu. So it's not a bad idea to always get some ideas from from there. If you're running low on steam after like 6 or 7 questions. But again, I would still make sure those questions align with the objectives

228
00:50:40.300 --> 00:50:51.200
Lauren Dana: align with the goal you're headed them for, and double check those answers to make sure that they make sense, and that none of them are like Tricky or Nitpicky, or, you know, like leading them down a frustrating path.

229
00:50:52.670 --> 00:50:56.380
Emily Haagenson: Yeah, I agree with Lauren. I feel like sometimes

230
00:50:56.750 --> 00:51:22.479
Emily Haagenson: avoiding the wrong. A specific wrong answer is as important as getting the right answer right. So if you not to trick them, but to say like, Oh, here's somewhere. I think they might get hung up, putting that as a wrong answer, so they know to avoid it right? Possibly with that feedback about why it might hang them up right is

231
00:51:23.310 --> 00:51:28.140
Emily Haagenson: more useful than like, Lauren said. Just kind of a throwaway answer.

232
00:51:31.160 --> 00:51:47.530
Lauren Dana: I know we have. I know we have a colleague that did a workshop on this. I'm trying to find the link through it, but we can always send that in the follow up email if I can't it. But it was, and I think it was an hour and a half on how to create effective multiple choice. And it was excellent. So I highly, I learned a lot I highly recommend.

233
00:51:55.700 --> 00:52:00.860
Lauren Dana: Okay, so any other questions or thoughts, we're always happy to give people 8 min back.

234
00:52:02.600 --> 00:52:08.399
Lauren Dana: I know. Here, I'll just sit in front of the A/C. It's really hot here in Baltimore. So thank you. Thank you, Amy.

235
00:52:10.510 --> 00:52:29.389
Lauren Dana: Great. Well, thank you so much for attending Emily. And I will send up a follow up email. And our emails will be in in the email itself. So feel free to reach out to us any feedback you have or any ideas that you come up with, or anything experiences you have in your class. We would love to hear about it. So thank you so much for attending.