WEBVTT 1 00:00:19.040 --> 00:00:32.930 BSPH CTL Teaching Toolkit: All right, let's get started. We'll come to join us, for today's workshop, and the fundamentals are for multiple choice question writing for better learning hosted by the Center for teaching or learning 2 00:00:32.940 --> 00:00:49.890 BSPH CTL Teaching Toolkit: a quick note on the logistics. This workshop will be recorded, and the slides of recording and other resources will be shared with all registerings afterwards, and and there will be a queue in session at the end. 3 00:00:49.890 --> 00:00:57.500 BSPH CTL Teaching Toolkit: where you will come to ask questions during the session by raising your virtual hands, checking them in the chat. 4 00:00:58.680 --> 00:01:14.620 BSPH CTL Teaching Toolkit: So first let's introduce myself. I'm a Luci, a an instructional designer. I ctl we also have 3 Co. Facility teachers, Mia Jennifer and Mariki. We would you like to introduce yourself. 5 00:01:15.280 --> 00:01:16.870 Mia M Lamm: Mia: Sure. 6 00:01:16.940 --> 00:01:27.630 Mia M Lamm: Hello, everybody. Good afternoon. My name is Mia Lam I'm, a senior instructional designer with the center for teaching and learning, and and associate with health policy and management. Welcome. 7 00:01:28.320 --> 00:01:29.780 Mia M Lamm: I'll pass it to Jennifer. 8 00:01:30.420 --> 00:01:41.260 Jennifer Applegate: Hi. Everyone. My name is Jennifer Applegate I'm, an assistant scientist in the Department of International Health. and also support these cells to society courses as one of the course facilitators. 9 00:01:43.480 --> 00:01:55.990 Marykate Bergen: Hi. Everyone. My name is Mary Kate Bergen. I'm. A second year, Dr. Ph. Student in Health, Equity and Social Justice. I'm also part of the course Facilitator Team and a ta for a number of courses excited to be here. 10 00:01:56.740 --> 00:01:59.970 BSPH CTL Teaching Toolkit: Yeah, thank you, mia. Jennifer and Mary Kate. 11 00:02:00.200 --> 00:02:15.510 BSPH CTL Teaching Toolkit: So first I would like to say that this introductory, multiple choice person writing workshop, we will explore how to create a effective multiple, to ask question that based on sounding learning objectives. 12 00:02:15.510 --> 00:02:28.190 BSPH CTL Teaching Toolkit: So, after completing this session you can achieve the listed for goals. We I will not read the all of them, so you can have a look at the those for today's workshop 13 00:02:30.620 --> 00:02:46.220 BSPH CTL Teaching Toolkit: multiple choice questions. One of the widely used assessment instruments before we dive in let's at first define what multiple choice questions are. So in this workshop we defend a multiple to us. Question 14 00:02:46.220 --> 00:03:04.690 BSPH CTL Teaching Toolkit: All multiple choice at home is why you wait to the learner is invited to select one alternative, the correct one being the key from a list of response options, including the K. And the distractors which are incorrect answers 15 00:03:04.770 --> 00:03:08.500 BSPH CTL Teaching Toolkit: in response to the question which is called STEM. 16 00:03:08.570 --> 00:03:25.870 BSPH CTL Teaching Toolkit: That's a standard multiple choice question. It will consist of a 3 core elements, the STEM, the distractor, and the key just to be efficient. We we use Mcq. You reference to multiple to us. Question in the following slides. 17 00:03:25.960 --> 00:03:30.730 BSPH CTL Teaching Toolkit: Next, I will turn it over to Jennifer and Marie Kate. 18 00:03:32.910 --> 00:03:50.710 Jennifer Applegate: Thanks so much, Lou. So first, we want to start with talking about some of the advantages of multiple choice questions for learning, and this is based on our review of the literature, as well as our experiences developing these questions, and also supporting faculty to develop these questions for their assessments. 19 00:03:50.870 --> 00:04:07.510 Jennifer Applegate: So first, it's an effective method of formative assessment. And what I mean by formative assessment is assessing how a student is learning during a class. It also facilitates timely feedback, which is helpful for monitoring student learning and identifying gaps and understanding 20 00:04:07.510 --> 00:04:20.420 Jennifer Applegate: which can then inform, help in inform. If you need to make a shift mid class, possibly revisit some concepts or dive deeper into particular areas. 21 00:04:20.880 --> 00:04:41.930 Jennifer Applegate: It also has provides excellent data for post-test discussion, meaning that you're able to look at. Where were there gaps to the student learning, and is that by student? Is it by the group of students? And this is something that the course plus feature quiz generator does quite well, and we'll talk about that in subsequent sessions more. 22 00:04:42.300 --> 00:05:06.430 Jennifer Applegate: And finally, it provides a scalable option for both formative and summative assessment, and by summative assessment I mean what has been learned by the students at the completion of the class, and this is a particularly important for large classes, where there could be a lot of grading that needs to come at once. So multiple choice questions give a a good option for those types of assessments 23 00:05:10.180 --> 00:05:11.600 Jennifer Applegate: next slide, please. 24 00:05:13.190 --> 00:05:32.280 Jennifer Applegate: So what are some of the disadvantages of using these for learning as Lou was good. I was touching on, and we're going to talk about more in subsequent slides. Poorly written distractors can confuse. Students also flawed. Construction of the STEM, or the distractors might result in unintended difficulty. 25 00:05:32.470 --> 00:05:49.200 Jennifer Applegate: Multiple choice questions might also encourage guessing, because students might be thinking if there are potentially more than one correct answer for the particular question. So, being very clear in your STEM and distractors is important. And again, we'll talk more about this in subsequent slides. 26 00:05:49.840 --> 00:06:09.080 Jennifer Applegate: Also multiple voice choice questions also target lower levels of learning. Typically so there ones that are easier to write, usually for a recall rather than like higher order, thinking and more complex concepts. And again we're going to touch a bit about how you develop strategies for this 27 00:06:09.080 --> 00:06:10.230 moving forward. 28 00:06:10.380 --> 00:06:29.370 Jennifer Applegate: And then, finally, another point that we will be discussing is that multiple choice questions can pose challenges for academic integrity. So we know that there are online websites where students can both test or there might be sharing across students so particular tests or answers to test. 29 00:06:29.370 --> 00:06:35.830 So this is again something else that we're thinking about, and want to give you some strategies to try to prevent 30 00:06:36.690 --> 00:06:37.870 Jennifer Applegate: next slide, please. 31 00:06:40.540 --> 00:06:53.510 Jennifer Applegate: So we don't want to take too much time, because we have a lot of material to cover. But we'd love to hear from you a little bit about what have been your experiences using multiple choice questions. And what's brought you to this workshop today? 32 00:06:53.510 --> 00:07:03.260 Jennifer Applegate: If you could just drop some of those reflections into the chat, we'd like to take just a few minutes to assess and talk about those. Also. 33 00:07:03.260 --> 00:07:19.570 Jennifer Applegate: We are going to use this first session in the workshop series as a bit of a needs assessment to identify what we should focus the subsequent workshops on to better give guidance to you on how to compose these assessments, and also the questions 34 00:07:20.140 --> 00:07:37.740 Jennifer Applegate: and what we're waiting for folks to drop some things in the chat. I wanted to ask Mary Kate if she could share some reflections from her student perspective on multiple choice questions and experiences, challenges benefits. But go ahead, Mary Kate. 35 00:07:38.000 --> 00:08:07.910 Marykate Bergen: Yeah, Thank you. Jennifer. As a Dr. Ph. Students in the Health, equity, social justice, concentration. Most of my student assessments have been written assessments as opposed to quiz questions or assessments, using these multiple choice questions. But my experience with them has been through the waiver exams that I take to satisfy some prerequisite requirements, and I've always appreciated when those questions are well done, and as we're going to talk about construction wise that the STEM or that 36 00:08:07.910 --> 00:08:25.770 Marykate Bergen: core of the question is another chance to review the material. I feel like the faculty and teaching team are really trying to make sure. I'm also learning while I'm taking the assessment. It's not just a way to make sure that I pass the course I really appreciate when there is a lot of effort that goes into the construction. 37 00:08:29.000 --> 00:08:51.230 Jennifer Applegate: And I see. So, Mary, thank you for your question. Is the strategy different from multiple answer versus multiple choice that gets into partial credit. So I think you're asking about if if there are, it could either be matching, or possibly like a multiple choice question where a and B could be correct, and there could be 38 00:08:51.240 --> 00:09:03.400 Jennifer Applegate: multiple answers. And we are going to get into that a bit, I think, for both the matching and multiple choice. The importance of having a clear STEM 39 00:09:03.420 --> 00:09:13.690 Jennifer Applegate: is one strategy we're going to cover as well as is it really necessary to have more than one correct answer. And what are the pros and cons of that? 40 00:09:19.700 --> 00:09:37.100 Jennifer Applegate: Yup, and I see Brent has added there. Yep, easy to write, automatically graded, and it seems like they don't challenge students for all the reasons that Yep, that we just discussed. And yeah, I think that they can be easy to write. I would also say they really do take time. 41 00:09:37.100 --> 00:09:56.720 Jennifer Applegate: I know one challenge that I have faced when writing these is typically we are setting up our courses. We have our lectures, and then we say, okay, how are we doing our assessment? We'll do some quizzes, but maybe we haven't put the quizzes yet together before the course starts. Or we're doing it a week or so before the course starts. 42 00:09:56.720 --> 00:10:03.900 and then that can lead to some challenges and writing the questions, and I think it also leads to 43 00:10:03.910 --> 00:10:21.630 Jennifer Applegate: falling back on easier habits that we're gonna discuss some of like asking like Which statement is true as opposed to giving a more, a more clear and like concise question, that doesn't take that doesn't require students to look for the incorrect Answer. 44 00:10:21.630 --> 00:10:48.280 Jennifer Applegate: But I agree with you. I think they're incredibly especially as we're moving online. So many of the courses and our class sizes are getting bigger and bigger. They are a really good option for doing different assessments and being able to get some valuable data that also makes the grading burden a bit away from faculty. But it actually just shifts the burden more towards the question. Writing a little bit later. 45 00:10:48.530 --> 00:11:02.410 Jennifer Applegate: Pamela, I mentioned writing to distractors. Yeah, and I think I completely agree that can be very difficult for a conceptual class. I think that when you have more quantitative data there are ways that you can 46 00:11:02.410 --> 00:11:15.510 Jennifer Applegate: provide distractors that are reasonable and logical, but for conceptual class it takes a little bit more thinking into how are we going to write distractors that aren't 47 00:11:15.640 --> 00:11:19.530 Jennifer Applegate: completely implausible, but also not tricky. 48 00:11:21.590 --> 00:11:34.700 Jennifer Applegate: And then we have, Katherine mentioned used to ask about half multiple choice, half open, ended. And now ask more open-ended because it's Yup! I think I agree. It's hard to write good multiple choice questions. 49 00:11:34.700 --> 00:11:59.300 Jennifer Applegate: especially when they seem too obvious or too tricky. Yeah, I completely agree. I think it can be difficult to fall back on making a question more tricky to make it more difficult or potentially using, like a what we call a double barreled question, where part of the statements true, and part of it's false. And you have to look, then, for the distractor that's all true, and that can get really confusing for students. 50 00:11:59.810 --> 00:12:19.520 Jennifer Applegate: And then Mary again, thank you for bringing up about multiple choice business as a pre test before the content is introduced and focus learners on the learning objectives. Yeah, I think that's a great point. We, as part of this workshop, had discussed getting into a little bit of pre test post test. However, we didn't have quite have enough time. 51 00:12:19.520 --> 00:12:40.760 Jennifer Applegate: So maybe that's a concept that we will carry forward to subsequent workshops. So thanks for raising that. Okay, so Thank you, everyone for sharing your reflections. We'll have more time towards the end to talk through some different questions and answers, and please feel free to bring more reflections as we're moving through the discussion. 52 00:12:42.640 --> 00:12:48.000 Jennifer Applegate: So now i'll hand it back over to Mia to take us through some of the strategies. 53 00:12:51.110 --> 00:12:54.320 Mia M Lamm: Thanks, Jennifer. Lou. Do you mind going to the next slide? 54 00:13:00.060 --> 00:13:01.300 Mia M Lamm: Great. Thank you. 55 00:13:01.670 --> 00:13:18.330 Mia M Lamm: So thank you that there's been some really great insights already being discussed, and, as has been mentioned already. Writing multiple choice questions is challenging. It's also time consuming, and many factors have to be considered so. 56 00:13:19.360 --> 00:13:22.730 Mia M Lamm: Weakness and question. Writing is frequently 57 00:13:23.100 --> 00:13:35.310 Mia M Lamm: not as much in the style of the question, but as a result of the misalignment of the learning objective, the activities, and the actual test item. So 58 00:13:35.620 --> 00:13:45.720 Mia M Lamm: if assessments are misaligned, and the objectives. or to with the objectives or instructional strategies, it can really undermine student motivation and learning. So 59 00:13:45.770 --> 00:13:55.060 Mia M Lamm: let's just just to sort of highlight. This, for example, let's consider a scenario, so your instructional learning objective is for students to apply analytical skills. 60 00:13:55.240 --> 00:14:01.680 Mia M Lamm: but your assessment assessment item measures only factual recall. and consequently the students 61 00:14:01.760 --> 00:14:15.150 Mia M Lamm: home, their analytical skills and their courts work, and the course activities on the material, but then become frustrated that the exam does not measure what they learned. And and we see this a lot, particularly from the international point of view. And we're talking about quiz improvement. 62 00:14:15.800 --> 00:14:31.430 Mia M Lamm: and ultimately it's educators. We want to design effective multiple choice items, so that the alignment makes sense for both, knowing whether students have attained the targeted learning objective, but also so that students feel they're being assessed fairly and accurately. 63 00:14:33.210 --> 00:14:34.390 Mia M Lamm: Next slide, please. 64 00:14:36.150 --> 00:14:50.370 Mia M Lamm: So to demonstrate what I mean by aligning a learning objective with your test question, Here's a very, very simple lower order, thinking example of an objective which is to recall the biological lifecycle biennial plant. 65 00:14:50.640 --> 00:15:08.570 Mia M Lamm: It's associated learning it, activity, which in the case that I've created is just a lecture going to watch the lecture about this topic, and then it's it's it's a. It's corresponding multiple choice question which is asking, how long does that biennial plan generally live with a series of response options? 66 00:15:08.570 --> 00:15:20.280 Mia M Lamm: I hope it's clear here when i'm trying to highlight the importance of aligning that cognitive level of your objective on the multiple choice question. They should both target the same level of learning. 67 00:15:20.500 --> 00:15:27.510 Mia M Lamm: And we're gonna go into that a little bit more. When we talk about higher order developing higher order questions in the session 68 00:15:27.870 --> 00:15:28.990 Mia M Lamm: next slide, please. 69 00:15:30.790 --> 00:15:45.080 Mia M Lamm: So, before you begin designing your assessments, we want to consider the purpose right. So how this assessment show you what students are. and does the assessment measure what it's supposed to measure relative to the course. Learning objectives 70 00:15:45.530 --> 00:16:04.490 Mia M Lamm: when there's a high degree of test validity this ensures it's relevancy to the curriculum, and the reliability will really depend on the consistency or dependability of a question, and the degree to which that test and and multiple choice questions consistently measures 71 00:16:04.710 --> 00:16:07.750 Mia M Lamm: the learning outcomes that you want to see. 72 00:16:10.950 --> 00:16:22.140 Mia M Lamm: So now let's dive a little bit. Now we've kind of created a little bit of that foundation. Let's dive into some best practices for the actual writing of our questions, some which has already been touched upon, which is great. 73 00:16:22.240 --> 00:16:24.580 Mia M Lamm: Next slide, please loop thanks so much. 74 00:16:24.800 --> 00:16:37.830 Mia M Lamm: So the STEM is the foundation of your multiple choice item, and after reading the STEM, the students should know exactly what the problem is right. So in other words, we want to make this done meaningful. 75 00:16:38.650 --> 00:16:41.660 Mia M Lamm: But the student has to infer the problem. 76 00:16:41.770 --> 00:16:58.200 Mia M Lamm: The item will likely measure more the students ability to draw inferences from being descriptions rather than achievement of their course objective. So you really want to concentrate on that clarity and that that STEM is a complete problem statement. 77 00:16:58.530 --> 00:17:11.619 Mia M Lamm: So the other thing I would like to mention is that a question and answer format usually works better than some other styles, such as filling the blank. Not that you shouldn't use them, but just to be aware that 78 00:17:12.089 --> 00:17:23.440 Mia M Lamm: a question and answer format is most similar to how we communicate in real life. Right? So when we ask someone for information, it's generally in that format, and it's a little bit of an easier cognitive load 79 00:17:23.500 --> 00:17:38.310 Mia M Lamm: and negatively worded items. Those are those statements where a student is instructed to identify the exception. the incorrect answer, or the least correct answer, and using negative such as not as one example 80 00:17:38.470 --> 00:17:53.030 Mia M Lamm: require students to keep reconstructing the response options in their minds, and trying to figure out what both what is not correct as well as what is correct at the same time, and it can pose some additional challenges in your quizzing. 81 00:17:53.210 --> 00:18:02.650 Mia M Lamm: So, because I know this is a common fit pitfall. The negative statements let's just take like 30 more seconds and talk a little bit more about that in the next slide. 82 00:18:04.870 --> 00:18:09.760 Mia M Lamm: Thank you. So negatively stated. Items are frequently used because 83 00:18:10.230 --> 00:18:20.490 Mia M Lamm: they're relatively easy to construct. They're easier. So developing one distractor rather than the 2 or 4 required for a positively worded item. It's just less time consuming. 84 00:18:20.750 --> 00:18:30.780 Mia M Lamm: And when researching for this workshop, I ran across this study. That's on the slide. It's titled negatively, worded, multiple choice questions and unavoidable threat. 85 00:18:30.820 --> 00:18:50.540 Mia M Lamm: and it was done at the University of Melbourne. And if you want to check it out. I've also linked it to our reference slides, and we'll we'll share these slides with you at the end of the when we follow, send our follow up email. Their data shows that negatively worded questions cause even well prepared students to select incorrect answers 86 00:18:51.030 --> 00:18:54.360 Mia M Lamm: and all that being said about Mego statements. Occasionally 87 00:18:54.440 --> 00:19:06.340 Mia M Lamm: we use them right, and sometimes they're appropriate. So you know, for just for one example. for objectives dealing with health or safety issues, where, knowing what not to do is important. 88 00:19:06.490 --> 00:19:20.080 Mia M Lamm: but for the most part it's a good idea to try and make a positive statement out of a negative. But if a negative must be used, the word should be emphasized by using underlying italics both face and capital similar to what you see on this slide. 89 00:19:20.190 --> 00:19:33.990 Mia M Lamm: In addition, being careful with those response options that they're not also negative. You want to phrase those positively to avoid forming a a confusing double negative item within, you know, with the STEM. 90 00:19:35.050 --> 00:19:42.890 Mia M Lamm: So, just to take a real quick moment, we do have a lot of material. We are gonna have an open Q. A. But are there any burning questions at this point? 91 00:19:47.770 --> 00:19:49.410 Mia M Lamm: Okay, let's move forward. 92 00:19:51.070 --> 00:19:58.590 Mia M Lamm: Let's talk about designing the response option a little bit. So when designing the response options. 2 or 3 distractors 93 00:19:58.650 --> 00:20:11.780 Mia M Lamm: are as effective as 4 or 5 options, and typically the fourth response, Option is the most difficult and time consuming to write and statistically contributes really very little information 94 00:20:11.890 --> 00:20:17.100 Mia M Lamm: from about what you want to gain from the students response to that item. 95 00:20:17.930 --> 00:20:23.780 Mia M Lamm: Numerous studies indicate that multiple choice items that have multiple correct answers. 96 00:20:23.930 --> 00:20:27.030 Mia M Lamm: I know this is a little bit controversial, but they 97 00:20:27.220 --> 00:20:32.100 Mia M Lamm: a proven to be lower in reliability, lower in discrimination 98 00:20:32.160 --> 00:20:44.480 Mia M Lamm: and higher difficulty when compared with the question with one correct answer. So i'll just put that out there. And another approach to crafting a strong multiple choice item is to avoid humorous distractors, really tempting to do. 99 00:20:44.630 --> 00:20:56.610 Mia M Lamm: But it's not only makes the question incredibly easy in some cases, or confusing, and others it it's really a good idea to make those other response options plausible. 100 00:20:56.650 --> 00:21:13.990 Mia M Lamm: and for another reason to do that is also because it can help diagnose where students went wrong in their thinking. So if you create plausible response options that you know kind of cover some gaps, it can also help identify where things are going. Maybe wrong 101 00:21:16.190 --> 00:21:17.740 Mia M Lamm: next slide. Thanks. 102 00:21:17.860 --> 00:21:21.140 Mia M Lamm: So here's an example. 103 00:21:21.990 --> 00:21:39.110 Mia M Lamm: Oh, sure, I can do that like. And so here's an example of what could be what could be improved in a multiple choice question items. So here's one example. This is a couple of flaws that i'd like to point out. But can you guess what they are? So you're welcome to chat real quick. Any ideas 104 00:21:41.540 --> 00:21:48.630 Jennifer Applegate: and mia. Well, folks are putting things in the chat. Brent asked if you could briefly summarize what those 4 quadrants mean 105 00:21:48.810 --> 00:21:53.050 Jennifer Applegate: reliability, dependability, and the other 2 106 00:21:53.990 --> 00:22:15.980 Mia M Lamm: sure. So the so, the the just going back to me. So the the right, the validity. Is that the the test question is, it's aligned? The question is solid. The statements are written Well, it it's a valid. It's going to have validity within that curriculum. So it's going to make sense in your course in a nutshell. There's a little bit more to it. 107 00:22:15.980 --> 00:22:35.190 Mia M Lamm: and then the reliability is really concerned with the consistency or dependability of a question, so that again it's it's falling into alignment. It's. It's going to be clear, and the the results that are coming out of that test question are going to be a true assessment 108 00:22:35.190 --> 00:22:36.800 Mia M Lamm: of students learning 109 00:22:37.580 --> 00:22:47.640 Mia M Lamm: right? And i'm happy to provide more in specifically about this. And this is great is a great question in our follow up email, or when we talk a little bit more in our Q. A. 110 00:22:52.900 --> 00:22:56.290 Mia M Lamm: So any ideas about this question i'll move on. If not. 111 00:22:58.250 --> 00:23:02.210 Mia M Lamm: okay. So let's go ahead and move forward. Let's look at 112 00:23:04.290 --> 00:23:05.260 Mia M Lamm: where we're at. 113 00:23:05.260 --> 00:23:25.530 Mia M Lamm: So the problem here with this multiple choice question is that the STEM isn't meaningful right? Which of the did? The following is a true statement. And what has happened in this construction is the some of the problem statement has been put into the response options. It also makes the response options already and also redundant right. So you have some 114 00:23:25.530 --> 00:23:30.910 Mia M Lamm: things like mitochondrial genomes repeated in each response, and let's take a quick look at the revision. 115 00:23:32.740 --> 00:23:52.550 Mia M Lamm: So the STEM should be meaningful. You could see the revision on the right. The STEM should be meaningful by itself, and to present a definite problem. So additionally, the alternative should be stated clearly and concisely with that redundancy removed, and you can see how it's a much more readable and easy question to to answer 116 00:23:53.090 --> 00:24:07.580 Mia M Lamm: right. So items that are excessively wording wordy really are really assessing the speed and ability of students reading rather than than their attainment of the of the learning objective and knowing what that content is. 117 00:24:09.350 --> 00:24:23.550 Mia M Lamm: Okay and to move forward. Thank you. So to wrap this section up. Let's briefly touch on some approaches beyond the writing part of that multiple choice questions and look at the resources and strategies needed for the assessment support. 118 00:24:24.730 --> 00:24:26.850 Mia M Lamm: It helps students if you avoid. 119 00:24:26.850 --> 00:24:47.800 Mia M Lamm: Not yet. Yeah, thank you. It helps students. If you avoid using trick questions and providing test parameters, for example, by being transparent with with your students about the content. The test is covering from the material they've studied, and providing clear instructions for any limits, restrictions, timing, or expectations you have for the students. 120 00:24:47.800 --> 00:24:52.510 Mia M Lamm: and it's also really helpful to use terminology that's familiar to all students in your class. 121 00:24:52.810 --> 00:25:02.780 Mia M Lamm: This is particularly vital for students whose primary language is not English right? So you might try to avoid just for one example, the use of idioms. 122 00:25:02.780 --> 00:25:17.500 Mia M Lamm: While these may seem like common knowledge. They are culturally and linguistically situated, and Don't always translate across a variety of languages right? So this makes them a potential challenge, and possibly even unfair for your students. 123 00:25:20.830 --> 00:25:21.990 Mia M Lamm: Next slide, please. 124 00:25:22.490 --> 00:25:36.020 Mia M Lamm: And i'd also like to mention the importance of leaning into an inclusive mindset when preparing your exam. So, in addition to some of the concepts that we've discussed some strategies for this might include offering office hours before high stakes, exam 125 00:25:36.180 --> 00:25:38.890 Mia M Lamm: organizing content review sessions 126 00:25:38.930 --> 00:25:53.740 Mia M Lamm: and prepare students if they need to interpret visuals, charts, or figures on an upcoming exam. and lastly, provide you study guides and vocabulary lists, or tests, so that everybody can kind of calibrate to the same page. 127 00:25:54.630 --> 00:26:01.490 Mia M Lamm: So now i'll turn it back to Lou to dive into our higher order, thinking questions. Thanks so much. 128 00:26:03.140 --> 00:26:05.120 BSPH CTL Teaching Toolkit: you mia. 129 00:26:05.500 --> 00:26:14.690 BSPH CTL Teaching Toolkit: when we try to write, cast we, P. You. You want to assize a range of things that students should be able to do as a result of the instruction. 130 00:26:14.800 --> 00:26:27.830 BSPH CTL Teaching Toolkit: So we also want to determine, if we remember important elements of the counting. But we also want to know if I, if we can use these elements to understand and solve problems. 131 00:26:28.090 --> 00:26:44.940 BSPH CTL Teaching Toolkit: it's a relatively straightforward to read multiple choice question, to determine whether students remember what we want them to remember. But it can be more challenging to write questions that cast hair, other skills of being able to use the information. 132 00:26:45.190 --> 00:26:47.420 BSPH CTL Teaching Toolkit: So as the first part of 133 00:26:47.470 --> 00:27:04.930 BSPH CTL Teaching Toolkit: the multiple Choice Question Workshop series, we will introduce the bloom of taxonomy, of combination from work, and as some strategies to converting some strategies, i'm converting lower other thinking, multiple choice questions to hire other thinking ones. 134 00:27:04.930 --> 00:27:10.790 BSPH CTL Teaching Toolkit: In the future we will have the third part, which we will get paper 135 00:27:11.100 --> 00:27:17.890 BSPH CTL Teaching Toolkit: and specifically AIM for guiding faculty and developing higher out of thinking. Multiple choice questions. 136 00:27:20.770 --> 00:27:31.550 BSPH CTL Teaching Toolkit: Almost taxonomy of cognition is a framework that's start with this 2 levels of thinking as important basis, remember and understand. 137 00:27:31.820 --> 00:27:36.870 BSPH CTL Teaching Toolkit: of pushing our brains to 5 other tear all the levels of thinking. 138 00:27:36.960 --> 00:27:47.480 BSPH CTL Teaching Toolkit: helping us move beyond remembering and the recording information, and then move it into application, analysis, evaluation, and a creation. 139 00:27:48.370 --> 00:28:07.240 BSPH CTL Teaching Toolkit: The magic comes with the associated verbs in the figure shown. We have about 10 verbs per category. But if you Google bloom with the verbs, you can find more and more. So. These verbs can be inspirational, as you think about how to ask the questions 140 00:28:07.280 --> 00:28:10.910 BSPH CTL Teaching Toolkit: to determine whether students can do sort of 141 00:28:11.000 --> 00:28:16.500 BSPH CTL Teaching Toolkit: vacation, and that is the evaluation which is appropriate for your teaching. 142 00:28:19.740 --> 00:28:37.340 BSPH CTL Teaching Toolkit: Here are some approaches we would like to share with you, to convert over all the thinking, multiple choice, to hire all the thinking, multiple choice. So the first approach is yield scenario based atoms, particularly one that 143 00:28:37.340 --> 00:28:40.740 BSPH CTL Teaching Toolkit: simulate reward experiences. 144 00:28:40.760 --> 00:28:51.030 BSPH CTL Teaching Toolkit: You read in this table with scenario. We should ask the students to make a choice that mimics how they would use their knowledge in a real world setting. 145 00:28:51.250 --> 00:29:02.600 BSPH CTL Teaching Toolkit: This should be an appropriate level of complexity tax team. All they should be able to use the information at this point in their educational progression. 146 00:29:05.260 --> 00:29:17.100 BSPH CTL Teaching Toolkit: The second approach is asking learners to analyze or interpret it. Information from they use, provided as a part of the atom stamp or response options. 147 00:29:17.200 --> 00:29:29.370 BSPH CTL Teaching Toolkit: In many cases we juice. So she has a job is charged cables with tables, data, size programs, and it graphs simulate the real world tasks. 148 00:29:32.280 --> 00:29:44.020 BSPH CTL Teaching Toolkit: The third approach is asking learners to select to the option with the correct answer supported with the logical reasoning based on the knowledge and the skills being assessed 149 00:29:44.060 --> 00:29:50.040 BSPH CTL Teaching Toolkit: like the example, you can give a several options, options, and a reasonings 150 00:29:50.120 --> 00:29:54.750 BSPH CTL Teaching Toolkit: can be a combination of wrong option. But corrective reasoning 151 00:29:54.830 --> 00:29:59.120 BSPH CTL Teaching Toolkit: all correct option but wrong reasoning as it these tractors. 152 00:30:02.110 --> 00:30:08.400 BSPH CTL Teaching Toolkit: Alright, so now it's rate of the following stamps of the multiple choice questions. 153 00:30:08.480 --> 00:30:24.860 BSPH CTL Teaching Toolkit: And let's suppose a minute for one for each of them to think about how to convert them from a lower level, multiple choice to hair level, multiple choice. You can use the approaches we just mentioned. 154 00:30:25.150 --> 00:30:27.610 BSPH CTL Teaching Toolkit: The first example is 155 00:30:27.690 --> 00:30:37.240 BSPH CTL Teaching Toolkit: they att to rate is susceptible. People who have been exposed to a primary case. They refer to as the blank. 156 00:30:37.970 --> 00:30:42.190 BSPH CTL Teaching Toolkit: so you can share your so in the chat 157 00:31:01.350 --> 00:31:05.150 BSPH CTL Teaching Toolkit: i'll bring to team. I mentioned you incidents. 158 00:31:10.330 --> 00:31:13.500 BSPH CTL Teaching Toolkit: Are you? Are you talking about 159 00:31:14.020 --> 00:31:20.200 BSPH CTL Teaching Toolkit: using this in this cases for this STEM, or you are filling the bank. 160 00:31:30.240 --> 00:31:31.930 BSPH CTL Teaching Toolkit: All right. 161 00:31:33.010 --> 00:31:36.020 BSPH CTL Teaching Toolkit: I' to move on. Let's see 162 00:31:37.020 --> 00:31:53.600 BSPH CTL Teaching Toolkit: one of the reviewing examples I I I picked so you can see the revision here. It's a revision is using a real world scenario. This revision not only made the STEM a complex statement 163 00:31:53.600 --> 00:32:05.240 BSPH CTL Teaching Toolkit: which make the stamp use meaningful and presence a definitely problem as me. Talk about this just now, but also it encourage you to apply the concept. 164 00:32:05.360 --> 00:32:10.000 BSPH CTL Teaching Toolkit: We'll set up 165 00:32:10.190 --> 00:32:20.920 BSPH CTL Teaching Toolkit: a factory sorry, and a secondary attacker to a real world case, which is a much beyond memorizing the definition of the secondary tax rate. 166 00:32:21.270 --> 00:32:32.280 BSPH CTL Teaching Toolkit: You can see that a use of verb calculate in the STEM, which is a so, say a. With the apply category in the bloom of the Texas solely. 167 00:32:34.980 --> 00:32:43.020 BSPH CTL Teaching Toolkit: Alright, Here is a second example, name 3 qualities of effective entrepreneurs. 168 00:32:45.690 --> 00:32:49.690 BSPH CTL Teaching Toolkit: You're welcome to share your revisions in the chat. 169 00:33:22.920 --> 00:33:42.650 BSPH CTL Teaching Toolkit: Okay. Now let's see one of the revision examples how to make this question as a hair, other thinking, multiple choice question. So the revenue is review the financial. So this is 3 start up companies which one would you invest again. And the why 170 00:33:42.970 --> 00:33:50.810 BSPH CTL Teaching Toolkit: this is revealing you, that of you know, like a student, same poly recall 3 qualities of effective entrepreneurs. 171 00:33:50.840 --> 00:33:56.080 BSPH CTL Teaching Toolkit: So which are the this. This revision, you 172 00:33:56.130 --> 00:34:01.620 BSPH CTL Teaching Toolkit: real work case, which are the financials of 3 companies. 173 00:34:01.770 --> 00:34:11.340 BSPH CTL Teaching Toolkit: also required students to apply the concept of qualities of a active entrepreneurs into the analysis of the financials 174 00:34:11.420 --> 00:34:16.889 BSPH CTL Teaching Toolkit: to make a selection. They decide on the corresponding reasoning and it. 175 00:34:17.989 --> 00:34:26.860 BSPH CTL Teaching Toolkit: Looking at those 2 examples, we know that constructing multiple choice atoms, sizing hair, Auto thinking is a possible. 176 00:34:26.870 --> 00:34:32.770 BSPH CTL Teaching Toolkit: Moreover, it is worse. Well investing the time. The resources. In doing so 177 00:34:33.110 --> 00:34:38.620 BSPH CTL Teaching Toolkit: I need to be challenging multiple choice Adams offer attractive balance. 178 00:34:38.750 --> 00:34:49.489 BSPH CTL Teaching Toolkit: They have a potential to simultaneously meet the needs of learners seeking to improve their learning through the media of assessment. 179 00:34:49.570 --> 00:34:56.110 BSPH CTL Teaching Toolkit: and the educators who wish to opt in meaningful information about students. 180 00:34:57.620 --> 00:35:02.230 BSPH CTL Teaching Toolkit: Alright, Next i'm gonna turn over to Murray Kate 181 00:35:06.760 --> 00:35:23.620 Marykate Bergen: Great. So academic integrity, obviously core to everything that we do as course, faculty and course, teaching teams and multiple choice questions certainly raise some issues related to academic integrity and the code of conduct here. So 182 00:35:23.620 --> 00:35:34.940 Marykate Bergen: we often hear from instructors that a concern of using multiple choice questions is that students may cheat, because obviously being largely online students, aren't generally taking these 183 00:35:34.940 --> 00:36:03.040 Marykate Bergen: exams in a setting where there's a lot of oversight. So we have a few tips that can help to promote academic integrity when you're using these types of assessments, and one is just reducing the weight of these types of exams in terms of the overall grades. So, taking the stakes down a little bit, and having smaller or more frequent exams that are worth less when it comes to the final grade. Really using that idea of a formative assessment. 184 00:36:03.040 --> 00:36:32.360 Marykate Bergen: You notice that Mary mentioned in the chat that she uses these tools to get an idea before the content is introduced, of what students know, and really getting a baseline, so that could be a way to use these as well. Honor statements can be very effective. I know as a student when I see those when i'm about to take an exam. It's just a good reminder for me. And then thinking about times where an open book exam could be appropriate, just another chance for students to engage with the course material. 185 00:36:32.360 --> 00:36:40.140 Marykate Bergen: or a chance to collaborate with other students, provided that the instructions are very clear around that next slide, please. 186 00:36:42.480 --> 00:37:12.390 Marykate Bergen: If you use course plus to generate your quizzes. The tool that's actually called course plus generator is very helpful. In this way. You can use things like question groups. So that allows you to pick a limited number of questions from the overall pool, so no 2 students will have the exact same exam at the exact same time. So that's a nice feature of that software that's available multiple versions of an Exam: always a useful tool. So that's multiple students in an in person. 187 00:37:12.390 --> 00:37:26.850 Marykate Bergen: The in class can have different exams as well. Other best practices include a time limit to keep students focused and invested in the exam as opposed to going outside. Of course, costs are are looking at other resources 188 00:37:26.900 --> 00:37:50.770 Marykate Bergen: and randomizing questions, or having different test versions as we've already talked about. With course plus, can be very useful and communicating with your students. You all are here. You're taking this workshop. I'm sure you all do that already. But really just taking down the intensity and and limiting the stress for students as much as possible, so that cheating feels a little bit less necessary or appealing. 189 00:37:50.770 --> 00:38:17.700 Marykate Bergen: And what I've learned from this course Facilitator team is having a quiz set for each course can be really helpful. So, having multiple questions that focus on the content that can be cycled in and out. So they're still very useful tools, but they align very well with the principles and goals around academic integrity. So we have just a couple of minutes for anyone to share tools, tips, and tricks in the chat 190 00:38:17.770 --> 00:38:27.400 Marykate Bergen: things you found helpful when it comes to academic integrity and your assessments, particularly around multiple choice questions, any best practices that you all have found. 191 00:38:33.830 --> 00:38:42.480 Mia M Lamm: I can actually speak to that. If you don't mind Pamela Berg just mentioned. I just did it in my class. 192 00:38:42.480 --> 00:38:57.310 Mia M Lamm: adding the in video questions. Those are the interactive lecture quizzes that you can add as part of online material, an online lecture using that tool in the quiz generator and course plus the platform. 193 00:38:57.520 --> 00:39:09.240 Mia M Lamm: So and it's a great tool for highlighting concepts. I actually just added it. We just added it in our class that we teach it teaching, learning, and leading. 194 00:39:09.240 --> 00:39:23.460 Mia M Lamm: and students love it, and I was so kind of surprised. I think they found it really helpful to just take just kind of flop, you know, a couple of little questions, or even a prompt just to help 195 00:39:23.610 --> 00:39:40.520 Mia M Lamm: them concentrate on those most valuable concept or high level concepts, or, you know, notes or topic areas where you want them to focus for whatever they need to apply in their assessment period. So I I yeah, I think those are. There's it's a great way to have some formative assessment. 196 00:39:40.520 --> 00:39:44.270 Mia M Lamm: There are some limitations in that. If you download the material 197 00:39:44.740 --> 00:39:59.340 Mia M Lamm: from course, plus, it's like. If the student downloads the Mp. For those quizzes, don't come with it. So it has some limitations, as far as being a graded assessment. But if you're really just trying to create like a knowledge that kind of experience and it works really great. 198 00:39:59.730 --> 00:40:02.770 Mia M Lamm: I think. Yeah, happy to talk more about that. 199 00:40:05.390 --> 00:40:23.140 Jennifer Applegate: I was just gonna add when talking about to making multiple quiz sets. And again, I think we're in a subsequent workshop. We're going to focus more on how you would think through this and do this with course. But as you're if you are thinking of doing that as a strategy. And you're writing your multiple choice questions. 200 00:40:23.140 --> 00:40:33.830 Jennifer Applegate: If you're able to write one question and then think about how do I test the same concept a little bit differently, because you're gonna want those questions in both 201 00:40:33.830 --> 00:40:52.700 Jennifer Applegate: of the different quizzes to be equitable and of equal weight and testing the same concepts to be fair. The students, so it can feel a little bit like double work. But if you're doing a question, I recommend doing it one to one, instead of like writing a whole quiz, and then going through and saying, now i'm gonna write another quiz set. 202 00:40:52.700 --> 00:41:11.550 Jennifer Applegate: So just thinking through. If you're writing one question, if you can ask it in a different way, go ahead and jot that down, and then maybe you'll end up with a a whole other set of quizzes. But another point with that is, it doesn't have to be every question that you have a separate 203 00:41:11.550 --> 00:41:25.860 Jennifer Applegate: question, for you might only do like 5 questions or a sampling of questions that are slightly different, and both of those can help to decrease the risk of sharing of answers among students. 204 00:41:31.740 --> 00:41:39.900 BSPH CTL Teaching Toolkit: Alright. Thank you very much for sharing the tapes, you know, to promote academic. 205 00:41:41.570 --> 00:41:43.440 BSPH CTL Teaching Toolkit: Now let's move on. 206 00:41:44.650 --> 00:41:54.570 BSPH CTL Teaching Toolkit: So before we end this session, please take a minute and reflect what are the key points. So take away from today's workshop. 207 00:41:55.400 --> 00:41:59.160 BSPH CTL Teaching Toolkit: No, i'm gonna cause the first 30 s. 208 00:41:59.480 --> 00:42:03.240 BSPH CTL Teaching Toolkit: you know. Welcome to share your, you know, takeaways in the chat. 209 00:42:15.430 --> 00:42:23.590 BSPH CTL Teaching Toolkit: Oh, i'm less share that she will reveal or quizzes for first term. Yeah, that's a very. 210 00:42:25.310 --> 00:42:26.270 BSPH CTL Teaching Toolkit: you know. 211 00:42:26.790 --> 00:42:33.150 BSPH CTL Teaching Toolkit: Thank you for you know, attending our workshop, and then, you know, take actions 212 00:42:35.260 --> 00:42:38.590 BSPH CTL Teaching Toolkit: so, and came share that 213 00:42:38.660 --> 00:42:51.120 BSPH CTL Teaching Toolkit: we will like Frame the stamp positively. Your reframe. A multiple choice fishing can move it up. Blooms text on yeah, very, very good 214 00:42:51.190 --> 00:42:54.610 BSPH CTL Teaching Toolkit: sauce on this. Yeah. So 215 00:42:55.240 --> 00:43:05.950 BSPH CTL Teaching Toolkit: as me, I mentioned it's very important to you know. Frame the STEM positively. And also you can, you know, make your multiple choice questions. 216 00:43:07.770 --> 00:43:14.410 BSPH CTL Teaching Toolkit: Our a range of cognitive levels from a Bloomus tax. 217 00:43:15.500 --> 00:43:21.250 BSPH CTL Teaching Toolkit: Alright, Thank you for sharing your takeaways 218 00:43:22.280 --> 00:43:32.660 BSPH CTL Teaching Toolkit: to rip up. What do we have covered in this workshop? There are some who and don't do so we win. If we want to construct a good multiple choice question. 219 00:43:32.940 --> 00:43:42.130 BSPH CTL Teaching Toolkit: So first, that we should avoid excessive use of none of us above, or in some taxes sentences. 220 00:43:42.660 --> 00:43:49.400 BSPH CTL Teaching Toolkit: we should award excessively use of a negative phrasing as a brand I just mentioned. 221 00:43:49.550 --> 00:44:02.180 BSPH CTL Teaching Toolkit: we should also avoid using culturally best and a female terminology. Lastly, we should avoid using and possible. These factors. 222 00:44:03.900 --> 00:44:11.490 BSPH CTL Teaching Toolkit: the highly recommended that we craft a clear and aligned questions that are meaningful. 223 00:44:12.030 --> 00:44:22.590 BSPH CTL Teaching Toolkit: You know we should use the strategies to support our learners especially, you know, learners whose first language is not English. 224 00:44:24.300 --> 00:44:37.860 BSPH CTL Teaching Toolkit: I think we should the right questions at a very cognitive level, driven by the learning objectives. Lastly, we showed the employee strategies to promote academic integrity. 225 00:44:40.970 --> 00:44:49.120 BSPH CTL Teaching Toolkit: All right. Now please feel free to ask any questions. We have an answer. Yeah. in the chat. 226 00:44:50.420 --> 00:45:07.130 BSPH CTL Teaching Toolkit: If you don't have any questions we would like to ask you like, what do you want to learn more about what we can improve for our workshops? Please share your thoughts in the chat to help us make our future workshops better 227 00:45:22.650 --> 00:45:38.620 Mia M Lamm: sure. So Brent has asked, Why select all that? Apply? Questions are not ideal, and could be explain why, and I again, I will follow up with. And I posted here in the chat some detailed 228 00:45:38.810 --> 00:45:47.760 Mia M Lamm: documents that might help with some further detail, but typically like as I mentioned. 229 00:45:48.330 --> 00:46:02.050 Mia M Lamm: and studies have shown that, having the multiple correct answers, well, not i'm not saying that they're completely wrong, but they are a little bit more confusing. So, depending on how clearly they're worded. 230 00:46:02.050 --> 00:46:14.280 Mia M Lamm: it can be confusing for students. It can also make the question a little bit easier. For example, once there's that select, i'll select all that apply in the you know. Usually, you know you have a a STEM at the. 231 00:46:14.540 --> 00:46:32.000 Mia M Lamm: and you might put select all it. Apply kind of the learning students that there's multiple correct answers, which is the appropriate way to go. But it also can make a question a little bit easier, because they already know they're looking for multiple multiple questions as well as just not being as 232 00:46:32.160 --> 00:46:44.720 Mia M Lamm: as reliable a measure versus a very zoom question with one correct answer. It could be a little bit easier sometimes. There's some concepts that are 233 00:46:44.730 --> 00:46:54.460 Mia M Lamm: hard to ungroup, and so it's a little harder to write, so that being said, it's not that you shouldn't use it, but sometimes it could be overused, and it's not 234 00:46:54.500 --> 00:47:00.340 Mia M Lamm: quite as good a strong of an assessment in most cases. 235 00:47:04.630 --> 00:47:05.720 BSPH CTL Teaching Toolkit: Thank you, mia. 236 00:47:15.060 --> 00:47:16.480 Mia M Lamm: That's true, Kathy. 237 00:47:24.260 --> 00:47:30.410 BSPH CTL Teaching Toolkit: All right. Seems we don't have any questions right now. 238 00:47:31.650 --> 00:47:35.830 BSPH CTL Teaching Toolkit: Yeah. So feel free to, you know. 239 00:47:35.890 --> 00:47:48.770 BSPH CTL Teaching Toolkit: Ask any question, even after the workshop we will send a follow up email, so hesitate to contact us. If you need any help writing your multiple to questions. 240 00:47:49.030 --> 00:47:51.970 BSPH CTL Teaching Toolkit: All right. So let's move on. 241 00:47:52.360 --> 00:48:04.210 BSPH CTL Teaching Toolkit: So today's workshop is a first part of our multiple choice Question workshop series. Please make sure to check out the part, 2 and part 3 in the near future. 242 00:48:04.440 --> 00:48:17.860 BSPH CTL Teaching Toolkit: The course past, quiz, generator, workshop will focus on you introducing this creation and settings as well as question, size, and how to use them to manage. Quiz questions. 243 00:48:17.880 --> 00:48:21.120 BSPH CTL Teaching Toolkit: address, academic integrity, concern. 244 00:48:21.470 --> 00:48:27.150 BSPH CTL Teaching Toolkit: and the part 3 is writing multiple choice question for hair a level thinking 245 00:48:27.720 --> 00:48:39.690 BSPH CTL Teaching Toolkit: we that deeper into creating multiple choice, that as in critical thinking, and we have faculty and tas with the opportunities for hands on exercise. 246 00:48:39.730 --> 00:48:43.380 BSPH CTL Teaching Toolkit: So you can check the days and registration links 247 00:48:43.480 --> 00:48:50.810 BSPH CTL Teaching Toolkit: on the events page of our teaching. Okay, website, speed, Ray and school events. Canada. 248 00:48:53.840 --> 00:48:59.510 BSPH CTL Teaching Toolkit: the center for teaching and learning is ready to help faculty and tas in many ways. 249 00:48:59.520 --> 00:49:14.640 BSPH CTL Teaching Toolkit: including but not limited to course, season course, development course, revenge and educational technology integration. So please feel free to reach out to us. Use any of the masses list here. 250 00:49:18.510 --> 00:49:31.700 BSPH CTL Teaching Toolkit: We also developed a session evaluation survey. We really value your fit act. This is survey. A survey will be sent in our follow up email as well. 251 00:49:35.110 --> 00:49:49.200 BSPH CTL Teaching Toolkit: Alright, Thank you very much for joining us today. I hope that you found this workshop information looking forward to seeing you all again in the future. Ctl: events have a good one. 252 00:49:52.750 --> 00:49:53.770 BSPH CTL Teaching Toolkit: Thank you.