WEBVTT 1 00:00:22.490 --> 00:00:24.679 Mia Lamm: Lou, I believe you're muted. 2 00:00:24.780 --> 00:00:25.870 Mia Lamm: Hi, everyone. 3 00:00:26.800 --> 00:00:28.110 Hossein Zare: Hi, good morning. 4 00:00:28.230 --> 00:00:46.439 BSPH CTL Teaching Toolkit: I'm sorry. All right. Yeah, thank you for joining us today. today's workshop is about leveraging the course plus Chris generator to promote academic integrity hosted by the center for teaching and learning. 5 00:00:46.530 --> 00:00:54.200 BSPH CTL Teaching Toolkit: this workshop will be recorded slides and recording will be shared with you all after worse 6 00:00:54.340 --> 00:01:02.039 BSPH CTL Teaching Toolkit: and and 0 will be our Q. A session at the end. however, you will come to ask questions during the session. 7 00:01:06.210 --> 00:01:20.390 BSPH CTL Teaching Toolkit: So all right. my name is Lucy. I am an instructional designer at Ctl. we also have a 3 co facilitators mia, Brian and Steph, would you like to introduce yourself? 8 00:01:21.560 --> 00:01:32.240 Mia Lamm: Hi, everyone! My name is mia Lam. I'm a senior instructional designer with the center for teaching and learning as well as an associate in the Department of Health policy and management. 9 00:01:33.880 --> 00:01:45.210 Brian Klaas: I'm Brian Kloss. I'm the associate director for technology here at the center for teaching and learning. I also have a faculty appointment here in the schools. R. 3, center for innovation in science education. 10 00:01:46.370 --> 00:02:02.320 Seth Blackburn: good morning, everyone. I'm Seth Blackburn, and I'm the center for teaching and learning self-test support specialists, or what we call Ctl help. this basically means on the main person responsible for responding to support tickets regarding course plus and also keeping up with the documentation on our CTO help website. 11 00:02:03.520 --> 00:02:19.650 BSPH CTL Teaching Toolkit: All right. Thank you. So let me start with the learning outcomes for this workshop. all these full learning outcomes. we will help you promote at them integrity in your multiple choice question assessment. 12 00:02:22.170 --> 00:02:35.650 BSPH CTL Teaching Toolkit: Is this a workshop? I includes the 5 sections, assessment quick question creation quick settings. quiz statistics and closing with extra resources. 13 00:02:38.040 --> 00:02:58.400 BSPH CTL Teaching Toolkit: One of the major job acts of a multiple choice question assessment, even if well designed, is that they are prone to cheating. So before we dive in to discuss how Chris Generator can help you reduce cheating. let's explore why students achieve. In the first place. 14 00:02:58.510 --> 00:03:04.200 BSPH CTL Teaching Toolkit: So here is the all is a share. Your thoughts. 15 00:03:08.830 --> 00:03:14.830 BSPH CTL Teaching Toolkit: It's a short answer poll question you can, you know. Give your answer like 16 00:03:15.010 --> 00:03:22.379 BSPH CTL Teaching Toolkit: a few worse. Have you give you about 30 s, and then we can share the poll. raise off 17 00:03:58.720 --> 00:04:04.359 BSPH CTL Teaching Toolkit: all right. let's in this call and then share the relots 18 00:04:09.020 --> 00:04:20.579 BSPH CTL Teaching Toolkit: so we can see that. the answer, like, I sure to succeed. is management of time competition. Yeah, those are a great answers. 19 00:04:20.820 --> 00:04:25.640 BSPH CTL Teaching Toolkit: so this is the research you found out. 20 00:04:27.470 --> 00:04:47.019 BSPH CTL Teaching Toolkit: So some research. There are 5 main reasons that students would like to cheat. So the first one is that students may have misunderstanding of what compromise academics is honest. They do to cultural all regional differences. 21 00:04:47.070 --> 00:05:00.790 BSPH CTL Teaching Toolkit: all they have misunderstanding of directions to all expectations. The second reason is that they have purchased management pro destination or this organization. 22 00:05:01.650 --> 00:05:18.530 BSPH CTL Teaching Toolkit: So the reason is that students may feel disconnected with the course, with the subject, with the, you know the instructor, or even the learning materials. So it's like inclusive learning. Environment is not provided. 23 00:05:18.950 --> 00:05:27.120 BSPH CTL Teaching Toolkit: First, reason is, students have the fear of failure or lack of confidence in their ability. 24 00:05:27.260 --> 00:05:34.610 BSPH CTL Teaching Toolkit: last one, maybe, you know, students have a 0 to depression or other mental health problems. 25 00:05:37.390 --> 00:05:47.830 BSPH CTL Teaching Toolkit: looking at this graph it, there is shows that students are more likely to engage academic dishonesty on 5 weighted assessments. 26 00:05:48.020 --> 00:05:52.400 BSPH CTL Teaching Toolkit: I 6 exams increase pressure on students 27 00:05:52.550 --> 00:06:02.449 BSPH CTL Teaching Toolkit: before. Therefore, the lack of of students being tempted to cheat, especially when students feel unsure about their ability. 28 00:06:05.140 --> 00:06:10.910 BSPH CTL Teaching Toolkit: So in the first part of this workshop I will talk about. I'll know the 29 00:06:10.930 --> 00:06:30.380 BSPH CTL Teaching Toolkit: and a formative multiple choice assessment can promote academic integrity before you're creating your multiple to ask questions. You need to first decide how to use it and include the method of assessment in your syllabus. Will this multiple to us question quizzes or exams to be graded. 30 00:06:30.450 --> 00:06:47.329 BSPH CTL Teaching Toolkit: so how much will be they? How much will it be weighted, and will will they be spread out through a term such as weekly quizzes? Or will they be assigned at a constraint to the times in the term, such as midterm and a final exam? 31 00:06:47.630 --> 00:06:59.160 BSPH CTL Teaching Toolkit: No sticks and a former tape assessments. Are you already used interchangeably? that you already have no no point value, as they monitor students learning throughout the term 32 00:06:59.380 --> 00:07:07.480 BSPH CTL Teaching Toolkit: it, improve the ads and the quality of the student learning, with the frequent opportunities for checking knowledge, is all skills 33 00:07:07.570 --> 00:07:13.470 BSPH CTL Teaching Toolkit: they provide ongoing feedback to instructors about how teaching of that learning. 34 00:07:15.830 --> 00:07:28.969 BSPH CTL Teaching Toolkit: So I do. We recommend using multiple to us questions as low sticks and a formative assessment wherever possible. Building company with a summer table assessment. there are a lot of benefits. 35 00:07:29.060 --> 00:07:35.280 BSPH CTL Teaching Toolkit: low steps and affirmative assessment provides a license for learning environment. 36 00:07:35.310 --> 00:07:40.119 BSPH CTL Teaching Toolkit: It lowers the students in also exam. Of course. 37 00:07:40.410 --> 00:07:55.689 BSPH CTL Teaching Toolkit: it up there. Confidence by small achievements, it connects them. There is a course counting monthly. There are some examples using multiple choice questions as a no stick in the formative assessment. Of course. 38 00:07:55.700 --> 00:08:00.149 BSPH CTL Teaching Toolkit: for example, in that trip is this, knowledge checks 39 00:08:00.180 --> 00:08:02.170 BSPH CTL Teaching Toolkit: free tasks and posts. 40 00:08:02.220 --> 00:08:07.209 BSPH CTL Teaching Toolkit: practice quizzes or review quizzes, and a self assessment. 41 00:08:08.720 --> 00:08:15.320 BSPH CTL Teaching Toolkit: All right next. I will send over to that to talk about question creation. 42 00:08:16.650 --> 00:08:29.799 Seth Blackburn: yes. Hello again. Everyone I I like Lu just said I'm going to be covering part 2 of this workshop which is about quiz question creation. there are 3 main tools within the question that I'll be going over today. Quiz that question blocks and question tagging 43 00:08:30.710 --> 00:08:31.979 Seth Blackburn: next slide, please. 44 00:08:33.110 --> 00:08:50.370 Seth Blackburn: So our first question today, what is the quiz set? quiz sets are literally a set of 2 or more quizzes that appear as a single quiz to students. a good real world comparison to quiz sets would be having different stacks of exams in a classroom, and randomly picking one of the exams to give to each. And as they walk into the classroom. 45 00:08:51.060 --> 00:08:56.339 Seth Blackburn: and before you decide, if you want to implement a quiz set for your zoom. There are some important things you want to keep in mind. 46 00:08:56.460 --> 00:09:18.229 Seth Blackburn: A quiz set is comprised of the quizzes that already exist in the quiz generator for your course, and each quiz that goes into the set must be set up before you can create your quiz set you can have as many quizzes in a set as you want. However, all the quizzes in the set must have the same total point value. This prevents you from having some students get assigned a quiz with 45 points, while other students get a sonic quiz to say 50 points. 47 00:09:18.630 --> 00:09:27.949 Seth Blackburn: one of the most efficient things about using quiz sets is that the settings for a quiz that override individual with settings, thus cutting down the amount of time you need to spend in the quiz generator. 48 00:09:28.420 --> 00:09:42.569 Seth Blackburn: Essentially, you don't need to spend time configuring the set of details of each quiz in a set, because the settings, the quiz itself will override these individual settings. This pertains to details such as the open and close dates, the time allowed to complete the quiz, and how results are displayed to the students. 49 00:09:42.870 --> 00:09:49.910 Seth Blackburn: Therefore, once you've credit, Quiz said, it is treated exactly like any other quiz or exam in the quiz generator. Manage everything as if for a single quiz. 50 00:09:50.050 --> 00:09:57.100 Seth Blackburn: The only time you need to drop down back to the individual please level after this point is when you manage the questions, or I need to see the answer key for individual quizzes. 51 00:09:57.650 --> 00:10:23.079 Seth Blackburn: and when it comes to the question of academic integrity, this is perhaps the primary reason to utilize questions, because they undoubtedly promote quick security by ensuring that not all students have this exact same exam. The ability for students to share questions and answers amongst each other greatly decreases. This applies to both current offerings of a given course, since peers may not receive the exact same exam, but also future offering. Since exam information that is shared online or elsewhere has less of a chance of being applicable to a future student. 52 00:10:23.600 --> 00:10:24.760 Seth Blackburn: Next slide, please. 53 00:10:26.900 --> 00:10:44.429 Seth Blackburn: a question block is a group of questions that will always appear together in a quiz. This means that even in a randomized quiz set up questions within a question block will remain together. Something important to know about question blocks is that though the block can be considered a single unit that will keep all of your associated questions together, the questions do still stand on their own. 54 00:10:44.540 --> 00:10:58.780 Seth Blackburn: This means that each question in the block still counts as one question in the quiz. For example, if you have 3 questions in a single question, Block, this will count as 3 individual questions in the quiz. The block format is mostly used to keep related questions together when it comes to displaying the order of questions in a quiz. 55 00:10:59.360 --> 00:11:06.959 Seth Blackburn: and this leads into the major benefit of utilizing question blocks, which is keeping questions that have related content, or a common reference together in your quiz. 56 00:11:07.080 --> 00:11:17.590 Seth Blackburn: So, for example, you might have questions which all pertain to a very specific topic, and don't want to spread them across your quiz. You may also have a table that students need to refer to when answering a handful of questions on the exam. 57 00:11:17.750 --> 00:11:25.249 Seth Blackburn: And both of these cases importing the related questions into a question. Block would be very helpful and making sure students don't become confused by the order of questioning. 58 00:11:26.120 --> 00:11:36.720 Seth Blackburn: So as far as academic integrity question blocks in and of themselves aren't necessarily designed to grow academic integrity, since their primary function is to keep clusters of questions together across your exams. 59 00:11:36.860 --> 00:11:52.169 Seth Blackburn: That being said, though the default behavior is to display questions in the order, they were added to the question block. There is also the ability to randomize the order of questions. If you are already randomizing the display order of questions in your exam turning on randomization within your question blocks and maintain the security benefits that randomization bring. 60 00:11:52.490 --> 00:12:03.990 Seth Blackburn: However, keep in mind that this should only be done at the order of questions within the block doesn't matter. One last thing that's important to remember about question blog is that by default all questions are displayed in the order that they were added. 61 00:12:04.010 --> 00:12:14.659 Seth Blackburn: or how they have been otherwise moved around in the question. Block, editor. you also have the ability to lock the first question into place, so that does, it does not get randomized while other questions in the block are displayed in the random order. 62 00:12:14.820 --> 00:12:21.789 Seth Blackburn: You might want to do this if you have a block of questions that are all related to a table or graph, and that table or wrap is displayed in your first question of the block 63 00:12:21.890 --> 00:12:23.040 Seth Blackburn: next slide, please? 64 00:12:25.590 --> 00:12:42.770 Seth Blackburn: And lastly, there's question tagging up to this point in the Qu question section we've covered quiz sets and question blocks with sets. Allow for disbursement of different exams and question blocks ensure that related questions always appeared together. However, what if you would like to achieve the randomness of quiz that while still maintain the cohesion of question blocks. 65 00:12:42.820 --> 00:12:50.969 Seth Blackburn: The best option to address this to them is to use a tag role-based quiz question tagging for a tag. Rule-based quiz can be best understood as a two-step process. 66 00:12:51.150 --> 00:12:58.309 Seth Blackburn: You will first need to assign tax to your questions. These can be anything you would choose to meaningfully describe a question such as anatomy or multiple choice. 67 00:12:58.370 --> 00:13:02.809 Seth Blackburn: The next step is to create roles that will dictate how questions will be pulled as well as how many. 68 00:13:02.970 --> 00:13:20.749 Seth Blackburn: for instance, though you may have 8 anatomy questions in your question bank, you can create a role that directs the place to only pull 4 of these questions with the anatomy tag, you can even use rules to account for multiple tags which allows for a more detailed pruning of questions. For example, you may want to specifically select for questions that have had with both anatomy and multiple choice. 69 00:13:21.540 --> 00:13:27.579 Seth Blackburn: Now, looking at the primary benefit of question tagging, they can help ensure that all desired topic areas are included on a given close. 70 00:13:27.750 --> 00:13:40.429 Seth Blackburn: Although the quiz generator supports simple, random selection of questions to include on a quiz. This round is likely lead to situations where some students mostly receive questions for only one topic area where some quest, some students have questions covering every topic area. 71 00:13:40.500 --> 00:13:48.650 Seth Blackburn: So to to account for this, you might assign each question a tag that describes its topic area and then create roles to govern. How many questions from each topic area should appear. 72 00:13:49.360 --> 00:14:03.589 Seth Blackburn: Another aspect to consider is the variety of question types in your question bank. your existing question bank make feature multiple choice on the blank as well as essay questions. If this bank is used in conjunction with simple random selection, the difficulty of an exam could very greatly across your students. 73 00:14:03.770 --> 00:14:13.540 Seth Blackburn: Therefore you might want to create a bank of questions that contains 10 multiple choice 10 fill in the blank and 10 to the said questions. But then create role, specifying that each student should only receive 3 of each question top. 74 00:14:14.220 --> 00:14:29.240 Seth Blackburn: As for the other benefits of question tagging, if you already have an understanding of push sets from earlier. A good way to think of this tool is that it is like quiz sets on a smaller scale. The key difference is that while quiz sets offer up different quizzes to students, question tagging offers of different questions to students within the quiz. 75 00:14:30.290 --> 00:14:37.380 Seth Blackburn: Lastly, because of their similar similarities. The ability of question tagging to promote academic integrity. Integrity is very close to that of 76 00:14:37.470 --> 00:14:50.709 Seth Blackburn: because question tagging will allow you to offer up completely different questions to students taking the same quiz. The likelihood of successful sharing of questions and answers greatly decreases. Again, this quick security can apply to both current offerings of a given course as well as any future offerings. 77 00:14:50.760 --> 00:14:58.689 Seth Blackburn: Peers will be less able to successfully share quiz information amongst each other, and it is much less likely that future students will benefit from any information that is share. 78 00:14:59.480 --> 00:15:07.719 Seth Blackburn: please feel free to ask any questions about anything I've just covered with that question lost in question tagging, but for now I'm going to turn it over to Brian to discuss close settings. 79 00:15:09.980 --> 00:15:33.399 Brian Klaas: Thanks, Seth. So Seth gave us a really awesome and useful high level overview of the sort of different tools that are in course, plus that. Let you use randomization strategies. That sort of help to enhance academic integrity in that regard. And I just want to walk you through this a little bit in the actual course plus site interface. I teach a number of courses, and I use these tools myself to help promote academic integrity among my students. I teach. 80 00:15:33.400 --> 00:15:58.400 course, for that's required of all M. Ph. Students that has about 150 students in it every single term. And I want to make sure that I'm employing randomization to properly assess the students, and ensure that they are not cheating or hopefully reducing any cheating that might be going on in those classes, because I don't have time to write entirely new quizzes every single term for my students. So there is a lot of content reuse, and I want to make sure that 81 00:15:58.400 --> 00:16:02.879 that that randomization is being effectively employed to help reduce cheating. 82 00:16:03.030 --> 00:16:28.030 Brian Klaas: We're opportunities for cheating among our students and the things I'm going to show you today. I know some of you have used Marie has used quiz sets in the past. Pablo and his team. use a question tagging in the Ep. 7, 21 series. Is that what it is? Public he's not in? Yes, so some of our faculty are already using this. because they are powerful and useful tools So let me just start sharing my screen here, and as I go along, if you have questions, feel free to ask them. 83 00:16:28.030 --> 00:16:53.029 Brian Klaas: I'm happy to ask the respond to those questions that you might have here. And I'm gonna start with sort of a this, some some non randomization on topics that might be helpful in terms of helping to increase academic integrity in quizzes or exams in your online or well based face classes and any of your classes at all. So here I am on the setup here. 84 00:16:53.030 --> 00:17:18.030 Brian Klaas: on the quiz generator, and right up at the top of the. There's many options in the setup here, and right up at the top of the options. Are some setups a bit that questions that apply to grading, and whether or not this quiz or exam, or whatever it is, is counted towards the final grade in the class. Let me zoom in a little bit here, and by enabling this option, a number of other options then become available to you that are specifically tied to things 85 00:17:18.030 --> 00:17:43.029 Brian Klaas: like academic integrity. Because if the quiz isn't counted towards the students, final grade in the class other tool things like, is it an open book or closed book, you know, quiz or is there, you know, time limit, or whatever or having students sign their name, are less important. So in order to see these options, the quiz does need to be flagged as counting towards the students final grade in the class, and the first option that appears here is this question of, is this an open book? Quiz 86 00:17:43.030 --> 00:18:07.769 Brian Klaas: Sam And really the only material impact this has on the actual delivery of the quiz or exam, and the quiz generator is a warning is a warning that was created by Jon Ernick and the office of academic integrity. And I'm gonna go ahead and say, Yes, here and save my changes real quick. Just so you can see what this looks like. So if I test the quiz or preview the quiz oops. Sorry I didn't mean to do that. 87 00:18:07.770 --> 00:18:32.769 Brian Klaas: Actually, it should be no, shouldn't it? Yes, here I am, messing up my own demo Here, let me go back to the setup. Sorry about that, folks. I should have left this on. no, it's not an open book, quiz or exam. Sorry. That's what I meant to do. It's not an open book, quiz or exam, because if you set that option and I say, test the quiz. You'll notice there is this reminder that appears up at the top of the page here, before the students are allowed to take 88 00:18:32.770 --> 00:18:57.759 Brian Klaas: a time or sorry. A closed book, quiz or exam. This text always appears as text is written by the office of academic integrity, and basically warn students that it can't be looking at outside stuff. If you have any questions, ask your instructor clicking around could send a message to course indicating that you were perhaps using things outside of the quiz or exam window, which could be a violation of that closed. But 89 00:18:57.760 --> 00:19:24.250 Brian Klaas: exam. So that's one thing that you can do to get students to think more about the academic integrity there, right is by saying, yes, it's an open book, exam or not. The other one I wanted to point out was this option to require students to sign their name to the academic Ethics code before submitting their answers. And so, if you enable this option, what happens when a student takes the quiz or exam, if I go into it here 90 00:19:24.510 --> 00:19:49.509 Brian Klaas: this, this box appears at the very top of the quizzer exam. there's no way to move it around And the, it's basically says that by typing your name you pledge that you did this work in accordance with the applicable academic Ethics code of the institution, and we don't specify Bloomberg School of Public Health, because, of course, while Bloomberg School of Public Health is is one place from where students students take classes. 91 00:19:49.510 --> 00:20:12.580 Brian Klaas: We also have students from other parts of the the university as well, and we want to make sure that we're covering all the academic ethics codes there. But the point is that a student can type their name here at the very beginning, before they start answering any questions, or they can type it at the end right before they submit. But they won't be able to submit their answers until they type their name here, if you enable that option that is a requirement there. 92 00:20:13.360 --> 00:20:22.239 Brian Klaas: So Florian wants to know that warning block be outlined. yeah. So what do you mean specifically by outlined Fran? If I may ask. 93 00:20:23.400 --> 00:20:39.810 Fran Burman: Hey, Brian? Sorry about that? so it's clarifying. not this, but not this required signature. That's great. But on the previous of warning blocks. But very late, like, yeah, it's a little light. it's very easy to read. It's a little 94 00:20:40.290 --> 00:20:43.390 Fran Burman: the maybe larger 95 00:20:43.680 --> 00:21:08.659 Fran Burman: text, or, you know, a larger font, or we can. We can look at that. I mean this, this warning block is actually been in there for about 2 years, and we haven't received any feedback from students or faculty that it's difficult to read or easy to overlook. But we can certainly consider. Consider doing. I went right by it. Well, okay, so that. But but but that that raises a really valid point as a faculty member myself, and as the the sort of product manager for cold, for 96 00:21:08.660 --> 00:21:31.520 Brian Klaas: course, plus as well, we all know that people don't read things right, I can tell you the number of times I have students. So one of my exams you know it says it's a it's a a 30 min exam, and if you leave the exam page, the countdown timer continues to run, and every term, every single term. I teach my communications primer. Course, I have 4 or 5 students who are like, I didn't know 97 00:21:31.520 --> 00:21:56.519 Brian Klaas: then. I only had 30 min exactly. Exactly. So that's a whole other challenge. And you know, study. Show that the more text you have on the screen, the less likely people are to read those instructions. It's like legal agreements. Nobody actually reads them But So we're trying to strike that that balance there. and it's definitely a challenging balance to set up Marie, you have a question or comment. 98 00:21:56.520 --> 00:22:21.509 Marie Diener-Weset: the required signature. So I have got an email in the past from students who said, Oh, you know, I timed out of the exam. But I forgot to put my signature in. And it seems like, that's okay, right? Right? You know, they're still saved right? Is it something they saw? I mean. So, having them type their signature, there is sort of based off the idea that you know if you are adding your name to something. You're seeing a reminder when you start to take a quiz 99 00:22:21.510 --> 00:22:46.510 Brian Klaas: for exam about ethics and ethical behavior more likely to behave ethically. Now, this is in part based on research by Dan Airily, who we have recently found out in the last couple of weeks, has apparently fudged the data a lot on a lot of his studies around ethical behavior. And and and these exact kind of say same things. But there's been other research that is sort of supported that a separate from Dan airlines work as well. But 100 00:22:46.510 --> 00:23:06.660 yeah, sometimes students forget to do this. it's you know that if time runs out, time runs out you know, I think that's the most important thing is that it doesn't sort of give students an extra couple of minutes to type their name when they could be changing answers or continuing to answer questions there, but that that can and does happen if a student chooses not to type their name, and time runs out 101 00:23:06.660 --> 00:23:07.749 Brian Klaas: that it's going to run out. 102 00:23:08.060 --> 00:23:32.920 Brian Klaas: That's an important clarification. So the other thing you can do another really powerful tool to in terms of ensuring academic integrity is providing a maximum time for taking the quiz or exam. And that's one of the last options that's available on the setup here. This maximum time allowed to complete the quiz, because what you ultimately want to do is try to properly time box your quiz or exam, so that a student is unable. 103 00:23:32.960 --> 00:23:57.109 Brian Klaas: A student who does not know the answers to the questions, who does not demonstrate strong knowledge of the content that's being covered cannot go out and look up at those those questions very quickly on the Internet or in their notes. that's really what you want to try to achieve with this kind of time limiting and time boxing. So, for example, in my quit in this particular quiz that I do in this communications. Primer class students have to answer 20 questions in 25 min. 104 00:23:57.110 --> 00:24:22.110 so that still gives them, you know, a minute and a few seconds to answer each of the questions in the quiz. But it doesn't give them enough time to go quickly, click through the lectures or look at their notes. If they don't know the answers to the questions. so you could definitely work with your course instructional designer, or if you don't have one assigned to your class. you can contact the instructional design team and Ctl, and they can work with you in terms of balancing out the the, the 105 00:24:22.110 --> 00:24:47.109 time you have for questions with, or the the content of your questions, and the difficulty of those questions with the amount of time that you allow students to have? Because what you don't want to do is say, Oh, you have 3, and's hours to answer these 5 questions. If they're not particularly difficult questions to answer right? Because that gives students plenty of time to cheat to, you know, watch lectures or do readings while they're answering the questions. And that definitely is a detriment to academic integrity in situation 106 00:24:47.110 --> 00:24:52.379 where you don't want students to be using other sources or have an open book, quiz or exam. 107 00:24:52.540 --> 00:25:16.829 Brian Klaas: and so from moving on from there. I want to focus now on randomization. A little bit more here. So there's a number of questions here in the quiz generator. that allow you to do different sort of options and settings around randomization and the first one that you'll probably notice I'll come back to this one about all questions, or subset in a minute The first one is on randomizing the display order of questions. So if you choose this, you have a 10 108 00:25:16.830 --> 00:25:41.829 Brian Klaas: question quiz. You choose this options. The questions are, gonna appear in a random order for every student in the the in the in the quiz, right? So it's random display order. It's not randomizing anything else but the display. Order the questions. You then have the option to randomize the display order to multiple choice questions, so that you would have each student would sort of have a different display of answers to A, B, C, d, and so on, and combined 109 00:25:41.830 --> 00:26:06.819 Brian Klaas: having randomizing, display order of questions with randomizing display of answers to multiple choice questions, really does help reduce the amount of cheating when students choose to work with each other, because it's very becomes more difficult for the student to say, oh, what was the answer to question 9. Because that's it could be a different question. 9 for each student. And you know the answer, C could be question. It could be different for each of those students right there. and Seth was talking about a question 110 00:26:06.820 --> 00:26:34.580 Brian Klaas: blocks as well as a way of holding questions together. So a question block just to recap would be something like the following, 3 questions are based on this table or the results from this study. And so you need to be able to refer to that first question in that block to answer sort of the next 3 questions in the block. Now you can with this option, randomize the display order of questions and blocks, because you may be like it doesn't matter. The other questions can appear in any order. 111 00:26:34.590 --> 00:26:37.529 Brian Klaas: but if you have 112 00:26:37.670 --> 00:27:01.499 Brian Klaas: a if you have a question block where it is the kind that it says. The next 3 questions are based on this table. You don't want that randomized right? You don't want that because you're the table could come as the third question in that block instead of the first question in that block. It's not what you want. So if you choose this option, there's a second sort of follow up randomization choice that says always display the first question assigned to a block first. 113 00:27:01.500 --> 00:27:20.289 and that will ensure that in a block, whatever that first question is, will always appear first. But the subsequent questions in that block may be randomized. Again, the more randomization you can have, particularly where there's multiple choice, true false match from 2 list questions involved the higher sort of level of academic integrity you might be able to achieve in that 114 00:27:20.290 --> 00:27:44.780 So really quick looking some questions here that came in the chat penguin wants to know. Can you share more about the purpose of question hints and examples of when you should use them, or when not to So question. Hints are an option that appear on each questions you create question. You can create a hint for answering that question. I would turn that over to on my instructional design colleagues to answer the question of 115 00:27:44.780 --> 00:28:09.779 when hints should be used, or when they should not be used in in quizzes. I know they're not frequently used, because a well designed and written question should be fairly clear in its intent, but sometimes you might need to add in a hint to say, clarify a point that might be a question for students without them having to reach out and ask you and answer that question via email. 116 00:28:09.780 --> 00:28:17.540 or come up to the front of class and do that. But, mia Lee, do either of you have any responses to Penguin's. inquiry about that. 117 00:28:18.730 --> 00:28:31.099 Mia Lamm: Well, I think you covered that pretty well, Brian, actually. But I think it really depends on the actual question and the actual test. But I agree with everything you said, and I'm not sure I have too much more to offer Luke 118 00:28:32.430 --> 00:28:33.930 BSPH CTL Teaching Toolkit: the same here. 119 00:28:34.320 --> 00:28:56.440 Brian Klaas: Okay, great And so, and if you think of others other reasons as we go along, we could definitely add that in and bill points makes a really important point about randomizing multiple choice, the answers to multiple choice questions. You can't just say things like A and B, or you know A and D as your responses. Because, again, if you're randomizing multiple choice answers. 120 00:28:56.550 --> 00:29:18.469 Pablo Martinez Amezcua: everybody's answers are going to be different. Right? So the answer for a for one student is not going to be answer A for another student. So you really can't use things like A, B, A and B, or A and D, and stuff like that in your multiple choice questions. There you can go ahead. Thank you, Brian. One comment about. So I some of the sometimes the order of the answers. 121 00:29:19.200 --> 00:29:36.919 Pablo Martinez Amezcua: yeah, I think I've read somewhere that it should be sequential. For example, if you have, the answers could be point 1 1 10, you should keep it that like that. So in those situations you recommend not to randomize right? Because you don't want someone to have 10, 122 00:29:36.920 --> 00:30:05.079 Brian Klaas: we want one right? So randomization, super powerful and super useful. But, Pablo, you make an excellent point that students can often get misled or easily misled. If you have a series of numeric answers that follow a logical sort of flow or a logical pattern like they start at point one and then point 5, and then point 7 5 and one. Let's just say those are the numbers there. it would be easy for students to get confused. 123 00:30:05.080 --> 00:30:30.770 Brian Klaas: They see a pattern developing that actually, isn't there through randomization right? If they saw like point 7 5 point, you know, or point 5.7 5 one, you know point 2 5, it might be easy for students to skip that point 2 5, because they're not paying attention to the pattern that's been developed in the in the display of the answers there. So that's that's definitely a concern that you should keep in mind when thinking about using randomization, particularly for multiple choice answers there. 124 00:30:30.770 --> 00:30:55.770 Pablo Martinez Amezcua: And and that randomization of the answers cannot be customized. Right? So and yeah, and so in the push generator, it's you either randomize the answers to all multiple choice questions or not, there's no way of like turning it on and off for individual questions. Unfortunately, there is a way to do that. If you happen to use a question type called the Combo question type, where you can choose to randomize the display order in combo 125 00:30:55.770 --> 00:31:20.630 Brian Klaas: types of the answers of the provided answers in the drop down list that appear in combat questions. So that's sort of a a separate thing. And I've got to move forward because I want to make sure we have time for mia to talk about statistics. But the other thing I want to show really briefly is a little bit about randomization of the questions that are selected and tag rule quizzes. So Seth talked about tag rule quizzes, and those are enabled by this question of 126 00:31:20.630 --> 00:31:45.630 Brian Klaas: display all questions or use a subset. So if you just play all questions in a quiz, you have 10 questions in your quiz. It's gonna show all 10 questions, but the subset option allows you to do one of 2 things randomly. Selecting a subset of questions from, say, 10 out of 20 questions in your quiz, you can have 20 questions in your quiz and let the quiz generator randomly pick 10 for a student, which again helps promote academic integrity, because not everyone is receiving the same questions 127 00:31:45.630 --> 00:32:10.630 Brian Klaas: in the quizzer exam, or to randomly select questions based on tag rules or tag based rules. So let me switch over to a different quiz here that actually uses this. And this is my actual sort of quiz that I have in in the the communications primer course. And if I look at the questions here, you'll notice I have a a lot of questions. Right? I have. I believe it's 42 different 128 00:32:10.630 --> 00:32:35.799 Brian Klaas: questions. Yeah, 42 different questions. But when students take this assessment, they're only going to see 20 of those 42 questions, because I've set up a series of tag based rules that enable me to ensure that I get adequate content, coverage, and ad adequate coverage of the kinds of question types. I want to be asking in this quiz, so that 129 00:32:35.800 --> 00:32:59.450 Brian Klaas: basically all the lectures are covered, the main content areas are covered. But I still get a high degree of randomization. And I do that by establishing tag rules. So a tag rule is simply a rule that says, match this tag or tags that are associated to questions. So if you saw back here and go back one screen, you'll notice that each of my questions has a series of tags associated to it like Schwabish from a reading by Jonathan Schwabish. 130 00:32:59.450 --> 00:33:24.350 Brian Klaas: True Falls for the question, type, lecture on cultural humility, and so on and so forth. So I can take these tags and then say, Okay, I want to make sure that the Schwabish reading, which is really important to this class. I ask a minimum of 4 questions. So I set up this tag rule that basically says, using the tags rubbish, go ahead and include 4 questions in the final output. I do the same thing for the reading by Joseph Campbell 131 00:33:24.350 --> 00:33:48.549 Brian Klaas: for the lecture on Cultural Humility, the lecture on Monroe's Motivated sequence. And then it's sort of a broader set of questions to mix up true, false, multiple choice questions from the lectures. some on the science of learning, some on the science of visual design. All told, I'm going to have 20 questions here, and those 20 questions again will be selected based on these rules, and I can see 132 00:33:48.550 --> 00:33:54.950 Brian Klaas: the output of these rules might be by clicking on this test rules button down here. 133 00:33:55.210 --> 00:34:18.280 Brian Klaas: so here we are. I click the button, and it shows me a random selection of the questions from my total of 42, I'm gonna get 20 questions. See? There's 20 questions right there. and I can to click this button again and again, and it will give me a series of different results based on the rules that I've established. 134 00:34:18.280 --> 00:34:43.280 Brian Klaas: So tag rules when you combine. If you think about combining the randomization of tag rules which are gonna show up? It's so like, yeah. So tag rules. Let you do random selection, of course, content based on rules that you provide a random selection of questions randomly, it displayed in a random order, and then I can add to that display of answers and multiple choice questions randomly, or use 135 00:34:43.280 --> 00:35:07.460 Brian Klaas: from 2 list questions which randomize the display order of the responses or use the Combo question type, which also randomizes the display order of responses, unless you tell it not to all of these things together make it much, much more difficult for students to be able to work together and cheat, or in some cases rapidly look up information from existing materials, but particularly when students are sharing answers or working in groups 136 00:35:07.460 --> 00:35:28.230 Brian Klaas: to potentially cheat on something like a quiz or exam. The tag rule of quizzes are super powerful and super useful. I know that a published team has used them in the the 721 course. and I believe also, I think, the happy clot every department to use them in the 7, 51 series as well, because they can be really powerful in terms of creating randomization with adequate content coverage at the same time. 137 00:35:28.370 --> 00:35:44.529 Brian Klaas: All right. So I realize I've gone a little bit over my time. I'm gonna hold off on questions at this point. If it's okay and turn things over to mia, who's gonna look at statistics, And using the statistical information about responses to questions to sort of further develop academic integrity. 138 00:35:44.900 --> 00:35:57.299 Mia Lamm: Thanks so much, Brian, that's really great. really clear. Thank you. So just to shift gears a little bit, I'm going to talk about what a data you can get from your quizzes, and how that can help you. 139 00:35:57.300 --> 00:36:22.299 Mia Lamm: support academic integrity. So to access the statistical information and quiz generator, you're going to navigate to the quiz generator, select the desired quiz whatever quiz you want to look at the data for, and then you'll find a top menu which is shown on the slide here. where you can. Access responses to questions, quiz completion, data, quiz, grade statistics and a discrimination 140 00:36:22.300 --> 00:36:25.770 Mia Lamm: value report. Go ahead and next slide. 141 00:36:27.350 --> 00:36:44.139 Mia Lamm: So, as you can see on the right side of this slide, the question responses report will provide the percentage of students who got an individual question correct. That's what's highlighted in green as well as the distribution of incorrect answers across the class. 142 00:36:44.370 --> 00:36:52.209 Mia Lamm: And in assessing the difficulty levels of the quiz questions we refer to it as the p-value or percent correct 143 00:36:52.540 --> 00:37:22.300 Mia Lamm: indicating the percentage of students who answered a question correctly and to just assist you engaging the difficulty level. I've incorporated a table on the left side of the slide which provides some general ranges for categorizing the ease or difficulty of quiz questions for students. So for this particular question that I've put on the the screen here. a around 70% of students answered correctly. And this is falling into an average range of difficulty. 144 00:37:22.600 --> 00:37:35.680 Mia Lamm: And this data can be helpful for seeing patterns or finding weak questions through the identification of patterns in the student responses, you can tailor your instructional approach to effectively address any knowledge gaps 145 00:37:36.370 --> 00:37:37.420 Mia Lamm: next slide. 146 00:37:38.880 --> 00:37:56.810 Mia Lamm: So, moving on to the quiz completion statistics. This feature provides you with a holistic overview of quiz completion rates and percentage scores for each student, and it also includes how many attempts the student made and how long it took each student to take the quiz. 147 00:37:56.990 --> 00:38:00.540 Mia Lamm: and I thought I saw somebody chatted a question. 148 00:38:03.170 --> 00:38:12.720 Mia Lamm: Oh, yeah, from lack of clarity. And thank you, Bill, that's a great question. I think I'm getting there with some of the next slides, if we don't mind holding on. 149 00:38:13.990 --> 00:38:38.830 Mia Lamm: thank you. And so we can also access the overall quiz grade statistics, and this offers a detailed analysis of the grades achieved by students in your quizzes, so you can explore the standard deviation, the mean median and range and by understanding the grade distribution you can assess the difficulty of your quizzes. identify where students are selling or struggling, and also make informed decisions about your grading and feedback. 150 00:38:40.300 --> 00:38:59.110 Mia Lamm: So let's go to the next slide. Thank you. And do just a really quick. I just want to hear from you. because this often comes up. sh! If you identify quiz questions that are too easy or too difficult. Do you throw them out of the quiz? 151 00:38:59.510 --> 00:39:02.940 Mia Lamm: And why or why not? Real quick? 152 00:39:09.950 --> 00:39:12.050 Mia Lamm: Take about 30 s 153 00:39:12.740 --> 00:39:14.949 Mia Lamm: before we move forward. 154 00:39:23.500 --> 00:39:25.399 Mia Lamm: Couple more seconds. 155 00:39:31.700 --> 00:39:32.730 Mia Lamm: Okay. 156 00:39:40.870 --> 00:39:54.500 Mia Lamm: so so thank you all for your responses. I'm gonna move forward a little bit. But as some of you are probably pointing out the decision of whether to exclude quiz questions. 157 00:39:55.900 --> 00:39:57.950 Mia Lamm: Sorry? 158 00:39:58.170 --> 00:40:08.420 Mia Lamm: that are too easy or too difficult depends on various factors, and I've highlighted here just a few perspective to consider if you don't mind going to the next slide. Lou. Thank you. 159 00:40:08.680 --> 00:40:19.560 Mia Lamm: So these are the things that I've highlighted. first ensuring alignment with learning objectives. If a question is significantly easier or harder than the desired level of mastery. 160 00:40:19.650 --> 00:40:37.909 Mia Lamm: it may not effectively measure students understanding of the targeted concept. So in such cases. where it's not aligned? Excluding these questions can help ensure that the assessment accurately reflects the intended learning outcomes and second, maintaining motivation engagement students be not fully engaged with 161 00:40:37.910 --> 00:40:52.100 Mia Lamm: Mit. Ctl. And or take the assessment seriously if they feel unchallenged by easy questions. So, on the other hand, you know, questions that are excessively difficult can also lead to frustration. So it's something to consider when you're deciding. And I don't think it's 150, 162 00:40:52.160 --> 00:40:58.350 Mia Lamm: necessarily black or white. It's just figuring out the balance of your of your assessment. 163 00:40:58.510 --> 00:41:22.349 Mia Lamm: and you might also consider reliability. A well designed quiz aims to measure students, knowledge and skills. And if questions are too easy or too difficult, they may not accurately differentiate between students, level of understanding, which is where you want to be right. So, lastly, you definitely want to ensure that you're accommodating different learning levels. such as in a diverse classroom, students may 164 00:41:22.890 --> 00:41:46.979 Mia Lamm: have a varying prior knowledge and skills. so excluding questions that are too easy or difficult can ensure that all students have a fair opportunity to demonstrate their understanding without being unfairly advantaged or disadvantaged by question difficulty. And ultimately, as I mentioned, striking a balance between appropriately challenging and accessible questions. 165 00:41:47.100 --> 00:41:53.410 Mia Lamm: can help create meaningful assessments that really fully support student learning 166 00:41:54.670 --> 00:42:03.670 Hossein Zare: my, a quick question. So your level of difficulties, more than 50%. Well, what about the level of 167 00:42:03.780 --> 00:42:08.129 Hossein Zare: hey percentage, and none of the students could answer the questions. 168 00:42:08.640 --> 00:42:14.140 Hossein Zare: In that case I would say, if I check it several times. 169 00:42:14.440 --> 00:42:19.560 Hossein Zare: and each summer you don't see any response. The the level of question is on 170 00:42:19.770 --> 00:42:30.860 Mia Lamm: yeah, And and actually, that where we're going to go right now is into some data that dies a little bit deeper. that that we're going to talk about. But I think 171 00:42:31.150 --> 00:42:41.410 Mia Lamm: you need to determine so. And as Bill kind of was asking about to some of the data that we've looked at so far. That's given to you is 172 00:42:41.530 --> 00:42:57.139 Mia Lamm: the tab shallow in the sense that it's not actually, maybe, looking at the question at a deep level. So if students, for example, if you're sure that content is covered in your class, and you have students that are not able to answer questions 173 00:42:57.300 --> 00:43:23.469 Mia Lamm: either. It's the students gap, and they haven't done what they need to do, or the material isn't the relaying in a way that they can understand clearly, or the questions are not really aligned with that content. So all these things kind of need to be looked at. It's not just like a a in in some cases not always a very simple, straightforward question, and I think if we moved into this next 174 00:43:23.480 --> 00:43:35.290 Mia Lamm: slide. Blue we could start talking about the data in the discrimination value report that will help you dive a little bit deeper into really examining each specific question. 175 00:43:35.320 --> 00:43:37.040 Mia Lamm: Right? 176 00:43:37.280 --> 00:43:58.980 Mia Lamm: So looking deeper at the available data the quiz generator, will provide this discrimination value report. So discrimination value is a measure used to distinguish between high and low performing students, right? So it helps to identify questions that accurately differentiate their levels of understanding versus just seeing that 177 00:43:59.310 --> 00:44:06.799 Mia Lamm: their students are performing poorly on a question. I don't know why, right? We I just I'm seeing a pattern, but I don't know why? 178 00:44:07.380 --> 00:44:25.349 Mia Lamm: so a discrimination value of point 2 or higher is typically considered desirable and quiz is exhibiting a normal distribution. So the discrimination value is typically calculated by analyzing the performance of students who scored well on the overall assessment 179 00:44:25.380 --> 00:44:33.340 Mia Lamm: and comparing it with the performance of those who scored poorly. So if a question exhibits a high discrimination. That's over that point 2. 180 00:44:33.590 --> 00:44:42.469 Mia Lamm: It means that students who performed well on the entire assessment were more likely to answer that specific question correctly. 181 00:44:42.640 --> 00:44:46.920 Mia Lamm: while those performed poorly were more likely to answer it incorrectly. 182 00:44:47.230 --> 00:45:02.759 Mia Lamm: And these questions are, are ideal. So we want that we want that balance in that distribution. by identifying utilizing questions with a higher discrimination value, we can enhance the reliability and validity of the assessments 183 00:45:02.790 --> 00:45:09.050 Mia Lamm: right? So as you can see on the excuse me on the quiz report on the slide 184 00:45:09.930 --> 00:45:14.739 Mia Lamm: circled in red on that last column under discrimination. Index 185 00:45:15.890 --> 00:45:26.220 Mia Lamm: question number 3 has a discrimination index of point 1 9, which is a little bit too low. Right? So I'm gonna flag that when I see it, and I'm gonna take a a closer look 186 00:45:26.350 --> 00:45:30.880 also circled is its associated difficulty. Value of 0 point 9 4 in that 187 00:45:31.300 --> 00:45:44.890 Mia Lamm: fourth column, which means 94% of students answered the question correctly. So this question. I believe, is too easy. And please step in Brian or Seth, if you if if I'm miss quoting anything 188 00:45:45.940 --> 00:46:02.349 Mia Lamm: so it doesn't this. In this question it didn't really matter whether there were low or high performers. This question really needs to be written or removed right? And it's important to note that the discrimination index should never be negative. Negative values suggest that the question is not 189 00:46:02.430 --> 00:46:09.770 Mia Lamm: effectively measuring learning, and should not be used or needs to be revised in some way. We got a next slide, please. 190 00:46:10.090 --> 00:46:17.540 Hossein Zare: I think, for the came back, please. The table. If if the level of difficulties is 0 to one, and then 191 00:46:18.190 --> 00:46:21.200 Hossein Zare: it should be one 192 00:46:21.600 --> 00:46:24.340 Hossein Zare: if you look at the previous slide. 193 00:46:25.170 --> 00:46:32.730 Mia Lamm: But the first question China student didn't answer the question, the level that the the level of difficulties is 194 00:46:33.230 --> 00:46:35.730 Hossein Zare: 32 percentage, right? 195 00:46:36.490 --> 00:46:49.249 Hossein Zare: Not 68%. If if they want to stand for the high difficult question. And you understand, for the very easy questions the level of you got to for question 3 is 6 percentage too easy. 196 00:46:49.410 --> 00:47:03.829 Brian Klaas: Yeah. So I think the label it probably just brought up that same issue in the labeling of that column that, instead of difficulty, should probably say ease or easiness. And I think that's that's that's an easy fix for us to make. Absolutely. It's great feedback. Thank you both very much. 197 00:47:04.930 --> 00:47:09.280 Mia Lamm: Very good. Thank you. So let's go to the next slide. 198 00:47:11.580 --> 00:47:40.239 Mia Lamm: So this is the second part of the report. the discrimination value report that course plus will give you and what's particularly helpful is that it provides additional clarification to make it easier for you. Right? So it calls out questions that fall outside the normal ranges. And here on the slide, you can see it has highlighted 2 questions. one of which isn't on the other slide. So I won't go into that. But if you are a call question number 3, which is red circles in red here? 199 00:47:40.250 --> 00:47:50.489 Mia Lamm: that question. Number 3 had a low discrimination value, and we can determine that it does not differentiate between low and high performers, and therefore I would consider it 200 00:47:50.860 --> 00:47:53.559 Mia Lamm: maybe not reliable assessment of learning. 201 00:47:54.130 --> 00:48:03.240 Mia Lamm: And just to note for those that have obviously the expertise, as as many of you do. It's also possible to download all the data to an excel spreadsheet. 202 00:48:03.280 --> 00:48:17.740 Mia Lamm: so that you can. You can work through that data and manipulate it yourself. And I've circled that link at the at the bottom in green. and this way you can. You can work with that data and gain more insight, as you need, into your quiz statistics. 203 00:48:18.470 --> 00:48:30.329 Brian Klaas: And I should also note that the all the responses and grades to quiz can be downloaded into excel format. So you can do your own statistical analysis against those responses and grades. In addition to the data that comes from the discrimination value report. 204 00:48:31.130 --> 00:48:31.960 Mia Lamm: Thank you. 205 00:48:33.520 --> 00:48:36.010 Mia Lamm: Okay. Next slide. 206 00:48:39.130 --> 00:48:58.539 Mia Lamm: So all that being said, let's just do one last poll before we sort of start wrapping up here? why? And how does the analysis of this data really matter? Specifically, not just for improving our quizzes, but actually supporting an academic integrity, and again open answered question. 207 00:48:58.900 --> 00:48:59.980 Mia Lamm: Thank you, Marie. 208 00:49:03.890 --> 00:49:12.160 Mia Lamm: Just take a few minutes. How do you? How could you imagine that this data analysis could help with 209 00:49:12.700 --> 00:49:14.080 Mia Lamm: academic integrity. 210 00:49:20.240 --> 00:49:21.660 Mia Lamm: More seconds. 211 00:49:29.520 --> 00:49:30.480 Mia Lamm: Okay. 212 00:49:33.070 --> 00:49:34.270 Mia Lamm: thank you. Everyone. 213 00:49:39.620 --> 00:49:41.419 Mia Lamm: And we can move to the next slide. 214 00:49:48.270 --> 00:49:49.290 Mia Lamm: Hmm. 215 00:49:51.080 --> 00:50:00.500 Mia Lamm: and I'm sorry I can't. I'm I'm having trouble seeing the the actual responses. So Lou, do you mind letting me know we had a couple of responses on this. 216 00:50:05.240 --> 00:50:06.600 Mia Lamm: And that's okay. I'm 217 00:50:07.090 --> 00:50:32.799 Mia Lamm: thank you. That's okay, but thank you. Everybody for for participating. I'm having trouble seeing it. But I've highlighted a few things. I'm just to just to move forward for short on time. Improvement of quiz. Questions can significantly support academic integrity by ensuring the assessment, fairness, validity, and clarity, as I'm sure. Some of you are aware. it helps to identify patterns. 218 00:50:32.800 --> 00:50:38.660 Mia Lamm: So this analysis may reveal unusual trends, such as increase a sudden increase in scores. 219 00:50:38.660 --> 00:50:48.810 Mia Lamm: or maybe an unusually high number of correct answers on specific questions, and recognizing these patterns can alert us to potential instance of academic misconduct, such as cheating. 220 00:50:48.810 --> 00:51:11.229 Mia Lamm: Right? So we're seeing trends. we know we have a problem. And particularly, we, we, I work with the self society courses. We have hundreds of students going through those courses. And we are we have a major project trying to get. We're using quiz sets in place because, we see some of these trends. Right? So it. It really does help. 221 00:51:11.800 --> 00:51:31.249 Mia Lamm: And regularly reviewing quiz to 6 allows us to identify areas of weakness or ambiguity in the exam questions clear and fair assessments reduce the likelihood of students resorting to dishonest practices, to compensate, maybe for unclear instructions, or maybe unfair evaluation methods 222 00:51:31.930 --> 00:52:00.150 Mia Lamm: also identifying topics or concepts that students consistently struggle with can help us tell our our instruction, and by proactively supporting student learning, we can create an environment that discourage academic, you know, discourages academic dishonesty. And finally, we can identify specific areas where students need improvement by reviewing incorrect responses. And this allows us to personalize our feedback. So when we see the gaps, we can actually address it. 223 00:52:00.150 --> 00:52:28.209 Mia Lamm: as students are taking, you know, as completing these examples, we can see those areas of misunderstanding and through careful assessment, design and enhancing our design, and tailoring instruction and the individualized feedback, we really can effectively deter or do a little better job. of deterring academic dishonesty while promoting a culture of honesty, integrity, and intellectual growth among our students. 224 00:52:28.790 --> 00:52:40.460 Mia Lamm: So Now we have a few minutes left to open the floor for more. Q. A. I appreciate everybody's patience. I believe I'm gonna hand it to Brian. 225 00:52:41.500 --> 00:52:50.269 Brian Klaas: Sure. So if anyone has any questions, you can go ahead and raise your hand. Unmute yourself. Type in the text chat The team here will gladly answer those questions for you. 226 00:52:53.100 --> 00:53:03.609 Hossein Zare: 1 one question regarding that regarding so the randomization I should randomize the question and the responses for each question, or just the questions. 227 00:53:03.610 --> 00:53:28.609 Brian Klaas: So it'll I. So you have this. You have different levels of control. Right? You can randomize the display order. Well, actually, so you can randomize the questions right? Which questions are included. If you use tag rule, quizzes, or you create sort of a bank of questions and say randomly, select 10 out of 20 in the bank. That's the First level. The second level that you have is randomization of the display order of those questions. Right? You could say, I only have 228 00:53:28.610 --> 00:53:44.999 10 questions, these only questions. I'm never going to ask my students. But go ahead and randomize the order in which those display. And then the Third level is on multiple choice questions. You can choose whether or not to display the randomized, the display of the answers to those questions. 229 00:53:45.050 --> 00:54:09.500 Brian Klaas: And there's a few other question types like I mentioned. with. if you do match from 2 list questions, the answers are always randomized for the students for each student. And if you use the combo question type, the options that are in the sort of drop down that gets inserted into your question at multiple points is also randomized. But that one you can turn off randomization on. So there's lots of yeah. So there's the various levels of randomization does answer your question. 230 00:54:10.240 --> 00:54:12.210 Brian Klaas: Yes, thank you. Great. 231 00:54:12.360 --> 00:54:34.720 Mia Lamm: I also see a quite a, a, a comment by Bill. that about some students actually ask for feedback on quizzes. the challenges when we have randomization students must be clear about the question wording, not the question number, and it brought to mind. That's a great point. I wanted to just mention a slightly different scenario again, back to the self society courses where? 232 00:54:34.720 --> 00:54:44.779 Mia Lamm: we have many hundreds of students going through those, those, those courses, and before and in some class some of the course is still missing. Quiz sets 233 00:54:44.780 --> 00:54:56.669 Mia Lamm: one strategy that the course facilitator team is taking is actually summarizing feedback. So they're looking for those gaps in the quiz data. And they're writing up 234 00:54:56.670 --> 00:55:16.260 Mia Lamm: summaries of and and references to where students can go back and look in the content to identify those those common trends and gaps. And it's working beautifully. And students love it. So it's just another strategy without maybe the specifics of each 235 00:55:16.260 --> 00:55:22.120 Mia Lamm: question. that can really help with that personalized feedback. 236 00:55:23.650 --> 00:55:40.139 Pablo Martinez Amezcua: I I also know there's a answered Key, which the students can see afterwards. Right? But can we? We have? We have had a issues with students, not seeing the answer key that we wrote, are there options to show or hide, or? 237 00:55:40.290 --> 00:56:09.879 Brian Klaas: Okay? I don't know. There definitely are. So in the sharing results section. And this is something you can work with your instructional designer on in the sharing results section of the quiz set up, you do have the option to display the answer. Key to students after a specific date, basically. Well, sorry after the deadline for the quiz is passed. You can then say, Yes, go ahead and display the answer. Key. You can even say no, we don't want to do that, and then change that like a week after 238 00:56:09.880 --> 00:56:33.720 Brian Klaas: the quiz or exam happens. I know in your case, Pablo, you've got lots of students who are in the 7 21 take the midterm and final, who are like, you know? I can't take it on those days. Can I take it a couple of days later? So you don't want the answer key to suddenly appear. But the other options available in the answer. Key is showing whether to decide whether or not students who have not completed the exam can see the answer key, because there's 239 00:56:33.720 --> 00:57:00.370 Brian Klaas: there's There's that option was added by default. Students who don't take the quiz or exam cannot see the answer. Key. There's a way to override that as well. so just yeah, I just wanna make sure that setup is available to, you know, is it? It's properly configured. And your your course, instructional designer or Ctl help can definitely walk you through that as well. But that is an option. So students do have like effectively. The answer. Key also shows them their quiz. 240 00:57:00.750 --> 00:57:24.750 Brian Klaas: If you're using randomization, whatever questions they they got, whatever randomization was done, it appears that same order to them as well. So they see exactly what they saw while taking the quizzer exam. So there's no concern. I mean, we hear it all the time. It's like I, I know. Course, plus change the order of these questions like, No, actually didn't. this is, everybody gets a little differently, and that is something you would have to explain right. If you do do a like a 241 00:57:24.750 --> 00:57:48.820 Brian Klaas: an answer key or exam review, you want to explain is, Bill was pointing out in the text chat there you're looking for feedback. You've got to like, really focus on the content of the question, not the numbers, because it really does confuse students very, very easily. in those in those sort of review sessions when you're like, okay for question 9, because in a tag rule, quiz or randomized display order quiz, you could have, you know, 9 different question. Nine's right 242 00:57:48.820 --> 00:58:00.209 Pablo Martinez Amezcua: erez agmoni depending on what the variations that occur during the execution of the quiz itself. So it's important thing to keep in mind. Thank you, Brian, and just to follow up on that, if there's some guidance on how to write 150 243 00:58:00.220 --> 00:58:05.519 Pablo Martinez Amezcua: and and spend an effective answer key that you could share. That would be great, because we've had 244 00:58:05.700 --> 00:58:11.300 Pablo Martinez Amezcua: different people writing questions, and they different styles to answer the answer key. 245 00:58:11.440 --> 00:58:20.169 Pablo Martinez Amezcua: So I know you should say, option a is to identify which one is the correct one, and why and why the incorrect ones are incorrect. 246 00:58:20.240 --> 00:58:22.820 Pablo Martinez Amezcua: What we done? We didn't have some 247 00:58:23.250 --> 00:58:31.919 Pablo Martinez Amezcua: standard way to do it in in the course, so if there is some guidance that you could share, that'd be great, if not, we can think about it. 248 00:58:32.040 --> 00:58:34.740 Brian Klaas: I will turn that over to to me and Lou. 249 00:58:36.160 --> 00:58:57.099 BSPH CTL Teaching Toolkit: Yes, please. They do. Look how our upcoming multiple trials question, workshop, part 3, it's it's specifically focused on writing multiple to a question for hair level thinking including work on how to create, you know, effective answered options and some working session. Yeah. 250 00:58:59.860 --> 00:59:04.510 Brian Klaas: great. Well, thank you. Everyone for your questions, and I'll turn it back over to Lou to wrap things up. 251 00:59:04.620 --> 00:59:06.160 BSPH CTL Teaching Toolkit: Oh, oops! Sorry. 252 00:59:06.340 --> 00:59:23.809 BSPH CTL Teaching Toolkit: all right. Thank you so much for joining us today. if you can. Could you feel all this this survey before you leave I help you from this workshop information when the helpful and looking forward to seeing you all in the future. Ctl, events have a good one. 253 00:59:27.780 --> 00:59:32.769 Hossein Zare: Thanks everybody. Thank you so much.