WEBVTT 00:03:30.900 --> 00:03:36.720 Elizabeth Stuart: Thank you so much for joining in this session of so again I'm 22 00:03:37.170 --> 00:03:43.830 Elizabeth Stuart: Liz Stuart and Associate Dean for education at the school. And this is, I think that third or fourth in a series of teaching workshops, we're doing 23 00:03:44.130 --> 00:03:52.890 Elizabeth Stuart: To help get us all ready for fall instruction and as I'm sure you all saw we are now officially fully online in the fall and so 24 00:03:53.820 --> 00:03:56.730 Elizabeth Stuart: You know, We're all figuring out how to make that happen. 25 00:03:57.270 --> 00:04:08.220 Elizabeth Stuart: And so today's session is going to focus on designing effective online assessments. At the end of today, I will also bring back and sort of remind us of some other resources and and other topics. 26 00:04:08.820 --> 00:04:19.920 Elizabeth Stuart: You can learn more. But today, our main focuses on online assessments and I'm thrilled that we have three really great presenters who will kind of give them each of them give their own perspective. 27 00:04:20.580 --> 00:04:28.890 Elizabeth Stuart: Brian class from CTL will give an overview of different sort of what we even mean by what is an effective online assessments. 28 00:04:29.460 --> 00:04:33.750 Elizabeth Stuart: And I give some examples, especially things that we can implement in course plus 29 00:04:34.410 --> 00:04:41.940 Elizabeth Stuart: Jennifer deal will then talk about her own experience with different sorts of assessments in a large scale epidemiology course. 30 00:04:42.330 --> 00:04:52.230 Elizabeth Stuart: And then your vertical give his perspective as the director of our Office of Academic Integrity, which, you know, it's still going to be a concern for and maybe even more of a concern for some people, for 31 00:04:52.950 --> 00:05:04.560 Elizabeth Stuart: When we're doing online assessments, so I'm excited to learn a lot. So again, here's I could have moved to the slide, but it was nice seeing the pictures on the previous one. 32 00:05:05.400 --> 00:05:10.410 Elizabeth Stuart: Just a note, just as we've done in previous sessions like this, please use, use the chat. I'm 33 00:05:10.770 --> 00:05:21.630 Elizabeth Stuart: One of my goals in these workshops, is to build a community of people thinking about these things and sharing ideas, so I really welcome people to post questions post ideas. 34 00:05:22.050 --> 00:05:28.440 Elizabeth Stuart: Give your own perspective in the chat. And at the end, I'll come back to that and talk more about how we can continue these conversations in different ways. 35 00:05:28.890 --> 00:05:39.690 Elizabeth Stuart: If we have time at the end. We might also be able to do some audio questions. And so at that point, you could raise your zoom hands. Then, but again, all sort of monitor the chat and save some of the questions for the end 36 00:05:41.160 --> 00:05:52.260 Elizabeth Stuart: So with that, I will turn it over to Brian who will be able to sort of give us an overview of kind of what again what we mean by effective assessments and what are some strategies we might think about using 37 00:05:53.520 --> 00:05:54.150 Elizabeth Stuart: Thank you, Brian. 38 00:05:58.620 --> 00:06:10.350 Brian Klaas: Thank you, Liz usually helps if I unmute myself before I get started. Okay. Hi there. I'm, I'm Brian class. I work in the Center for Teaching and Learning. I also have an instructor appointment, am I, and I teach a couple of classes online. 39 00:06:11.040 --> 00:06:15.600 Brian Klaas: Around communications for Health Sciences professionals and I'll be talking a little bit about that as we go along. 40 00:06:16.140 --> 00:06:26.400 Brian Klaas: But I what I wanted to start with today is just basically trying to define what do you mean by effective when we speak about creating effective online assessments. What do we mean by effective and the guidance that I'm going to share next 41 00:06:26.820 --> 00:06:27.660 Brian Klaas: Is really, I think, 42 00:06:27.990 --> 00:06:41.700 Brian Klaas: What most of the instructional design team in the Center for Teaching and Learning among others would sort of provide and say, is an effective online assessment. So to kick it right off and effective online assessment is one of course that assesses students fairly 43 00:06:41.940 --> 00:06:46.470 Brian Klaas: And the most important thing to think about here is that it's aligned with the learning objectives of the course right 44 00:06:46.650 --> 00:06:54.450 Brian Klaas: You want to make sure that the assessments are actually assessing what the students are supposed to be learning as defined in the objectives you provide them. 45 00:06:54.780 --> 00:07:01.770 Brian Klaas: So this gets around or gets away from assessments that are the gotcha questions or look at the footnotes or what have you, that you're assessing 46 00:07:02.070 --> 00:07:12.810 Brian Klaas: Students fairly based on what you're telling them, they're supposed to be learning in the class. And it also means that variations on the assessment, meaning a student has to take an exam later on because of some 47 00:07:13.380 --> 00:07:21.450 Brian Klaas: issue or problem or requirement in their life that those variations are equitable that you're not giving people extra work and more difficult exam, simply because they can't take them. 48 00:07:21.690 --> 00:07:31.530 Brian Klaas: At the time at which everyone else in the class is taking them so assessing students fairly is the first, the second is that effective online assessments are implementable in the current environment. 49 00:07:32.190 --> 00:07:37.230 Brian Klaas: So let me give you an example of what I mean by this. So in one of the classes that I teach around communications. 50 00:07:37.620 --> 00:07:45.720 Brian Klaas: For Health Science professionals, one of the activities is an elevator pitch and this class is taught on campus. And the goal of this exercise is to 51 00:07:45.930 --> 00:07:53.880 Brian Klaas: get students to speak clearly and concisely about the research that they are doing in a very short amount of time period while walking and talking 52 00:07:54.180 --> 00:07:58.800 Brian Klaas: And you think of walking and talking. That's not a problem. We do it all the time. It is extraordinarily difficult 53 00:07:59.100 --> 00:08:04.470 Brian Klaas: To focus on another person and get a clear and concise message that you have practiced and honed 54 00:08:04.710 --> 00:08:13.710 Brian Klaas: Across while you're moving. This is really tough to do and the students in the class often find this to be one of the most valuable exercises in the course because it really gets them thinking about 55 00:08:14.100 --> 00:08:18.930 Brian Klaas: How they approach people how they speak to people and speaking on a certain in a certain way improvisational late 56 00:08:19.560 --> 00:08:26.100 Brian Klaas: Now this class was caught in the fourth term and in the fourth term, this year we couldn't do this right, we couldn't literally because what we would do is we'd 57 00:08:26.340 --> 00:08:34.620 Brian Klaas: Leave the classroom and get the elevator and walk around the hospital and walk around the East Baltimore campus and avoid cars and other people and you know 58 00:08:35.370 --> 00:08:41.040 Brian Klaas: Trash cans and things like that and you can't do this. We couldn't do this in the coven environment. 59 00:08:41.490 --> 00:08:45.810 Brian Klaas: So what I did instead is I said hey look, you all have smartphones, so 60 00:08:45.960 --> 00:08:56.970 Brian Klaas: Figure out a way to walk and talk and record yourself with a smartphone. If you have a little selfie stick use that if you have someone you're living with currently that's safe to do where it's safe to do so. Walk and talk and 61 00:08:57.180 --> 00:09:05.790 Brian Klaas: Give your elevator pitch pre recorded and do it. So we still got to the core goal of the activity, which was speaking clearly and concisely, while moving 62 00:09:06.030 --> 00:09:12.780 Brian Klaas: We just couldn't do it in the sort of traditional way. So that's one way of thinking about is something implementable in the current environment. 63 00:09:13.020 --> 00:09:22.500 Brian Klaas: Now this might also mean the in terms of your classes the resources that are available if assessments of exams. If assignments require specialized software. 64 00:09:23.040 --> 00:09:33.120 Brian Klaas: Time and the wet labs, whatever it is that may not be possible for your students and it's important also to think about sort of the home and work considerations that students are now currently in 65 00:09:33.360 --> 00:09:42.600 Brian Klaas: We have students who live with seven other people in a small two bedroom apartment and they can't necessarily be in a quiet space and so doing a two hour exam. 66 00:09:42.810 --> 00:09:48.510 Brian Klaas: Might be extraordinarily difficult for them because they can't have that space and time to focus while doing an exam. 67 00:09:48.690 --> 00:09:59.250 Brian Klaas: They might live in a place where they only get internet for a few hours a day or electricity. A few hours a day. So again, doing these kind of long exams or long projects that take a lot of time may not be possible for them. 68 00:09:59.460 --> 00:10:03.150 Brian Klaas: And then of course you also have work considerations, particularly for students. 69 00:10:03.480 --> 00:10:14.430 Brian Klaas: Or learners who are in the healthcare industry and the time they may be spending and hospitals and difficulties and challenges that they have there. So just think about is my assessment implementable in the current environment. 70 00:10:15.030 --> 00:10:20.970 Brian Klaas: And next up, it's important that you're on the effective online assessment to be effective needs to be meaningful to the students right 71 00:10:21.150 --> 00:10:28.110 Brian Klaas: And I think what we see time and again on the evaluations that we see their, their schools course evaluation system, that students respond 72 00:10:28.320 --> 00:10:34.470 Brian Klaas: very positively to assessments that mirror something they'll need to do in the real world, when they graduate. 73 00:10:34.710 --> 00:10:44.160 Brian Klaas: Wherever you can align it with a real world activity students react very positively and get a lot out of it a lot more than, say, Did I have the right answer, or did I not have the right answer. 74 00:10:44.370 --> 00:10:54.300 Brian Klaas: And even in those situations where you're maybe trying to assess more on base knowledge tried to find opportunities to put theory into context. Put that base knowledge into context. 75 00:10:54.540 --> 00:11:00.630 Brian Klaas: Maybe you have 25 multiple choice questions, but then you end it with one sort of question that's more about synthesis 76 00:11:00.810 --> 00:11:09.150 Brian Klaas: And critical thinking and putting all that base knowledge into a real world context, they can apply and then that makes it a lot more meaningful for the students. 77 00:11:09.810 --> 00:11:16.050 Brian Klaas: It's also important for an online assessment to be effective, that's feasible for you to both design and deliver 78 00:11:16.440 --> 00:11:21.300 Brian Klaas: This is a big one right can you build the assessment and the time you have a lot of us are reevaluating 79 00:11:21.750 --> 00:11:25.140 Brian Klaas: The assessments that we have can, you know, and maybe you want to do something different. 80 00:11:25.590 --> 00:11:36.600 Brian Klaas: Maybe with a new quiz set feature in course. Plus, you say, Okay, I'm going to be able to five exams and then students are going to get randomly assigned one of those exams using the quiz that feature. Well, can you build five 81 00:11:37.080 --> 00:11:47.490 Brian Klaas: equally balanced and difficult exams in the time that you have to do that work that may not be possible. It's also important to think about, is this a reasonable expectation of the students time 82 00:11:47.910 --> 00:11:55.110 Brian Klaas: Well, students are home so they don't have these other pressures, so they can do project that is a lot more complicated involves a great deal of research. 83 00:11:55.770 --> 00:12:02.070 Brian Klaas: You know is that research possible for the students, given their current context and the time that they have you really have to think about 84 00:12:02.280 --> 00:12:10.080 Brian Klaas: Does my assessment fit into the credit hours that are assigned to this course is important consideration for all the students and then 85 00:12:10.560 --> 00:12:20.730 Brian Klaas: Perhaps most importantly, can you deliver meaningful feedback in a relatively short time period because I'm sure you've seen it. And we see it and Center for Teaching and Learning, I see it in the classes that I teach 86 00:12:21.300 --> 00:12:32.970 Brian Klaas: You know if feedback doesn't come back very quickly students get very upset and it's important, right, that they get feedback in a timely manner so they can incorporate that feedback into the future work that there'll be doing in your class. 87 00:12:33.420 --> 00:12:41.250 Brian Klaas: So for me, one of the classes I teach is an introductory communications course for all the MPH students and I have at students a term. 88 00:12:41.520 --> 00:12:52.200 Brian Klaas: And one of my big challenges is grading at presentations in less than a week and finding the time for me to do that and figuring out ways that I can that has been really challenging, but I need to do that. 89 00:12:52.470 --> 00:13:00.420 Brian Klaas: So students can then move on to incorporating that feedback into the rest of the work they do based off that presentation in the class, given the eight weeks that we have 90 00:13:00.690 --> 00:13:10.950 Brian Klaas: In our schedule so meaningful feedback timely delivered is a super important part of considering how you build effective and delivering effective online assessments. 91 00:13:11.370 --> 00:13:20.130 Brian Klaas: And yon will, of course, talk about this in just a few minutes. But I want to address it really briefly here but effective online assessment also ideally limits the risk of academic ethics concerns. 92 00:13:20.730 --> 00:13:22.980 Brian Klaas: Again, rather than multiple choice, necessarily. 93 00:13:23.520 --> 00:13:32.130 Brian Klaas: Assignments or exams that require critical thinking or projects versus multiple choice questions project based assignments, where you can still deliver that feedback in a timely manner. 94 00:13:32.550 --> 00:13:42.030 Brian Klaas: Using tools that are available in course plus around randomization around time boxing to help sort of make sure that the students really have to have mastery of the material in order to 95 00:13:43.230 --> 00:13:50.070 Brian Klaas: Complete the exam or the assignment of time allowed but more importantly, and I'm sure you on repeat this be really clear. 96 00:13:50.490 --> 00:13:58.740 Brian Klaas: Not only just use open book assignment formats, but be very clear about what open book means what sharing information means what collaborative work means 97 00:13:58.980 --> 00:14:07.590 Brian Klaas: Because students will carry on expectations from even one assignment to another class, there are some classes where students can collaborative work collaboratively. 98 00:14:07.890 --> 00:14:13.410 Brian Klaas: Collaboratively if I can speak work on homework. But when it comes time for a paper or exam. They can't do that. 99 00:14:13.620 --> 00:14:21.300 Brian Klaas: And if you aren't making those expectations really clear students will carry over those expectations and potentially cause academic ethics violations so 100 00:14:21.480 --> 00:14:30.360 Brian Klaas: Thinking about how you design your assessments to limit the risk of academic ethics concerns is a good thing for them and for you and and yon will certainly talk about this a little bit more 101 00:14:30.990 --> 00:14:40.530 Brian Klaas: So fortunately in course. Plus, there's a number of different tools that you can use to help build online assessments. Now that you have an idea of what an effective online assessment is 102 00:14:40.920 --> 00:14:46.380 Brian Klaas: And there's a couple of them. And some of them you may, some of you are probably quite familiar with, but the one I'll talk about first is the quiz generator 103 00:14:46.650 --> 00:14:56.250 Brian Klaas: Many people use this to deliver quizzes exams home works surveys capture information by students and it's used for anything, lots of different kinds of assessments. 104 00:14:56.790 --> 00:14:58.410 Brian Klaas: In courses at the school. 105 00:14:58.920 --> 00:15:08.040 Brian Klaas: It has support for multiple question types. There are very powerful randomization tools you can randomize the quiz. That's actually given to the student randomized questions that are included display options. 106 00:15:08.280 --> 00:15:12.330 Brian Klaas: It'll automatically grade many question types for you except for essays. 107 00:15:12.720 --> 00:15:22.410 Brian Klaas: So that significantly can speed the grading of assessments that are there, and there's lots of tools to manage special permissions and concerns when students have technical problems. 108 00:15:22.620 --> 00:15:28.410 Brian Klaas: Students can't take exams during the normal time window. All of those things are addressed by the tools in course plus 109 00:15:28.980 --> 00:15:40.350 Brian Klaas: The quiz generator is really powerful, it's getting more powerful. We actually this summer have introduced. I think the count is we're up to 12 new features in the quiz generator since the end of the last academic 110 00:15:40.830 --> 00:15:46.740 Brian Klaas: Year and late May. So we're putting a lot of effort into this and it keeps getting better and better. 111 00:15:47.250 --> 00:15:56.400 Brian Klaas: We did a whole session. Back in April on the quiz generator and all sort of ins and outs and things to consider when building exams, in particular, and the quiz generator 112 00:15:56.820 --> 00:16:05.340 Brian Klaas: That recording of that workshop is linked on the teaching toolkit site, which I know will be shared with everybody later on in this presentation. 113 00:16:06.030 --> 00:16:07.950 Brian Klaas: There are some things have been added to it. 114 00:16:08.430 --> 00:16:15.270 Brian Klaas: Since that workshop happened. You can follow the CTO blog. You can see the link on the screen in front of you. There you can script subscribe via email. 115 00:16:15.450 --> 00:16:25.170 Brian Klaas: You can subscribe via a newsreader client. And that's where all the latest stuff about the quiz generator and everything else in course plus is being posted so quiz generator quizzes exams homework. 116 00:16:25.620 --> 00:16:33.000 Brian Klaas: Next up is the Dropbox. Many people do use this. But from an assessment perspective, it's good for collecting Patriot papers. 117 00:16:33.360 --> 00:16:37.470 Brian Klaas: Projects presentations bundles of code. If you're having students right stuff in our 118 00:16:37.950 --> 00:16:48.540 Brian Klaas: Or doing other sort of statistical analysis their organization and reminder tools that are there for you so you don't have to get 50 emails that I'll have an attachment name to paper. 119 00:16:48.960 --> 00:16:52.680 Brian Klaas: Or homework and then sort through them. It handles all of that for you. 120 00:16:53.190 --> 00:16:58.050 Brian Klaas: And there's a feature of the Dropbox that not many people are aware of. It's called the timed Dropbox. 121 00:16:58.260 --> 00:17:04.710 Brian Klaas: And that's a Dropbox, where a student, you'd set it up you provide some kind of instruction or homework or exam file. 122 00:17:04.920 --> 00:17:15.450 Brian Klaas: And then you say okay, once you unlock this file, you have exactly two hours to complete this or 90 minutes or six hours or one day or whatever you want it to be and the student has to submit 123 00:17:15.660 --> 00:17:26.940 Brian Klaas: Their assignment back to the Dropbox in that time allotted, and that's really useful for when answer capabilities of the quiz generator in particular aren't enough so maybe students are generating our code or 124 00:17:27.300 --> 00:17:38.550 Brian Klaas: Have to generate a bunch of histograms in our or have to hand draw things that take pictures of and put in a Word document, because the quiz generator doesn't do file collection, the Dropbox is what does file collection. 125 00:17:38.880 --> 00:17:50.490 Brian Klaas: So it's an interesting tool. It's used in some of the comprehensive exams at the school and something to consider when the capabilities of the quiz generator aren't enough or can't quite address your assessment needs. 126 00:17:51.000 --> 00:18:01.800 Brian Klaas: The peer assessment tool has been around for a number of years. And this is a very valuable tool because it allows for rubric based evaluation of student work. And I'll talk a little bit more about rubrics in just a minute. 127 00:18:02.160 --> 00:18:11.310 Brian Klaas: But what it lets you do is have students evaluate each other's work and collates those assessments into sort of a final grade that you can accept or change or alter as you see fit. 128 00:18:11.760 --> 00:18:19.260 Brian Klaas: So, for example, right now, in the MPH program. There's a core course that all the 200 or so students are taking on public health policy. 129 00:18:19.560 --> 00:18:24.480 Brian Klaas: And rather than the faculty and to read 200 papers in less than a week and a half because they have to 130 00:18:24.750 --> 00:18:34.920 Brian Klaas: give feedback and then students have to make iterative iterative versions of those those policy papers they have fellow students evaluate everybody so each of the 200 students evaluates three three of their peers. 131 00:18:35.400 --> 00:18:42.030 Brian Klaas: And they evaluated online and do it via a rubric and what's great about the peer assessment tool again. Is it collects all that stuff for you. 132 00:18:42.570 --> 00:18:47.670 Brian Klaas: The assignment can be random. So you can say you just randomly assigned people or you could say if you're in a group. 133 00:18:47.940 --> 00:18:55.410 Brian Klaas: You're going to evaluate this group or there's someone's doing a group presentation evaluate this group or everyone evaluate this group for groups, evaluate each other's lots of tools for that. 134 00:18:55.650 --> 00:19:08.130 Brian Klaas: And of course, it can be linked to Dropbox of submissions so that students do have an artifact a paper, something like that to submit it's automatically linked as part of their peer assessment in the rubric itself that students fill out. So it's a very, very useful tool. 135 00:19:08.880 --> 00:19:15.660 Brian Klaas: The discussion forum. People tend to think of for just students asking questions, but it is also a really good tool for assessments as well. 136 00:19:15.930 --> 00:19:22.800 Brian Klaas: You can evaluate students based on reflective discussion that they have that you give them a prompt and they have to respond and you evaluate that response. 137 00:19:23.040 --> 00:19:30.960 Brian Klaas: You could have debates between groups of people or even the entire class where students can point in counterpoint each other and you evaluate them based on their posts there. 138 00:19:31.230 --> 00:19:37.530 Brian Klaas: You can do journal article analysis or even peer analysis where a student will present an idea present a president is some sort of proposal. 139 00:19:37.650 --> 00:19:45.060 Brian Klaas: Or some sort of statement about a topic and then multiple students would be responsible for responding that student. And everyone would see it and you could evaluate that work. 140 00:19:45.300 --> 00:19:51.780 Brian Klaas: Because all of the posts in a great book category can be graded and those grades them automatically imported into the gradebook. 141 00:19:52.200 --> 00:19:56.670 Brian Klaas: It's also important to note that discussion forum can also be used for small group work private group work. 142 00:19:56.790 --> 00:20:07.980 Brian Klaas: So you could have maybe six groups in your class and then internal debates between each of the groups and you would evaluate them separately so that each group does not see each other's work there. Again, very powerful tool for 143 00:20:08.400 --> 00:20:15.390 Brian Klaas: For developing online assessments and then finally there's lots of presentation and performance sort of tools for presentations and performance. 144 00:20:15.780 --> 00:20:25.650 Brian Klaas: That you might have students use in your assessments. We've got zoom, of course, great for presentations either individual or group presentations for students doing debates debating one another. 145 00:20:25.830 --> 00:20:29.370 Brian Klaas: And also for student led discussion student led discussions are a great way 146 00:20:29.580 --> 00:20:38.550 Brian Klaas: Of helping students demonstrate mastery of material, right, because you have mastery of material is the faculty, they have to demonstrate mastery of material if they're going to be leading discussions. 147 00:20:38.820 --> 00:20:48.480 Brian Klaas: Another tool that's used fairly often certainly in the online courses is voice thread. This is a tool or students can pre record a presentation again either an individual or group presentation. 148 00:20:48.780 --> 00:20:57.930 Brian Klaas: And then once it's published other students or you can then actually make recordings directly on the presentation itself so powerful tool for sort of 149 00:20:58.140 --> 00:21:03.270 Brian Klaas: Pre recorded presentations that wouldn't necessarily take up time in your live synchronous sessions. 150 00:21:03.660 --> 00:21:12.510 Brian Klaas: And don't forget the smartphones that pretty much all of our students have are very powerful content creation tools, like I said, gave that example of students recording themselves delivering an elevator pitch. 151 00:21:12.840 --> 00:21:27.120 Brian Klaas: There's lots of tools they know how to use them. It's very easy for them to record video or audio, they could do local site visits. Again, as long as it's safe for them to do so in the current climate, again, it's a powerful creation tool for doing some very creative work around assessment. 152 00:21:28.560 --> 00:21:35.400 Brian Klaas: So I'd like to wrap up, just by giving some advice. Some things we've learned that are effective in delivering 153 00:21:35.970 --> 00:21:41.100 Brian Klaas: Assessments online that we've seen time and again come up and and some things we think are effective. 154 00:21:41.730 --> 00:21:51.060 Brian Klaas: We are. We certainly get the feedback and know to be effective from our experience in online assessments. First of all, if you're delivering an exam. Consider splitting that exam into two different parts 155 00:21:51.540 --> 00:22:01.170 Brian Klaas: The first part would be where you have your multiple choice, you're automatically graded questions. And the second part would be where you would include your essay or open ended questions because it does a couple of things. First of all, 156 00:22:01.380 --> 00:22:09.300 Brian Klaas: It gives students a break. Again, you have students who will be in parts of the world, who will not be able to sit for two hours solid. They simply won't because of family issues because of 157 00:22:09.510 --> 00:22:14.250 Brian Klaas: resource issues in our connectivity electricity, whatever it is they won't be able to do that. 158 00:22:14.880 --> 00:22:21.840 Brian Klaas: And this gives them a break. And that's certainly something that students appreciate and helps them to focus better in both parts of your exam. 159 00:22:22.050 --> 00:22:27.810 Brian Klaas: What it also does with that sort of, you know, the first part being all the automatically graded question, I say, multiple choice here, but it could be 160 00:22:28.020 --> 00:22:32.430 Brian Klaas: True, false, or match from two lists or numeric value, whichever one you want to use. 161 00:22:32.700 --> 00:22:43.770 Brian Klaas: It allows because they're automatically graded students get immediate feedback and they know what to think about and consider before they do the essay oriented part of the exam where more critical thinking and synthesis is involved. 162 00:22:44.040 --> 00:22:49.830 Brian Klaas: And what's nice about that is, again, that second part that essay part allows for authentic student responses. 163 00:22:50.010 --> 00:22:58.140 Brian Klaas: It's not just simply repeating facts and confirming the have basic factual knowledge, it allows them to show off their ability to synthesize that knowledge. 164 00:22:58.260 --> 00:23:08.760 Brian Klaas: And creatively use it and potentially or ideally real world scenarios. So consider that if you're doing exams. It has worked out very well in the class that have done it with very positive student feedback on that front. 165 00:23:09.450 --> 00:23:18.180 Brian Klaas: The next one is use rubrics. So I mentioned rubrics before rubrics are great reports are awesome. I in all my classes use rubrics for every single one of my assignments. 166 00:23:18.510 --> 00:23:22.320 Brian Klaas: Except for the one quiz that I have in the class and the classes. Excuse me. 167 00:23:22.650 --> 00:23:28.560 Brian Klaas: Each one has one quiz, but the others all the other assignments are rubric breaks, because what it does is allows you to set very 168 00:23:28.770 --> 00:23:33.150 Brian Klaas: Very clear expectations about what is required for excellence. 169 00:23:33.390 --> 00:23:43.530 Brian Klaas: By posting this rubric ahead of time students know what they need to do in order to succeed and do an excellent and a level job on a paper on a project on a presentation, whatever it is. 170 00:23:43.800 --> 00:23:56.490 Brian Klaas: It significantly and I mean significantly reduces arguments about grading because you have been very explicit and very upfront about what it takes to get the best possible score in that class. Now students 171 00:23:56.730 --> 00:24:00.630 Brian Klaas: Who are going to argue are going to argue. But by having this out there that significantly 172 00:24:00.870 --> 00:24:10.680 Brian Klaas: Reduces that administrative work of saying, well, here's why you got something because you have the rubric to explain why they receive the gray. The gray. They did across multiple criteria. 173 00:24:11.250 --> 00:24:17.670 Brian Klaas: Now this is the instructional designers in the Center for Teaching and Learning be happy to talk with you about developing rubrics, they do take time to develop 174 00:24:18.090 --> 00:24:21.630 Brian Klaas: But they certainly are very powerful instrument for delivering 175 00:24:22.140 --> 00:24:29.880 Brian Klaas: really effective assessments to students and the peer assessment tool has a type of assessment called faculty slash TA 176 00:24:30.210 --> 00:24:35.820 Brian Klaas: And that's a type of assessment, where basically students never see the rubric. Your you or your TA 177 00:24:36.150 --> 00:24:45.690 Brian Klaas: Are the only ones who ever see it, you get one rubric per student per assignments and you can just go in and use that same rubric to fill it out and provide your comments and feedback and again 178 00:24:45.990 --> 00:24:49.950 Brian Klaas: There's tools in there to send it back to the students to automatically import those grades in the gradebook. 179 00:24:50.190 --> 00:24:58.830 Brian Klaas: Rather than having ad copies of a rubric and a Word file and having to duplicate it for one student fill it out duplicate it for another student fill it out. 180 00:24:59.070 --> 00:25:12.660 Brian Klaas: Just do it in the faculty tip or assessment and that way everything is done digitally. It saves you a lot of time and hassle and the embarrassment. As I haven't counted personally in the past of sending the wrong rubric to the wrong student always a bad thing. 181 00:25:14.070 --> 00:25:22.500 Brian Klaas: Use the data that's available in course plus to determine the quality of the questions you're asking in your exams or quizzes and and determine the fairness of them. 182 00:25:22.920 --> 00:25:30.360 Brian Klaas: The grading data is available to you in Excel. You can export all the responses and grades and the scores to excel for every single quiz. It's in the quiz generator 183 00:25:30.840 --> 00:25:35.820 Brian Klaas: There are statistics that are provided to you that tell you how many students got the question right or wrong. 184 00:25:36.000 --> 00:25:48.180 Brian Klaas: And there's something called the discrimination value report, which tells you basically how difficult or easy. A question was, or how easy it was for a person who did poorly on the test to get that answer right or get full points for that. 185 00:25:48.780 --> 00:25:57.210 Brian Klaas: This data is available on every single quiz and every single exam and the quiz generator. You can use it to examine students across the class and individual class, but more importantly 186 00:25:57.390 --> 00:26:05.970 Brian Klaas: To examine that data across offerings in the class, the multiple times your classes offered. So you can see over time. Ideally, the questions that are perhaps imbalanced. 187 00:26:06.180 --> 00:26:14.070 Brian Klaas: Questions that need to be revised, or maybe even eliminated in your quiz generator based assessments. And then finally, 188 00:26:14.730 --> 00:26:20.220 Brian Klaas: It's important. Also, that basically you know you're prepared right there's little details and always sort of get in the way. 189 00:26:20.460 --> 00:26:27.630 Brian Klaas: And just some final reminders on that. First of all, it's really important to provide very clear instructions, not just about the assignment itself, but when it's do 190 00:26:28.020 --> 00:26:36.450 Brian Klaas: The requirements, technically, and who to contact for problem. So again, if it's an exam, who are the students supposed to contact if they run into a problem like their computer crashes. Right. 191 00:26:36.630 --> 00:26:45.840 Brian Klaas: Do they contact CT. I'll help who really can't make that determination if they should be allowed back into the exam or they contact a TA, or you check those open do close dates. 192 00:26:46.470 --> 00:26:54.630 Brian Klaas: Really make sure there's alignment between whatever's in the tools and whatever's on the schedule page or if you're using the class sessions tool. Make sure that it's clearly labeled there. 193 00:26:54.840 --> 00:26:59.310 Brian Klaas: Be very careful that the dates are listed in if you send out emails that they are correct. 194 00:26:59.490 --> 00:27:05.700 Brian Klaas: You know, because you will get a lot of students pushback. If you say, Oh yeah, the papers due Friday, but the Dropbox says it's do Thursday and you're like, oh, I misspoke. 195 00:27:06.000 --> 00:27:09.450 Brian Klaas: Students won't give you any leeway on that right so double check those dates. 196 00:27:10.140 --> 00:27:20.520 Brian Klaas: Understand the special permissions that are tools that are available in tools like the quiz generator that allow you to let students back into their exam in case they accidentally submitted it 197 00:27:20.970 --> 00:27:29.820 Brian Klaas: Before they were really done or if you have students who need to take it outside the normal dates understand those tools CT. I'll help can help you understand and work with a special permissions tools. 198 00:27:30.180 --> 00:27:32.160 Brian Klaas: And then finally, with your quizzes and exams. 199 00:27:32.550 --> 00:27:41.640 Brian Klaas: Ask somebody a preview it you know your TA maybe somebody who took the exam took the class in a previous year, because maybe graduated. You can give them guest access to your course and 200 00:27:41.850 --> 00:27:51.750 Brian Klaas: Let them take that quiz or exam ahead of time to give you feedback I can help find a little problems or big problems that you may have in those quizzes or exams before the rest of the students in your class do 201 00:27:52.650 --> 00:27:58.110 Brian Klaas: So that's pretty much it for what I wanted to cover. I just want to put one more last shout out that if you have questions about 202 00:27:58.710 --> 00:28:05.580 Brian Klaas: The tools and course plus CCL help is there to answer those questions for you. If you have questions about designing effective assessments. 203 00:28:05.850 --> 00:28:10.920 Brian Klaas: Will something work will it not at work. Is this a good rubric. How can I improve the rubrics that I'm currently using. 204 00:28:11.100 --> 00:28:21.750 Brian Klaas: Instructional Design team in the Center for Teaching learning is also there for you. So please use their resources. They're very valuable and they'll be happy to help you. So I'm going to turn it back over to Liz now. 205 00:28:21.990 --> 00:28:23.220 Brian Klaas: And stop sharing 206 00:28:23.220 --> 00:28:23.640 Brian Klaas: My screen. 207 00:28:25.650 --> 00:28:31.170 Elizabeth Stuart: Great. Thank you so much. That was a great overview of the possibilities. I've used the discussion, not doing this. 208 00:28:31.710 --> 00:28:44.130 Elizabeth Stuart: Discussion where I'm a quiz generator for quite a few years now. And yes, it has made my life easier, but I certainly can be more creative with other other assessments in my class. So I'm now going to turn it over to Jennifer deal who's 209 00:28:44.220 --> 00:28:46.710 Elizabeth Stuart: Assistant Professor in the Department of epidemiology. 210 00:28:46.740 --> 00:28:56.130 Elizabeth Stuart: And she can give us sort of a perspective on from someone who has tried to implement a bunch of these things, and she'll kind of just talk through her perspective. I don't think I'll put the slides up so that we can 211 00:28:56.970 --> 00:28:58.380 Jennifer Deal: See her because she's just gonna 212 00:28:58.500 --> 00:29:00.390 Elizabeth Stuart: Going to talk so. Thanks, Jennifer. 213 00:29:01.320 --> 00:29:09.030 Jennifer Deal: Yeah, thank you, Liz. Thanks so much. So I teach a very large introductory FP course so epidemiologic and Princeton public health one 214 00:29:09.630 --> 00:29:17.010 Jennifer Deal: And we have multiple sections of the course, we've actually had a fully online sections of course now for over six years. 215 00:29:17.460 --> 00:29:27.510 Jennifer Deal: So we've been implementing and having online exams for quite a while. I think in this era though we've really stopped to rethink how we implement them. 216 00:29:28.230 --> 00:29:35.070 Jennifer Deal: And so I'm going to talk a little bit today. We've had a lot of discussions as an instructional team, a lot of discussions with Cpl. 217 00:29:35.760 --> 00:29:46.500 Jennifer Deal: About the best way that we can implement in the areas coven moving forward. And our goal is always to maintain fairness across the different sections of the course. 218 00:29:47.310 --> 00:30:04.050 Jennifer Deal: As well and I'll say this is large. So our online course, we typically have around 170 students enrolled, but our first term course typically in person. We've had up to 300 and I'm not sure yet what the enrollment will be for first term, but we're looking at definitely a large course. 219 00:30:05.640 --> 00:30:17.730 Jennifer Deal: So as I said, you know, our goal is to be fair across the different sections of the course in the course and thinking about the era of coven in the past, you know, traditionally we've always had proctored exam. 220 00:30:18.840 --> 00:30:28.590 Jennifer Deal: And realizing now, you know, in this situation, we have a lot of students with additional professional responsibilities additional home responsibilities. 221 00:30:29.340 --> 00:30:40.050 Jennifer Deal: A lot of additional potential stress. We also are in a situation where we can't. I think ethically asked to people who have not been living together to get together in the same room. 222 00:30:40.530 --> 00:30:50.100 Jennifer Deal: To have that proctored environment and so that leaves us with the option as an online proctor, but especially for a course our size, there is a cost. 223 00:30:50.520 --> 00:30:54.390 Jennifer Deal: For many students who use kind of outside online proctored exam. 224 00:30:55.110 --> 00:31:04.350 Jennifer Deal: Which it could be an additional burden for students. And if we really just didn't feel like it was feasible, it wasn't really tenable anymore to maintain this proctored status. 225 00:31:04.860 --> 00:31:18.120 Jennifer Deal: So with that in mind, we decided to go on proctored and so have made some changes. And our goal is really to maintain our standards that we set for the course that we've been having for many years now. 226 00:31:19.020 --> 00:31:26.550 Jennifer Deal: Again, the fairness and and really, we also wanted to think carefully about how we might minimize the opportunity or the temptation. 227 00:31:26.970 --> 00:31:33.750 Jennifer Deal: For academic dishonesty, especially in a setting of an on proctored exam. And so we as a group. 228 00:31:34.350 --> 00:31:46.530 Jennifer Deal: made several decisions that we're implementing this summer starting on Friday when we, when we have our mid term. So our exams now are open book, open notes. These two because that closes. 229 00:31:47.370 --> 00:31:59.460 Jennifer Deal: But they still are an individual assignment. We have other assignments in the course that are more collaborative and we are very clear in terms of what the expectation guard and so 230 00:32:00.420 --> 00:32:07.110 Jennifer Deal: Both on our syllabus in the exam instructions that we send to students via email. Prior to be exam. 231 00:32:07.650 --> 00:32:11.610 Jennifer Deal: When we have the exam instructions that they see when they first pull up the exam. 232 00:32:11.970 --> 00:32:19.470 Jennifer Deal: We define what we mean by open book, open notes. So we tell them they can refer to the course material. So that's everything that they have four plus 233 00:32:19.830 --> 00:32:27.960 Jennifer Deal: Their own personal notes as well as their textbook to complete the exam, but we tell them that has to be done individually, which means that they may not discuss 234 00:32:28.500 --> 00:32:38.010 Jennifer Deal: The exam with anyone else during the exam or with anyone else after the exam until the the grades have been returned and so that is clearly spelled out 235 00:32:39.210 --> 00:32:44.190 Jennifer Deal: And then we do also have, there's an option for an exam for students to check the 236 00:32:44.880 --> 00:32:57.600 Jennifer Deal: You can check a box that students have to sign an academic ethics statement before they take the exam. And we also let them know that we're going to be requiring them to do that. And I'm sure young we'll talk a little bit more about that option as well. 237 00:32:58.860 --> 00:33:10.440 Jennifer Deal: Another thing that we have changed, or at least thought more carefully about is the time the amount of time that we offer for the exam. So Brian mentioned time boxing and that is something we're using 238 00:33:10.920 --> 00:33:15.120 Jennifer Deal: And he also mentioned the course plus data that you can get on the quizzes. 239 00:33:15.540 --> 00:33:24.270 Jennifer Deal: And, you know, having administered this class now for several years, we were able to go back and look and look at the average time that it took students to complete the exam. 240 00:33:24.750 --> 00:33:37.500 Jennifer Deal: And so what we tried to do is figure out what's the time that's going to encourage masters and material, but still be fair and make sure that students have enough time. So we actually have some data to help us guide that is that decision. 241 00:33:39.390 --> 00:33:49.020 Jennifer Deal: We also are really utilizing those randomization options. So we have students randomized from exam. So Brian mentioned with set 242 00:33:49.560 --> 00:33:57.480 Jennifer Deal: For this particular exam. We have three different versions that we've created. So each student will be randomized to receive one of those exams. 243 00:33:58.230 --> 00:34:07.380 Jennifer Deal: When they get that exam, the order of the questions will be randomized and then for the multiple choice questions, even those multiple choice answers will be randomized 244 00:34:07.770 --> 00:34:18.810 Jennifer Deal: And so that just minimize the possibility of being able to talk to someone during the exam about the content. So we were really emphasizing that too and 245 00:34:19.800 --> 00:34:30.570 Jennifer Deal: We also want to be completely fair in terms of the time that we offer the exam so many of our students now are not in Baltimore are all over the world. 246 00:34:31.140 --> 00:34:43.860 Jennifer Deal: And so we have chosen three times on Friday when the exam will be open and we tried to pick times that would make it easy for people in different time zones around the world. So we've got three different times on Friday. 247 00:34:44.460 --> 00:34:53.280 Jennifer Deal: And then for our students, our learners who may need that time perhaps they are they're working, especially those who are involved in encoded 248 00:34:54.210 --> 00:35:06.300 Jennifer Deal: We're also offering a time on Saturday morning. So for possible options for them to take the exam, but again just limiting the number of the amount of time that the ACM is open as well. 249 00:35:08.100 --> 00:35:16.170 Jennifer Deal: So really that's kind of the practical decisions that we've made in terms of content. We've really thought about that. So in the past. 250 00:35:16.530 --> 00:35:23.220 Jennifer Deal: We use all multiple choice exams that was very straightforward to grade, especially for the number of students that we have 251 00:35:23.910 --> 00:35:28.860 Jennifer Deal: But really thinking about having exams and having assessments that are meaningful for students. 252 00:35:29.400 --> 00:35:38.910 Jennifer Deal: We free thought that. And so we organized. Some of our, our questions and we actually now have a couple of short answer questions are on our behalf as well. 253 00:35:39.510 --> 00:35:47.400 Jennifer Deal: We have thought very carefully about the rubric for this. So I second Brian's, you know, kind of recommendation as a rubric. There's they're critical 254 00:35:47.730 --> 00:35:52.020 Jennifer Deal: And we've worked very hard with some of our teams to come up with example. 255 00:35:52.770 --> 00:36:01.500 Jennifer Deal: We sell out how many points are given for each idea we sell out if students should be, you know, if you should take points off if there's a wrong answer. 256 00:36:01.980 --> 00:36:14.460 Jennifer Deal: Etc. So we thought carefully about that. The other option. And I think this is a really fun option that I was only recently made aware of. So it for a calculation question and course plus 257 00:36:14.970 --> 00:36:24.000 Jennifer Deal: You can actually have the student insert the answer, instead of have it in multiple choice and you can give an upper bound or a lower bound, so you can account for grading error. 258 00:36:24.270 --> 00:36:31.260 Jennifer Deal: And then, of course, plus will automatically breathe out for you. So our calculation questions or quantitative questions have now been moved to that format. 259 00:36:31.680 --> 00:36:40.170 Jennifer Deal: I think the only caution I would say is with that format, you have to be very clear to the students about what units you expect them to use 260 00:36:40.530 --> 00:36:44.730 Jennifer Deal: Otherwise course possible kind of wrong. So for example, if you have a percent. 261 00:36:45.180 --> 00:36:51.450 Jennifer Deal: Say, the answer is 85% a student could write in at five or they could write in point eight five. 262 00:36:51.780 --> 00:37:05.490 Jennifer Deal: Folks would be correct in terms of their understanding of the material. But of course, plus is only going to be able to count one of those answers right so we're just really careful in our communication with students about those units and what the expectations are. 263 00:37:06.990 --> 00:37:10.320 Jennifer Deal: So that's really kind of thinking overall in terms of the content. 264 00:37:10.920 --> 00:37:21.030 Jennifer Deal: I'll say an option in course plus that we're really excited about and we started investing in that we will use for future exams is the tagging option. 265 00:37:21.510 --> 00:37:30.330 Jennifer Deal: And so in course. Plus, you have the option to go in and tag this question and I think it makes sense in terms of, you know, you could tag it in terms of the type of content. 266 00:37:30.870 --> 00:37:35.520 Jennifer Deal: But we've also been Vic keep thinking very carefully about how we night when use these tags. 267 00:37:35.880 --> 00:37:42.930 Jennifer Deal: And so we've been labeling questions in our question bank with the year that the question was administered 268 00:37:43.260 --> 00:37:49.890 Jennifer Deal: Which section of the course was the question administered. And so all of these things will allow us to sort of questions. 269 00:37:50.460 --> 00:37:55.260 Jennifer Deal: And pull from those questions to create teacher exam. And so I think that will be 270 00:37:56.070 --> 00:38:04.560 Jennifer Deal: Easier for us moving forward. I'll say, you know, we did the quiz that this time around, and it really was. It was tough for us to get three exams. 271 00:38:05.040 --> 00:38:13.920 Jennifer Deal: That we felt were equal in terms of fairness and how hard the questions were across the board. So we're really looking forward to that. That tagging option. 272 00:38:15.420 --> 00:38:22.800 Jennifer Deal: And so I you know in terms of kind of thinking about key takeaways and things I would encourage you to think about when designing these assessments. 273 00:38:23.250 --> 00:38:34.680 Jennifer Deal: I would urge you to carefully consider whether it's really necessary to have a proctor for your exam. Again, I think this is maybe just an undue burden for a lot of students at this time. 274 00:38:35.580 --> 00:38:45.900 Jennifer Deal: Even though, that means, then, you know, again, you might want to take some extra steps to discourage academic dishonesty. I think the course has a lot of great tools that allow you to do that. 275 00:38:47.100 --> 00:38:58.440 Jennifer Deal: I talked a lot about exams, because that's really where our minds have been recently, but we have other assignments for the course that I also think are incredibly effective and they aren't even necessarily great it 276 00:38:58.950 --> 00:39:04.320 Jennifer Deal: So we have through the quiz generator questions that we give to students following each lecture. 277 00:39:04.980 --> 00:39:10.560 Jennifer Deal: That are really self assessment, we want to make sure that they understand what the key takeaway points of that lecturer. 278 00:39:10.860 --> 00:39:19.290 Jennifer Deal: And these are not graded, they can access them as many times as they like. But it's just one additional learning tool for them. The other thing I would say 279 00:39:20.070 --> 00:39:29.280 Jennifer Deal: Encouraged in terms of thinking about is sometimes individual assignments are great, but we also have some assignments that each student has to individually complete 280 00:39:29.910 --> 00:39:36.930 Jennifer Deal: But we encourage collaborative discussion and for those questions to be worked out with their peers with their colleagues in the class. 281 00:39:37.440 --> 00:39:45.630 Jennifer Deal: And part of that is because epidemiology is a very collaborative discipline and part of that is because we have different learners with different strengths 282 00:39:46.260 --> 00:39:50.850 Jennifer Deal: You know, we have students in our class who are masters level coming straight out of undergrad. 283 00:39:51.240 --> 00:40:01.050 Jennifer Deal: All the way up into students who are in doctoral programs. The degree of quantitative strength that our students have is different. And so that really gives students a chance to actually learn 284 00:40:01.800 --> 00:40:12.000 Jennifer Deal: What the material is, I think, as opposed to just worrying about whether or not they have the answer right. So I'd encourage thinking about, you know, kind of outside the box in terms of those types of things, too. 285 00:40:13.170 --> 00:40:22.830 Jennifer Deal: And also just put in a plug for Ctr. So we've been working very closely with them and Brian, you and your team are wonderful. I know one of the new changes. 286 00:40:23.370 --> 00:40:31.320 Jennifer Deal: To the course plus plus generators based off a suggestion that we had. And so we appreciate it. I think for spouses incredibly responsive. 287 00:40:31.710 --> 00:40:35.610 Jennifer Deal: And really likes to hear your thoughts and make changes constantly 288 00:40:36.270 --> 00:40:47.280 Jennifer Deal: So please don't hesitate to reach out to them. The guidance for guidance or if you find that there's something that you would like to see that you don't have the option to do. I think they're very happy to think through that with you. 289 00:40:48.000 --> 00:41:01.470 Jennifer Deal: And I think the final thing I would just say, you know, no matter what you decide in terms of your assessment. I felt as though the ideas being very clear in terms of the communication what's meant by open book, open notes or closed book close notes. 290 00:41:02.490 --> 00:41:10.050 Jennifer Deal: You know, how should should students be, you know, giving their answers in terms of the answer key so 291 00:41:11.190 --> 00:41:18.900 Jennifer Deal: We have actually created. We had a series of questions that were practice questions for our exam in the past. 292 00:41:19.410 --> 00:41:23.100 Jennifer Deal: And we actually took some of those and modified them into the new format. 293 00:41:23.550 --> 00:41:31.860 Jennifer Deal: Right, so we didn't have to write any new questions. We just modified what we already had. But that really gives students a chance to see what our expectations are. 294 00:41:32.460 --> 00:41:41.970 Jennifer Deal: And what they might be facing on the day of the exam so that they feel prepared. So I think those are my my key points. So I'll turn it back to us. 295 00:41:44.190 --> 00:41:52.740 Elizabeth Stuart: Great. Thank you so much, really good to hear about how a very large class is tackling this good luck on Friday, Saturday. 296 00:41:54.240 --> 00:41:54.780 Jennifer Deal: Thank you. 297 00:41:55.650 --> 00:41:56.640 Elizabeth Stuart: Maybe we can do a 298 00:41:56.700 --> 00:41:57.390 Elizabeth Stuart: Check them later. 299 00:41:58.140 --> 00:42:00.690 Elizabeth Stuart: So I'll now hand it over to john for Nick. 300 00:42:00.990 --> 00:42:02.190 Elizabeth Stuart: Who is a professor in the Department 301 00:42:02.190 --> 00:42:07.140 Elizabeth Stuart: Of Health Policy and Management, but today we'll be talking from the academic ethics ethics. 302 00:42:07.170 --> 00:42:10.950 Elizabeth Stuart: Perspective and I think actually some of his comments will build nicely on 303 00:42:11.010 --> 00:42:11.730 Jennifer Deal: What we just heard. 304 00:42:12.840 --> 00:42:23.610 Jon Vernick: Thank you, Liz and everyone for being here. My lights are flickering. It is raining hard here. So if I crash. I'm, you know, building an ark and gathering the animals together so 305 00:42:24.480 --> 00:42:26.280 Jon Vernick: Apologies. I 306 00:42:26.520 --> 00:42:38.970 Jon Vernick: Let me emphasize right off the bat, no Proctor and as I put in the chat know proc during is going to be feasible this year. This is something that Brian and Liz and Laura and Ellen and I have all discussed 307 00:42:39.270 --> 00:42:54.660 Jon Vernick: Voluntary proc during as some of you have probably used in the past, is it feasible in a world where those voluntary Proctor's were ordinarily outside of one's household and there are a whole host of problems with 308 00:42:55.860 --> 00:43:08.550 Jon Vernick: With commercial products and services, especially for our students in other countries, and you can't proctor some students and not others and maintain equity as, as Brian was saying. 309 00:43:10.440 --> 00:43:24.210 Jon Vernick: I know, then we're all trying to take the class that we had and turn it into a virtual or or online version. I'm here to tell you that it's not necessarily going to be easier for you to simply do 310 00:43:24.420 --> 00:43:40.830 Jon Vernick: What you have been doing in the past in an online version nine courage you to think in the same way. Jennifer has about how to modify accordingly. One reason, it will not be easier to do what you've always done, for example, give multiple choice. 311 00:43:42.330 --> 00:43:56.340 Jon Vernick: Exams is I can tell you from an ethics perspective, if you have one academic ethics try all that we have to have that will take more of your time than reading dozens 312 00:43:56.640 --> 00:44:06.630 Jon Vernick: Of authentic assessments would so whereas you think maybe you're saving yourself some time. You're actually not dozens might be an understatement actually 313 00:44:07.500 --> 00:44:22.020 Jon Vernick: Ethics events are simply a disaster, a disaster. Obviously for the student but a disaster for for you and your time and your stress level and your use of migraine medication and everything else and 314 00:44:22.590 --> 00:44:37.500 Jon Vernick: I encourage you to do your best. As a result, to avoid closed book exams and that typically means thinking hard about as both Brian and Jennifer have said authentic assessments. 315 00:44:37.890 --> 00:44:40.560 Jon Vernick: And that's such a buzzword. We all hate it but 316 00:44:41.190 --> 00:44:55.650 Jon Vernick: If you prefer. Think of assessments, again, as Brian said that better mirror what the students will actually be asked to do in a real world setting thats related to your class and just an example. I teach 317 00:44:56.370 --> 00:45:13.470 Jon Vernick: 175 undergraduate students up at Homewood and that's regularly and in person class. And so, of course, I've always given multiple choice exams or at least partly multiple choice exams, because my gosh, who wants to grade 175 318 00:45:13.920 --> 00:45:28.410 Jon Vernick: Questions. But when that went to online in the fourth term I decided I can't give I can't give them multiple choice exam and feel confident about academic ethics issues. 319 00:45:29.130 --> 00:45:43.140 Jon Vernick: And so I crafted and policy analysis. It's a policy course that I teach are real world real life policy analysis I craft a little story where the students were asked to think of themselves as 320 00:45:43.530 --> 00:45:53.580 Jon Vernick: Being given an assignment in their first job as a policy analyst and I kind of tell you when I got the student feedback they loved the authentic. 321 00:45:53.940 --> 00:46:08.850 Jon Vernick: Me who were ever heard of an undergraduate praising and exam, they, they love the authentic assessment and let me tell you, for the three or four years prior to that, I never once had a student. Tell me, do you love the multiple choice exams, so 322 00:46:10.080 --> 00:46:16.650 Jon Vernick: You, you can do it, you can do authentic assessments, you can make them timed 323 00:46:17.940 --> 00:46:27.210 Jon Vernick: In a variety of different ways. Again, that Brian and Jennifer have talked about. That's what I did. I gave them a certain amount of time from the moment that they opened the exam. 324 00:46:28.320 --> 00:46:38.820 Jon Vernick: And they had a 24 hour period to open the exam. So time zones weren't an issue. And trust me, I had undergrad students who had decamped back to their 325 00:46:39.360 --> 00:46:50.490 Jon Vernick: their family's home country, wherever that was and all worked out just fine. Of course there's more grading, but there wasn't a single academic ethics problem. 326 00:46:51.000 --> 00:46:57.750 Jon Vernick: You've got to give those clear expectations, how many times, if we said this now and it can't be in a single email. It's got to be 327 00:46:57.960 --> 00:47:05.910 Jon Vernick: In three different places. We know that people don't read emails as much as we wish. They don't read the syllabus as carefully as we wish. It should be 328 00:47:06.270 --> 00:47:13.170 Jon Vernick: In my opinion in the syllabus in an email and in the exam itself in the instructions to the exam when they 329 00:47:13.980 --> 00:47:27.180 Jon Vernick: When they open it. So I urge you to avoid as much as you possibly can close book exams to if you can avoid multiple choice and substitute authentic assessments. 330 00:47:27.540 --> 00:47:36.900 Jon Vernick: Very last thing I'll say, so I have lots of time for discussion is and maybe Brian can help me here as well. There was a brief allusion to 331 00:47:37.470 --> 00:47:47.130 Jon Vernick: Give the students instructions about what to do if they have computer problems. Well, we have told the students in the context of academic ethics training. 332 00:47:47.550 --> 00:47:53.760 Jon Vernick: Is if your computer crashes in the middle of the exam. Keep working 333 00:47:54.420 --> 00:48:07.800 Jon Vernick: Keep working and once you have had Internet access restored and not your computer crashes, but you lose Internet access once Internet access is restored all of the work will have been say 334 00:48:08.760 --> 00:48:22.110 Jon Vernick: Now, if it turns out you are some place where Internet access is coming and going coming and going. That's a different problem. And the students have been instructed to contact you. 335 00:48:22.680 --> 00:48:32.940 Jon Vernick: And we presumably by their phone, and we encourage you to be as flexible and reasonable as you can. Under those circumstances. 336 00:48:33.660 --> 00:48:48.990 Jon Vernick: And we're just all going to have to accept that there's going to be more of these unusual circumstances than we've seen in the past with more students under more different circumstances, taking say, Brian, anything you'd like to add to that. 337 00:48:50.310 --> 00:48:51.210 Brian Klaas: Nope, that's great. Young 338 00:48:51.840 --> 00:48:58.560 Jon Vernick: Okay. Well, as I said, I'll shut up so that there's lots of time for for good questions and discussion. 339 00:49:00.540 --> 00:49:06.630 Elizabeth Stuart: Great. Thank you so much. This was really helpful. I've, I've learned a lot already. And one thing I want to flag. 340 00:49:07.080 --> 00:49:13.020 Elizabeth Stuart: Is it if you're not following the chat. There's Salinas posting some really helpful links to different resources so 341 00:49:13.710 --> 00:49:24.240 Elizabeth Stuart: Keep a look over there on. I have a few questions that people have posted that I'm going to try to get through the first one I think is a quick one. But I think it's so timely because 342 00:49:25.200 --> 00:49:36.660 Elizabeth Stuart: It timely because it's about time zones. I think we, you know, there's lots of considerations this fall about time zones and where students are going to be located. So, Jennifer. This is really a question for you, which is 343 00:49:37.440 --> 00:49:37.950 Elizabeth Stuart: Sort of 344 00:49:38.220 --> 00:49:48.990 Elizabeth Stuart: Can you talk through why you all decided to do these sort of set blocks of to our windows. The four to our windows versus just opening up an exam for 24 hours and letting students are to 345 00:49:49.260 --> 00:49:50.640 Elizabeth Stuart: Choose which two hour window. They want to 346 00:49:50.640 --> 00:49:51.000 Use 347 00:49:52.140 --> 00:49:53.460 Elizabeth Stuart: Right, so we used to 348 00:49:53.460 --> 00:50:05.340 Jennifer Deal: For this course, we would open the exam up on a Thursday, and we would close it on a Saturday, and it was an open open for an entire three days and the students could take it and it was a two hour time block, but they could take it whenever they chose 349 00:50:06.510 --> 00:50:11.100 Jennifer Deal: Their concern with keeping that kind of schedule was got that as a lot of time. 350 00:50:11.730 --> 00:50:17.370 Jennifer Deal: And a lot of access to the exam, which could potentially encourage some discussion between students 351 00:50:17.700 --> 00:50:25.890 Jennifer Deal: Who maybe have taken the exam already and not taking it and that could not even be intentional. Right. Sometimes students just want to talk about how they felt the exam with 352 00:50:26.460 --> 00:50:37.020 Jennifer Deal: It so we felt like if we tried to be understanding of time zones, but limit the time that the exam was open that could potentially reduce some of that chatter, or at least potential chatter 353 00:50:38.820 --> 00:50:40.440 Elizabeth Stuart: Great, thank you. I think 354 00:50:41.970 --> 00:50:44.760 Brian Klaas: The other point to point about that is that it's also about the availability of the faculty and 355 00:50:45.510 --> 00:50:48.630 Brian Klaas: Students will expect you to be awake at three o'clock in the morning when they're taking the exam. 356 00:50:48.960 --> 00:50:52.950 Brian Klaas: And get back to them immediately if they have any kind of problem whatsoever. 357 00:50:53.190 --> 00:50:59.940 Brian Klaas: And by putting these sort of time slots in there at which people are available, you can get back to them immediately. You can be monitoring your email and say, 358 00:51:00.120 --> 00:51:07.050 Brian Klaas: Okay, I gave you another access to the another time to access the exam. It'll if it's a one one time attempt. So I think that's a really important consideration as well. 359 00:51:08.340 --> 00:51:08.940 Elizabeth Stuart: Yeah, thank you. 360 00:51:11.010 --> 00:51:11.400 Elizabeth Stuart: So, 361 00:51:12.420 --> 00:51:20.940 Elizabeth Stuart: This one is a big, you know, we only have like six minute or want to make sure we end, I very time they want to make sure we end by five, five and I have a few announcements. I want to make at the end. 362 00:51:21.300 --> 00:51:28.170 Elizabeth Stuart: So this next topic is huge and it's about rubrics. There's been sort of a lot of chat in the chat box about rubrics and 363 00:51:29.040 --> 00:51:35.970 Elizabeth Stuart: You know, one angle maybe to make it yon responded, a little bit, but I think, you know, one angle ON THIS IS JOHN If you could 364 00:51:36.510 --> 00:51:47.610 Elizabeth Stuart: Sort of talking about how you did manage to grade the hundred and 75 policy analyses, but more generally. And so just, you know, brief thoughts from any of you. 365 00:51:48.000 --> 00:51:59.460 Elizabeth Stuart: On this tension of, sort of, how do you make a rubric that is kind of detailed enough to be helpful as a rubric, but not so detailed that it kind of constraints, the creativity of the students and sort of 366 00:51:59.730 --> 00:52:14.280 Elizabeth Stuart: Some people in the chat kind of comment to that they've had experiences where it sort of feels like the students just check the boxes for rubric. And so, Brian. Maybe I'll start with you and just see again, we don't have a lot of time. But if you just have some quick reactions to that. 367 00:52:14.700 --> 00:52:19.530 Brian Klaas: Sure, this is an ongoing problem for me because part of my courses about design as well as delivery. 368 00:52:19.980 --> 00:52:25.110 Brian Klaas: How you design the information that you're presenting to people that's super important. That's half the story that you're telling 369 00:52:25.350 --> 00:52:32.190 Brian Klaas: And we, I do have students who just like, Well, I did this and this and this and this and it looks terrible or no effort was put into that. 370 00:52:32.340 --> 00:52:37.200 Brian Klaas: And it frustrates me, but I have to give them a decent grade because the rubric tells me that I have to give them a decent grade. 371 00:52:37.380 --> 00:52:44.370 Brian Klaas: So this is an area that I've been exploring a bit recently. And there was a really good session in the Delta symposium earlier this year from a faculty member 372 00:52:44.820 --> 00:52:48.570 Brian Klaas: At Homewood who does is kind of an interdisciplinary science courses where she talks about 373 00:52:48.870 --> 00:52:56.580 Brian Klaas: Talks about introducing creativity into her rubrics and how you assess creativity during with rubrics and there's a bunch of good resources on the web. 374 00:52:56.970 --> 00:53:09.360 Brian Klaas: There's a bunch of educational exchanges that have these I'm still trying to figure out the best way to do that and not make it a check the boxes, kind of thing because creativity is only one part of the rubric and i think you know this girl had mentioned you want to 375 00:53:11.010 --> 00:53:23.340 Brian Klaas: You know, sort of leave things a little bit vague, a little bit open ended not using you know exactly this criteria with this quantitative amount so that students have some flexibility to work with, but I'd love to hear from either saline or Kathy. 376 00:53:23.880 --> 00:53:29.850 Brian Klaas: Who are here from Center for Teaching and learning about really quickly. If you have any ideas on that front. And I see the Jennifer Weinstock 377 00:53:30.660 --> 00:53:41.880 Brian Klaas: You know mentioned three share rubrics to help with these problems. And I think that's actually a really great idea, sharing some examples to see what works in other classes and, you know, steel what works. There's nothing wrong with that. Except not when you're taking a test. 378 00:53:45.660 --> 00:53:51.390 Celine Greene (CTL): I'm this is Celine. I just want to point out that in terms of shared rubrics. For one thing, 379 00:53:52.440 --> 00:54:01.650 Celine Greene (CTL): The peer assessment tool inside course plus that Brian mentioned before, can be used just for faculty and teachers as well as peer to peer and also self assessment. 380 00:54:01.980 --> 00:54:12.180 Celine Greene (CTL): We actually have sample rubrics in the peer assessment tool, including holistic and analytic rubrics and if you don't know the difference of holistic and analytic 381 00:54:12.630 --> 00:54:18.420 Celine Greene (CTL): Because of the timeframe. I'm just going to tell you go to the events page under past events on CTS toolkit. 382 00:54:18.810 --> 00:54:25.260 Celine Greene (CTL): We had some phenomenal workshops on. I apologize that names are escaping me right now. 383 00:54:25.620 --> 00:54:39.030 Celine Greene (CTL): But talked about how to build a rubric and talked about the different types of rubric and what you're actually evoking from the students using the rubric as a motivational as opposed to a punitive or checklist tool. 384 00:54:39.480 --> 00:54:50.010 Celine Greene (CTL): So that's all I would really want to do is both push the fact that there are some sample rubrics in the peer assessment tool we encourage faculty sharing and I think Liz is going to be mentioning 385 00:54:50.370 --> 00:55:01.470 Celine Greene (CTL): peer mentors. Before we leave today and that might be one place to start, as well as go ahead and please visit our events page and look for the recording under past events for our toolkit on rubrics. 386 00:55:02.250 --> 00:55:09.630 Kathy Gresh: Yes, this is I'm Cathy crash, just the follow up with slowly and I did pull it up on saline, and there was a really great workshop 387 00:55:10.410 --> 00:55:27.750 Kathy Gresh: Last September. Um, it was a feature guest, Dr. Sarah pointing from the School of Medicine is actually written a book on rubrics and there's a recording of her presentation. It's just really fantastic. And also, if you need any help with rubrics, you are free to 388 00:55:28.980 --> 00:55:37.080 Kathy Gresh: You know, set contact that instructional designer, they'll give you some feedback on your rubrics. There's a lot of misunderstanding about how Roberts work sometimes 389 00:55:37.950 --> 00:55:46.110 Kathy Gresh: So feel free to contact us and as links out there is great examples. If you go to a peer assessment and look at the rubric area. 390 00:55:50.910 --> 00:55:53.160 Elizabeth Stuart: You're under Jennifer. Do you want to add anything 391 00:55:54.330 --> 00:55:57.570 Jon Vernick: No, I'd rather have others ask questions, if there are any others. 392 00:56:02.550 --> 00:56:10.140 Elizabeth Stuart: There, there was a there were not a lot of other questions there. And let me. Sorry, I got myself disorganized and 393 00:56:12.840 --> 00:56:13.290 Elizabeth Stuart: The 394 00:56:14.550 --> 00:56:25.110 Elizabeth Stuart: I will just maybe just. Well, I do want to save two minutes at the end, but maybe a quick comment just for people who didn't see it in the chat yawn. If you could summarize. There's a little back and forth about 395 00:56:25.860 --> 00:56:37.710 Elizabeth Stuart: non English speakers and sort of how to think about the length of an exam for people who might struggle have more struggles with English and sort of how to think about that. So yeah, maybe you could just quickly summarize 396 00:56:37.860 --> 00:56:49.380 Jon Vernick: No, I really, really appreciate that. It's, it's incredibly important from a fairness and ethics perspective, not to make individual decisions. 397 00:56:49.770 --> 00:57:00.360 Jon Vernick: jump around and give some students more time than others on the exam because of your assessment about the their ability to complete in a reasonable amount of time. 398 00:57:00.780 --> 00:57:04.890 Jon Vernick: Because of English issues. Now, if they have an accommodation 399 00:57:05.850 --> 00:57:21.270 Jon Vernick: That the university or school has approved for extra exam time, then you can go into course plus for a timed exam and one by one. Set the additional amount of time that they get. And that's what I have 400 00:57:21.600 --> 00:57:27.390 Jon Vernick: Done. But I strongly urge you, in fact, we could get in trouble if you start individually. 401 00:57:28.230 --> 00:57:37.710 Jon Vernick: Giving additional time to specific students if you're worried that that students don't have enough to whose English is not flawless us English 402 00:57:37.950 --> 00:57:56.040 Jon Vernick: To complete an assessment, then maybe that assessment isn't giving students enough time and you should instead give all students more time and create the assessment in a, in a way that can be completed within the the reasonable amount of time. Thank you for that. 403 00:57:57.780 --> 00:57:59.700 Elizabeth Stuart: Oh, I'll just do that too, if that's 404 00:57:59.970 --> 00:58:01.980 Jennifer Deal: If that's okay. That's just to jump in quickly. 405 00:58:02.220 --> 00:58:09.690 Jennifer Deal: We have a large team. So we're lucky that we usually have at least one non native English speaker, as our th. But we usually like to try to 406 00:58:10.020 --> 00:58:23.520 Jennifer Deal: RTA practice our exams, before we get to that. And so by having a native English speaker, and a non native English speaker that can also be helpful in terms of understanding and then once you decide on the time completely agree. It should be the same for everybody. 407 00:58:25.950 --> 00:58:26.580 Elizabeth Stuart: Great. 408 00:58:27.660 --> 00:58:32.430 Elizabeth Stuart: Thank you so much. So we, I think we didn't quite get to all the questions. I see. 409 00:58:32.700 --> 00:58:39.600 Elizabeth Stuart: Stan has his hand up. But again, this is really meant, just to be the start of the conversation. So I want to take just a minute to kind of 410 00:58:40.020 --> 00:58:46.980 Elizabeth Stuart: highlight some additional ways to continue this. So first, a big thanks to the presenters. Again, I certainly learned a lot. 411 00:58:47.520 --> 00:58:56.520 Elizabeth Stuart: And I think a great example of the expertise we have in house please email me with any questions comments or suggestions we have these sessions, for example, scheduled through 412 00:58:57.630 --> 00:59:02.550 Elizabeth Stuart: Sort of mid late August. But if there's interest we can continue them into the fall 413 00:59:03.150 --> 00:59:12.930 Elizabeth Stuart: I really want to highlight this CL virtual teaching resources page. I'm calling this the one stop shop and it really is a place where a lot of these things are sort of there. 414 00:59:13.590 --> 00:59:20.790 Elizabeth Stuart: And to have we've tried to assemble kind of the key things people are going to need to be using and thinking about this, this fall. 415 00:59:21.510 --> 00:59:26.400 Elizabeth Stuart: And then there's two other sort of resources. I want to highlight one is this Microsoft Teams group. 416 00:59:26.640 --> 00:59:40.170 Elizabeth Stuart: Where we've set up a team site for virtual teaching. And the idea is that, that can be a place to share ideas that could be a place for you to share rubrics and have ways continue. Some of these conversations and then select briefly mentioned the 417 00:59:41.220 --> 00:59:48.540 Elizabeth Stuart: Faculty peer teaching mentors and senior teaching assistants, hopefully you all saw announcements about those. And I'm very pleased to say 418 00:59:48.990 --> 00:59:59.970 Elizabeth Stuart: I think as of yesterday. We've pretty much finalized our list. We're gonna have orientations for those people on Monday, and every department has representation and there 419 01:00:00.870 --> 01:00:04.500 Elizabeth Stuart: Is now a PDF online Celine will post in the chat a link to it. 420 01:00:05.130 --> 01:00:15.840 Elizabeth Stuart: Where you can find out who your kind of representatives are in your department, although hopefully they will also be reaching out to you if you're teaching it first term, hopefully there'll be reaching out to you soon. Second term might wait a little bit 421 01:00:16.500 --> 01:00:24.900 Elizabeth Stuart: But the idea is that they are going to be a resource for people for all the faculty who are teaching virtually in the fall to help with these sorts of questions. 422 01:00:25.470 --> 01:00:36.840 Elizabeth Stuart: And then we have two more workshops August six, that one, which is sort of a general promoting discussion online and then August 18 is sort of a just general 423 01:00:37.560 --> 01:00:45.600 Elizabeth Stuart: readiness for the fall, kind of like little quick tips and strategies, maybe, including some like zoom tips and for that one. We have some really great presenters who 424 01:00:45.930 --> 01:01:00.000 Elizabeth Stuart: Have been teaching this with the summer, winter for the summer and can kind of share some of the strategies that they've been using this summer. So with that, again, I see some helpful Selena has posted a ton of helpful. 425 01:01:01.050 --> 01:01:16.110 Elizabeth Stuart: Things in the chat. So please, you know, cut and paste there before we leave. And a huge thanks I'm thrilled. We had such good attendance again and I'm excited to continue these conversations in lots of different ways. So thank you so much. Have a good evening.