Unknown Speaker  0:00  
Hi there. I'm Brian Klaas. And today I'm going to cover in a very brief manner, the topic of designing effective online assessments. And I think it's important to start out by asking the question, Well,

Unknown Speaker  0:14  
what do we mean by effective, right? If we're going to talk about effective online assessments, we need to know what effective means well, effective has a number of sort of different lenses. And the first one is that I think effective means you're going to assess students fairly, that the assessments are aligned with the learning objectives, not only for the course, but that particular topic or module in your class. And also that any variation in the kind of assessment that you do is equitable, making sure that everyone gets effectively the same assessment, even if there's a randomization, different kinds of assignments assigned or topics assigned to students that there's equity across all the different kinds of assessment that you're doing for that particular assessment. Effective also means that it's implementable in the current environment, especially right now, you know, making sure that this is an assessment you can actually do. So for example, I teach a number of classes on communication for health science professionals, and in one of these classes, one of the major activities is doing an elevator pitch where you have to actually physically walk around the School of Public Health School of Medicine, East Baltimore campuses, and pitch your research to your peers and to the others in the class. And this is a really exciting activity that students get a lot out of, because sometimes just walking and talking can be very difficult in and of itself. But in the current environment where there's a pandemic going on, we simply cannot do a physical in person elevator pitch, so it's not implementable in the current environment. So instead, what we did was we did some online versions of this where people walked around with family members, that people they were they were in a bubble with, while recording themselves, that's what we did instead. So really, it's about making sure that you can actually do it in the current environment, and that the students have the resources to do it, that you have the resources to do it, right, that if you're asking them to do work in a wet lab, that they actually have access to a wet lab, or if you're going to be doing an essay, that you have the resources to grade that essay and give feedback in a timely manner, because that's one of those things we see time and again, in student evaluations is they want timely feedback. So making sure you're you're able to do it. And then finally, don't forget the kind of home and work considerations, especially right now, when students are in a pandemic environment, they may be living in a part of the world where electricity is not regular, or internet connectivity is not regular, or it's intermittent or slow at best. So doing a two hour online exam may not work well for them. Or they may be living in an apartment with five other people and not really having any private space to focus. And that can definitely be a challenge for them. In terms of doing the assessment. Next up, it's important that your assessments are found meaningful by the students right? assessment should mirror something they're going to need to do in the real world when they graduate students want practical implementations of what they're learning in the class. This is why things like sometimes presentations, op eds, are really important and meaningful for students or debates, any number of non sort of regurgitated activities, where they actually have to synthesize things put theory into context, into the kind of thing that they might do in the real world, it becomes more meaningful for students and that makes it more effective. Make sure your assessment is feasible to design and deliver it right. You know, if you're thinking, Okay, I have to translate my on person in in person assessment to an online environment, can you actually do that with your schedule and get it up and running and ready to go and sort of tested and thought through? Clearly before? You know, you you put it out there that you're going to do that? Is the assessment a reasonable expectation of the students time, right? Again, I talked earlier about students who might be in environments where they don't have reliable internet access is a 90 minute exam is a two hour exam reasonable for them? Or if you're like, oh, we're not going to do an exam, you're going to write a 14 page paper in the next week? Is that reasonable of their time given all the other things that they're doing and all their other classes? And again, can you deliver meaningful feedback in a short time period? This is super important to students. As I mentioned earlier, they really want feedback as close as possible to when the assessment is delivered to help make course corrections and adjustments for the rest of the class. And effective online assessment definitely limits the risk of academic ethics concerns, right? We want assessments that require critical thinking rather than just restating facts. This is why project based assignments can be really effective to help limit academic ethics risks, versus a multiple choice question where students can easily look that up. If you are going to be doing exams, look at exam randomization and time boxing. I'll be talking about those a little bit later in this presentation, and employ open book assignment formats whenever possible, even on exams. There's ways of making These exams are equally as challenging with an open book format as a closed book format. But it does reduce students encouragement or incentive to cheat when you're using open book assignment formats. So of course, plus offers a number of online assessment tools that you can use beyond just quizzes and exams, which we'll certainly talk about. But there's a bunch of other tools that you may not be aware of. So I quickly want to cover those. The quiz generator, of course, is probably the one that most people are familiar with, right? It supports multiple question types, there are seven different eight different question types. There's very powerful randomization tools, where you can randomize the display order of questions, you can randomize the display order of multiple choice answers, you can provide multiple exams and have the student be randomly assigned to one of those exams. And there's even something called tag rule quizzes where you assign tags, like lecture one, or readings or cellular biology to a question. And then you say, hey, I want you know, three questions from week one, two questions from the lectures five questions from the readings based on these tags. And it randomly assigns and creates a quiz for students based off that very, very powerful. In terms of timely feedback, there's automatic grading of many different question types there. So you can get feedback back quickly to students. And there's of course tools for managing special permissions for students who may need to take an exam again, or take it out of the norm, that sort of standard dates for the class, the CTL blog, the link is on the screen here. The CTL blog has lots of different posts about the quiz generator, and a lot of new features that were added this year. So if you want to learn about that, you can do that there. The Dropbox is great for turning in papers, projects, bundles of code, for those of you who might be taste teaching statistics classes. And it's great for turning in homework rather than doing it via email, because email is problematic and difficult. In terms of organizing all of this, it has organization and reminder tools for you and for the students as students can get reminded. And it's easy for you to sort of collect everyone's homework in one place and download it in a single zip file. There's a variation on the dropbox called the timed Dropbox, where you can say, okay, you only have as a student, two hours to open this Dropbox, look at the instructions, grab the file, and then submit your file, file your file within two hours. And this is really powerful when the answer capabilities of the quiz generator generator aren't enough, when you have them doing something, say an Excel or they have to hand draw things or use another program to create output that the quiz generator wouldn't be able to accept. Time, Dropbox can be very, very useful for that. Pure assessments are great, right? Because students evaluate each other and it takes a burden of the assessment by the faculty off their shoulders, and has students do that work instead. And the great thing about peer assessments is that they are rubric based evaluations of pure work. And rubrics are very powerful and useful for students for knowing what's expected of them and how to achieve excellence on a particular assessment. You can do random assignment, you can do group based assignment, you could say, you know, everybody has to grade two papers from somebody else, you can even link to Dropbox submissions, so you can say, Okay, everyone turn in your file and a Dropbox and you're going to randomly get assigned two of your peers to do that. There's other kinds of assessments for faculty and ta so that they can do a rubric based evaluation of student work as well. The discussion forum is great for assessment because it allows a conversation and for students to in a more written way, engage in deep thought and in conversation and debate with one another. You can use it for reflective discussion, you can have debates among groups of students, you can do journal article analysis, you can do peer analysis of each other's posts, and all these posts can be graded inside the gradebook themselves, sorry, inside the discussion forum themselves, and then imported into the gradebook. You can also allow for private small group work. So you can break people into small groups and have them do those same act discussion forum activities inside of small groups that can only be seen by those groups. And not everyone in the class. presentations or other kinds of performance based assessment are great. You can use zoom to do presentations, debates, student led discussions. Again, in my communications classes, we use present this a lot for all different kinds of performance based assessments. VoiceThread is another tool that you can use to have students present or give lectures and have students comment on those lectures or presentations that they do their. And don't forget the students smartphones are powerful creation tools. They can create all sorts of interesting content, video, audio, there's lots of other apps that are out there that help them create content that would allow you to assess their mastery of a particular set of knowledge. So I'd like to wrap this up here with a couple of tips around about designing effective online assessments that we've learned in the Center for Teaching and Learning that we'd like to pass on to you that we think can be helpful if you're going to be looking at an exam during a large exam. Say 6090 minutes, two hours, consider splitting that into two different parts. The first part can be a multiple choice part of it so that students can do that work in one environment, maybe say in 60 minutes, and then have a second part where the essay and open ended questions are there. And what this does that tends to work out very well is that gives students an opportunity for a break in the middle of their assessment, right? They're doing the multiple choice one quickly, it's time box, they don't have lots of time to do it, right, you're only giving them 60 minutes instead of say, two hours. And if they don't know the material, they wouldn't be able to answer the question in the time allotted. But it gives them an opportunity for a break. And that also allows for immediate feedback to students. Because multiple choice can be automatically graded by the quiz generator. And then the essay portion is there as the second part where you would manually grade that, and the essay portion allows for authentic student responses, do they really understand can they synthesize the content that we put together they can, can they apply it to real world scenarios, it's much more authentic in terms of their learning, and, and and their, your assessment of their knowledge of the exam. So having a examined two parts definitely helps us rubrics, rubrics are great. The Center for Teaching and Learning of law has lots of resources on rubrics about how useful they are. Because they set clear expectations about what is required for excellence. I use rubrics in all of my classes, I know a number of other faculty have moved to that as well. And in those classes, in many cases, in most cases, I would say it's significantly reduces arguments about grading, because the rubric spells out in no uncertain terms, exactly what students need to do to achieve excellence on their assessments. There's no arguing it's like it's in the rubric, that's what you're being evaluated on, you didn't follow the rubric. Therefore, you know, you didn't achieve this level of excellence on this particular criteria. There is in the pure assessment tool, something called the faculty slash ta peer assessment

Unknown Speaker  11:53  
type.

Unknown Speaker  11:54  
This allows you to apply rubrics to any kind of assessment really, students don't have to turn files into a Dropbox, they don't have to evaluate each other, even though it's in the peer assessment tool. They don't have to evaluate each other. It's a faculty TAA peer assessment, where you can use rubrics to evaluate student work, and then also use the data that is given to you by course, plus to determine the quality of the assessments you're creating and the fairness of the questions. This is particularly the case in the quiz generator. You can there's all the grading data for stuff in the quiz generator, pure assessments, surveys, and and of course, the gradebook is all available in Excel format for you to do data analysis on your own about this. There are statistics in the quiz generator that talk about you know, the number of percent of students who got a question right or wrong, and that's broken down by answer by question. They're scoring statistics. And there's something called a discrimination value report in there as well, that shows you the the the sort of variants in between low scoring students on an exam or high scoring students on exam and, and whether or not they got specific questions right or wrong, and how that might help you sort of rethink those questions which are maybe too easy or too difficult. You can have all this data, it's available on all of your classes. So you can examine students, across students in a given class, as well as across offerings. Of course, this is something that's been done by a number of the large epi and biostats classes to help sort of figure out how they can do better in terms of creating effective assessments online, and effectively and equitably assessing the students as well. And then finally, you know, just make sure that you're prepared and the students are prepared for any kind of assessment provide clear instructions for who to contact for what right, if it's a technical issue, contact CTL. If you CTL help, if you have a question about the content of the assessment, instructions, contact a TA or you as the faculty, whoever it might be, provide really clear instructions so students aren't left in a panic, check those open and do or close dates in the system, make sure they're the right ones, the last thing you want is to have an assessment close early, like an exam close early, because the wrong date or time is in there. Or leave it open, you know, for a week instead of a day, you know, as you had wanted there, understand how special permissions work, particularly in the quiz generator. They're very powerful, very useful for making sure that students who can't take an assessment outside of normal days and times can can do that work. And they're also useful for cases where students can have accidentally submitted their answers or they lost their internet connection. There's tools in there to help you sort of get them finished with their exams. And then also remember to just preview them have your if you have a TA have them tested, have a colleague test it just to make sure it makes sense that it works correctly as you expected. The questions make sense to you and others you might be working with on your course instructional team. And finally, if you need help you have questions, contact CTL help. They're great for how do I do this in course plus questions. They will gladly answer those Questions about how do I set up this exam? How do I do special permissions? How do I create a rubric in the peer assessment for faculty ta assessment, all those things are there for you and the CTL instructional design team. The whole group of them is more than happy to work with you in moving your maybe traditional face to face assessments into an online environment. They have lots of examples. They work with lots of different faculty, they can provide good war stories, things to look out for. They're happy to work with you just reach out to them and ask them for that.

Transcribed by https://otter.ai