WEBVTT

1
00:00:00.030 --> 00:00:02.360
Lauren Dana: Great greeting consistency

2
00:00:02.520 --> 00:00:04.700
Lauren Dana: across teaching teams.

3
00:00:04.810 --> 00:00:22.519
Lauren Dana: I did also want to let you know that the session is going to be recorded which you probably just got alerted to and during the session we also encourage you to post any questions in the chat. But there will be multiple times throughout the workshop, where we stop for question and answers. So thank you again for coming.

4
00:00:25.200 --> 00:00:35.630
Lauren Dana: I'm 1st going to start to introduce 1st myself. I'm Lauren Dana. I am an instructional designer at the center for teaching and learning. I'm going to quickly pass it over to my colleague, Amy.

5
00:00:35.850 --> 00:00:40.870
BSPH CTL Teaching Toolkit: Hi! I'm Amy Pinkerton, and I'm a senior instructional designer at the center for teaching and learning.

6
00:00:41.400 --> 00:01:04.800
Lauren Dana: Thank you, Amy. And today we actually are very excited because we have some special guest speakers. We have faculty and teaching teams from the Bloomberg School of Public Health, who are going to talk about their own experiences with grade norming. In their quite large courses. They will introduce themselves more thoroughly during their portion of the workshop. But I did just want to introduce you to our guests, Michelle, Bedoya.

7
00:01:05.379 --> 00:01:07.009
Lauren Dana: Annalise, brochure

8
00:01:07.770 --> 00:01:12.170
Lauren Dana: and and Lily, so we want to thank them for spending some time with us today.

9
00:01:14.790 --> 00:01:31.039
Lauren Dana: Alright, before we get started, I want to quickly go over our learning objectives for today. So after attending and viewing this workshop, you will leave being able to define grade norming, explaining both the benefits and the process, as well as examining 2 of our Bsph case studies.

10
00:01:31.170 --> 00:01:35.390
Lauren Dana: and you will hopefully leave with some tips and resources that will help you in your course.

11
00:01:35.780 --> 00:01:37.850
Lauren Dana: I'm now going to hand it over to Amy.

12
00:01:38.420 --> 00:01:47.040
BSPH CTL Teaching Toolkit: Thanks, Lauren. So I'm going to start by briefly defining what we mean by grade norming and then explaining or identifying some of its benefits and then going over the process.

13
00:01:47.550 --> 00:02:11.349
BSPH CTL Teaching Toolkit: So what is grade norming, so ensuring that all graders are consistent in their assessments is crucial for fairness. In grading, for example, in a large class, a student shouldn't have to worry about if it's going to be fair or not, or their grade shouldn't be determined by who is grading their paper, whether it be the instructor versus a teaching assistant, or even across different instructors for a course.

14
00:02:11.550 --> 00:02:37.219
BSPH CTL Teaching Toolkit: But to get this consistency can be a challenge for teaching teams. So one way that you can gain that consistency in grading is through grade norming. So what is grade norming grade norming is a or grade. Calibration is a process that brings a group together to decide how to assess. Students work consistently, so that, regardless of which person grades the work. The rating falls within a close range.

15
00:02:38.450 --> 00:02:43.819
BSPH CTL Teaching Toolkit: So why do we want to do this? What are the benefits? Well, let's start with the benefits for students.

16
00:02:43.950 --> 00:02:56.559
BSPH CTL Teaching Toolkit: 1st is fairness and equity. All students are assessed using the same standards, reducing the risk of bias and favoritism among different graders, and that's very important for both students and instructors.

17
00:02:56.750 --> 00:03:09.439
BSPH CTL Teaching Toolkit: Grade norming also creates consistency and feedback. So all students receive more uniform feedback, making it easier to understand the expectations for the assignments, and then also what they did well and what they can improve.

18
00:03:09.880 --> 00:03:25.429
BSPH CTL Teaching Toolkit: and probably, most importantly, is the increased trust in the assessment process. When students know that that grading is consistent and fair, they're more likely to trust the overall assessment process leading to greater satisfaction with their learning experience.

19
00:03:25.960 --> 00:03:37.390
BSPH CTL Teaching Toolkit: Speaking of trust, that is also a benefit for teaching teams, because all members of a teaching team are all graders, are accountable to the same grading standards and practices, so they have trust in the process

20
00:03:37.700 --> 00:03:52.509
BSPH CTL Teaching Toolkit: for teaching teams. This also grade norming also encourages collaborative learning. When teaching teams work together, they typically share best practices and resources which enrich their learning environment for them and their students.

21
00:03:53.100 --> 00:04:06.820
BSPH CTL Teaching Toolkit: and finally grade norming can save you time. It saves you time while you're grading, because everyone's been trained through the grading process, and then it also reduces the likelihood of time spent on grade disputes with students.

22
00:04:07.460 --> 00:04:18.529
BSPH CTL Teaching Toolkit: So with all these benefits in mind, let's talk about how this actually works. What is the grade norming process? And while grade norming doesn't have a 1. Size fits. All. This is a general basic process.

23
00:04:19.050 --> 00:04:33.390
BSPH CTL Teaching Toolkit: First, st of course, you have to select the assessment that you're grade norming. Usually these are your major assignments, but you can go all the way down to things like discussion, forum posts and daily check-ins. So start with. Usually you start with the biggest assignments and you work your way down.

24
00:04:34.076 --> 00:04:50.640
BSPH CTL Teaching Toolkit: Once the assignment is selected. The instructor is going to have need to create a rubric, or, at the very least, a set of standards that you want to be used by the teaching team when they're grading that assignment, and these are usually tied to the learning objectives for that assignment or competencies.

25
00:04:50.770 --> 00:04:55.810
BSPH CTL Teaching Toolkit: And we have a lot of resources on how to create a rubric that we'll share in the follow up email

26
00:04:56.810 --> 00:05:24.460
BSPH CTL Teaching Toolkit: after the rubric or standards are created, the instructor is going to model what grading looks like. So they'll either collect anonymized collections of student work, or you might use something like chatgpt to create student mock student work at different levels. And then the instructor is going to explain why that work receives the grade that it receives. So if it's a c paper, A B paper an a paper. They'll explain why, while applying the standards.

27
00:05:26.170 --> 00:05:39.800
BSPH CTL Teaching Toolkit: And then after that, the teaching team will actually practice grading individually. So they'll go through the standards. And at this point it's key that the teaching team writes down anything that needs clarified, or questions that they might have about how standards are applied.

28
00:05:40.080 --> 00:05:51.879
BSPH CTL Teaching Toolkit: And then finally, they'll come together to compare and calibrate their grading. So they'll compare the grades that they came up with for the anonymized student work, and then they'll identify

29
00:05:52.700 --> 00:06:03.469
BSPH CTL Teaching Toolkit: how they need to adjust their grading or calibrate their grading to make it more consistent until the everyone across the teaching team is grading pretty consistently across the range.

30
00:06:03.650 --> 00:06:14.169
BSPH CTL Teaching Toolkit: And again, that's the general process. But we'll look at. I'm excited to show the or see the 2 case studies that we have today to see to see how they applied the process for their courses.

31
00:06:14.597 --> 00:06:18.329
BSPH CTL Teaching Toolkit: But before we do that, are there any questions before we move on.

32
00:06:19.860 --> 00:06:22.759
BSPH CTL Teaching Toolkit: then I don't see anything in the chat. Okay.

33
00:06:23.680 --> 00:06:26.779
BSPH CTL Teaching Toolkit: all right. In that case. I'll pass it to Lauren.

34
00:06:26.780 --> 00:06:52.580
Lauren Dana: Great. Thank you, Amy. So, as Amy mentioned, we are now going to have some of our faculty and teaching teams present a case study on how they implemented grade norming within their own courses. So whether you are brand new to grade norming or have done it before, we're hoping you will think about how you might apply some of the strategies that our teams are going to discuss. So I'm going to 1st pass it over to our teaching team for public health policy. So Michelle and Annalise, thank you.

35
00:06:53.080 --> 00:07:07.980
Annelise Brochier (she/her): Thank you, Lauren, and thanks everyone for attending today's workshop. My name is Annalise, and I use she her pronouns. I served as a co-lead ta with my colleague, Michelle Badoia, for this past summer's public health policy course. And just for a bit of context.

36
00:07:07.980 --> 00:07:23.700
Annelise Brochier (she/her): the goal of this course was to build students skills with regard to identifying and understanding health policy decision makers, framing policy problems and policy questions and developing and evaluating policy options and advocating for change.

37
00:07:24.100 --> 00:07:31.129
Annelise Brochier (she/her): So this is a required course for incoming Mph students, and we had 326 this past summer.

38
00:07:31.200 --> 00:07:40.369
Annelise Brochier (she/her): The course is taught by Dr. Josh Sharfstein, and in addition to Michelle and me, we had 10 tas to help grade assignments and address student concerns throughout the term

39
00:07:41.190 --> 00:07:45.850
Annelise Brochier (she/her): slide. Please thank you. So why did we choose to implement grade norming?

40
00:07:45.890 --> 00:07:52.340
Annelise Brochier (she/her): Well, with such a large teaching team? It was important to us to have an activity where we could level set with all of the Tas.

41
00:07:52.370 --> 00:08:06.929
Annelise Brochier (she/her): with the exception of one multiple choice quiz. All of the assignments for this course were writing assignments, and therefore there was some degree of subjectivity or interpretation involved in applying the rubric criteria to the written assignments.

42
00:08:07.120 --> 00:08:22.569
Annelise Brochier (she/her): About two-thirds of our tas, myself included, had also never even served as tas before. So we really wanted an activity like the norming exercise, to help build our confidence and give us an opportunity to practice before we were let loose to grade student assignments.

43
00:08:23.200 --> 00:08:44.070
Annelise Brochier (she/her): And then, lastly, and perhaps most importantly, we really wanted to respond to feedback that we had heard through prior year's course evaluations. A common theme was that students had concerns about inconsistent grading across different ta sections, and Ctl. Had suggested that in order to address this concern. We try norming for this summer's course to get ahead of this issue.

44
00:08:46.670 --> 00:09:14.200
Annelise Brochier (she/her): We completed the norming exercise for the 1st written assignment of the course which is called assessing a health policymaker. I understand this slide is very small, but it pictures the rubric for the assignment as well as the assignment instructions. So for this assignment. Students were asked to identify a policymaker, summarize that policymaker's background and describe their public health areas of interest, summarize their work in that public health area and describe a specific policy they sought to address.

45
00:09:15.290 --> 00:09:16.579
Annelise Brochier (she/her): And for

46
00:09:16.890 --> 00:09:21.989
Annelise Brochier (she/her): all of these assignment criteria they were baked into this rubric that you see pictured on the screen

47
00:09:22.070 --> 00:09:22.825
Annelise Brochier (she/her): and

48
00:09:24.030 --> 00:09:32.579
Annelise Brochier (she/her): the rubric and the assignment instructions pictured here were available to students before they submitted the assignment, and then, of course, available to tas as they were grading

49
00:09:34.610 --> 00:09:55.849
Annelise Brochier (she/her): before Michelle gets into the details. I'll just provide a quick overview of our norming process. So in order to prepare our teaching team for the exercise. We actually introduced it at our very 1st ta meeting, so long before the 1st assignment was submitted, we thought that this early introduction would build our team's confidence in our ability to achieve uniformity and grading of that 1st assignment.

50
00:09:56.440 --> 00:10:18.410
Annelise Brochier (she/her): Then, as the assignment deadline was approaching, we converted the rubric into a Google form. And we found that this electronic data capture really enabled Michelle to run some cool analytics that she'll get into in a little bit that summarized our team's grading decisions and helped us identify areas where we could work on calibrating and and reaching consensus in our approach.

51
00:10:19.050 --> 00:10:41.270
Annelise Brochier (she/her): After each of the Tas graded 2 student submissions using this Google form, we, you know, we all graded the same 2 student submissions. We then disseminated the results of Michelle's analysis along with some key takeaways over email. So I just want to highlight that this activity was entirely asynchronous in order to accommodate the scheduling constraints of our team.

52
00:10:44.590 --> 00:11:01.770
Michelle Bedoya: All right. Thank you, Annalise. I'll go into the nuts and bolts on how the norming exercise worked and what we did, step by step. So first, st after students submitted their assignment, and before Tas began grading, we began by randomly selecting 2 assignments from the pool of submissions.

53
00:11:01.820 --> 00:11:23.810
Michelle Bedoya: and then, as Annalise mentioned, we created a Google form. Well, we had created the Google form. But we used the Google form with 4 6 point categories and 2 3 point categories adapted from the rubric, and this form really provided a mechanism for us to complete the norming exercise asynchronously due to time constraints and conflicting schedules as Annalise mentioned.

54
00:11:24.160 --> 00:11:31.760
Michelle Bedoya: So each ta was asked via email to use the Google form to grade each assignment by selecting a score for each category.

55
00:11:31.770 --> 00:11:52.379
Michelle Bedoya: And here the higher score, the higher the score, the more the points for the students. And so we gave Tas 48 h to complete the norming exercise, and once the 48 h had passed, we actually received 11 responses from Tas, although we had a total of 12 tas. So we had about a 92% response rate, which is great

56
00:11:52.380 --> 00:12:11.180
Michelle Bedoya: and we would have had 100%. But one of the Tas was in the process of defending her Phd. Proposal in that 48 h window. So it happens, but overall with 6 domains, 11 Tas. And 2 assignments, we had a total of 1, 32 observations for the norming exercise.

57
00:12:14.330 --> 00:12:41.729
Michelle Bedoya: So for data collection, we used Google forms. And what's great about Google forms is that it allows for 2 options to to view the Google form responses, and these are really great. To provide this comprehensive view of the grading data, and the 1st option allows for viewing color coded bar graphs for scoring in each domain or category. And this option was helpful for visualizing the variability in scores across different categories of the rubric.

58
00:12:41.750 --> 00:13:10.430
Michelle Bedoya: And these bar graphs really allowed us to see patterns such as which categories may have had higher or lower scores, and how the scores differ across them. So really all to say that we were able to quickly determine which categories were of potential concern with regard to consistency and uniformity and grading, and really allowed Annalise and I to start thinking about why there was that variability and to start asking questions about why this variability was occurring

59
00:13:12.130 --> 00:13:13.070
Michelle Bedoya: next slide.

60
00:13:13.680 --> 00:13:21.260
Michelle Bedoya: The second option to view the grading data in Google forms. Was this like really cool button which is circled in red on the slide?

61
00:13:21.320 --> 00:13:35.000
Michelle Bedoya: And this button produces an excel spreadsheet of all the Google form responses. And in our case the rows represented the tas that responded, and the columns represented each category across the 2 example assignments.

62
00:13:35.180 --> 00:13:52.810
Michelle Bedoya: and this really allows for more detailed analysis because it gave us the information we needed to create some descriptive statistics like the mean or the Median score for each domain. And these, in creating these statistics, we can, we can try to understand the central tendency and the spread of scores.

63
00:13:52.870 --> 00:14:19.849
Michelle Bedoya: and also, more importantly, since the rows in the spreadsheet detailed the tas associated with the scoring, we're able to analyze individual trends and variations among Tas and identify possible tendencies of certain tas to grade more strictly or leniently, and this provided an avenue for Annalise to work with specific tas and provide pointed feedback. If there were tas that had outlier grading practices

64
00:14:23.400 --> 00:14:37.960
Michelle Bedoya: so overall our norming exercise can be described in 2 steps. And that's data collection and analysis. And ultimately the goal was to identify key insights, to disseminate to the Ta team, to keep in mind as they graded their assignments.

65
00:14:38.130 --> 00:14:45.420
Michelle Bedoya: And 1st we looked at the raw scores from Tas and visualize the data using the bar graphs to explore those trends.

66
00:14:45.450 --> 00:14:51.480
Michelle Bedoya: And this kind of exploratory analysis showed us that there was a need for further calibration.

67
00:14:51.840 --> 00:14:55.440
Michelle Bedoya: And then, through this exploratory analysis, we extrapolated

68
00:14:55.940 --> 00:15:05.889
Michelle Bedoya: some insights on the notable differences among tas, and ultimately we found that there were some varied interpretations or expectations on the grading criteria.

69
00:15:06.765 --> 00:15:15.270
Michelle Bedoya: But I really wanna point out that the data we collected didn't really tell us why the differences occurred. So

70
00:15:15.440 --> 00:15:39.619
Michelle Bedoya: our process did involve some qualitative assessment, and Annalise and I dedicated time to talk with one another at our individual meetings. To speak with Dr. Sharpstein, the instructor for the course, to engage with students and ask questions, and feed and ask and solicit feedback on the assignment, and then also meeting with Tas individually and collectively at our weekly meetings to understand

71
00:15:39.620 --> 00:16:04.749
Michelle Bedoya: what specifically about the domains may have resulted in a lack of uniformity. So our process was much more than just the data collection that we pointed out through the Google form responses. So once we had this good handle on, why these differences in grading might have occurred. We sent an email to the teaching team with this feedback and with very specific recommendations, all with the aim to standardize the grading process.

72
00:16:08.230 --> 00:16:31.340
Michelle Bedoya: So here is a screenshot of the email we sent to the teaching team. I know it looks very small. So I'm happy to send a copy of this email to anyone here. But I include it just to provide an outline of what we sent. So it it tracks our data collection and our analysis process. We provided the mean score across categories for each example, and then we provide recommendations on how to approach each category

73
00:16:31.340 --> 00:16:43.950
Michelle Bedoya: along with some specific tips on why categories may have certain variability in grading, and what the tas can do to ameliorate concerns on consistency.

74
00:16:47.120 --> 00:17:02.769
Michelle Bedoya: So Annalise and I have many, many takeaways from this exercise, but we'll discuss a few. The 1st is that the exercise really enhanced teaching team confidence, which was our goal. So it was a very positive outcome, because it clarified expectations

75
00:17:02.770 --> 00:17:20.849
Michelle Bedoya: and and helped identify grading tendencies for Tas to have this self awareness on how to approach, not just this assignment, but assignments moving forward, and for many tas. As Annalise mentioned, this is their 1st time grading an assignment ever in their capacity as tas, so it helped them practice and get feedback.

76
00:17:21.190 --> 00:17:22.999
Michelle Bedoya: and then the tas

77
00:17:23.170 --> 00:17:38.907
Michelle Bedoya: were also made aware of which domains might require extra attention while grading, and so knowing in advance where the inconsistencies were were like allowed the tas to be particularly vigilant in those areas

78
00:17:39.430 --> 00:17:44.680
Michelle Bedoya: which ultimately supported a fairer or more balanced evaluation process. Moving forward

79
00:17:45.334 --> 00:18:12.769
Michelle Bedoya: another key takeaway was that the norming exercise were largely influenced by contextual factors. One being the level of ta engagement, which was crucial for successful norming, and that engagement was something that we tried to inculcate from the very beginning. As Annalise mentioned, we discussed the norming exercise at our 1st meeting, just to get the Tas ready to understand that this was very important, and that we were going to be dedicating time just to increase that engagement.

80
00:18:12.880 --> 00:18:27.390
Michelle Bedoya: Another contextual factor relates to the assignment itself. So, in speaking with students, we noted that there are varying interpretations of the instructions which may have influenced the variability in categories, and so

81
00:18:27.570 --> 00:18:37.339
Michelle Bedoya: it highlighted the need for clearer instructions and additional guidelit guidance, and like, it really shed light on how the variability may not be a result of grading

82
00:18:37.650 --> 00:18:41.160
Michelle Bedoya: inconsistency, but maybe on vague instructions.

83
00:18:41.300 --> 00:18:56.500
Michelle Bedoya: and so on. That end. The the exercise provided insights for improvement and improvements on how the rubric was drafted, and how the assignment instructor instructions could be refined or be made more clear.

84
00:18:59.090 --> 00:19:15.700
Annelise Brochier (she/her): So overall. We really feel that norming is not a 1 time activity, but an ongoing process. Ideally, of course, we would conduct a formal norming session for each assignment. But it just wasn't feasible giving our teaching teams competing demands, not to mention the challenge of coordinating such a large group.

85
00:19:15.740 --> 00:19:29.439
Annelise Brochier (she/her): but still starting the term with a norming activity proved to be very effective. It allowed us to establish a common standard for grading expectations, and really align on what constitutes overly harsh or perhaps overly lenient grading.

86
00:19:29.730 --> 00:19:47.590
Annelise Brochier (she/her): Even though we didn't complete formal norming activities for subsequent assignments, we did implement some additional measures to ensure grading consistency across the teaching team throughout the course. So we created annotated rubrics that more clearly defined the criteria. For, say, a 1 point versus 2 point deduction.

87
00:19:48.020 --> 00:20:03.729
Annelise Brochier (she/her): We also established a group chat with where Tas were encouraged to post questions like, Would you give this a 4 or 3 if they were really uncertain what to give it, and so that enabled us, as a group to reach consensus and then apply consistent grading standards moving forward.

88
00:20:04.270 --> 00:20:07.860
Annelise Brochier (she/her): So with that, I'll hand it back to Amy and Lauren. Thank you.

89
00:20:08.510 --> 00:20:22.209
Lauren Dana: Great. Thank you so much, Annalise and Michelle, and I'm sure many of you have questions that you want to ask the teaching team, so we will have Q. And A after our next case study. But I would really like to introduce Ann Lilly for fundamentals of program evaluation.

90
00:20:24.270 --> 00:20:35.430
Anne Lilly: Hi, thank you very much. Great to be here again. My name's Ann Lilly. I'm a senior research associate in population, family, and reproductive health in school, public health.

91
00:20:36.034 --> 00:20:41.730
Anne Lilly: And I co-teach fundamentals of program evaluation. In 1st term.

92
00:20:43.350 --> 00:20:51.539
Anne Lilly: So I wanted to give some context for our course, and then I wanted to talk through some of the great norming practices that we have adopted.

93
00:20:52.900 --> 00:21:04.838
Anne Lilly: before doing that I wanted to say a couple of things. First, st I I do not consider myself an expert. I think this is something I am still learning as a co-instructor for a course.

94
00:21:05.510 --> 00:21:09.606
Anne Lilly: we do our best in this area. But I feel like we are continually learning

95
00:21:10.120 --> 00:21:25.100
Anne Lilly: and I also just want to elevate what Annalise said earlier, like, this is an ongoing process. It is not like. Set it and forget it. It is something that you really need to kind of work on. Within the course.

96
00:21:25.120 --> 00:21:27.830
Anne Lilly: But then also, even year to year.

97
00:21:28.240 --> 00:21:35.399
Anne Lilly: you can always identify improvements, for you know the next offering of the course. Whether that be

98
00:21:36.345 --> 00:21:38.010
Anne Lilly: changing your rubric.

99
00:21:38.030 --> 00:21:45.309
Anne Lilly: modifying the instructions like you do notice patterns. As you go through these kinds of processes. So

100
00:21:45.590 --> 00:21:47.859
Anne Lilly: I just want to affirm those things.

101
00:21:48.877 --> 00:21:55.299
Anne Lilly: So just to explain this course. This is the online version of an in person. Course.

102
00:21:55.470 --> 00:21:57.440
Anne Lilly: it's offered 1st term.

103
00:21:57.856 --> 00:22:03.880
Anne Lilly: Most of the course, is asynchronous, and each year we have 80 to a hundred students.

104
00:22:04.337 --> 00:22:13.259
Anne Lilly: And we, we tend to have somewhere between 3 and 4 tas. So this is kind of a smaller scale than what Michelle and Annalise were discussing.

105
00:22:13.985 --> 00:22:17.449
Anne Lilly: And across the course we have 4 written assignments.

106
00:22:19.610 --> 00:22:22.620
Anne Lilly: yeah. 4 written assignments lot to grade.

107
00:22:23.044 --> 00:22:32.020
Anne Lilly: So on the right hand of the slide, I really just provide the sort of the different grade norming practices that we have adopted.

108
00:22:33.540 --> 00:22:42.130
Anne Lilly: I will describe these verbally, and then, on subsequent slides, I have some like some visuals. To give you a sense of what I'm really talking about

109
00:22:42.830 --> 00:22:45.769
Anne Lilly: the 1st thing I listed here is ta selection.

110
00:22:46.540 --> 00:22:53.679
Anne Lilly: So we this year. We actually implemented a performance task when selecting tas

111
00:22:54.279 --> 00:23:02.539
Anne Lilly: just really to make sure that folks applying to be a ta had a basic grasp of the core content of our course.

112
00:23:03.150 --> 00:23:07.290
Anne Lilly: So we essentially provided them with

113
00:23:07.490 --> 00:23:15.270
Anne Lilly: kind of an example of prior student work, of course anonymized and asked them to quickly grade

114
00:23:15.300 --> 00:23:18.330
Anne Lilly: part of that assignment from a prior year.

115
00:23:18.740 --> 00:23:22.109
Anne Lilly: To really just see kind of like what they picked up on

116
00:23:22.480 --> 00:23:26.559
Anne Lilly: and that really kind of informed our selection of tas for this year.

117
00:23:27.603 --> 00:23:42.500
Anne Lilly: I think Annalise mentioned. Kind of having annotated rubrics. But our grading guides are just that. It's basically the rubric for each assignment with detailed notes.

118
00:23:43.300 --> 00:23:44.200
Anne Lilly: As to

119
00:23:44.220 --> 00:23:50.030
Anne Lilly: what type of response would warrant? What kind of score in different areas of the rubric?

120
00:23:52.250 --> 00:24:07.230
Anne Lilly: and the 3rd thing I have listed here is our teaching team rubric so comparable to the process that Michelle and Annalise were describing. Before we actually move into grading each assignment.

121
00:24:07.914 --> 00:24:13.349
Anne Lilly: We have one example, assignment graded by the whole teaching team.

122
00:24:13.440 --> 00:24:17.430
Anne Lilly: People grade independently, they fill out their own rubric.

123
00:24:17.550 --> 00:24:24.429
Anne Lilly: And then people are asked to put their scores into the shared teaching team rubric.

124
00:24:25.254 --> 00:24:28.989
Anne Lilly: And that's really the tool we use to discuss

125
00:24:29.000 --> 00:24:33.819
Anne Lilly: grading. That example assignment. And I have a visual of that

126
00:24:36.380 --> 00:24:47.680
Anne Lilly: The 4th tool that I describe here is our greeting, reliability worksheet. This lives in onedrive. Everyone has access to it. And we basically ask that

127
00:24:48.060 --> 00:24:51.350
Anne Lilly: all of our tas as they are grading.

128
00:24:51.510 --> 00:24:56.190
Anne Lilly: Put the scores for the rubric into that grading reliability worksheet.

129
00:24:56.960 --> 00:24:58.760
Anne Lilly: This is just a

130
00:24:59.320 --> 00:25:10.399
Anne Lilly: it's a very transparent way for each ta to to know themselves if they seem to be grading more leniently or more harshly than their peers.

131
00:25:10.710 --> 00:25:15.529
Anne Lilly: and because of the way it's laid out, they can visually just pinpoint.

132
00:25:15.959 --> 00:25:21.070
Anne Lilly: You know. Is it the 1st part of the assignment? Is it the second part like, where are they?

133
00:25:21.120 --> 00:25:24.479
Anne Lilly: Very different than their peer graders.

134
00:25:26.700 --> 00:25:36.229
Anne Lilly: very similar to what Annalise was describing. We also have. We just have, like a culture in our teaching team that like, when you have a question, you should pose it to everyone.

135
00:25:36.731 --> 00:25:40.168
Anne Lilly: So we have a Whatsapp group that we use.

136
00:25:41.020 --> 00:25:58.880
Anne Lilly: people. If it's easier, people also will just email, the entire teaching team. But really, just this idea that we need to have open communication with one another, and that if questions, if a question arises for one person, it's likely someone else is wondering this, too. So let us just share questions with everyone.

137
00:25:58.890 --> 00:26:00.789
Anne Lilly: Everyone can see the same answer.

138
00:26:00.930 --> 00:26:02.330
Anne Lilly: We're all on the same page.

139
00:26:03.375 --> 00:26:13.400
Anne Lilly: And then I will say, my co-instructor and I. We review rubrics before we deem them final. So we have a grade norming process.

140
00:26:13.480 --> 00:26:18.829
Anne Lilly: We also do a final check of rubrics before they are released to students.

141
00:26:19.626 --> 00:26:23.880
Anne Lilly: So many, many steps. You go to the next slide.

142
00:26:26.780 --> 00:26:36.270
Anne Lilly: So this is just a visual. And again, I know this is not really legible. But this is a visual, a visual of our grading guide. So

143
00:26:36.510 --> 00:26:40.039
Anne Lilly: it basically takes the rubric and then on the very right.

144
00:26:40.050 --> 00:26:44.112
Anne Lilly: it gives explanation of what we're looking for.

145
00:26:44.810 --> 00:26:48.309
Anne Lilly: It's very funny because Michelle is in this class right now.

146
00:26:48.822 --> 00:26:52.950
Anne Lilly: So, Michelle, you're under the hood. You're under the hood.

147
00:26:53.860 --> 00:26:55.420
Anne Lilly: This is how we grade

148
00:26:55.823 --> 00:27:09.889
Anne Lilly: but basically, we give examples of like, what would be an acceptable response for each part of the rubric, and then we give tips of like. If you see this, give 5 points. If you see this give 4 blah blah.

149
00:27:10.410 --> 00:27:14.729
Anne Lilly: So we try to give as much explicit instruction as possible. Because,

150
00:27:15.170 --> 00:27:22.199
Anne Lilly: you know, people kind of come to being tas from different backgrounds, different levels of skill. It could be their 1st time being a ta so

151
00:27:22.480 --> 00:27:25.530
Anne Lilly: trying to help with the whole process.

152
00:27:25.810 --> 00:27:27.369
Anne Lilly: Can you go to the next slide

153
00:27:29.460 --> 00:27:45.380
Anne Lilly: cool. So this is the teaching team rubric that I was talking about. So again, you know, we as a teaching team will all grade one assignment to kind of work toward reliability. People do that individually. And then they're asked to enter their scores in this team rubric.

154
00:27:46.000 --> 00:27:53.819
Anne Lilly: So basically, we come to the meeting with and basically you'll see on the right hand side. This is again a rubric.

155
00:27:54.330 --> 00:28:09.120
Anne Lilly: but on the right hand side you'll see Alds, km, this is our teaching team. So everyone enters their scores. And then, as we are discussing, we come to agreement on that very last column agreed upon score.

156
00:28:09.380 --> 00:28:11.290
Anne Lilly: So we have open discussion.

157
00:28:11.940 --> 00:28:20.790
Anne Lilly: and you know, oftentimes it's the case, like in the 1st row here everyone agreed that that should be 5 points. So we're like great. It's 5 points. We don't need to talk about this.

158
00:28:20.880 --> 00:28:24.940
Anne Lilly: But this allows us to just know exactly where everyone stands

159
00:28:25.398 --> 00:28:27.150
Anne Lilly: and to discuss easily

160
00:28:27.980 --> 00:28:29.120
Anne Lilly: next slide.

161
00:28:32.100 --> 00:28:42.989
Anne Lilly: Okay? And then this is the grading reliability worksheet which this may look kind of overwhelming. But I think this has been a helpful tool. So basically, just to explain.

162
00:28:43.482 --> 00:28:53.190
Anne Lilly: our course uses different program examples for the assignments. And each program example is graded by one ta. So, for example, like

163
00:28:53.210 --> 00:28:58.980
Anne Lilly: all of these scores in purple are one ta blue is another. Ta

164
00:28:59.060 --> 00:29:00.999
Anne Lilly: yellow is our 3rd ta

165
00:29:02.500 --> 00:29:05.189
Anne Lilly: But what this basically does is for

166
00:29:05.810 --> 00:29:07.849
Anne Lilly: this is for assignment one

167
00:29:08.302 --> 00:29:10.880
Anne Lilly: this breaks out every part of the rubric.

168
00:29:11.070 --> 00:29:15.150
Anne Lilly: So people as they're grading, they can put their scores in here, and they can see

169
00:29:15.330 --> 00:29:18.532
Anne Lilly: how they are doing. I will say,

170
00:29:19.120 --> 00:29:24.861
Anne Lilly: This is when all is said and done. This is when we've already done our review. This is

171
00:29:25.640 --> 00:29:32.369
Anne Lilly: cause, I mean, you'll see their average scores are actually pretty pretty comparable. Especially blue and yellow.

172
00:29:33.081 --> 00:29:36.209
Anne Lilly: Purple is maybe a bit more harsh. But

173
00:29:36.940 --> 00:29:39.830
Anne Lilly: this just allows people as they're grading to know kind of

174
00:29:39.960 --> 00:29:41.390
Anne Lilly: how their greeting fares

175
00:29:42.120 --> 00:29:43.340
Anne Lilly: next slide.

176
00:29:44.830 --> 00:29:47.759
Anne Lilly: Okay? So then just some key takeaways.

177
00:29:49.100 --> 00:30:03.792
Anne Lilly: I mean, I think Michelle and and Elise touched on this as well. But this takes a lot of work. Frankly. I mean, I just want to put it out there. I think it is. Well, it's well deserved, like it's it's important work. We do want things to be fair.

178
00:30:04.370 --> 00:30:10.509
Anne Lilly: that's valued. But it does take time, and it takes effort to kind of think through what will make the most sense.

179
00:30:11.358 --> 00:30:24.940
Anne Lilly: I also would really advocate employing multiple strategies. You know, we do grade norming as a group. But then we have the grading reliability worksheet. Then we also review the rubrics before they're final.

180
00:30:25.354 --> 00:30:28.720
Anne Lilly: I think having those like stop gaps is good.

181
00:30:29.500 --> 00:30:50.220
Anne Lilly: We use onedrive a lot, and I know, like a lot of people use onedrive. And maybe this is like a patently obvious comment. But it's just. It can be really powerful because people can be in the same document, at the same time making edits, and you don't have any conflicted copies. It's really easy to share, really easy to share information. And to share documents.

182
00:30:52.440 --> 00:30:56.520
Anne Lilly: and then just emphasizing the importance of open communication.

183
00:30:58.590 --> 00:31:05.270
Anne Lilly: you know, having this way of the tas, knowing how their grade, how their grading compares to the other tas

184
00:31:06.011 --> 00:31:10.628
Anne Lilly: and also having a strategy for getting questions addressed quickly.

185
00:31:11.310 --> 00:31:15.330
Anne Lilly: you know. So people are not kind of left wondering like, how am I going to integrate this.

186
00:31:15.640 --> 00:31:18.420
Anne Lilly: So, having that open communication

187
00:31:18.450 --> 00:31:19.700
Anne Lilly: is very helpful.

188
00:31:20.651 --> 00:31:24.830
Anne Lilly: And I think that's it for me. I will pass it back to Lauren and Amy.

189
00:31:26.260 --> 00:31:38.510
Lauren Dana: Great. Thank you so much, Anne. So now we are going to pause for questions. If anyone has questions for our guest speakers, I know they gave us a lot of really important and vital information.

190
00:31:38.670 --> 00:31:50.650
Lauren Dana: And why everyone thinks about that. I'm going to go back to our chats. We did have a chat question from Celine. Can you estimate how long the entire iteration of grade norming takes? So not just the 48 h that you gave

191
00:31:50.660 --> 00:31:56.450
Lauren Dana: the Tas to grade as individuals, but to do the actual norming. So I don't know if both of our

192
00:31:56.550 --> 00:32:00.429
Lauren Dana: courses want to jump in here. If you do give a general.

193
00:32:01.470 --> 00:32:09.380
Annelise Brochier (she/her): Yeah, I can speak to this for our public health policy course. So for the Tas who were

194
00:32:09.830 --> 00:32:14.860
Annelise Brochier (she/her): participating in the activity it, I imagine. It just depends on the intensity of

195
00:32:15.010 --> 00:32:16.520
Annelise Brochier (she/her): whatever assignment

196
00:32:16.560 --> 00:32:41.679
Annelise Brochier (she/her): you're using for your grade norming. So for us, it takes about 15 to 20 min to grade and assessing a policymaker assignment. So each ta probably took 20 to 25 min to grade each of them just because it was the 1st one they had seen. So that's a little under an hour of work on the part of each ta. And then we devoted maybe 20 min of our next ta meeting to discussing the results.

197
00:32:42.297 --> 00:32:47.872
Annelise Brochier (she/her): And then Michelle and I were talking in the chat, and we would guess that we spent

198
00:32:48.550 --> 00:32:52.699
Annelise Brochier (she/her): combined a few hours preparing

199
00:32:52.940 --> 00:33:09.979
Annelise Brochier (she/her): the the key takeaways email doing the analysis of the Google form, creating the Google form all of those more behind the scenes aspects. So lead Tas took us a few hours, the rest of the teaching team maybe an hour hour, 15.

200
00:33:11.010 --> 00:33:11.900
Annelise Brochier (she/her): Celine.

201
00:33:12.330 --> 00:33:17.577
Celine Greene: Can I ask a follow up to that? And that is, I think you kind of answered yes to this

202
00:33:18.120 --> 00:33:24.120
Celine Greene: Was it affected by the type of assignment and the number of of individuals. Because.

203
00:33:24.310 --> 00:33:25.560
Celine Greene: I mean.

204
00:33:25.590 --> 00:33:30.810
Celine Greene: I'm thinking, if you you said you had 10 others besides the 2 of you right?

205
00:33:30.860 --> 00:33:38.340
Celine Greene: 12 people. It's kind of an easy thing to come up with an average. But if it was only 3 or 4 people, there might be strong arguments

206
00:33:38.660 --> 00:33:44.359
Celine Greene: for norming, and you know, for the norming, it's not such an easy average like one person's

207
00:33:45.280 --> 00:33:50.630
Celine Greene: explanation of why they ranked something so low a criteria so low

208
00:33:50.670 --> 00:33:52.250
Celine Greene: and performance

209
00:33:53.160 --> 00:34:06.410
Celine Greene: it sounds, you know, harder to do so. I don't know, maybe, and can also speak to this. But so it definitely sounds like a yes, it takes longer for the type of assignment. But what about for the number of participants in the norming activity? Does

210
00:34:06.440 --> 00:34:11.249
Celine Greene: 12 people versus 3 or 5 people. Does that make a difference?

211
00:34:11.500 --> 00:34:12.370
Celine Greene: Thank you.

212
00:34:16.330 --> 00:34:23.390
Annelise Brochier (she/her): So. And my only experience is with this large teaching team. So I wonder if if you've used norming for

213
00:34:24.498 --> 00:34:26.650
Annelise Brochier (she/her): teams of different sizes.

214
00:34:27.989 --> 00:34:28.779
Anne Lilly: I mean.

215
00:34:29.109 --> 00:34:30.369
Anne Lilly: I think

216
00:34:31.369 --> 00:34:37.299
Anne Lilly: certainly. Well, so it's funny. Michelle and Annalise, did you guys did you meet to discuss

217
00:34:37.459 --> 00:34:46.149
Anne Lilly: like the dispersion on any given part of your rubric, or it was like you analyzed. You sent information to people by email. But did you have, like a group discussion.

218
00:34:47.130 --> 00:35:03.060
Annelise Brochier (she/her): So Michelle and I had a discussion initially sent the information by email. And then in our, we had weekly Ta meetings. So at our Ta meeting. Following the norming exercise, we discussed what we had disseminated over email.

219
00:35:04.190 --> 00:35:04.910
Anne Lilly: Okay.

220
00:35:06.337 --> 00:35:13.542
Anne Lilly: Cause, Celine, I take your point. I mean, it really kind of depends on how complicated the assignment was, and like how long. The assignment is

221
00:35:14.090 --> 00:35:16.049
Anne Lilly: but I do think by

222
00:35:16.300 --> 00:35:20.779
Anne Lilly: by having everyone grade in advance and like share their scores in advance.

223
00:35:21.431 --> 00:35:27.750
Anne Lilly: You can just kind of like get to business in the discussion and kind of walk through stuff.

224
00:35:30.420 --> 00:35:34.080
Anne Lilly: I would also say, maybe we're too like hierarchical. But

225
00:35:34.580 --> 00:35:36.657
Anne Lilly: when there's disagreement

226
00:35:37.760 --> 00:35:42.706
Anne Lilly: The decision kind of then falls to my co-instructor. And I, you know, like

227
00:35:43.180 --> 00:35:52.749
Anne Lilly: not that we don't want to be democratic, but there are just times when the tas are relatively new to this material. They're new to this process, and we will just say, like.

228
00:35:52.970 --> 00:35:54.039
Anne Lilly: it's a 10,

229
00:35:54.250 --> 00:36:06.480
Anne Lilly: or you know, whatever the case may be, so we don't. We certainly want to hear people out. But at the same time, it is like, okay, let's keep it moving like. Let's move to the next part of the rubric. So

230
00:36:06.880 --> 00:36:09.474
Anne Lilly: I don't know if that addresses your question, though, Celine. But

231
00:36:10.430 --> 00:36:20.419
Celine Greene: It? It does. It's I mean, actually, what you're saying about sometimes having it. Not so normative. I mean, Norming is important, and it brings discussion that's important.

232
00:36:20.490 --> 00:36:23.380
Celine Greene: But understanding that sometimes there are going to be

233
00:36:23.910 --> 00:36:28.700
Celine Greene: their situations and assignments where Norming is influenced

234
00:36:28.710 --> 00:36:38.189
Celine Greene: by the more knowledgeable slash, hierarchical aspects of the faculty team. So thank you. Thank you. Folks.

235
00:36:39.090 --> 00:37:01.669
Michelle Bedoya: I I do want to quickly point out that in addition to the time that at least discuss, I mean, there are like, perhaps unaccounted for conversations that we've had with Dr. Sharfstein, or students that did inform that process. So while, like yes, we devoted certain time to the actual exercise, there are conversations after class or during office hours. Where we

236
00:37:01.670 --> 00:37:12.200
Michelle Bedoya: collected information that ultimately informed that final email and those tips that we gave. So it's it's hard to put a timeframe around it, but just keep that in mind.

237
00:37:14.430 --> 00:37:25.440
Lauren Dana: Great. Thank you to our teams. We have another question in the chat. How quickly do you release grades on assignments for students? Are you able to norm with your teaching teams and release feedback weekly. That's a great question from Teresa.

238
00:37:29.530 --> 00:37:33.430
Anne Lilly: I mean, I can jump in, and then, you know, I welcome Michelle and Annalise. But

239
00:37:33.910 --> 00:37:40.129
Anne Lilly: This is a really important question, because in our course our assignments build off of one another. So

240
00:37:40.200 --> 00:37:47.519
Anne Lilly: when you're working on assignment 2, you actually really need to know how you did an assignment one and parts of assignment. 2 are actually

241
00:37:47.700 --> 00:37:50.259
Anne Lilly: almost like mirrored from the 1st assignment.

242
00:37:50.718 --> 00:37:55.449
Anne Lilly: So we do want people to be able to react to the feedback. So we have.

243
00:37:55.950 --> 00:38:08.709
Anne Lilly: And I don't know. I think our team is like this because the expectations are just really clear. But we have a very tight grading timeline. We go through our norming process in our teaching team meeting.

244
00:38:08.930 --> 00:38:13.159
Anne Lilly: And then we basically, that's Monday afternoon.

245
00:38:13.370 --> 00:38:19.060
Anne Lilly: And then we ask the tas to be done with their grading by Wednesday midnight.

246
00:38:19.180 --> 00:38:21.070
Anne Lilly: so they have, like 2

247
00:38:21.150 --> 00:38:23.929
Anne Lilly: 2 days and an evening to grade.

248
00:38:24.080 --> 00:38:30.130
Anne Lilly: and then ideally. Thursday, my co-instructor and I do our review of the rubrics.

249
00:38:30.650 --> 00:38:38.519
Anne Lilly: Any reconciliation that needs to happen happens on Friday and Friday end of day. We try to have everything out. It is like

250
00:38:38.660 --> 00:38:42.209
Anne Lilly: it is a little bit of a sprint, but with the 8 week term

251
00:38:42.420 --> 00:38:45.219
Anne Lilly: it, I don't know. It's a challenge to.

252
00:38:46.690 --> 00:38:52.640
Teresa Pfaff: Yeah, I'm from. I'm from school of nursing. We're at 6 weeks. So we're in a very, a very quick.

253
00:38:52.780 --> 00:38:59.679
Teresa Pfaff: quick sprint. And what happens with us is that with our our tas we have, we do 6 weeks, and then play and repeat

254
00:38:59.760 --> 00:39:14.070
Teresa Pfaff: so kind of making sure, not only from that one section that we're norming, but also to the second section. So and with the the high volume about we had 88 in one section and I think another

255
00:39:14.440 --> 00:39:20.605
Teresa Pfaff: 76. So it's been. It's been a lot. But this is extremely helpful. So thank you. And tightening down

256
00:39:21.080 --> 00:39:24.969
Teresa Pfaff: that grading process, timeline, I think is really great feedback. So I appreciate that.

257
00:39:27.670 --> 00:39:43.690
Michelle Bedoya: I just wanna echo the 8 week terms is a challenge. And for this term this past term we only did one norming exercise, and I don't know with the process that we had, that we would have done it. We would have been able to do it for every assignment, just because we wanted to have grades with back within a week.

258
00:39:43.967 --> 00:39:54.849
Michelle Bedoya: And so this is something for Annalise. And I to think about is like, how do we streamline the norming process so that it's not as work intensive. But I don't know that it would have been feasible with the process that we described.

259
00:39:58.700 --> 00:40:02.620
Anne Lilly: Alright, Celine, I see your hand. But, Teresa, just one other thing to say.

260
00:40:02.930 --> 00:40:04.414
Anne Lilly: cause you mentioned

261
00:40:05.430 --> 00:40:17.220
Anne Lilly: kind of like validation across groups, too, which is like a whole other hidden caboodle. But I did just want to say what we've also done in the past. Which we didn't do so much this year, but

262
00:40:17.340 --> 00:40:23.270
Anne Lilly: to kind of like prime the Tas. We've also taken graded assignments from last year.

263
00:40:23.610 --> 00:40:31.530
Anne Lilly: anonymize them and shared them with the tas, as like Hi, like, these are some examples of how you might go about grading

264
00:40:32.138 --> 00:40:42.330
Anne Lilly: cause. It just gives more context and depending on how people's brains work like, you know, seeing more and more examples can help people be primed to do the work. So

265
00:40:42.430 --> 00:40:47.099
Anne Lilly: that's also a potential strategy, especially if you have, like, a lot of volume of assignments.

266
00:40:47.775 --> 00:40:53.089
Anne Lilly: And if you can like, review quickly and say, like, Yeah, I agree with how these were graded.

267
00:40:53.640 --> 00:40:57.910
Anne Lilly: you can share them as like, hey, this is like a guide.

268
00:40:58.334 --> 00:41:00.159
Anne Lilly: To how you could grade so.

269
00:41:01.300 --> 00:41:26.959
Teresa Pfaff: Thank you. Yeah. And that's something that we've done. Our onedrive is very robust, which I I really enjoy. And that's what that's something that we've done. We haven't taken it, I will say, for like to the next level. So this is so helpful. But like that gritty and reliability worksheet. I'm already thinking. Now we're in. We're into week 2. And I'm like, I can quickly like, get this together, for I'm like, Okay, I mean, it. It looks great, but thinking that everybody can visualize and see, because over on our side of the world

270
00:41:27.310 --> 00:41:35.620
Teresa Pfaff: you can't see those grades. Everyone like there's some availability to to see those norming across in our in our platform. But it's not

271
00:41:35.730 --> 00:41:37.529
Teresa Pfaff: that robust like this.

272
00:41:40.800 --> 00:41:43.470
Lauren Dana: Great. Thank you. That's a great question, Teresa Steline.

273
00:41:44.490 --> 00:41:48.783
Celine Greene: Well, it kinda goes toward this. This is why I was. I raised my hand.

274
00:41:49.280 --> 00:41:55.939
Celine Greene: So Teresa's over school in nursing, and other people might be from other universities. But at Bloomberg school and course, plus.

275
00:41:56.700 --> 00:42:00.180
Celine Greene: we have this peer assessment tool that student groups can use.

276
00:42:00.800 --> 00:42:11.890
Celine Greene: And I'm already tell. I'm telling you. I'm already writing a course plus feature request, and I would welcome everybody to. If you think this is a good idea to do a plus one or something

277
00:42:12.070 --> 00:42:28.229
Celine Greene: like, why not have a faculty? We have the rubric as a grade book, but I've already had faculty individually. Ask me, can't we do that where we're all feeding into the rubric. But it's only one individual can grade the one rubric that's tied to the grade book. Item.

278
00:42:28.580 --> 00:42:49.173
Celine Greene: And so if there was a separate instead of a peer assessment, it was almost like a teaching team assessment page. It would allow for a lot of that discussion, because you could have those comments in the rubric, but similar for Teresa or somebody even right now, outside, of course, lessons. We can't have things just because Celine just dreams them up.

279
00:42:50.180 --> 00:43:05.129
Celine Greene: I think with Microsoft forms you can do this right. I mean, we can build an ex with Microsoft forms. We can. It goes into an excel spreadsheet so you can have your averages, and you can have your open comments with each thing. It's just

280
00:43:05.170 --> 00:43:10.689
Celine Greene: a little clunkier than the it's, you know, we can't design the form just

281
00:43:11.010 --> 00:43:18.330
Celine Greene: with Microsoft forms the way that I've envisioned it that I can do it. Old school, when I do. Microsoft office form letters and things like that.

282
00:43:18.810 --> 00:43:23.749
Celine Greene: but I think we can get there. But I think it would be really nice for us to not just have

283
00:43:24.030 --> 00:43:32.639
Celine Greene: the rubric with performance criteria wherever you're collecting your data in the grade norming, or actually, during the term when you're actually employing it.

284
00:43:32.790 --> 00:43:34.400
Celine Greene: But to have

285
00:43:34.470 --> 00:43:45.222
Celine Greene: private to the teaching teams, those feedback comments. And then you decide what you might want to share with the students. And I just I I love the ingenuity that you're all employing to get

286
00:43:45.600 --> 00:43:51.230
Celine Greene: at these criteria performance and criteria ratings down you.

287
00:43:51.520 --> 00:44:00.780
Celine Greene: It's ingenious. I wanna like make it easier for you. So I'm like thinking, I wanna I'm I'm with. I'm with Lauren and Amy at the teaching learning center.

288
00:44:00.790 --> 00:44:07.990
Celine Greene: But again, I don't know if anybody else has any other aspects to the feature that they might want. Besides

289
00:44:08.150 --> 00:44:10.760
Celine Greene: rubric criteria, performance, level

290
00:44:10.870 --> 00:44:15.870
Celine Greene: and open feedback for quite individual criteria and or the whole.

291
00:44:16.030 --> 00:44:17.799
Celine Greene: the whole kit and caboodle.

292
00:44:18.350 --> 00:44:24.370
Celine Greene: and you could overwrite it just like the peer assessment tool. In course, class is my thinking.

293
00:44:27.060 --> 00:44:29.229
Celine Greene: Seems like that's what we're doing right.

294
00:44:30.570 --> 00:44:33.749
Teresa Pfaff: One thing with ours I just wanna mention from, like the the.

295
00:44:34.230 --> 00:44:52.869
Teresa Pfaff: the nursing side and clinical aspects is that we have a lot of potentially identifiable information for our students linked to some of their building assignments. So thinking of Ferpa, this is just something that I've been digging into with the the Hopkins AI use. So

296
00:44:52.910 --> 00:44:55.430
Teresa Pfaff: I've been a little hesitant to

297
00:44:56.270 --> 00:45:16.109
Teresa Pfaff: I think to like build and leverage a lot of like AI. So this is thinking, thinking through, not dissimilar, but like what tools we can use for building some of this. If we're putting, for for example, like for large chunks of feedback data for ours because they are associated with specific student sites, experiences, and clinical settings.

298
00:45:16.220 --> 00:45:23.848
Teresa Pfaff: So I don't know that takes it on a on a separate tangent. But like as we're building these, not only for the teaching teams and examples, but even if they're de identified

299
00:45:24.548 --> 00:45:28.821
Teresa Pfaff: I guess I'm like highlighting like if there's any discussion about Ferpa and

300
00:45:29.180 --> 00:45:30.639
Teresa Pfaff: and concerns with that

301
00:45:32.650 --> 00:45:33.839
Teresa Pfaff: more for AI.

302
00:45:43.143 --> 00:45:56.210
BSPH CTL Teaching Toolkit: Actually, I had a question as well for the for our guests. And that is so for context, I'm 1 of the co-facilitators of the Teaching Assistantship Training course that Ctl provides for bsphtas.

303
00:45:56.210 --> 00:46:21.189
BSPH CTL Teaching Toolkit: And because of that in that role I tend to. I answer questions to tas especially students who've never ta'd before. So that's sort of the context for my question. So now that you've gone through this grade norming process, and I'm sure that you've given advice and guidance to Tas, who have gone through these grade norming processes. What advice would you give to a brand new ta who is grading for the 1st time, especially if

304
00:46:21.190 --> 00:46:25.089
BSPH CTL Teaching Toolkit: their grading with a teaching team for the 1st time.

305
00:46:28.330 --> 00:46:49.319
Michelle Bedoya: I can start, I would say, ask questions and be curious. I know I mentioned that contextual factor about ta engagement, and it's really important that the teaching team is aware of what the concerns are and where there's potential ambiguity. And I would say that no conversation is in vain. We've considered everyone's feedback, and we actually improved

306
00:46:49.520 --> 00:46:57.309
Michelle Bedoya: other assignment, rubrics and instructions based on that. So I would just be be communicative, be engaged, be involved, and ask questions.

307
00:47:03.080 --> 00:47:08.189
Anne Lilly: I mean, I I would just echo that as well. I mean, especially if someone's relatively new.

308
00:47:11.330 --> 00:47:19.558
Anne Lilly: It's probably gonna sound really patronizing. But in our course. We also create a lot of guidance documents for tas

309
00:47:20.000 --> 00:47:23.320
Anne Lilly: based on how some people grade. It doesn't seem like they read them.

310
00:47:23.940 --> 00:47:30.852
Anne Lilly: So I might patronizingly advocate that people read the

311
00:47:31.820 --> 00:47:34.270
Anne Lilly: resources shared with them.

312
00:47:34.700 --> 00:47:36.609
Anne Lilly: Yeah.

313
00:47:37.070 --> 00:47:39.799
Anne Lilly: but questions absolutely. Engagement. Yes.

314
00:47:40.880 --> 00:47:54.770
Michelle Bedoya: Can I just can. I just jump off that real quick? Another thing that we asked Tas to do is like, actually look at the slides. You don't even have to go to class. But look at the slides being presented to the students, and you'll have some insights on what

315
00:47:55.060 --> 00:48:12.920
Michelle Bedoya: Dr. Sharpsteen and teaching team means when there is a specific word like what is a policy. And there were so many times where Annalise and I said, well, did you go back and look at the slides? Go back and look at Slide 23 from lecture 2. So yes, just it's it sounds patronizing, but I think it's necessary to remind them.

316
00:48:15.620 --> 00:48:18.949
Teresa Pfaff: And I can share something that maybe we've piloted.

317
00:48:19.080 --> 00:48:24.000
Teresa Pfaff: This actual semester that's been really helpful is we have set

318
00:48:24.270 --> 00:48:48.150
Teresa Pfaff: standard emails that we prime and send every Friday to our teaching. Tas, hey? Module 3 opens up this week. Here's what they're learning in theory. Here's what the clinic is on on Tuesday. Here's what Lab is on Wednesday. Here's what we're moving towards, how to integrate everything. Here are the best resources, like, if you have to do like a quick like if you can't get to all the content which, of course, we're like, you're

319
00:48:48.270 --> 00:48:54.350
Teresa Pfaff: really need to know the content. But if you need, like the the cliffs notes. There's even summary guides that we create.

320
00:48:54.600 --> 00:48:55.759
Teresa Pfaff: which is a lot.

321
00:48:56.010 --> 00:48:56.790
Teresa Pfaff: But.

322
00:48:59.860 --> 00:49:12.020
Anne Lilly: No, that's a great, I think that's a great idea. I mean, cause we I know Michelle and Annalise mentioned this earlier, but in our course evaluations, especially last year, we got feedback that like, there's a huge amount of variation in grading.

323
00:49:12.150 --> 00:49:16.120
Anne Lilly: But what came to pass like what we started to realize was like

324
00:49:16.180 --> 00:49:26.940
Anne Lilly: some of our Tas did not really have a complete command of the course material which was like slightly disturbing. But yeah, I I appreciate this idea of like

325
00:49:27.200 --> 00:49:29.659
Anne Lilly: this is what's in the course this week.

326
00:49:30.216 --> 00:49:34.643
Anne Lilly: Hi! Maybe you want to go to course, plus and look at the slides

327
00:49:35.080 --> 00:49:37.290
Anne Lilly: to Michelle's point. That's a great idea.

328
00:49:41.040 --> 00:49:42.570
Lauren Dana: Thank you, Teresa.

329
00:49:42.710 --> 00:50:02.660
Lauren Dana: and I know, Teresa, you did have a question, I think Ann and Michelle sort of addressed it. But building off that. How do you structure ta orientation after hiring? So I know Anne talked a little bit about how they've actually had the tas complete a practice grading round before they hire. But what do you do after they're hired? That's a great question.

330
00:50:06.961 --> 00:50:08.970
Anne Lilly: I can jump in. So

331
00:50:09.190 --> 00:50:12.949
Anne Lilly: I I think structure is really important. So we

332
00:50:13.350 --> 00:50:17.289
Anne Lilly: in our 1st teaching team meeting, which is actually like before the term even starts.

333
00:50:17.340 --> 00:50:29.407
Anne Lilly: We just talk through roles and expectations. And part of that is like our teaching, like our grading timeline grading expectations. But

334
00:50:29.920 --> 00:50:43.916
Anne Lilly: just kind of like the different roles that they have as tas like. We we rotate leading our team meeting like, but things like that. We orient them to also the schedule of the term, like when things are due.

335
00:50:44.770 --> 00:50:52.559
Anne Lilly: But yeah, and I mean, I'd be happy to share, just like an example of our meeting agenda. Which basically, like.

336
00:50:52.730 --> 00:50:55.880
Anne Lilly: has all the information for orientation.

337
00:51:02.780 --> 00:51:15.520
Annelise Brochier (she/her): Sorry we don't have much to add beyond that. Just we had. We had weekly T Ta. Meetings, and in our 1st meeting we went over an overview of the course just primed them on what the assignments would be.

338
00:51:19.370 --> 00:51:24.280
Michelle Bedoya: I wish I wish we had more structure. But but it's something we're working on.

339
00:51:25.330 --> 00:51:25.830
Michelle Bedoya: and.

340
00:51:25.830 --> 00:51:47.449
BSPH CTL Teaching Toolkit: I think that's a really good point, too, that this is always an ongoing process. And of course, design and development period is ongoing. And then within that grade, norming and grading and ta development, all of that happens over time. So it's okay. If you don't have a guide right away. But it could be a like a 3 year or 2 year old goal or something for the future. And that's okay. Thank you.

341
00:51:50.360 --> 00:52:00.820
Lauren Dana: Great. This has been a great discussion. We do want to just be mindful of your time. I know it's Friday afternoon, so I'm sure people have things they want to be doing. Do we have any final questions.

342
00:52:02.611 --> 00:52:06.918
Anne Lilly: I just wanna ping off of something, Teresa said earlier.

343
00:52:07.920 --> 00:52:18.580
Anne Lilly: using AI for grading I don't know when that will start to be something that is a norm. But and I'm not trying to put all the tas out of a job

344
00:52:19.310 --> 00:52:28.539
Anne Lilly: at the same time, though, when it comes to norming and like, even when I am creating grading guidance for tas, I'm like I could be telling a computer to do this.

345
00:52:30.830 --> 00:52:38.620
Anne Lilly: and again. I love Rtas. They're a great team, but I'm just like mindful of time, and like how time is spent. And so I just

346
00:52:38.700 --> 00:52:46.239
Anne Lilly: I don't know if there's like an upcoming session about that, or could be, or and I totally hear Teresa on privacy concerns, and like not wanting to

347
00:52:46.770 --> 00:52:50.550
Anne Lilly: divulge like identifiable information. But if there is a

348
00:52:50.830 --> 00:52:53.090
Anne Lilly: protected way

349
00:52:54.230 --> 00:52:58.279
Anne Lilly: that AI could be harnessed to assist in grading or to

350
00:52:58.390 --> 00:53:01.940
Anne Lilly: conduct grading, I would be really interested in like

351
00:53:02.350 --> 00:53:07.680
Anne Lilly: what that looks like? What are norms? Also like what is best practice and like, how can we protect

352
00:53:08.475 --> 00:53:10.290
Anne Lilly: our students? But

353
00:53:11.240 --> 00:53:12.469
Anne Lilly: just going to put that out there.

354
00:53:14.790 --> 00:53:17.140
BSPH CTL Teaching Toolkit: Yeah, that's a great point. Thank you.

355
00:53:19.740 --> 00:53:22.180
Lauren Dana: Yeah, great, thank you. And we're hopefully, we'll see

356
00:53:22.280 --> 00:53:36.362
Lauren Dana: maybe some future workshops on how AI could be used to help in the grade norming process, maybe not necessarily grading like you're saying, but how to just make it more efficient and provide samples, or even provide feedback to tas on the grading they gave using the rubric.

357
00:53:37.220 --> 00:53:41.240
Lauren Dana: okay, I'm gonna pass it over to Amy to to close us out for today.

358
00:53:41.240 --> 00:54:05.170
BSPH CTL Teaching Toolkit: Yes, thank you. So before we go, I just wanted to ask. We would love to hear your feedback for today's session. So please take a few minutes to complete this anonymous survey by going to the link on the screen or scanning the QR code. And again, that's an anonymous very short survey, just to let us know we do take that feedback and use it when we plan future workshops. So we appreciate that feedback.

359
00:54:05.729 --> 00:54:28.590
BSPH CTL Teaching Toolkit: I also wanted to let you know that we had an activity planned, but we thought the discussion was better, so we didn't end the QA. Session to do the activity. But we'll be sharing a worksheet with you in the follow up email along with some resources that we talked about in today's session. So you'll find that in the follow up email, there's a worksheet that can take you through a mock assignment

360
00:54:28.880 --> 00:54:42.360
BSPH CTL Teaching Toolkit: grade norming exercise. So if you want to dive more into this. I recommend that you check that worksheet out. And again, that'll go out in a follow up email. But otherwise, thank you so much, and we hope that you

361
00:54:42.720 --> 00:54:58.789
BSPH CTL Teaching Toolkit: are that you use this information to do some grade norming in the future, and we hope that you found this valuable. And and again, it's another special thanks to our guests. Thank you so much for joining us today. You truly enriched this workshop with your with your expertise. So thank you.

362
00:55:01.340 --> 00:55:02.330
Annelise Brochier (she/her): Thanks. Everyone.

363
00:55:02.330 --> 00:55:04.439
Anne Lilly: Thank you. Thank you again. Bye.

364
00:55:04.440 --> 00:55:05.780
Michelle Bedoya: Have a great weekend.