WEBVTT

1
00:00:04.100 --> 00:00:06.610
(Amy Pinkerton) BSPH CTL Teaching Toolkit: Alright, and Brian. You can start at any point.

2
00:00:06.860 --> 00:00:32.960
Brian Klaas: Okay. Hi, there! My name is Brian Kloss. I work in the center for teaching and learning. Many people around the school knows me know me as that course, plus Guy, because I do head up the team that manages and runs course plus. But for those of you who don't know. I actually also have a faculty appointment here at the school, and I've been teaching for about 8 years. I teach courses primarily on health science communication for non expert audiences. I teach a class that's required

3
00:00:32.960 --> 00:00:49.819
Brian Klaas: of all the Mph. And Msp. Students. But I also last year started teaching with my colleague David Dowdy. If you don't know David, he's our sort of the head of academics here at the school the Vice Dean on that front. And sorry executive Dean and

4
00:00:50.160 --> 00:01:14.080
Brian Klaas: we started teaching a class last fall on using generative AI in sort of daily public health practice. And that's why I think I'm here today to talk about specifically 3 ways to use generative AI in your public health teaching. So that's what today is going to be about. This is the last of the picnic Pd sessions. Very happy to be here and want to thank the team who put together the whole picnic Pd series this summer for inviting me

5
00:01:14.080 --> 00:01:29.489
Brian Klaas: to talk and I'm gonna be talking through most of the next about 25 min or so. If you have questions. Please feel free to type those in the text chat. There'll be open a. A. QA. At the end, and also an opportunity for maybe you to play around with some of the prompts I'm going to be sharing today

6
00:01:29.610 --> 00:01:57.809
Brian Klaas: and ask questions of me. During that time as well. So for many of us. For many of us here in higher education. Generative AI means one thing and one thing only, and that is the homework apocalypse the fact that from the release of Chat Gpt. In December of 2022, every student in every class was going to use generative AI to cheat on all their homework and have generative AI do it for them. And while there's certainly some truth to it

7
00:01:57.810 --> 00:02:22.769
Brian Klaas: that generative AI is great, especially if you're writing things. It's not necessarily sort of the end of of homework and assignments, and I'll talk about at the end a very specific way, sort of the 3rd technique I'm gonna talk about today about how you can use generative AI to sort of blow up that idea that it's useful to for assignments, particularly at the graduate level. It might be much easier if you're in the K through 12 space.

8
00:02:22.770 --> 00:02:37.850
Brian Klaas: maybe even undergraduate. But at the graduate level it's a bit harder to use it effectively to generate content effectively. And I'll talk more about that a little bit later on before that, though. Before we get started, I just wanna launch a quick poll. So I can see from everybody here.

9
00:02:38.274 --> 00:03:01.589
Brian Klaas: What what genre of AI tools you're currently using. If you are using any go ahead and vote in there. If you're using a tool that I don't include in this list. By all means, please type in the text chat. I'd love to see it again. I'm gonna focus on generative AI. I know many faculty here at Bloomberg. Use machine learning, traditional machine learning techniques in their research. I am not talking about traditional machine learning.

10
00:03:01.590 --> 00:03:26.570
Brian Klaas: Generative AI is a subset of what we sort of call a AI in general artificial intelligence. I'm focusing on generative AI here today, not machine learning, and not the kind of sort of statistical analysis, tools that are very popular and powerful that you can use in tools like our or our studio. Maggie also uses perplexity, which is very cool. I'm gonna focus mostly, though, on generative AI,

11
00:03:26.570 --> 00:03:50.320
Brian Klaas: and it looks like most people ha have or are using either chat, Gpt or Microsoft Copilot, then followed by Google Gemini as well. Nobody's using Claude, which is kind of a disappointment, because Claude is actually pretty awesome. Particularly if you're working with documents like Pdf files, and you want to ask questions about the document or summarization. Claude is really really good.

12
00:03:50.600 --> 00:04:04.510
Brian Klaas: So thank you for voting. I appreciate that. That's very helpful for me to know where people are in terms of the tools they're using oops. But I want to frame this, and and and try to ask you to think about using Generve AI as kind of like a personal assistant.

13
00:04:04.530 --> 00:04:31.749
Brian Klaas: It is not a replacement for you. It's not a replacement for your doctoral students. It's not a replacement for most human beings. And certainly people who have a certain kind of domain expertise. It's kind of like a personal assistant. A personal assistant will still makes mistakes. They're still learning but that personal assistant can kind of offload some tasks especially when you're pressed for time, or you're tired, especially tasks that come around things like

14
00:04:31.750 --> 00:04:55.350
Brian Klaas: summarization and ideation. In particular, I'm gonna talk a lot about ideation during today's session. And but no matter how you choose to use generative AI, just always make sure, especially in the educational space, that it's aligning with your teaching goals. It's not only about saving time. It's about making sure that how you're using it lines up with your learning objectives in your class, or for an assignment

15
00:04:55.350 --> 00:05:13.269
Brian Klaas: or for company season, a degree program, or for your own personal goals, in terms of teaching a class, or offering a training or a workshop, or whatever it is, always make sure to align to those goals. It shouldn't just be about saving time. It should be about making sure that your students, your learners, your participants, always have the best experience possible.

16
00:05:13.360 --> 00:05:24.320
Brian Klaas: So before I get into sort of talking about the specific 3 strategies for using generative AI in your public health teaching. I want to take just a little bit of time to talk about the importance of good prompting.

17
00:05:24.520 --> 00:05:48.910
Brian Klaas: because writing good prompts is a skill and writing good prompts is the key to effectively using generative AI really pretty much, no matter what your purpose is. Writing good prompts is really critical. You know, it's a skill. And like most skills, you can become better at it with practice and a lot of times it takes practice. There's a lot of stuff on the Internet. If you go read, it's like, Oh, you should do this thing. Or if you use these magic words.

18
00:05:48.920 --> 00:06:11.759
Brian Klaas: yeah, magic words are magic words. I'm not gonna help you actually get work done at the end of the day. But there are some. There's a small body of research that's coming out. Now, that's sort of saying, here are some things that actually lead to better results in your prompting. And I wanna talk about 4 specific things that you can do to write better prompts and therefore get better results from generative AI, because this will definitely affect

19
00:06:11.760 --> 00:06:36.019
Brian Klaas: the the techniques that I talk about a little bit later on. So the 1st one is, you gotta be specific, really. I mean, if you go to an auto mechanic and you're like, oh, my car is dead. Fix it! Well, that's not gonna be really helpful to the mechanic. Right? Without specific details. The mechanic can only guess as what a problem might be. But if you go to that same car mechanic and you say, well, my car started vibrating when I was driving it, and then the engine started sputtering, and it's the engine stopped, and I managed to pull over.

20
00:06:36.090 --> 00:06:45.280
Brian Klaas: and then, after like I don't know. A few seconds, I got the engine started again, but when I got it started again it sputtered some, and then stalled out again. After a couple of seconds.

21
00:06:45.490 --> 00:07:00.490
Brian Klaas: That gives the mechanic a lot more information to go on, and they'll start by looking at the fuel injector, because that's what their internal mental model and their experience tells them is the most likely culprit. So you want to be specific. So let me give you an example of this. So let's say you work at a nonprofit.

22
00:07:00.750 --> 00:07:07.409
Brian Klaas: and you're looking at ways to drive up donations to that nonprofit. So our basic prompt is, I work at a nonprofit. How can I get more donors?

23
00:07:07.460 --> 00:07:22.140
Brian Klaas: Well, you're going to get a response right? You always get a response from tools like Chatgpt or Copilot or Claude. But the response is kind of generic here right? It may not specifically address the real problems that your nonprofit or your Ngo is having in terms of fundraising.

24
00:07:22.230 --> 00:07:43.559
Brian Klaas: So if instead, I create a prompt. That's much more specific about the problems that I'm having. If we're focusing on a nonprofit in Baltimore and who's worried about a donor death spiral, we provide more context and give more specific instructions. So in this case, I'm saying also give me 3 strategies that I can use to avoid this donor death spiral, and then contrast the strengths and weaknesses

25
00:07:43.710 --> 00:08:07.729
Brian Klaas: of those strategies. We get something that's actually quite helpful, as someone who's worked in the nonprofit space for a long time, even before working at Hopkins. This is like, yeah, this actually is like, really, really helpful. This is a much more effective response from the the AI in this case, or the large language model is what I should say, because underneath every tool, like Chatgpt, copilot, Claude, there's a specific computer piece of computer software known as a large language model that does this work.

26
00:08:07.730 --> 00:08:15.509
Brian Klaas: So you're going to get a much more effective response from a large language model or Llm. Than a broad sort of general question. So being specific is important.

27
00:08:15.510 --> 00:08:39.179
Brian Klaas: And if you're learning, you need to learn about something, we all need to learn about things. We may not have the full set of knowledge in our research about smaller aspects of our research or disciplines that are sort of parallel to ours. And generative AI can actually really help in that regard. It's not always again going to be right, but it can help us figure out how to learn better when we are needing to

28
00:08:39.179 --> 00:08:48.430
Brian Klaas: learn about a topic. So again, if I ask a really broad question, I'm going to get a response that's not really necessarily specific to my

29
00:08:48.460 --> 00:09:17.880
Brian Klaas: needs, my aims or my objective. So if I just say, how can I learn about the chemistry of opioids, I'm gonna get this really broad thing that covers everything. But that may not be the knowledge that I need to accomplish my task right? So if instead if I provide more context that I'm going back to school, and I need to work on my chemistry knowledge, then ask for a structured and comprehensive approach to learning this topic, it actually gives me a much more concrete outline that I can follow for the next 3 months. Again, this is much more useful, much more actionable for me.

30
00:09:18.330 --> 00:09:30.980
Brian Klaas: And, by the way, don't worry. I'm gonna make sure that everybody gets a copy of the whole slide deck. So you have all these prompts. You'll see all of them there. You don't need to worry about screen capturing or scribbling stuff down. You'll get all the prompts I'm going to share today.

31
00:09:31.040 --> 00:09:50.510
Brian Klaas: All right. Next up with your prompting, start with an action word, or put an action word at the beginning of a sentence, create, write, make generate these action. Words direct the large language, model again, chatgpt Claude copilot to generate useful and actionable output in most cases, if you don't include these kinds of action words.

32
00:09:50.510 --> 00:10:02.010
Brian Klaas: The Llm. Will give you information that you might be relevant. But it's not going to be specifically useful to solve a problem or take action which is really in many ways for most of us one of the reasons why we use these tools.

33
00:10:02.010 --> 00:10:26.990
Brian Klaas: So here's a brief example of this. I asked Microsoft Copilot. Can you tell me about economic policy? And it gives me all sorts of general information, sort of really high level stuff, and it doesn't. If I'm writing a research paper and I need to sort of brush up on economic policy in a specific way, it doesn't help me. So I I wanna be specific. And I say instead, create a 1 week learning plan to learn basic economic policy. So I can write about it in an article focusing on wage gaps in the Us.

34
00:10:26.990 --> 00:10:45.820
Brian Klaas: And how they affect long term health outcomes. You know, I get a step by step, outline to this process. It might be a little bit broad, but this is much more actionable for me. Than the that 1st prompt without one of those verbs sort of at the beginning of the prompt saying, Here's what I want you to do, create something, make something, generate something.

35
00:10:45.980 --> 00:10:47.870
Brian Klaas: Next up is role play.

36
00:10:48.060 --> 00:11:10.340
Brian Klaas: or maybe more specifically, I should say, tell the AI who it is, and therefore what it can do. And there's a growing body of evidence of research that says that telling generative AI. Who it is in that moment of your question, and what its areas of expertise are, can really greatly improve the perceived quality and usefulness of the output.

37
00:11:10.340 --> 00:11:34.039
Brian Klaas: So tell the generative AI what its expertise is, and don't be afraid to also tell it who its audience is, who it's speaking for creating content for right whether it's an audience of high schoolers or researchers at a conference on microbiology or policymakers in DC. Tell it who its audience is, tell it who it is, what its expertise is, and you'll get much better responses. So here's an example. In this case, I said, you know, pretends you have a master's in social work.

38
00:11:34.040 --> 00:11:47.330
Brian Klaas: read the study, summarize the methods section, and come to a definite conclusion about the validity of the methods and why they're good or bad. And so the results, I think here are actually really useful, because it has a context that says, this is who you are.

39
00:11:47.330 --> 00:12:07.570
Brian Klaas: from that. It kind of infers a set of skills and focuses on domain knowledge that's really specific to that. And it will also many times when you say, you know, pretend that you're a professor, or you have a doctorate in this subject. It'll infer kind of writing style that's often much more appropriate for academia, and then it does what you want, and I think at the end of the day gives you much more useful results.

40
00:12:07.630 --> 00:12:14.189
Brian Klaas: So really, briefly, of my like, how to write better prompts, remember, provide useful context, be specific.

41
00:12:14.230 --> 00:12:24.188
Brian Klaas: start with an action word, and then role play, and including all 4 of these elements, will make your prompts better, and the output from the generative AI much more useful to you.

42
00:12:24.490 --> 00:12:52.080
Brian Klaas: And again, think about your prompting a lot of people I see, who use generative AI just go in and they ask a question like, you know, I don't know like, what's the best Shwarma in London. Well, great! But that's not necessarily going to be like going to get you the answer of What's the best shawarma in London? We have to be a bit more intentional. Provide context. And again, in terms of teaching, we want to make sure we align with our course or learning or assignment objectives and our teaching goals there.

43
00:12:52.340 --> 00:13:01.420
Brian Klaas: Okay, so that's my like better prompting 1. 0, 1 thing, because you'll see some of these techniques. In fact, all these techniques used in the example prompts moving forward.

44
00:13:02.760 --> 00:13:22.620
Brian Klaas: But let's get to the 1st of the 3 ways of using generative AI into Public Health Force because it's already 1213 and the 1st one I want to talk about is assignment, creation, and ideation. So large language models like Chatgpt, Claude Copilot are really good at brainstorming or ideating, coming up with ideas, lots of ideas.

45
00:13:22.700 --> 00:13:25.869
Brian Klaas: I'm a super busy person. I have a job I teach.

46
00:13:26.060 --> 00:13:46.199
Brian Klaas: I teach in every single academic term. I've got kids, and I'm moving to Puerto Rico, and my life is very busy, and I don't often have the mental bandwidth to come up with 50 ideas for something. But what I do have the bandwidth for is letting a tool like Chatgpt or Claude come up with 50 ideas, and I can go through and be like that one's interesting.

47
00:13:46.240 --> 00:13:49.310
Brian Klaas: Let me think about that and follow up on that.

48
00:13:49.400 --> 00:14:02.260
Brian Klaas: This is where this is what I'm talking about. When I talk about assignment, creation, or ideation, having the large language model help you either come up with ideas or take ideas that you already have, and then sort of iterate through them.

49
00:14:02.420 --> 00:14:05.079
Brian Klaas: and maybe hopefully make them better in the process.

50
00:14:05.100 --> 00:14:18.929
Brian Klaas: So one thing that's really easy to do is to make assignments from course materials. You got course materials, whether it's readings, or your own Powerpoint slides or assignment notes or assignment descriptions, P-d-f's word documents. Whatever is, you've got course materials

51
00:14:18.930 --> 00:14:27.990
Brian Klaas: and coming up with interesting assignments can be really challenging, sometimes coming up with complex in-depth assignments that are also engaging for students.

52
00:14:27.990 --> 00:14:52.229
Brian Klaas: So what we can do is we can sort of follow a sort of prompt outline that I'm gonna share with you in just a second that does a bunch of things you're gonna role play. You're gonna tell the large language model what department or course you're teaching in your department. You're in the course you're teaching providing files for analysis. So it knows what you wanna sort of base your assignment off of a target audience, a time limit. This is really important, because otherwise large language models would be like.

53
00:14:52.230 --> 00:14:58.169
Brian Klaas: here's an assignment that will take 2 years to complete, and that does not really help us in our 8 week terms here at public health.

54
00:14:58.170 --> 00:15:23.019
Brian Klaas: and again providing learning objectives so that you're aligning with your teaching goals. And so a prompt that I've used in this case is your professor from whatever department or center at Hopkins. You're teaching on this topic. Analyze the attached files, then create a comprehensive and engaging assignment for master students, doctoral students, groups of 3 to 5 people whatever it is, the assignment should take no more than whatever hour.

55
00:15:23.020 --> 00:15:40.689
Brian Klaas: an hour to complete and align it to these objectives, and I included comprehensive and engaging, because sometimes adding, a few adjectives like that that are sort of positive oriented, can actually help the large language model create something that is much more engaging. That isn't simply a dry regurgitation of the knowledge that you would expect students to have.

56
00:15:40.690 --> 00:15:53.030
Brian Klaas: So I did this. Here, here's an example of this about teaching techniques for investigating the outbreak of communicable diseases diseases. In this case this was the sir model

57
00:15:53.060 --> 00:16:03.150
Brian Klaas: that was being covered in the lecture, and I said, create a comprehensive and engaging assessment for groups of 2 to 3 students, and it should take no more than an hour to complete. And here's the learning objectives. So

58
00:16:03.460 --> 00:16:30.190
Brian Klaas: I do this. And it comes up with this idea, and it could come up with one idea. But I could also say, create 10 different assignments for me, and it would create 10 different assignments for me, and I could go through those and be like, oh, this one's useful, or pick and choose the bits and pieces that I like the most from that assignment, or whatever assignments in there. I'm not sitting there brainstorming all this stuff. I'm the one who has to say, yeah, this is accurate. Yeah, this will actually measure what I want the students to get out of this.

59
00:16:30.200 --> 00:16:37.779
Brian Klaas: That's something that that large language model can evaluate. But where it can do the work is to brainstorm and come up with a different bunch of ideas for you.

60
00:16:37.920 --> 00:17:02.919
Brian Klaas: Another area that I've used personally. Chat. Gpt, well, yeah, actually, mostly chat Gpt for is coming up with discussion questions. This is not my area of strength in my classes. It really is not so. What I did instead was. And we're talking about looking at different kind of readings or papers. And I say, you know, analyze this and focus on the role of intermediaries in research, uptake and then create 5 discussions questions suitable for master students to lead a discussion

61
00:17:02.920 --> 00:17:13.070
Brian Klaas: on their own. Each question should be complex enough, that it has no single answer, but scoped tightly enough, that a discussion about the question can conclude with reasonable answers in under 10 min.

62
00:17:13.069 --> 00:17:19.650
Brian Klaas: So here I said 5 questions. But again I could have said 50, or I could have said 10, and I can go through this and be like, that's an interesting question.

63
00:17:19.720 --> 00:17:41.110
Brian Klaas: I want to ask my students to lead a discussion based off that question, or I can ask that question on the discussion Forum or in the classroom again, by looking at sort of the documents, by looking at the content that I already have in my class, and I'm sharing with my students. I can ask the large language model to do that work much more rapidly than I could, and then I can decide what I want to use and what I don't want to use.

64
00:17:41.110 --> 00:17:54.739
Brian Klaas: If you already have an assignment assignment transferred transformation can definitely be something that the large language models can use. So, for example, I could say, modify this lesson to include more opportunities for cooperative learning, changing it from an individual assignment into a group. One

65
00:17:54.940 --> 00:18:19.810
Brian Klaas: redesign this activity to focus on inquiry, based investigation or case based investigation is another way to say that right or I could say, modify the following project description to better suit the needs and interests of doctoral students. Right? This is a real challenge sometimes in our classes is, we don't have the bandwidth to say doctoral students should be operating over here, so I'm going to give them an entirely different assignment than I would my 1st year, master students.

66
00:18:19.810 --> 00:18:36.700
Brian Klaas: So the large language model can do this work. And then again, you're the one who says, Yes, this is right or no, that's wrong, or this is a good idea. But I'm also going to do this right. You still have to evaluate it. It's your assistant you have to check their work, but they're doing that kind of grunt work for you so that you don't have to.

67
00:18:36.780 --> 00:19:00.629
Brian Klaas: Another area. That, I think, is great for assignments is having a large language model. Look at clarity and organization in the assignments is something I do in all my classes. Now, 1st and foremost, review this assignment description, and point out areas where the writing can be simplified to ensure clarity to non-native English speakers, or I should just say, to ensure clarity right? I can write lots of big giant run on sentences right? Most of us in academia can do the exact same thing.

68
00:19:00.750 --> 00:19:19.300
Brian Klaas: but clarity is really important, and large language models are really good right now at rewriting, content to make it simpler, to make it more clear, we might lose subtlety and nuance, and that might be an issue. But again, just like an assistant. That's your job to check it. You don't have to do the hard work of actually doing the rewriting.

69
00:19:19.310 --> 00:19:42.519
Brian Klaas: Another thing that I do. This is great. If you have a really complicated project that students need to follow. In terms of, like the you have an assignment or a project that has lots of steps. Tell the large language model to create a checklist right, and to go back and make sure that all the required components are completed in there, because sometimes, if you tell the large language model, check your work, it actually checks its work and gives you much, much more accurate answers.

70
00:19:42.520 --> 00:19:49.999
Brian Klaas: So instead of you creating a checklist, it can create the checklist. And that helps you also say, Wait, this one step is missing. Why is that?

71
00:19:50.000 --> 00:20:14.971
Brian Klaas: And then go back and figure out why that one step is missing from this checklist? Because it wasn't maybe clearly articulated in your assignment description. This all helps to reduce student questions, student confusion, and arguing over grades when the time comes for grades to come out. And speaking of arguing over grades. You could also use a large language models to create rubrics for assignment. Rubrics are great. I love rubrics. In all my classes that I teach

72
00:20:15.540 --> 00:20:36.279
Brian Klaas: I use rubrics for all of my grading, except for multiple choice. True, false matching quizzes like the sort of auto graded quizzes. In course, plas. Rubrics are great because they tell students what they need to do, and they reduce arguments about grades. I teach probably 7 800 students a year. Now, because I teach this class. It's offered in a bunch of the terms.

73
00:20:36.280 --> 00:20:59.949
Brian Klaas: If I didn't have rubrics I'd be fighting over so many points with students. And then the way rubrics really have reduced that for me. I know the instructional designers at the Ctl. Team love rubrics. We all love rubrics. They're great. Think about how you're going to use them. So this is an example prompt you can use for creating rubrics in here, and you know I will say that the rubrics that I've generated out of Claude and out of Chatgpt

74
00:20:59.950 --> 00:21:24.320
Brian Klaas: have been pretty good, but they need human work, and I'm not trying to give Amy and and Lauren and Emily. More work. But I'll tell you. The the instructional designers in the center for teaching and learning are more than happy to work with you on refining rubrics and making sure they really align with the goals of an assignment. And again, the the large language model can get you started, and then maybe you could take that and work with an instructional designer to really refine it.

75
00:21:24.320 --> 00:21:30.910
Brian Klaas: Make it clear and excellent, because a little human intervention on rubrics goes a very, very long way.

76
00:21:31.280 --> 00:21:55.700
Brian Klaas: but no matter what approach you take with assignments, always again be intentional, align it with your course, assignment, or course or assignment objectives, or your teaching goals, whether that's inclusion, skill, building, meeting, accreditation requirements. In the course case of some of my classes. Make sure you're aligning it with your goals. There, that's super important, all right. Next up. Quiz building. Yes. So

77
00:21:57.000 --> 00:22:01.729
Brian Klaas: yeah. Well, I'll get to that in a second. I'm thinking that so good got derailed there. Quiz building. So

78
00:22:02.160 --> 00:22:04.450
Brian Klaas: coming up with good questions for quizzes

79
00:22:04.460 --> 00:22:29.600
Brian Klaas: or exams can be hard. And I'm not gonna make the argument that large language models are really good at synthesizing lots of content into complicated and complex questions that you might find on like you know your I don't know your Ep. 7 21 exam. Or, you know, in Marie Dean or West Stat methods class on the final exam. I'm not gonna say that they're quite that good, because that does require a lot of sort of thought and careful planning.

80
00:22:29.600 --> 00:22:39.229
Brian Klaas: But for all the things that you might want to otherwise create quizzes or questions for review assessment, knowledge building questions for large language models can be great.

81
00:22:39.500 --> 00:23:03.410
Brian Klaas: So you can do it. For example, for essay questions for brainstorming, different essay questions right that might go into a series of content. Here, I said, you know you work at your professor on Hbs, at Bloomberg. You teach courses on social justice, intersectionality, and public health in Baltimore create 5 essay questions that allow master students targeting a specific group to demonstrate mastery of applying systems, thinking tools to food deserts in Baltimore

82
00:23:03.410 --> 00:23:28.399
Brian Klaas: pretty specific. But again, I don't have to come up with 5 different questions and then figure out what's the best one? The Llm. Does it for me, and then I can be like this is a good question. I will use this, or I like this element from this question and this element for this question, and you put them together, and you have an entirely new question. Again, brainstorming is a really powerful way of creating the large language models are really good at kind of like brainstorming

83
00:23:28.400 --> 00:23:29.720
Brian Klaas: in that regard.

84
00:23:30.110 --> 00:23:47.659
Brian Klaas: So another really powerful way of using large language models for sort of quiz making is to help you quickly create review questions that will help the students to help make sure that students understand course content.

85
00:23:47.960 --> 00:24:03.220
Brian Klaas: And I want to take a small detour, but not really a detour. It's actually one of the main reasons I'm doing this to talk about a brand new feature that we just launched in course, plus on Friday, and that is the course plus AI review quiz maker.

86
00:24:03.250 --> 00:24:07.119
Brian Klaas: So what this tool does is that, in

87
00:24:07.140 --> 00:24:32.050
Brian Klaas: course, plus if you have a Ctl hosted lecture. That means a lecture that appears on a page builder page. It has that lecture materials box. If any of you've done online courses. That's a a Ctl produced lecture with a lecture materials box. If you use those online lectures in your on campus courses, the that's a anything in the lecture materials box. That's a Ctl produced lecture. It doesn't currently work with Panopto doesn't work with Zoom Meeting recordings.

88
00:24:32.427 --> 00:24:40.739
Brian Klaas: Might be something we'll consider in the future. But right now, if you have a lecture, in course that has, it has transcripts, because they all do.

89
00:24:41.000 --> 00:25:02.629
Brian Klaas: you will see a button for calling. Now the AI Review quiz maker. And the reason that we've done this, we added this tool is that we wanted to help faculty and tas create more opportunities for formative assessment in their classes, because the literature on effective teaching shows time and again

90
00:25:02.630 --> 00:25:22.029
Brian Klaas: that practice testing is the single, most effective way of helping students, remembering core didactic content. The the reason why you give lectures, the reason that you're going to expect them to watch them helping students. And you can help students remember that and demonstrate that they understand that content through review quizzes

91
00:25:22.120 --> 00:25:32.469
Brian Klaas: the reality is that most of us don't have the time to do that. We're super busy people. We don't have a chance to. We don't have the time to make a review quiz for all 16 lectures in our course we don't have the time.

92
00:25:32.510 --> 00:25:49.289
Brian Klaas: So this tool will help you and will facilitate that process. Whether you're gonna make a standard review, quiz that students can review before at the end of every week, or for midterms finals or you're making in lecture quizzes which you can do and put those inside the videos that you're actually doing itself.

93
00:25:49.330 --> 00:26:15.749
Brian Klaas: So if you go to course plus. And you see your lecture materials box on any lecture page and again, this is only available to faculty and tas cause. This is on the editing view of a lecture not available to students. You'll see this button that, says, AI Review quiz maker, there's if you click on that. It shows you some text about how the tool works, and you can click the button that says, generate a review quiz. So I just wanna talk a little bit about how this process works, and some really important points. 1st and foremost.

94
00:26:15.750 --> 00:26:19.370
Brian Klaas: your data, including the entire lecture. Content is private.

95
00:26:19.370 --> 00:26:41.960
Brian Klaas: It is not used in any way, shape or form, to train, chat, Gpt, or Claude, or any large language model. The legally binding agreement that we sign with Openai says that the content that we put into this kind of private version of Chat Gpt cannot ever be used to train, chat, Gpt, or any future large language model. So your data is completely private

96
00:26:42.230 --> 00:26:56.120
Brian Klaas: the entire time it gets uploaded, processed, and then immediately deleted. As soon as the large language model generates the quiz for you. And again, this is totally optional. Nobody has to do this, nobody is forced to do this. This is a completely optional tool.

97
00:26:56.450 --> 00:27:13.789
Brian Klaas: and what you'll get is an email. This isn't done in real time. This isn't like using Chat Gpt, where you give it a prompt, because all the work is done behind the scenes for you. And what you'll get is an email that has the questions, the answers, and then an explanation of why the large language model reasoned that particular

98
00:27:13.880 --> 00:27:37.720
Brian Klaas: explanation, or that answer was correct. So here's a really quick screenshot of one example set of questions that came from the AI Review quiz maker here that gets sent in email. You can see there's the question, the answer. And then an explanation of why? Again, the large language model reasoned that that was the right answer. And you can run this once you can run it 10 times you can get. You're always going to get different sort of different answers. I mean, eventually, you'll run out of

99
00:27:37.720 --> 00:27:44.519
Brian Klaas: different questions to ask, especially if you have like a 10 min lecture. But you're going to get excuse me different answers and questions

100
00:27:44.720 --> 00:27:49.729
Brian Klaas: each time, and it's also important to note that no quiz, in course, plus is created

101
00:27:49.810 --> 00:28:13.869
Brian Klaas: because someone with expertise meaning a faculty member, a ta. Somebody needs to review that and be like, yes, that's a good question. No, that's not quite right, whatever it is before it goes and becomes a quiz. But once you have those questions and answers generated, it becomes very easy to literally copy and paste into a regular quiz or an in lecture, quiz, in course, to make things much easier for you, and to create that that sort of formative assessment

102
00:28:13.870 --> 00:28:19.859
Brian Klaas: for students to help them make sure they understand the key ideas and concepts in your lectures.

103
00:28:19.860 --> 00:28:37.819
Brian Klaas: And again, I will say, if you're gonna use this tool. Just be intentional about it. Make sure it aligns with your goals. It's not really a substitute for thinking through and creating complex questions for a midterm or final in most cases, although that's that's for you to decide. I would say that. But that's for you to decide. If you use this tool or not?

104
00:28:38.050 --> 00:28:40.829
Brian Klaas: All right, running out of time, and I'm a little bit later than

105
00:28:40.950 --> 00:29:08.049
Brian Klaas: practicing, and I've been like 5 min under. But I'm not. Last thing I want to talk about is last ways of using generative AI in your class, you're teaching is to really focus on student-centered reflection through AI. And this is about combating, cheating by actually having students use large language models in intentional ways and asking them to reflect on the process because, Chatgpt. They're not always right, right?

106
00:29:08.050 --> 00:29:11.090
Brian Klaas: They're not always right. They're sometimes limited in scope.

107
00:29:11.150 --> 00:29:30.480
Brian Klaas: And by having students use these tools they begin to see their failings, and certainly their possibilities, but certainly their failings. Why, they can't just use the tool as is why it doesn't really cover this specific domain because it doesn't have training knowledge around that. And what's great about having students use the generative AI tools directly

108
00:29:30.480 --> 00:29:45.449
Brian Klaas: is that using them critically and reflecting on that use really requires some degree of mastery of the subject domain by the students themselves, so they're able to see the flaws in the output or the limitations in what the generative AI can do for them.

109
00:29:45.530 --> 00:29:52.160
Brian Klaas: So a really simple way to do this is, you have an assignment. You're like, Okay, you're going to work with a generative AI on this.

110
00:29:52.370 --> 00:30:00.220
Brian Klaas: and then you're going to reflect on it, and the grading is on the reflection, not on the direct output from the large language model.

111
00:30:00.250 --> 00:30:25.240
Brian Klaas: So you can ask questions like, was the suggested content useful? What new avenues were opened up to you by the generative AI output? This is something that David and I do in our generative AI class. And the students find these experiences to be really powerful. And and I think on some level transformative because they're like, yes, I can see how this works. And yes, I can see how better prompting works. And yes, I can see the limitations. And yes, I can see where I should not be using this, or why I shouldn't be using this.

112
00:30:25.240 --> 00:30:46.459
Brian Klaas: but I can also see where it can help me out as a professional, because again, students are going to be out there in the real world working, and they're going to be probably very shortly if they aren't already expected to use this stuff. So by focusing and putting the grading on the reflection, the students really have to think about what they learned, how they learned, and why content was created in the way that it was oops.

113
00:30:46.650 --> 00:31:10.600
Brian Klaas: Right? So another way to do this really one of the strongest ways to avoid, you know, letting AI do all the work is to build an assignment through stages, from ideation to outline to final product, and then again focusing on reflection, is part of the the grading process. You know. What did you learn along the way? What would you change right? What would you change in the process. And by splitting this across multiple days or weeks.

114
00:31:10.860 --> 00:31:38.090
Brian Klaas: you sort of use some of the limitations of what an AI can remember. It doesn't really remember anything. It has data stored in a database, but that data gets cleaned out every 3 every few days. So it's more difficult for students to sort of. Say, I'm just going to continue on a conversation forever and ever on a really complicated project, and let it do all the work for me, because eventually you'll run into that context window limitation where Chat Gpt or Claude will start forgetting things. And again, AI enabled assignments really at the end of the day

115
00:31:38.090 --> 00:32:04.670
Brian Klaas: may require much more synthesis of information by students, right instead of just summarization or citing right? So if you're going to do this specifically, ask for reflection on the appropriateness of using generative AI in the process. That's a really important part to get students to think critically about it and specifically ask for reflection on potential biases. In the analysis, because generative AI was used and generative. AI reflects biases that we find in our common culture.

116
00:32:05.300 --> 00:32:29.079
Brian Klaas: So my hope is that. And that's it. So that's that's those are the 3 techniques that I want to talk about. I hope you go chat with the Gpt. Take what you learn from today and go chat with the Gpt. And these are really powerful tools for ideation, summarization, rewriting. Just make sure that you remember these are prediction engines, not perfect oracles, the future or the past, and again, always be intentional. Think about how you want that large language model to assist you.

117
00:32:29.080 --> 00:32:35.529
Brian Klaas: Make sure it's not solely about saving time, but really achieving a better learning experience for your students where possible.

118
00:32:35.800 --> 00:32:45.920
Brian Klaas: So that's it for me. I'm gonna turn it back over to Amy. Really, briefly, I realize we're a couple of minutes are all over. Sorry I get passionate, and I just run my mouth. And but I'm gonna turn it back over to Amy before we start the QA.

119
00:32:46.450 --> 00:32:58.449
(Amy Pinkerton) BSPH CTL Teaching Toolkit: Thank you so much, Brian. I just wanted to let everyone know that. I'm posting the Pdf of today's workshop slides in the chat, and those will also be shared in our follow up email to all zoom registrants

120
00:32:58.590 --> 00:33:20.950
(Amy Pinkerton) BSPH CTL Teaching Toolkit: and then on the screen you'll see we have a QR code. And then also we'll post the link in the chat to this anonymous workshop evaluation form. We really do appreciate your feedback on these workshops. We use that feedback to plan future workshops. So please fill out the workshop evaluation again. It only takes a few minutes, and your feedback is anonymous.

121
00:33:21.310 --> 00:33:23.600
(Amy Pinkerton) BSPH CTL Teaching Toolkit: and then next slide, please.

122
00:33:23.980 --> 00:33:50.020
(Amy Pinkerton) BSPH CTL Teaching Toolkit: We also encourage you to continue your professional development with the center for teaching and learning. If you're a faculty member at Bsp, we have an online, self-paced modular course called the essentials, of course, design, development and teaching at Bsp, and if you're not a Bsp faculty, you can still check it out. There's some helpful modules that can be applied that can be applicable to faculty, not just at Bsp.

123
00:33:50.517 --> 00:34:04.120
(Amy Pinkerton) BSPH CTL Teaching Toolkit: we also have an online self-paced course for teaching assistants called the Ctl. Teaching Assistantship Training Course, and students who complete that course get a certificate of completion at the end of all of the assignments.

124
00:34:04.635 --> 00:34:13.180
(Amy Pinkerton) BSPH CTL Teaching Toolkit: And then we also just have general workshop on demand videos and our Ctl blogs. So if you're looking for just in time or really specific topics.

125
00:34:13.505 --> 00:34:37.570
(Amy Pinkerton) BSPH CTL Teaching Toolkit: kind of training check those out, including. I wanted to mention. This picnic pd, session is the final session in our picnic Pd series, and all of our all of the recordings from the picnic Pd series are posted on our on demand videos and workshops. Page. So check that out. If you missed any of the picnic Pd sessions that we've offered this summer, but otherwise thank you so much, Brian.

126
00:34:37.570 --> 00:34:45.599
(Amy Pinkerton) BSPH CTL Teaching Toolkit: And please stick around if you have questions and we'll pass on to the open. QA. Thank you so much.

127
00:34:45.940 --> 00:34:52.950
Brian Klaas: Thanks. So if you have questions for me please feel free to. You could raise your hand type in the text, chat, happy to answer them either way.

128
00:34:54.270 --> 00:34:59.449
Brian Klaas: and thank you for the kind words folks in the chat appreciate it, Josh. You have a question.

129
00:34:59.700 --> 00:35:02.310
Joshua Francisco: Yes, Hi, so

130
00:35:05.400 --> 00:35:06.940
Joshua Francisco: I feel like.

131
00:35:07.010 --> 00:35:11.840
Joshua Francisco: or at least from my knowledge, like course, plus is kind of like the 1st integration.

132
00:35:11.870 --> 00:35:27.238
Joshua Francisco: like in an Lms that we have, that I'm I'm aware of like in Johns Hopkins. I was wondering if there were other pushes like, let's say in canvas, because, like, you know, other other schools are. And like what that conversations around those

133
00:35:27.790 --> 00:35:30.619
Joshua Francisco: are happening. I just wanted to ask that. But.

134
00:35:30.620 --> 00:35:58.649
Brian Klaas: That is a great question. So I can tell you from what I know. That the answer is, there is a lot of discussion around using tools, large language, model tools, generative AI in canvas across the enterprise as well. There's sort of 2 primary discussions that I'm aware of right now, the 1st one is enabling co-pilot, Microsoft copilot inside of Microsoft 365. So it's word Powerpoint. Excel outlook things like you know.

135
00:35:58.700 --> 00:36:22.410
Brian Klaas: write me our email response that says no in a polite way, inside of outlook or create a Powerpoint presentation. From this outline I created in word or I have to write a Grant proposal, and I need help writing the introduction inside of word so that's what copilot inside of Microsoft 3, 65 would allow you to do

136
00:36:22.772 --> 00:36:25.997
Brian Klaas: the other sort of tool that is being looked at.

137
00:36:26.500 --> 00:36:42.829
Brian Klaas: I wanna say, it's i 5. It's basically sort of kind of like the review quiz maker. It's a tool that integrates into canvas that you can take files in your canvas course site, and then from those files generate a quiz, a summarized document, an assignment.

138
00:36:42.830 --> 00:37:07.170
Brian Klaas: sometimes an assignment description, those things. So that's being looked at pretty heavily as well. The big challenge for an institution like Hopkins. And it's funny that you bring this up, Josh, because I was having this exact conversation with some folks who work for adobe just the other day some technical technical product leaders at adobe just the other day. And you know, I think the challenge of a place like Hopkins is that

139
00:37:07.170 --> 00:37:33.819
Brian Klaas: you have to meet lots of different requirements, check a lot of boxes around the law because you have not only issues around students and Ferpa and data privacy on the sort of academic side you have the whole clinical side to deal with, and that leads to all sorts of complicated challenges. In terms of meeting all those requirements and being fully compliant to all the things you have to be compliant because people work on both sides of the the stand right, both on the academic

140
00:37:33.820 --> 00:37:44.819
Brian Klaas: and the clinical side. And then there's the real, very, very real question of cheating. If you open these tools up and you say, sure, every student at Johns Hopkins gets this tool for free

141
00:37:45.100 --> 00:38:14.040
Brian Klaas: on some level, you could be enabling significant amounts of cheating, and that gives a lot of people pause as well. So I think what you're going to find at Hopkins broadly, is enablement of AI or gender AI tools that are focused more on creating content by faculty and tas or by clinical directors, subject matter experts, and less on students and learners themselves. Because I think that's going to be a massive hurdle to overcome over time. Does that answer your question, Josh?

142
00:38:14.340 --> 00:38:16.040
Joshua Francisco: Yeah, yeah, thanks. So much.

143
00:38:16.380 --> 00:38:17.720
Brian Klaas: Carolyn, you have a question.

144
00:38:18.580 --> 00:38:24.279
Caralynn Wilczewski: Hi, thank you, for that was really helpful. I've tried using AI a couple of times, but one thing I've

145
00:38:24.280 --> 00:38:49.240
Caralynn Wilczewski: constantly run into. I was primarily trying to use it to generate quiz questions. And every time the quiz questions that I get out are pretty basic. They're really just like knowledge driven and so do you have any suggestions on how to better design prompts to try to get the AI tool, to generate more like higher order thinking questions, because I have gone to some of the other Ctl lectures where we know it's really important for these

146
00:38:49.240 --> 00:38:55.690
Caralynn Wilczewski: questions to more accurately kind of model what we expect the students to do out in the real world, which is not just like.

147
00:38:56.080 --> 00:38:58.680
Caralynn Wilczewski: you know, vomit facts.

148
00:38:58.680 --> 00:39:24.770
Brian Klaas: Right? Exactly. Exactly. So yeah, that's a really good question. So in in terms of how we sort of went about creating the AI quiz generator the AI review quiz maker. Excuse me, we really wanted to focus more on the kind of like basic facts based basic ideas because it can be really challenging, for I think at this point, without super detailed prompting. For a large language model to come up with that sort of higher order. Thinking kinds of questions. Because

149
00:39:24.770 --> 00:39:38.580
Brian Klaas: we have to. When you build a tool like this, make it pretty straightforward and standardized. That applies to lots of different kinds of context. The great thing about doing this on your own is that you have the opportunity and option to add a lot of that context and a lot of that detail.

150
00:39:38.580 --> 00:39:58.360
Brian Klaas: So there are things there are ways you can in add things you can add to your prompt that will help focus it on higher order thinking, saying, like actually describing, like Bloom's taxonomy, I don't know if you're familiar with Bloom's taxonomy and saying, Here's Bloom's taxonomy. I want you to focus on these levels in bloom, taxonomy in all caps. Right? Do not ask questions about specific facts

151
00:39:58.360 --> 00:40:22.940
Brian Klaas: in the sample materials, whether it's a Pdf. Or a transcript or Powerpoint file. That's a really good way, actually, of helping the large language model to avoid just sort of regurgitation of didactic facts. In your in, in, in your question making and the other thing I would say is, is again, don't just ask for like one ask for like 20? Because I think at the end of the day, in my experience, you're gonna have to piece together

152
00:40:22.990 --> 00:40:46.669
Brian Klaas: the best ideas that you get from a large language, model and or say, this is a good core of an idea, but I need to add my own touch to it. I need to rewrite this so that it uses a more complicated data set or it uses a more complicated scenario. It's good. This again, they're good for foundational stuff. You just kinda have to add more to it. Is that helpful Carolyn at all?

153
00:40:46.670 --> 00:40:59.150
Caralynn Wilczewski: Super helpful. Thank you. I didn't realize I just really need to be like, I was kind of trying to be more, I guess, vague, and how I was trying to get it to do what I wanted. But I really just need to say, Go here, do this thing.

154
00:40:59.400 --> 00:41:20.239
Brian Klaas: Yeah, very much. So very much. You gotta be specific, although I will say there is some. There is some evidence. It's not really nothing that I've seen reproduced. But people who work with large language lot models, a lot in terms of like engineering. I've read a bunch of blog posts from different engineers who are like, look, I've been doing this for a year and my company. And here's what we do that sometimes too much instruction

155
00:41:20.390 --> 00:41:40.820
Brian Klaas: can work negatively against you. So there is a balance to be struck. I'm not saying like, and by too much instruction, I mean, like you know, 3,000 words of instruction. That's not. That's too much, but you know 500 words. That's not too much at all, and you can cram a lot of detail and a lot of specific context into 500 words or less.

156
00:41:40.980 --> 00:41:41.500
Brian Klaas: Really.

157
00:41:41.500 --> 00:41:42.730
Caralynn Wilczewski: Great. Thank you so much.

158
00:41:42.730 --> 00:41:45.000
Brian Klaas: You're welcome. Thanks, great question. Thank you for asking

159
00:41:47.370 --> 00:41:48.810
Brian Klaas: other questions.

160
00:41:50.760 --> 00:42:10.590
Brian Klaas: So I will say, by the way, if you are a faculty member who is on this call, and you are interested. David and I last year set up a sort of copy of our generative AI. Course. You're welcome to go in there and take a look at it. All you need to do is just email me. I'll add you to that course site. If you want to review all the materials we just had to make a separate copy

161
00:42:10.590 --> 00:42:21.780
Brian Klaas: to respect student privacy laws so that you weren't seeing student work in there. But that option exists if you are interested in in doing, in in taking a look at our generative AI course and all the content that's in there.

162
00:42:21.830 --> 00:42:22.760
Brian Klaas: Sukom.

163
00:42:23.910 --> 00:42:37.609
S Kanchan: So I just want to follow up what you that Caroline ask, and that we talk about. So so Caroline can say, Oh, yeah, you you? We have 20 questions right? And these are 10 questions that I pick.

164
00:42:37.720 --> 00:42:45.020
S Kanchan: So go back and tell the AI that you picked these 10 questions and ask for more questions

165
00:42:45.040 --> 00:42:48.420
S Kanchan: that reflects those 10 that you show us.

166
00:42:48.600 --> 00:42:48.980
Brian Klaas: Yeah.

167
00:42:48.980 --> 00:42:52.810
S Kanchan: That would help train the AI to go to the next level.

168
00:42:52.810 --> 00:43:00.240
Brian Klaas: Yeah, in that conversation. Because again, like, when we're dealing with the review, quiz the AI review quiz maker, it's not really a conversation. It's kind of like a 1 off

169
00:43:00.240 --> 00:43:24.080
Brian Klaas: one shot prompt, and then it generates the content. But if you're actually having that discussion, providing examples or saying this thing that you made was really good do more of these really is a very effective, prompt technique for large language models. They like a little stroking. They like a little positive reinforcement. Believe it or not. We don't know why researchers don't know why, but a little positive reinforcement and positive examples

170
00:43:24.080 --> 00:43:27.810
Brian Klaas: tend to move that conversation very much in the direction you want it to go.

171
00:43:30.910 --> 00:43:31.870
Brian Klaas: Okay?

172
00:43:33.520 --> 00:43:47.209
Brian Klaas: Well, it doesn't look like there's any more questions coming in. So I will say, thank you all for being here. I'm glad that you found it useful, and if you have questions about generative AI and stuff, feel free to reach out to me, or you want to take a look at that class that that David and I teach you can also reach out to me as well.

173
00:43:50.590 --> 00:43:51.860
Brian Klaas: Thanks everyone.