WEBVTT 1 00:00:02.600 --> 00:00:29.419 BSPH CTL Teaching Toolkit: All right, and we will begin So Hello! And welcome everyone to this center for teaching and learning, teaching toolkit workshop. On diving deep into course data My name is Amy Pinkerton, and I'm a senior instructional designer at Ctl. Or the Center for teaching and learning, and I'm joined by my colleague. I'll let Lauren introduce herself. Thank you, Amy. Hello! My name is Lauren Dana, and I'm an instructional design support specialist at CTO. 2 00:00:29.700 --> 00:00:38.660 BSPH CTL Teaching Toolkit: Thank you. And as I mentioned a moment ago, today's workshop will be recorded and the recording will be shared with the Zoom registrants. 3 00:00:38.770 --> 00:00:45.650 BSPH CTL Teaching Toolkit: if you have any questions. Please post them in the Zoom chat, and also there will be opportunities for Q. A. Throughout. 4 00:00:45.660 --> 00:01:15.180 BSPH CTL Teaching Toolkit: and before we begin to look at course, data, and learning analytics. I wanted to mention that we'll be covering data analytics that are available in the Bsph learning management system course plus But the types of data that will be covering can also be on in other management systems like canvas or blackboard. So if you're not from Bspa. If you're from a different school at the University. there's still should be applicable information in the workshop. it just would be access differently. 5 00:01:15.180 --> 00:01:17.830 BSPH CTL Teaching Toolkit: depending on your your learning management system. 6 00:01:18.210 --> 00:01:32.689 BSPH CTL Teaching Toolkit: All right. So with that out of the way. I wanted to first mention our agenda, so we'll be first we'll define what a learning management system is, as well as what our learning analytics. then we'll look at how, 7 00:01:32.830 --> 00:01:48.360 BSPH CTL Teaching Toolkit: specifically, you would access analytics through course, plus and then we'll look at how quantitative and qualitative data can be used for course management and course redesign and then we'll have a quick simulation activity, followed by Q. A. And a survey at the very end. 8 00:01:50.650 --> 00:02:01.600 Lauren Dana: Great. I'm just going to quickly go over our learning objectives for today. So after attending this workshop, you will be able to define learning management system, analytics and learning analytics 9 00:02:01.680 --> 00:02:05.510 Lauren Dana: identify the learning analytics accessible through course plus 10 00:02:05.710 --> 00:02:18.429 Lauren Dana: analyze how Lms. Analytics can be used for both course management and course redesign, and then you will actually evaluate sample mls, analytics and develop actionable steps for course management and course redesign. 11 00:02:20.560 --> 00:02:25.409 Lauren Dana: All right. So let's start by getting our feet wet by reviewing some key definitions. 12 00:02:25.450 --> 00:02:31.259 Lauren Dana: The first definition that we'll discuss is, what do we mean by learning management system data. 13 00:02:32.760 --> 00:02:45.599 Lauren Dana: Just to review a learning management system or Lms is a software or web-based technology that houses learning blackboard canvas and our very own course, plus are all examples of an Mls. L Lms 14 00:02:45.600 --> 00:03:02.810 Lauren Dana: Lms data is all the data that the Lms tracks and provides to its users. We will go into a more exhaustive list later. But some examples of the data that course, plus tracks are, how often a student logs, in what time they access lectures, what time they submit assignments. 15 00:03:04.330 --> 00:03:16.129 Lauren Dana: And now that our feet are just a little bit, we, we're gonna jump in a little bit deeper. The next logical question for faculty is, what do we do with this data? And this is where learning analytics comes in. 16 00:03:17.120 --> 00:03:34.769 Lauren Dana: The official definition as determined by the first International Conference on learning, analytics and Knowledge is the measurement, collection, analysis and reporting of data about learners and their contacts for purposes of understanding and optimizing learning and the environments in which it occurs 17 00:03:34.770 --> 00:03:49.239 Lauren Dana: in Planar terms, learning analytics is the analysis of the data your Lms can provide with the ultimate goal of using your analysis to make data in forms, not data driven, but data informed, pedagogical decisions. 18 00:03:49.390 --> 00:04:10.910 Lauren Dana: Basically, you'll be asking yourself, how can the data course plus collects help improve your experience and the experience of your students. And our goal today is to help you find and interpret lms data as well as student evaluations and provide some practical suggestions on how this analysis can guide both course management and course redesign. 19 00:04:12.490 --> 00:04:28.920 Lauren Dana: Hey, we're going to first discuss the quantitative analytics available to you through course, plus, as Amy mentioned, earlier, similar analytics are available and all Lms is. But since Bsp uses course plus, we're going to use course plus as the example Lms. In this presentation 20 00:04:29.040 --> 00:04:41.429 Lauren Dana: to access these quantitative analytics, you will first go to faculty tools on the right column you will see a section labeled administrative tools and the quantitative data we will be working with is under student activity reports. 21 00:04:41.530 --> 00:04:51.929 Lauren Dana: You will see that course, plus offers data on lecture, content, access, live, talk attendance. If your course is fully virtual discussion, forum and site access. 22 00:04:52.060 --> 00:05:01.220 Lauren Dana: we will now give a brief snapshot of these different data points, as well as some of our suggestions on how to use this data for course management and course redesign. 23 00:05:01.350 --> 00:05:14.250 Lauren Dana: All of the data points that we will go through are available to download in Pdf or Excel format. and we'll be showing screenshots of data pages from our course for teaching assistance. But all student names have been removed. 24 00:05:14.350 --> 00:05:18.419 Lauren Dana: I'm going to pass it over, Amy. To begin with our first data point. 25 00:05:22.120 --> 00:05:28.649 BSPH CTL Teaching Toolkit: Sorry I forgot I was muted. Thank you, Lauren. so our first data point will be site access data. 26 00:05:28.670 --> 00:05:57.709 BSPH CTL Teaching Toolkit: And if so, if your course is an island, then your course, your site. Access reports are like the flight logs of who is coming and going from your course site. And this data can be very helpful for both both course, management and course redesign because it overviews how students are accessing and when they're interacting with your course site by showing the dates and times of individual students, first and last visits to the course site as well as their total number of visits overall. 27 00:06:00.200 --> 00:06:24.220 BSPH CTL Teaching Toolkit: So for course, management we recommend that faculty check their their course site activity reports. to make sure that all students have logged in by the end of the first week of an academic term. So, for example, if you're a teaching a summer Institute course that started this week. Then I would recommend that you check your student site. Access reports. no later than Monday of next week to see if there are any students who haven't logged in yet. 28 00:06:24.710 --> 00:06:31.040 BSPH CTL Teaching Toolkit: And that way you can kind of keep it. Keep a gauge on how your students are interacting with the course site 29 00:06:31.630 --> 00:06:41.859 BSPH CTL Teaching Toolkit: reviewing course site access during the term, like throughout the term, may also allow you to identify students who, might be infrequently logging in, and might 30 00:06:41.960 --> 00:06:46.579 BSPH CTL Teaching Toolkit: be at risk of falling behind and need some additional contact or support. 31 00:06:47.010 --> 00:06:56.490 BSPH CTL Teaching Toolkit: so when you're using this side access data, some questions that you can ask yourself are are there any students who have not logged in after the first week of the term? 32 00:06:56.640 --> 00:07:01.329 BSPH CTL Teaching Toolkit: Are there any students who have not locked, who have logged in infrequently throughout the term? 33 00:07:01.450 --> 00:07:07.689 BSPH CTL Teaching Toolkit: And are there any students who are struggling with parts of the course and perhaps side access can help 34 00:07:07.750 --> 00:07:10.900 BSPH CTL Teaching Toolkit: provide some additional context if the students are struggling. 35 00:07:12.360 --> 00:07:19.389 BSPH CTL Teaching Toolkit: So that's how you can use that that data for course management. Now let's look at how you can use this data for course redesign. 36 00:07:19.990 --> 00:07:37.190 BSPH CTL Teaching Toolkit: So for course, redesign, you might notice that the pacing or workload of your course, is uneven. If your students are spending too much or too little time in the course site throughout the term, so there might be uneven distribution across the term and how students are accessing the course 37 00:07:37.290 --> 00:07:48.729 BSPH CTL Teaching Toolkit: you may also want to review. If there's increased course site access. if that's connected to higher student achievement or vice versa. So is there a correlation between site, access and student achievement. 38 00:07:48.900 --> 00:07:50.850 BSPH CTL Teaching Toolkit: And then, finally, 39 00:07:51.060 --> 00:08:13.359 BSPH CTL Teaching Toolkit: I did want to mention that if you are looking at site access that doesn't include anything that the students are doing outside of the course website. So that that doesn't include time spent on reading materials or time spent on downloaded lectures, working on projects or papers and other applications. So it's just that access to course, plus 40 00:08:13.940 --> 00:08:33.689 BSPH CTL Teaching Toolkit: And this is just one piece of information that can help you inform your class management and redesign. So when using this data for course, redesign. Some questions to consider are, again, does engagement with the course increase, decrease, or main consistent, and is higher or lower interaction with the course site reflected or correlated with student achievement. 41 00:08:35.900 --> 00:08:52.979 Lauren Dana: All right, I'm Gonna pass things back to Lauren. Thanks, Amy. We are now going to move further away from shore. As we explore the different analytics available regarding lecture and content access, and then discuss ways that you can use this data when both your course is running. And when you're reflecting upon your most recent offering 42 00:08:53.980 --> 00:09:07.690 Lauren Dana: before we begin to discuss the data available on lecture and content access, I just want to clarify that course, plus can only collect data on lecture materials that are in the lecture materials box on the course plus lecture page. 43 00:09:07.890 --> 00:09:22.299 Lauren Dana: While Ctl does encourage you to use our production services. You can request to have self recorded materials that adhere to accessibility standards to be uploaded to the lecture materials box. So you, too, can use this data that we are about to explore. 44 00:09:22.730 --> 00:09:34.379 Lauren Dana: Chris, plus first offers a variety of data on lecture, access and views, as you can see in the image. First, you can access data on individual lecture sections where you'll see how often a student has viewed each section. 45 00:09:34.620 --> 00:09:46.260 Lauren Dana: if they downloaded the Pdf. M. P. 3, M. P. 4, and transcripts of each lecture, and finally, there's a timestamp for the first and last. Viewing of the lecture by each student. 46 00:09:51.140 --> 00:10:01.610 Lauren Dana: You can then view cumulative access for lecture materials where you can see the total views per each lecture section in the course. And this data is shown in both a bar graph and a chart. 47 00:10:03.530 --> 00:10:23.639 Lauren Dana: Hey? Then you can then access requests for lecture materials by day of the week, by the hour of the day, and your top 10 busiest hours. The data is shown in both the bar graph as well as a chart, and you'll see the total amount of times each lecture section has been viewed just as a reminder. This is when the lectures are viewed within the lecture box, not downloaded. 48 00:10:25.400 --> 00:10:30.290 Lauren Dana: Hey? Well, having all this data on your lectures is nice. What do you do with it as a faculty? 49 00:10:30.380 --> 00:10:43.639 Lauren Dana: We're going to go over a few questions to ask, as you analyze your own data for course management. You'll want to see if struggling students are viewing all the lectures. If not, this might be a good discussion point to raise with a struggling student. 50 00:10:43.760 --> 00:11:05.350 Lauren Dana: If you notice any lectures that have a high rewatch number, this may indicate that students need extra support with the content covered in this lecture. Once your course has ended, you can also use this data to help inform any changes you might want to make in your course. The first connection to make is to see if there is any relationship between accessing lectures and students achievement. 51 00:11:05.530 --> 00:11:13.639 Lauren Dana: If the students who are watching the lectures are doing well on your assessments, then your lecture, objectives and assessments are most likely well aligned. 52 00:11:13.690 --> 00:11:28.210 Lauren Dana: If students are watching the lectures, but still doing portly on the assessments, or vice versa. You may want to examine. If the assessments really embody the learning that occurs in the lectures, or determine if your lectures still meet the course, objectives. 53 00:11:28.560 --> 00:11:33.620 Lauren Dana: You can also examine similarities with lecture content, that is, or not being accessed. 54 00:11:33.680 --> 00:11:48.600 Lauren Dana: You may notice that students really like watching your videos, or if that a lecture is over 60 min long. Students are not watching all the sections of a lecture, or that your Ctl. Recorded lectures are watched more than self reports. These are all things to consider. 55 00:11:48.730 --> 00:12:06.489 Lauren Dana: Finally, you can view when students are watching the lectures in comparison to other due dates. If students seem like they're cramming. You may want to consider opening the lectures up earlier to allow for better time management, or add more formative assessments throughout the lectures to help encourage time, management and knowledge retention. 56 00:12:08.880 --> 00:12:18.140 Lauren Dana: In addition to lecture access data, you can view how often students access individual course pages and activities and course plus, as you can see in this screenshot. 57 00:12:18.550 --> 00:12:33.219 Lauren Dana: This, too, can help with course management and course redesign. Similarly to lecture access. If you have a student who is struggling, you can see if this student is not interacting with important course resources and activities. And again, this could be a starting point in your discussion. 58 00:12:33.380 --> 00:13:00.900 Lauren Dana: For course, redesign. You will want to begin by taking stock of which core resources are most viewed, and see if there's any connections between formats of content and views. For example, you may find your students really enjoyed working on your wiki, but they seem less engaged in the weekly discussion Forum, similar to lectures. You want to see if student achievement is tied to accessing course pages and activities. If not, you will need to go back and review your objective alignment. And these resources and activities. 59 00:13:01.050 --> 00:13:15.230 Lauren Dana: Finally, when students are accessing different, when students are accessing different activities and assessment can help you with course pacing, you may consider breaking up a project into parts. If students are just starting to access a big assessment the day before it's due. 60 00:13:15.440 --> 00:13:28.270 Lauren Dana: I do just want to remind you again that course, plus cannot provide data on any work students are doing outside, of course, plus. But all of this data that I just reviewed can really help you revise and improve your course. Content. Outside of lectures. 61 00:13:29.160 --> 00:13:43.240 BSPH CTL Teaching Toolkit: I'm not gonna pass it over to Amy for online library data. Thanks, Lauren. So, I wanted to briefly mention that the online library captures data in a similar way to the recorded lecture pages that Lauren mentioned. 62 00:13:43.240 --> 00:14:03.199 BSPH CTL Teaching Toolkit: the online library data can be helpful for in person classes that don't have recorded lectures to track, but still want to know if students are engaging with the Content posted on the course site. so it it won't show that a student actually read the file, but it will show that they accessed it through the course online library tool. 63 00:14:03.610 --> 00:14:22.909 BSPH CTL Teaching Toolkit: and you can view that information about how many times an individual file or link work clicked as well as specific students who have clicked on each file posted in the online library. So, for example, you can see if all of your students have read or access something that has been marked required in the online library. 64 00:14:26.020 --> 00:14:47.220 BSPH CTL Teaching Toolkit: So I also wanted to mention live talk data. if you teach an online course. So not a blended hybrid or face to face in person. Course? Then you might use live talks in your course, and these are live synchronous zoom sessions that take place in a again a dot, 8, one or dot 8, 6 course an online course. 65 00:14:47.340 --> 00:14:56.669 BSPH CTL Teaching Toolkit: then, if you do, and you use the live talk page to set up your live talks, then you'll have some data available to you. 66 00:14:57.210 --> 00:15:24.809 BSPH CTL Teaching Toolkit: So for example, you can view which students attended your live talk session live by clicking on the button on the live talk page, or you can see which students downloaded your live talks as the recording file and when they downloaded the recording this information can be used to track attendance as part of a participation grade, for example, or it can also help you determine what topics or live synchronous activities students like to engage with for future live talks. 67 00:15:27.420 --> 00:15:33.890 BSPH CTL Teaching Toolkit: All right. So now I'm going to pivot a little bit and talk about some of the discussion forum data that's available. 68 00:15:35.810 --> 00:15:53.029 BSPH CTL Teaching Toolkit: So you can access the discussion Forum activity Report, which will be which is broken down by student name, their last post, their total postings, and the total size of their posting. So the character count topic started and the posts reacted to. 69 00:15:53.160 --> 00:16:06.859 BSPH CTL Teaching Toolkit: and you can view. You can also view the total number of visits that each student has made to the discussion Forum, and you can see that see their first visit, their most recent visit, and the total number of visits that they've made to the discussion. Forum 70 00:16:08.470 --> 00:16:38.400 BSPH CTL Teaching Toolkit: You can also access each discussion from category and topic response report. So here you can see the total number of how many students responded to each topic within a category as well as how many students did not respond. So if you select an individual category or topic, then you'll see the actual names of the students who did and did not respond to that category in the topic. And that can help you identify if students need to. if students are participating, or if you need to reach out to a student for not participating. 71 00:16:39.310 --> 00:16:58.400 BSPH CTL Teaching Toolkit: So while what does this data mean for course management? for again, the analytics make it really easy for you or your Tas to determine students participation in the discussion. Forum. So so, for example, if you're using the discussion from participation as a a form of grading participation. 72 00:16:58.400 --> 00:17:14.129 BSPH CTL Teaching Toolkit: then you can easily see who's participating a quick note. If you're going to use this data to grade students. then I recommend that you set those expectations with students in your syllabus, and at the beginning of the course that they know that that's how they're going to be graded. 73 00:17:14.750 --> 00:17:36.730 BSPH CTL Teaching Toolkit: you can also get a feel for student engagement and interest by seeing if there are certain discussion topics that are more popular than others. So, for example, you might find that discussion, forums that are centered around current topics might be more popular, have higher student engagement than, say, a discussion forum on a topic, on a theory or reading 74 00:17:37.010 --> 00:17:51.859 BSPH CTL Teaching Toolkit: and then in that way you can perhaps alter future discussion, form topics to match with students interest. students may also be more engaged in posts where there's a high faculty or ta interaction with students, especially in online courses. 75 00:17:52.210 --> 00:18:04.600 BSPH CTL Teaching Toolkit: So when using this data again to help manage your course, you can ask yourself these questions. how does individual student postings, the frequency in length match with your grading requirements and expectations. 76 00:18:04.640 --> 00:18:14.859 BSPH CTL Teaching Toolkit: And are there any discussion forum posts that seem more popular than others that you could maybe, use in increasing engagement in future offerings? 77 00:18:17.180 --> 00:18:29.010 BSPH CTL Teaching Toolkit: Similarly, for course, redesign. you can go back and analyze your discussion from data through to more thoroughly to look at the frequency and quality of your discussion forum posts. 78 00:18:29.030 --> 00:18:42.219 BSPH CTL Teaching Toolkit: So, for example, you can analyze the discussion forums that have high engagement and see if you can replicate that in other posts you can also analyze low performing discussion forums, and either remove or rework them to improve their engagement. 79 00:18:42.580 --> 00:18:53.490 BSPH CTL Teaching Toolkit: And again, a question to ask yourself is, Does the frequency or quality of the discussion. Posts vary across topics or modules to help you identify that sweet spot where you have student engagement. 80 00:18:56.120 --> 00:19:04.749 BSPH CTL Teaching Toolkit: So far, we've looked at a lot of quantitative data. So now we're going to pivot and move on to qualitative data analytics that are available through course, plus 81 00:19:06.960 --> 00:19:14.500 BSPH CTL Teaching Toolkit: so 2 sources of qualitative data that we'll be looking at today are emails and student course evaluations. 82 00:19:15.990 --> 00:19:38.300 BSPH CTL Teaching Toolkit: So emails as a data source, the communications that you have with your students are a qualitative data source that you can review to help you. sort of check the pulse of your course. you can review emails that you send to students by using the email archive feature in course, plus And note that this only tracks emails that you send to students 83 00:19:38.300 --> 00:19:52.749 BSPH CTL Teaching Toolkit: in which you when you select the email archive setting in your in your course email tool, this won't track emails that are sent outside, of course, plus or emails that aren't saved to the archive. 84 00:19:53.090 --> 00:20:13.720 BSPH CTL Teaching Toolkit: you can also consider gathering information from any emails that you receive from students like, frequently asked questions or reported issues. So, for example, if a particular lecture has a frequently asked question associated with it, then you can post the Q. A. On the lecture page, or address it in the course for future offerings. 85 00:20:13.910 --> 00:20:24.180 BSPH CTL Teaching Toolkit: If students report an issue with an assignment like unclear instructions, or perhaps an external link is broken. then that's also something that you can go in and and fix right away 86 00:20:25.060 --> 00:20:38.769 BSPH CTL Teaching Toolkit: so things to look for. again. You should look for trends and email sent to students, frequently asked questions that you receive from students and reported issues that you receive from students to look for patterns or trends. 87 00:20:41.370 --> 00:20:45.999 BSPH CTL Teaching Toolkit: And now let's talk briefly about student course evaluations 88 00:20:46.240 --> 00:21:08.769 BSPH CTL Teaching Toolkit: or I'll I'll call them Eval for short. So Emails are vital to the learning process for both students and faculty. but it can be difficult to analyze the qualitative, open-ended comments from students. so I wanted to show you a method for analyzing student responses that are open ended, or those qualitative responses. 89 00:21:09.250 --> 00:21:28.970 BSPH CTL Teaching Toolkit: So first, you're going to sort your feedback from students by type, and the types are I'll go through each type. So comments that are unrelated to teaching and learning, or that are non-specific. So they're vague. those can be discarded because they're not not going to help you. improve your course. 90 00:21:29.360 --> 00:21:52.739 BSPH CTL Teaching Toolkit: Next, you would consider your positive comments. So these comments tell you what students think that you or your course are working well. and first enjoyed the positive comments, and then compare them to look for common themes. and you can also compare and contrast them with less positive comments to look for areas of agreement and disagreement among your students. 91 00:21:53.640 --> 00:22:19.990 BSPH CTL Teaching Toolkit: then consider any actionable suggestions from students So these are comments that shed light on a pain point in the course or offer a specific change that you could make within your course. that you can consider, and for that you should look for similar suggestions across students, and then consider any trade offs associated with making that change. So, for example, students might have a really good idea for something. 92 00:22:19.990 --> 00:22:34.590 BSPH CTL Teaching Toolkit: but the trade off would be higher administrative time or more. you know, a level of work or level of effort. So there might be reasons to accept student suggestions or not accept student suggestions. 93 00:22:35.150 --> 00:23:03.099 BSPH CTL Teaching Toolkit: And then finally, look at your non actionable suggestions. And these are these are suggestions that shed light on pain points. But to things that you don't have control over within the context of your course. So these might be Us. School level, like policy suggestions or changes to the way that course plus friends, so things that you wouldn't have control over. But what you could do is you can consider passing these comments along to individuals who could 94 00:23:03.330 --> 00:23:22.740 BSPH CTL Teaching Toolkit: potentially make that change? Or could you find that information useful? So, for example, you could send that to your academic or program coordinator classroom facilities or Ctl. so again, this is a method that you can use to systematically organize and respond to your student evaluations that are the qualitative student responses. 95 00:23:23.000 --> 00:23:30.180 BSPH CTL Teaching Toolkit: we could spend an entire workshop just on student course evaluations. And Lauren and I felt that 96 00:23:30.250 --> 00:23:43.040 BSPH CTL Teaching Toolkit: that this deserved its own workshop. So keep an eye out in the future for another workshop, just covering student course evaluations and how to analyze and apply that data. And that will be happening later this year. 97 00:23:45.070 --> 00:23:53.929 BSPH CTL Teaching Toolkit: All right. So I'm gonna quickly pause here and ask, are there any questions before we move on? you can post them in the chat, or you can raise your hand. 98 00:23:57.610 --> 00:24:22.510 Marie Diener-West: Amy. Thank you. This was really really helpful, and I'm really glad to be here. I do have a question, though, about you know, in terms of some of the quantitative analytics. When you have a very large course, you know, or one that just has, you know, more students, then it's easy to summarize at a glance is there, you know. Is there a summary measure of sort of the average, you know time of? First look, or you know the average length. 99 00:24:23.200 --> 00:24:39.599 Marie Diener-West: Just just curious. If there was any kind of summary statistic associated with the long listings of students. 100 00:24:39.600 --> 00:24:57.539 BSPH CTL Teaching Toolkit: What you're like, for example, if you want to contact all students who haven't access the course site within the first week. For as an example, if you're using the course email tool, you can select that as a criteria. And then course Plus will automatically bring up the students who you would need to email for that. 101 00:24:57.540 --> 00:25:06.970 BSPH CTL Teaching Toolkit: So that's one way that you could. You could use course, plus to kind of get around that. But as far as I know, I don't think there's like an average report for each 102 00:25:07.280 --> 00:25:16.410 Marie Diener-West: for each of the activity reports. But that tip about, you know, being able to select those from the The access is great. So thanks a lot. 103 00:25:16.410 --> 00:25:37.489 BSPH CTL Teaching Toolkit: of course, and then for the lectures as well, you could get an over like an at like. How many times a certain lecture has been accessed again, the data could be skewed at. One student could be viewing it a hundred times and one student 0. But I think it gives a pretty good snapshot, so it's easy to see. For example, Lecture 5 has been viewed significantly more than 104 00:25:37.490 --> 00:25:52.509 Lauren Dana: lecture one same with the course the activity pages. so it gives you, I think, a bigger holistic idea. Maybe not that nitty, gritty ability to to dig deeper. But I think you can get an overall view of it. 105 00:25:52.610 --> 00:26:04.069 BSPH CTL Teaching Toolkit: Yeah, no great ideas, thanks. And then I see that there's a question that came in in the chat, which is, could you review where we go to see the online library course, analytics and course, plus 106 00:26:04.070 --> 00:26:23.709 BSPH CTL Teaching Toolkit: for that, you'll go to the the online library page and you'll click on, manage the online library, and then there should be a button that says like, show tracking and if you click that you'll see each of the the the reports that are available for the online library. looks like 2 more race, their hands 107 00:26:24.440 --> 00:26:27.510 BSPH CTL Teaching Toolkit: not sure who raised their hands first, though. 108 00:26:28.970 --> 00:26:31.860 BSPH CTL Teaching Toolkit: Oh, let's see. 109 00:26:32.250 --> 00:26:37.810 BSPH CTL Teaching Toolkit: is it Mika? Or Misha? Yeah, yeah, maybe, Boris, you had an answer to 110 00:26:37.830 --> 00:26:41.689 Mija Ververs: to some of the people or not. You might want to go first. 111 00:26:41.730 --> 00:27:01.270 Mija Ververs: No, it's okay. Go ahead with your question. Then I can add a little bit. Thank you. 112 00:27:01.520 --> 00:27:07.259 Mija Ververs: we can trace whether they watch it or not, and some are not listed at all. What do we do wrong 113 00:27:09.500 --> 00:27:10.700 BSPH CTL Teaching Toolkit: for 114 00:27:10.770 --> 00:27:38.990 BSPH CTL Teaching Toolkit: so so for the lecture activity some lectures are are being tracked, but others are not. Is that is that correct? Okay, So the most likely reason for that would be If you're using some lectures are using our studio detail studio produced so there would be in the they would be in the lecture materials box versus something like a self recorded lecture. that's not in the lecture materials box. That would be one reason why it wouldn't be tracked 115 00:27:39.450 --> 00:27:48.940 BSPH CTL Teaching Toolkit: But if that's not the case, I would think I'd have to look at your specific course site to see why it wouldn't track something. But that's the that would be the most the most 116 00:27:49.150 --> 00:28:14.149 Mija Ververs: likely answer is that it might be a self recording And that's why it's not being tracked. 117 00:28:14.150 --> 00:28:28.989 Mija Ververs: is there a navigation video done by Ctl or new to's course, plus for people who do not have that habit being a student at Hopkins, because it's hard for them. 118 00:28:28.990 --> 00:28:52.559 BSPH CTL Teaching Toolkit: so especially. yeah, that? That's a good question. So the first thing that came to mind was the introduction to online learning course which students take to introduce them to course plus. And then also, there's a course plus guide specifically for students. available through Ctl help. And I can share the link to that in the chat. 119 00:28:52.560 --> 00:29:03.399 BSPH CTL Teaching Toolkit: There's the faculty guide, but then there's also a student guide. So here, let me pull up the link to that, and I'll put it in the chat. but you can share that link with students to 120 00:29:03.880 --> 00:29:21.069 BSPH CTL Teaching Toolkit: who are having technical issues. And of course, if students have any questions about course, plus or how to use one of the tools. they can also contact detail help which is available at the same place where I just posted the chat the link to in the chat. 121 00:29:21.260 --> 00:29:33.219 BSPH CTL Teaching Toolkit: They can also work when when Ctl. Will do our best to work with them to help get them you either addressed. If they have a question, or if they're having some kind of technical issue, we can help them navigate that. 122 00:29:33.260 --> 00:29:36.350 Mija Ververs: Okay, great. And and the video is for 123 00:29:36.380 --> 00:29:40.740 Mija Ververs: at the let's say, course, plus for dummies. Kind of 124 00:29:40.880 --> 00:30:01.969 BSPH CTL Teaching Toolkit: a little bit. Yeah, yeah, and again, like the guides, the guides are helpful. But if you have questions about like a specific tool or need even more guidance. Then. if you contact detail help, they'll connect you with this, an instructional designer, and we can work one on one with you to get you up to speed and get you the resources that you need. Thank you. Thank you. 125 00:30:01.980 --> 00:30:09.330 BSPH CTL Teaching Toolkit: Yeah, thank you. And then, I think, Boris, did you have a comment, and otherwise I don't see any other hands raised. 126 00:30:11.930 --> 00:30:13.430 Boris Buia: Yeah, 127 00:30:14.490 --> 00:30:32.249 Boris Buia: quickly So Maddie did that there is a way to download an excel. 5 of these reports, for to track for students that are watching in terms of lectures, and probably the Excel file is the richest form of documents you can generate. I can share the external size from last year. 128 00:30:32.800 --> 00:30:37.550 Boris Buia: We don't have average, but we have total numbers. And they 129 00:30:37.610 --> 00:30:38.780 Boris Buia: I'm not gonna 130 00:30:38.800 --> 00:30:54.870 Boris Buia: say that you don't have in your team someone we can generate average. But we can generate obviously for you if you need to, and so I can share the excel 5 from last year. And may I? If you have questions about some of the polls courses that I'm working with, please just reach out and we'll 131 00:30:54.960 --> 00:30:56.639 Boris Buia: address this issue 132 00:30:58.210 --> 00:31:11.350 Marie Diener-West: for us. Thank you for the response about the Excel file. I mean, in the okay, I came here to sort of learn what I'm not doing. And so this is, this is great. Thanks so much. Thank you for us. That was great. 133 00:31:12.540 --> 00:31:17.220 BSPH CTL Teaching Toolkit: all right. And then, if there are no other questions. Then we'll move on. 134 00:31:17.800 --> 00:31:46.390 BSPH CTL Teaching Toolkit: So, assuming there are no other questions. So we're actually going to do a quick, a a short scenario based activity. So while Lauren goes over the instructions, I'm going to create some breakout rooms. Great, thank you, Amy, all right. So now we're actually gonna try to put to practice and apply everything that Amy and I have gone through on a very small basis. So what we're hoping is for you to practice using the Lms analytics for both course management and course redesign. 135 00:31:46.390 --> 00:32:11.570 Lauren Dana: So Amy and I have created 3 role playing scenarios. in which you will be reviewing Mls. Analytics to inform some course management and course redesigned decisions. so before we begin, I just wanted to review with the next few minutes will entail. So Amy and I will present a slide that has a scenario on it. And you will either discuss this scenario in this large group, or we'll put you in small breakout rooms. 136 00:32:11.570 --> 00:32:20.850 Lauren Dana: and then we're hoping some of you will be brave enough to share out to the group group via zoom, or even be a chat, just some of the thoughts you had. because we learn from you as well. So 137 00:32:20.960 --> 00:32:41.219 BSPH CTL Teaching Toolkit: I'm gonna pass it over to Amy to review our first scenario. Yes, thank you, Lauren. So for our first scenario, we're gonna stay in our main group together. just so we can kind of go through one scenario as a as a whole group, and then for the other 2 scenarios that we have, we'll break into small groups. So. so here's our first scenario. 138 00:32:41.220 --> 00:32:56.340 BSPH CTL Teaching Toolkit: it is. So. Imagine that you're teaching an in person course. So it is a week. 4 of the academic term. You're teaching an in person course, and you notice that students are missing deadlines that are posted on the course content page 139 00:32:56.340 --> 00:33:14.699 BSPH CTL Teaching Toolkit: so you can raise your hand or post in the chat. what course plus data might you seek out to sort of find out what is going on here. So again, you can raise your hand or post in the chat. what course plus data might you seek out next? When you find that students are missing deadlines. 140 00:33:22.860 --> 00:33:24.320 BSPH CTL Teaching Toolkit: any ideas? 141 00:33:30.880 --> 00:33:57.650 BSPH CTL Teaching Toolkit: All right. Well, we have 2 ideas in the chat, Amy. So we have. how many which students have downloaded the syllabus course, site access and access to course pages as they have to take a look at. Yeah, those are all really great suggestions. And actually, that leads us to our next slide. So So, after looking at the student site, activity report, so course side access, which Marie Dean or West mentioned in the chat. 142 00:33:57.650 --> 00:34:21.869 BSPH CTL Teaching Toolkit: you realize that students are not logging into the course site. So they're not seeing the important dates that are posted on your content page. So, knowing that students are not logging into the course site. And remember, this is a face to face in person course, how would you address this? So again, you can raise your hand or post in the chat. How would you address your in person, students not logging into the course site? 143 00:34:30.429 --> 00:34:31.900 BSPH CTL Teaching Toolkit: Any ideas? 144 00:34:35.210 --> 00:34:42.829 BSPH CTL Teaching Toolkit: All right? Well, I have 2 recommendations off Yup, we have send an email to them. That's a great idea. So remind them an email. 145 00:34:43.219 --> 00:35:02.340 BSPH CTL Teaching Toolkit: And since it's a face to face. Course. sometimes students forget that they need to go and return to the course site. So sending an email to them or giving them reminders during the live class in person. Class sessions can be helpful for them to again. Remind them that they need to check the course site. and 146 00:35:02.340 --> 00:35:27.060 BSPH CTL Teaching Toolkit: if you are using the schedule builder tool for your content page versus the class sessions tool. you can also set due dates that automatically remind students about important assignments and and deadlines coming up. So again, if you're using the schedule builder tool, you can set due dates on the schedule, on the Content page itself. And that will automatically remind students that deadlines are coming up 147 00:35:27.940 --> 00:35:39.429 BSPH CTL Teaching Toolkit: all right. So now that we've done one scenario together. Now we're going to look at another scenario. and this time we'll break out into small groups. But first let's see what the scenario is. So this is our second scenario. 148 00:35:39.870 --> 00:35:48.129 BSPH CTL Teaching Toolkit: You're teaching an online course. And you notice that many students are emailing you. And you're feeling overwhelmed by the volume of emails. 149 00:35:48.160 --> 00:35:52.969 BSPH CTL Teaching Toolkit: So what might you do with the qualitative data that you pull from the emails 150 00:35:53.340 --> 00:36:05.919 BSPH CTL Teaching Toolkit: and what we're gonna do is we're gonna break into small groups. And you can think about what? What would you do with the qualitative email or the qualitative data that you get from your emails? So you should see 151 00:36:06.600 --> 00:36:07.670 BSPH CTL Teaching Toolkit: a 152 00:36:08.540 --> 00:36:10.910 BSPH CTL Teaching Toolkit: breakout room button 153 00:36:11.440 --> 00:36:17.094 BSPH CTL Teaching Toolkit: and break into your small groups. And I'm going to put the scenario details in the chat. so you don't have to write down anything. 154 00:36:25.164 --> 00:36:28.073 BSPH CTL Teaching Toolkit: All right. It looks like most people are back. 155 00:36:31.884 --> 00:36:38.143 BSPH CTL Teaching Toolkit: All right. So Do we have any volunteers to share what they how they address the scenario. 156 00:36:40.064 --> 00:36:42.884 BSPH CTL Teaching Toolkit: and you can raise your hand or post in the chat. 157 00:36:51.594 --> 00:36:56.533 BSPH CTL Teaching Toolkit: all right, and that no volunteers. Oh, okay, we got a volunteer Kathy. 158 00:36:57.434 --> 00:37:06.913 Kathy Gresh: Yes, yes, I was holding out to our group 1 one really good idea was end of the week. email, 159 00:37:07.024 --> 00:37:17.324 Kathy Gresh: covering some of those points. Another is having a policy in the course of always posting your questions. Discussion forum so everyone can benefit from them. 160 00:37:17.654 --> 00:37:22.653 Kathy Gresh: maybe making an FAQ for next year if it's still pertinent. 161 00:37:23.154 --> 00:37:30.324 BSPH CTL Teaching Toolkit: Oh, very good. Those are. Those are all really great recommendations. oh, we have another hand raised. 162 00:37:30.514 --> 00:37:31.434 BSPH CTL Teaching Toolkit: Go ahead. 163 00:37:33.874 --> 00:37:36.653 BSPH CTL Teaching Toolkit: Oh, they just lowered their hands. So maybe not. 164 00:37:37.764 --> 00:37:42.213 BSPH CTL Teaching Toolkit: But yeah, those are all really great suggestions. So thinking about looking for 165 00:37:42.244 --> 00:37:52.033 BSPH CTL Teaching Toolkit: trends and patterns in the emails that you receive, and then also like thinking ahead, like maybe having a course. Q. A. Or 166 00:37:52.544 --> 00:38:01.314 BSPH CTL Teaching Toolkit: even a FAQ later on. So so, and and again, like some recommendations that that 167 00:38:01.314 --> 00:38:28.754 BSPH CTL Teaching Toolkit: Lauren and I came up with for the scenario since there's 2 issues. One was the volume of email. So again, our first recommendation is to address the volume of email. So we also came up with create a course. Q. A. Discussion forum where students can post their questions about course content and assignments. And this is akin to having students raise their hand and ask a question in Class and the benefit there is that all students get to see the questions that were asked, and then the answers that are posted. 168 00:38:28.904 --> 00:38:45.514 BSPH CTL Teaching Toolkit: and then, The second thing is, note any trends, or frequently asked questions that can then be proactively answered for the next offering. So things like clarifying ambiguous assignment instructions, or even creating a a course FAQ page that addresses some of those frequently asked questions. 169 00:38:45.974 --> 00:38:50.153 BSPH CTL Teaching Toolkit: All right. So now I'm going to pass it to Lauren for our next scenario. 170 00:38:52.624 --> 00:39:06.324 Lauren Dana: Great, thank you, Amy. all right. So now we're going to take a focus into course redesign. So after your course for the first term has finished, you decide to review the average quiz score for each 171 00:39:06.434 --> 00:39:22.893 Lauren Dana: average put score for each quiz. You notice that students are doing very poorly on the first quiz. The first quiz is based on lecture one. So before I break you into breakout rooms with a larger question. We're going to start with the question of what course, plus data might you seek out next? 172 00:39:23.624 --> 00:39:40.903 Lauren Dana: and we are going to for the purposes of this scenario. Avoid the data and quiz generator. That's a whole. Another Ctl workshop, because there's so much in there. So just using course plus data, not in quiz generator. What might be some data that you seek out. 173 00:39:41.474 --> 00:39:44.074 Lauren Dana: Excellent. Did they access? Lecture one 174 00:39:46.174 --> 00:39:49.234 Lauren Dana: anything else you might just look at in general, in addition to data. 175 00:39:52.884 --> 00:40:01.543 Lauren Dana: Yeah, you might make sure they're taking the quizzes. I'm looking at objectives. But definitely, I think, looking at the access. so head to the next slide. 176 00:40:03.804 --> 00:40:13.693 Lauren Dana: Okay, so bad news, our worst fear. You realize that almost none of your students are watching lecture one. So that access is incredibly low. Maybe some of the sections 177 00:40:13.714 --> 00:40:37.958 Lauren Dana: and unfortunately, you did run out of time last term, and you updated lecture one with a self record, and since you were press for time it was not broken into section. So the lecture is 60 min long. Continuous So I'm gonna put you in your small groups in just a minute. But the question I want you to think about are, what are some ways, you might redesign your course to help improve learning, acquisition and retention, as reflected in quiz one. 178 00:40:38.476 --> 00:40:54.085 BSPH CTL Teaching Toolkit: All right. It looks like most participants are back in the main room. So, Lauren, I'll hand it back. I'll pass it back to you. Great. Would anyone like to share some ideas that their group came up with to help improve the quiz grades on lecture one as well as lecture access. 179 00:40:54.486 --> 00:40:59.565 Lauren Dana: either in the chat or yeah via zoom. 180 00:41:00.026 --> 00:41:21.725 Marie Diener-West: Well, I guess go for it. So you know, we did talk about the the fact that breaking that lecture from 60 min into smaller chunks would be one actionable item. You had indicated that we couldn't use the quiz generator, but but if we had practice quizzes it would be nice to go back and see. You know, if they had access practice quizzes. 181 00:41:22.166 --> 00:41:31.955 Marie Diener-West: awesome. Those are 2 great suggestions. Yeah. So I think a low hanging fruit is definitely to break the lecture up into different manageable sections. 182 00:41:31.966 --> 00:41:39.066 Lauren Dana: definitely, if there's any practice quizzes and take a look if they are working or using that, or what questions? They're getting wrong. 183 00:41:40.426 --> 00:41:42.456 Lauren Dana: Good. Any other thoughts for this one. 184 00:41:44.736 --> 00:41:54.136 Kathy Gresh: Sometimes you have to reevaluate the lecture and against the quiz questions and see where they actually covered, and the clarity of the questions as well. 185 00:41:54.676 --> 00:42:14.946 Lauren Dana: Great thanks, Kathy, yeah. The quiz questions might be not meeting the objectives of the lecture. So maybe students are finding it not even to be helpful at all. So why watch the lecture? And so I think that's a a great point as well as you want to make sure the lecture actually meets the course objectives as well as the quiz questions and Carrie how to greet 186 00:42:15.196 --> 00:42:35.956 Lauren Dana: suggestion in the chat as well as we do offer in lecture quizzes. So, as the students are watching your lecture, it'll stop. Ask them a question. It could even be a question that you adapt for your quiz just to make sure that they are understanding and retaining knowledge as they go. so that would definitely be for a Ctl recorded lecture. You can add those in lecture quizzes 187 00:42:35.956 --> 00:42:46.856 Lauren Dana: great, or pull questions and live talks right? So you could help prep them for the quiz questions as well, maybe not obviously the exact same, but similar questions. So they're getting used to to working on it. 188 00:42:47.346 --> 00:42:56.195 Lauren Dana: Great. This is a great holistic approach, I think, to how data can help you. And you can use a lot of different data to kind of come up with some of these solutions. Excellent. 189 00:42:58.556 --> 00:43:20.595 BSPH CTL Teaching Toolkit: all right. So so now, for the sake of time, we're gonna hit. Skip ahead to our workshop evaluation. So please let us know how we did by completing this workshop. And then this workshop survey. It should only take a a few minutes. and we'll also post the survey link in the chat. 190 00:43:38.046 --> 00:43:47.836 BSPH CTL Teaching Toolkit: Alright, thank you, Lauren, for posting the link in the chat so you can either scan the QR code, or follow the link in the chat to let us know how we did. we'll give you about 5 min. 191 00:43:48.156 --> 00:43:50.476 BSPH CTL Teaching Toolkit: and then we'll conclude our session. 192 00:44:29.906 --> 00:44:45.625 BSPH CTL Teaching Toolkit: Oh, yes, Elizabeth just added a great point in the chat where she said, Keeping recorded mirror material up to date. Yes, that is very, very important and very integral for student engagements. Because students, I think they can tell when something is out of date 193 00:44:45.736 --> 00:44:49.626 BSPH CTL Teaching Toolkit: or has been around for a while, even if it's not necessarily out of date 194 00:44:50.316 --> 00:45:06.086 BSPH CTL Teaching Toolkit: so the recording will be shared with everyone, who registered in the zoom after the session in a follow up email along with some of the resources that we we went over today. and 195 00:45:06.126 --> 00:45:12.106 BSPH CTL Teaching Toolkit: the recording will probably also be posted on our toolkit website. If that's the case, we'll share the link to that as well. 196 00:45:14.416 --> 00:45:22.686 BSPH CTL Teaching Toolkit: Oh, we have another question. I'm also a student. And this came up in a student group discussion. Oh, about the recorded lectures I'm assuming. 197 00:45:23.546 --> 00:45:38.525 BSPH CTL Teaching Toolkit: all right. So I think it looks like everyone's pretty much done with the course, evaluate or the workshop evaluation. because, of course, we can't do a workshop on data without collecting our own data. So thank you again for completing that quick survey. 198 00:45:39.586 --> 00:45:50.705 BSPH CTL Teaching Toolkit: and then again, I just wanted to thank everyone for their time today. Lauren. And I hope that you found today's workshop valuable as you manage and redesign your courses. using your your course. Data. 199 00:45:50.706 --> 00:46:13.255 BSPH CTL Teaching Toolkit: Lms, data is a great resource for you and we want to end with this final pro tip. again, if you're using any kind of Lms data to grade. So, for example, a participation grade. then we highly recommend letting your students know exactly what data you're using to grade them and and how you're tracking them just to avoid any kind of misunderstandings. 200 00:46:13.476 --> 00:46:31.995 BSPH CTL Teaching Toolkit: if you'd like to know more about any of the Lms data reports that we went over today, or if you'd like to sit with an instructional designer from Ctl to go over the specific data for your course. then you can contact Ctl help and they'll connect you with with either Lauren or me or another id on our team. we're here to help you. 201 00:46:31.996 --> 00:46:53.846 BSPH CTL Teaching Toolkit: the Ctl help. Page also contains the faculty slash ta and student course, plus guides that we mentioned earlier. so you can share that resource with your students as well. And again, the links will be in the chat. I also wanted to mention our self paced course for faculty who want to learn more about course, management or 202 00:46:53.846 --> 00:47:04.666 design and redesign and other pedagogical topics. That course is called essentials, of course, design, development, and teaching at Bsp, and then the link for that will also be in the chat. 203 00:47:04.666 --> 00:47:27.456 BSPH CTL Teaching Toolkit: And then finally, you can stay up to date on other future workshops and Ctl events, and including the future workshop on Quiz, generator and student course evaluations that we mentioned in today's workshop. by visiting our events page. And again, the links are in the chat. So thank you so much for coming to our workshop today, and we hope that you found it a valuable experience. 204 00:47:27.456 --> 00:47:32.696 BSPH CTL Teaching Toolkit: So I'm gonna go ahead and stop sharing and stop the recording. 205 00:47:32.746 --> 00:47:33.466 Marie Diener-West: Thank you.