Section 6.2. Managing Workshops

6.2. Managing Workshops

Once you've set up your workshop, you will need to manage student submissions and evaluation. Fortunately, the workshop makes it easy to track student activity as it happens. You can also choose how much feedback you give to students at any given time.

6.2.1. Workshop Phases

The six tabs on the workshop screen represent the six phases of a workshop. We've just completed phase 1. Phases 2 through 4 allow students to interact with the workshop. Phase 2 allows students to upload their submissions and perform instructor and self-assessments but does not distribute submissions for peer review. Phase 3 allows students to do everything in phase 2, but also distributes submissions for peer review. Phase 4 allows only self- and peer-assessments.

You may not want to use all three phases. There are two strategies for managing workshop phases. The easiest, but not necessarily the best, is to use only phase 3 after you've set up the workshop in phase 1. This gives students maximum flexibility. They can upload their assignments at any point before the due date. The problem occurs when students begin to peer-review each other's work. If students are permitted to peer-review before all of the submissions are in, the submissions won't be distributed randomly. The students who submit and review early will see only other early submissions. Late submitters will see only late submissions. Students who submit too close to the deadline may not give their peers enough time to review their work.

The alternative strategy uses only phases 2 and 4. You can set an interim deadline for student submissions and only accept student work before this due date. After the submission deadline, you'll set the workshop to phase 4 and students will only be allowed to perform peer reviews. This strategy will help ensure an even distribution of student work for evaluation and ensure that everyone has sufficient time for review.

6.2.1.1. Student submissions

Once you've activated the Allow Student Submission tab, students can begin to interact with the workshop. If you've required students to assess instructor examples, they will need to complete that assessment before they can upload their own work.

The student view of the workshop is very different from your own. Students see a gradual process in which they must first evaluate the instructor examples. Once they've completed these, they will then see the interface to upload their own assignments. Only after they've uploaded their assignments can they access the link to perform self-assessments (see Figure 6-8).

Figure 6-8. Student view of a workshop

---

Warning: Student submissions do not immediately appear in the instructor's workshop interface. The workshop module depends on a process on the server that runs on a regular basis (called a cron job) and won't update the interface until the cron job is complete. Your system administrator can tell you how often the process runs and how often you can expect the workshop to be updated.

---

6.2.1.2. Student assessments

Your view of student activity takes place on the main workshop page. Below the six workshop process tabs, you'll see three links that will help you track the workshop shown in Figure 6-9:

Ungraded Assessments of Teacher Submissions

From here, can access student evaluations of your example submissions. You give each evaluation a grade and give the students feedback.

Ungraded Assessments of Student Submissions

Like the assessment of instructor examples, you can grade the students' peer-review evaluations.

Student Submissions for Assessment

Use this link to grade the students' submissions directly. Later, you can use your rating to influence the final grade for the workshop.

Figure 6-9. Student activity links

Each of these grading opportunities is optional. Which you choose depends on how you want to calculate the final grade for the workshop. Each of these grades can be used in calculating the final grade, which I'll discuss below.

When you assess a student assignment, you'll see the same scoring guide you set up in the beginning of the workshop (see Figure 6-10). You'll use it to give the student a score for his work. Later, you can choose to include your assessment directly as part of the final grade or include it with the other peer reviews.

6.2.1.3. Administration

The administration area serves two important functions. First, below the tools, there is a complete record of all student submissions and evaluations. The system first lists your submissions and their assessments. Below your submissions is each student's assessment. It's easy to see the ratings each student has completed and get a quick overview of their scores. Finally, you'll see a list of each student's submissions with corresponding assessments.

Figure 6-10. Assess student work

The administration area also holds two important tools shown in Figure 6-11. The first tool sets the over allocation level. Over-allocation is the amount any given student submission will be assigned for peer review over the level needed for an even distribution of assignments. If the over-allocation is set to 0, then each submission is assigned the same number of times. Over-allocation greater than 0 allows each assignment to be assigned one or two more times more than the even distribution level.

Figure 6-11. Workshop administration tools

If you allow submission and peer review at the same time, increasing the over-allocation level will increase the chances a student won't have to wait for all the submissions before she receives her full quota of peer-review submissions. The disadvantage is that some students will receive one more or one less review of their assignment than their peers.

The other tool in the administration area creates a league table of student submissions. Like the standings in a sports league, a league table lists the highest-scoring papers. You can set the number of submissions listed and whether the students can see the names of the submitters.

6.2.1.4. Calculation of final grades

Once the workshop is completed, you can configure the calculation of the final grades by moving to phase 5. There are a number of factors you can weigh when assigning the final grade for the workshop, as shown in Figure 6-12. Each item is assigned a relative weight on a scale from 0 to 50. If each of the items is assigned a weight of 1, they will all factor equally into the score. The same will hold true if all the items are set to the same nonzero number. An item set to a weight of 10 will have 10 times the impact on the student's final grade than an item with a weight of 1, but 1/5 the impact of an item with a weight of 50.

Figure 6-12. Calculation of final grades

Weight for Teacher Assessments

Teacher assessments are your grades for the students' submissions. Set this weight to a nonzero number to include your score as a separate factor.

Weight for Peer Assessments

Assign a weighting factor for the scores from the peer-review process.

Weight for Bias

This factor measures the bias of the student receiving the grade. Bias is measured as the difference between the student's rating of peer submissions and the peer averages for those assignments. So if a student consistently rates their peers higher or lower than the average, they will receive a lower bias score. The workshop developer recommends not giving the bias factor a high weight.

Weight for Reliability

Reliability is another measure of how well the student's peer reviews compare to the average reviews. The reliability measure averages the difference between the student's grade and her peers' average grades after discounting for bias. So if a student closely follows the average score for all the assignments she assessed, she will get a good reliability rating.

Weight for Grading of Assessments

This factor calculates how much weight your grades for student assessments will carry in the final calculation.

Option for Peer Grade: Include Teacher's Grade

You can choose to roll your own grade of the student's submission into the peer assessments. If you feel the averages for peer assessments are too high or too low, you can add your own score for a student's submission into the average.

Once you save your final weights, you'll see a grade sheet listing each student in the class, the scores of each factor, and a final grade. If you're not happy with the grade calculation, you can go back and change the weighting before you determine the final grades.

Under the grade-weighting table, you'll also have the opportunity to create a new league table that will be visible when you publish the final grades.

After you've completed the grade weighting, you can display the final grades to the students by moving to the sixth and final phase, Display of Final Grades.

After the final grades are displayed, students can view their grades and the detailed score for each factor by clicking on the workshop link again.