PAF - Moderation Case Studies - eLearning

Refer to the PAF assignments – View and moderate results guide for information on the purpose of PAFs, SAPAs and Ratios and how they are calculated.

Moderation is an essential part of the process. Unjustified scores can significantly affect student grades, and can make responding to a student query (or appeal) difficult.

Consider limiting the PAF to a maximum of 1.10 to ensure students are not over rewarded for leadership skills within the group compared to other assessment criteria. If you do this, you will need to include it in your Electronic Course Profile and let the students know.

Note: You will need to cap your PAFs after you have downloaded them via Excel or through Grade Centre.

It is recommend that you use a Limit set between 15% and 10% to highlight student scores that may not reflect actual student performance. This will help you identify those groups where moderation is required.
All SAPAs over 1.06 should be checked. If there is no other student who agrees with the student’s self score, it is recommend that you remove the self score. You might find that you are more comfortable with a higher limit (e.g. SAPAs >0.8) but in any case set a limit and apply it consistently across groups.
All SAPAs less than 0.94 should be checked. If there is no student who agrees with the student’s self score, we recommend that you remove it. Again, find a limit that you are comfortable with and apply it consistently across groups.
When making a decision as to whether to remove a score from the PAF calculation, take into account all evidence including student comments.
Occasionally you will come across a student whose scores are outside your set Limit for all members. Consider removing all the scores given by this student. Often the scores will have a significant but unjustified effect on the PAFs for the entire group.
It is recommend that you also check low PAFS to ensure that they are justified.
If in doubt, ask another member of your teaching team (or talk to the group themselves). PAFs can significantly influence student grades depending on the weight of the group task and therefore moderation should always be justifiable.

Case 1 - A Group in Agreement

With the Limit set to 10% no scores are highlighted in this group showing a high level of agreement regarding contribution to the assignment within this group.

Group assessment results table

Reviewing the PAFs only one student Anne Benton has a low factor of 0.9. She has a SAPA of 1.00 showing she agrees with the judgement of other group members of her contribution to the assignment. The comments also support the scores given.

Review one student's results

The SAPAs have little variation from 1.00 (1.00, 1.01, 1.00, 1.02, 0.98, 1.02) showing all students have made realistic judgements on their own contribution to the assignment.

Moderation

Due to the agreement within the group no changes will be made to the results in the moderation process.

Review PAF and SAPA

Use of PAFs

If the group assignment result was 70 / 100, and you have set up the course (documented in the Course Profile and students informed at the start of semester) such that PAFs are used to modify group marks, each group member would receive the following marks:

Anne Benton = 0.90 x 70 = 63

Omari Brown = 1.05 x 70 = 73.5

Greek Cholevas = 1.04 x 70 = 72.8

Callum Collins = 1.00 x 70 = 70

Miguel Duan = 1.05 x 70 = 73.5

Glen Horvitz = 0.96 x 70 = 67.2

Case 2 - A Student Submits Unjustified Scores

With the Limit set to 15% four of five scores allocated by Maisie Jones are not in agreement with the rest of the groups allocations as is shown by the highlighted cells.

Unjustified scores

The ‘Individual v Average’ Ratios show that she has undervalued two students (Jia at 0.72 = 72% and Refina at 0.79 = 79%) in comparison to the rest of the group. In addition, Maisie has given herself a 20% bonus and Chet a 28% bonus (Individual v Average ratio = 1.20 and 1.28). These scores will have a significant effect on the PAF.

Effect on 'individual v average' score

Maisie’s SAPA is very high at 1.20 indicating that she thinks that her contribution is much greater than the rest of the group thinks it is and/ or she is trying to “game” the system by giving herself more marks (and thus getting a higher PAF).

High SAPA

Moderation

In the moderation process, Maisie’s unjustified allocations were all removed. The remaining scores are in agreement even when the Limit is tightened to 10%. The New PAFs are supported by the comments given.

Moderation results

Use of PAFs

If the group assignment result was 77 / 100 each group member would receive the following marks (New PAFs are used):

Jia Huang = 0.98 x 77 = 75.5 (before moderation = 0.92 x 77 = 70.8)

Maisie Jones = 1.07 x 77 = 82.4 (before moderation = 1.11 x 77 = 85.5)

Ben Kazmi = 0.98 x 77 = 75.5 (before moderation = 0.97 x 77 = 74.7)

Ruifina Lane = 0.97 x 77 = 76.2 (before moderation = 0.93 x 77 = 71.6)

Chet Lee = 1.01 x 77 = 77.8 (before moderation = 1.06 x 77 = 81.6)

This case clearly illustrates how a single student’s scores can have a significant effect on grades. Moderation is essential in the provision of justifiable and fair assessment.

Case 3 - Individuala Scores are not Justifiable

With the Limit set to 15%, two scores are highlighted as falling outside the group’s average. Janya Meas has given herself 26% more than the rest of the group (SAPA = 1.26) and Afia Newhouse has given her 69% of the average group score (Individual v Average Ratio = 0.69).

individual scores are not justified

Moderation

The comments were checked and there was no justification for these scores. The score Janya allocated by herself was removed (note this was also the highest score allocated to her).

Moderation

When thinking about how to moderate this, there are two initial actions:

read across the line for scores for Janya. They are (in order): 41, 54, 55, 59, 61, 68. The two outliers (41 and 68) are obvious as is the fact that the other student’s marks fall in two clusters: 54-55 and 59-61; and
read the comments that students have made to justify their scores.

You can also talk to your teaching team should you have any doubts or confusion but in this case, the comments indicated that there was no justification for Janya and Afia’s scores and so both were removed.

Moderation - removing scores

Limit 10%

With the Limit set to 10% one score allocated to Sarah Smith by Afia (Individual v Average = 1.15) is not agreement with the rest of the group’s allocations. This is probably a consequence of Afia allocating a low score to Janya as this meant she had ‘extra’ points to distribute and has chosen to give them to Sarah.

Note Sarah has underestimated her own contribution to the assignment (SAPA = 0.90).

Moderation

The comments were checked and there was no justification for either of these scores and so both were removed.

Moderation - remove scores

With the Limit set to 10% no scores are highlighted showing a high level of agreement regarding contributions to the assignment within the remaining group members. The New PAFs are realistic and are supported by the comments given.

Although it could be argued that removing the highest and lowest scores for both Janya and Sarah has resulted in a New PAF that is fairly close to the original, the moderation is recommended as:

it is a justifiable process that has been applied to all scores consistently and therefore a valid case can be made should a student query their marks; and
the small change in PAF could have a significant effect if the final mark for the course is on the borderline of a grade.

Use of PAFs

If the group assignment result was 80 / 100 each group member would receive the following marks (New PAFs are used):

Janya Meas = 0.86 x 80 = 68.8

Erin Murphy = 1.00 x 80 = 80.0

Afia Newhouse = 1.07 x 80 = 85.6

Helen Porter = 1.00 x 80 = 80.0

Luis Robinson = 1.00 x 80= 80.0

Sarah Smith = 1.05 x 80= 84.0

Case 4 - "Messy" - A Group in Trouble

The score allocated to Katherine Col (Individual v Average ratio = 0.77) by Katrina Turner is not in agreement with the rest of the group’s allocations.
The score Robert Watts (SAPA = 1.19) allocated himself is not realistic.
There is a lot of variation in scores allocated to Mathew Withers (Individual v Average ratios: 1.25, 1.25, 0.88, 0.88, 0.52, 1.32)

Scores table

Moderation

The score allocated to Katherine by Katrina was removed as there was no justification of the score in the comments or in the comments and scores provided by the rest of the group.
The score Robert allocated himself was removed as there was no justification in the comments and scores provided by the rest of the group.
- As the SAPA is quite large (>1.10), we need to check whether Robert has taken marks across the board or taken them from an individual. In this case, he has taken them from Daryl Luton (Individual v Average ratio = 0.90). This score also needs to be removed as it is due to Robert inflating his own score and not due to Daryl’s actual performance.
Based on the comments there is disagreement in the group about the level of Mathew’s input. The teaching team were consulted and it was felt that the outliers were not indicative of his performance. Therefore the highest and lowest scores were removed.

Moderation - remove scores

Limit 10%

With the Limit set to 10%, four of five remaining scores allocated by Katrina are not in agreement with the rest of the group’s allocations. Focusing on the students who have not been affected by the earlier moderations:

Mary Savin has a PAF of 1.06 (based on comments this is too high)
The score Katrina (SAPA = 1.14) allocated herself is not realistic given the rest of the group’s scores and comments.

Reviewing other students

Moderation

As no justification was found, the following scores were removed:

the score allocated to Mary by Katrina; and
the score Katrina allocated herself.

Moderation - remove scores

Use of PAFs

If the group assignment result was 90 / 100 each group member would receive the following marks (New PAFs are used):

Katherine Col = 1.01 x 90 = 90.9

Daryl Luyton = 1.00 x 90 = 90.0

Mary Savin = 1.04 x 90 = 93.6

Katrina Turner = 1.06 x 90 = 95.4

Robert Watts = 1.07 x 90 = 96.3

Mathew Withers = 0.84 x 90 = 75.6

Case 5 - Non-contribution Student

Firstly we need to deal with scores outside the 10% limit. Two scores allocated to Cheung Min by Christina Cowin (Individual v Average = 0.86) and Emily Barlott (Individual v Average = 1.11) are not in agreement with the rest of the group’s allocations.

Non-contributing student

Moderation

The comments were checked and there was no justification for the score allocated by Christina. The score Christina allocated Cheung is removed. Once Christina’s score was removed, the score allocated by Emily to Cheung fell within the 10% limit. It was justified by comments and hence was not removed.

Non contributing student

Now we can look at the effect of the non contributing student: Mark Willis. Comments indicate he stopped attending meetings in week 3 of Semester, but he has not withdrawn from the course. As a consequence, all group members PAFs are inflated as the 100 points was divided between 5 group members in practice not 6.

Therefore all PAFs will need to be corrected by multiplying by the following:

number of contributing group members / actual number of group members

which in this case is 5/6.

Note: If a group member drops out late in semester, the increase in PAFs across the group may be justified and the course coordinator will need to make this decision.

Use of PAFs

If the group assignment result was 80 / 100 each group member would receive the following marks (New PAFs are used):

Emily Barlott = 1.16 x 5/6 x 80 = 0.97 x 80 = 77.3

Christina Cowin = 1.18 x 5/6 x 80 = 0.98 x 80 = 78.7

Cheung Min = 1.14 x 5/6 x 80 = 0.95 x 80 = 76

Nicola Salom = 1.22 x 5/6 x 80= 1.02 x 80.0 = 80.3

Michael Schubert = 1.34 x 5/6 x 80= 1.12 x 80.0 = 89.3 OR

Michael Schubert = 1.10 x 80.0 = 88.0 *If PAFs are capped to 1.10

Note: This adjustment will need to be made manually in Grade Centre or in Excel.