Overview of the Buddycheck adjustment factor (Original)
In Buddycheck, students are asked to evaluate their peers' and their own contributions to a project according to the course coordinator's criteria. This can done in two ways:
- a Likert scale
- divided points (achieved by splitting 100 marks between the members of the group)
For each student, an adjustment factor (formerly known as PAF - Peer Assessment Factor) is calculated. After the student responses have been moderated, the adjustment factor can be used to re-calculate individual student assignment marks based on their peers' evaluation of their contribution to the group assignment.
i.e. If the group assignment receives a mark of 80 out of 100 and students receive adjustment factors of
- Student 1: 0.85
- Student 2: 1.00
- Student 3: 1.05
- Student 4: 0.95
- Student 1’s assignment mark = 80 x 0.85 = 68
- Student 2’s assignment mark = 80 x 1.00 = 80
- Student 3’s assignment mark = 80 x 1.10 = 88
- Student 4’s assignment mark = 80 x 0.95 = 76
Warning: The adjustment factor in Buddycheck should only be used to adjust individual students’ results if the process has been clearly explained to students and included in the Course Profile. Moderation of the student responses is essential to ensure that the final adjusted mark accurately represents their performance and contribution to the group. A small minority of students will try to “game” the system to give themselves a higher adjustment factor than is warranted, or potentially to penalise other group members.
Calculation of the adjustment factor (formerly PAF)
The factor is calculated by the sum of all scores attributed to a student individual by the number of criteria multiplied by 100.
The above formula assumes all students have completed the assessment. If a student has not completed the assessment the equation becomes:
Interpretation of the adjustment factor
A student who pulls their own weight in a team and who does similar amounts of work to everyone else will achieve a peer assessment factor (PAF) of 1.0. Students who lead the team and who are recognised as putting in extra work will achieve a PAF of above 1.0. It should be noted that in order for students to reward someone that they believe is doing extra work, they have to take points from another student (divided points option) who may not be ‘free riding’ but who is perceived to not be doing quite as much as the rest of the team. Therefore it is quite common for students to receive PAFs slightly below 1.0 but not to be in danger of failing as their group members have ‘robbed Peter to pay Paul’.
| PAF | Interpretation | Comments |
|---|---|---|
>1.5 | Alarm! Team failure | Something has gone wrong – either there is a student who is not participating at all or this student has taken all the work home and done it by themselves. Either way, learning objectives are probably not being achieved. |
| 1.15 - 1.5 | Super Leader | The team balance probably needs to be addressed as to achieve such a high score other students must not be participating or this student is doing far too much. |
| 1.05 - 1.15 | Leader | The student is showing definite leadership qualities and/or has been putting in significant extra effort. |
| 1.00 - 1.05 | Good teamwork | The student is working well with the group and has been recognised as pulling their weight (1.00) and perhaps a little more (>1.00). |
| 0.95 - 1.00 | Acceptable teamwork | This student has probably only been penalised because another team member has shown leadership and put in extra effort. |
| 0.85 - 0.95 | Social Loafer | Any PAF below 0.95 is unacceptable. Social loafers who lie in this band can usually be mentored with the group’s help and become productive members of the group. |
| 0.75 - 0.85 | Super Social Loafer | As above and below. |
| < 0.75 | Alarm! Individual failure! | The individual is in grave danger of failing the course. Much work is required for this student to be accepted back into the group and there will be trust issues with allocating this student any work. |
Average with self
Average with self gives an indication of how realistically students judge their individual contribution to a group project. It compares the score a student allocates themselves against the average that all the other students have given them. These averages can also be used to highlight students who are trying to “game” the system.
Average with self is the score that the student has given themselves over the average that all the other students have given them.
| Average with self | Interpretation |
|---|---|
| > 1.0 | Indicates that the student believes they do more in the group assignment / project than their group members think. The student may also be trying to “game” the system to improve their PAF. |
| 1.0 | Student has a realistic judgement of their contribution to the group assignment / project. |
| <1.0 | Indicates the student undervalues their contribution to the group assignment / project. |
Limit
The limit is used to highlight groups where there is disagreement between members with the scores allocated to individuals.
The limit is set to highlight groups where there is deviation from the average (1.0) of the individual Average with self and PAFs.
The default is Limit = 15%. i.e. Highlights groups where there are individual Average with self and PAFs greater than 1.15 and less than 0.85.
Moderation
It is recommended during the moderation process that all student results are checked.
The highlights (based on the Limit) and the Labels help to identify groups where there is disagreement between members with the scores allocated to individuals.
Check unusual PAFs by taking into account all of the contributing factors including the comments that students gave for their division of points. Tutors may also be able to provide supporting knowledge for unusual PAFs. In some cases it may be necessary to meet with groups or individual students to check on the contributions of group members to the assignment / project.
It is important that you can justify PAFs if they are contested.
The tool allows you to remove individual score students have allocated or all the scores given by a particular student. The tool will automatically recalculate the new PAFs. i.e. Students who try to “game” the system can have scores they allocated removed from the calculation. Recommendation: It is good practice to inform student that scores that are outside the Limit set (default 15%) can be moderated.
Based on: A peer assessment tool for teams, Associate Professor Lydia Kavanagh