Get Started: BuddyCheck

Below you will find an introduction to how Buddycheck works, and then step-by-step instructions with videos for setting up an evaluation for your Blackboard course space. Click the following link for a PDF with full written instructions: Buddycheck Guidance.

The FSE eLearning team conducted a pilot project to understand the potential for Buddycheck to be used as standard across the Departments, as a way to ensure a more rigorous approach to assessing individuals contributions to group work. The full report can be accessed here: Buddycheck Report.

Before setting up your evaluation, it is highly recommended that you work through the ‘Preparation’ section of the staff guidance and develop your peer evaluation plan and timeline alongside one of our eLearning team. This is to ensure that both you and your students receive the support you need to complete an effective peer evaluation. Book a support session via this link.

We have also created a page for student guidance on peer assessment, including a ‘What to expect’ guide to Buddycheck. Please share this with your students and use as a template for your own communications around peer assessment: Student Guidance: Peer Assessment Tools.

 

What does Buddycheck do?

How does the Buddycheck mark adjustment work?
Peer scoring variables

Setting up an evaluation

Adding a Buddycheck link to your Blackboard course
Creating a peer evaluation
Adding and editing evaluation questions

Peer evaluation in progress

Monitoring Results
Managing evaluations

After the evaluation

What does Buddycheck do?

 

The software allows group members who have completed and submitted a piece of group work to review and score each member (including themselves) by answering a number of questions based on 5 categories. Once students complete their peer scoring task, the results can be used to:

  • Adjust an existing grade in your Blackboard Course.
    I.e. all members of a group have been given the same grade for a group project. Buddycheck could then apply an adjustment factor to alter this grade on an individual level, based on the results of the peer scoring process.
  • Provide a personalised report to individual members of a group that displays their peer evaluation results compared to the average of that group.
  • Provide raw scores for an instructor to further manipulate.
  • Give the instructor an overview of group performance to easily pick up on conflicts or issues within groups.

You can choose to use any of the above outputs; they are all optional.

How does the Buddycheck mark adjustment work?

Team members’ contributions are assessed based on specified criteria and within a 1 to 5 rating scale, where 1 is the lowest and 5 is the highest.

The 5 default criteria based on the CATME framework used to evaluate individual performance are:

  1. Contributing to Team’s Work
  2. Interacting with Teammates
  3. Keeping the Team on Track
  4. Expected Quality
  5. Having Related Knowledge, Skills, and Abilities

 

If you would prefer to use your own set of criteria you can easily add and remove questions when setting up your evaluation, however we recommend using the default options as the CATME framework is a research-based quantitative method.

Peer scoring variables

Adjustment Factor

An Adjustment Factor value is a calculated number that attempts to show the contribution of a given student relative to the other members of their group. Specifically, this Adjustment Factor is the average rating of the student divided by the overall average rating for all members of the group. For full details of the adjustment factor calculations please download the Buddycheck Calculations guide through the following link: Buddycheck Calculations.

With or without self

‘With Self’ means that the student’s self-ratings are factored in. In some cases, exceptional conditions such as ‘Cliques’ may have an impact on the validity of the Adjustment Factor values, and in these cases the Adjustment Factor columns will be highlighted.

Original or capped

Depending on your chosen settings, you can have an original or capped adjustment factor. By default, the capped Adjustment Factor value is capped at a maximum value of 1.05. Also, values above 0.95 are rounded up to 1.00 (as findings are that values in this range are just ‘noise’). You can get an explanation of the capped adjustment factor by hovering over the switch on top of the results table.

Setting up an evaluation

Before you can create a Buddycheck Peer Evaluation for your students to complete, you must create a link within your Blackboard course.

  1. Open the relevant Blackboard course and turn Edit mode on.
  2. Click Tools > More Tools > Buddycheck.
  3. Edit the title. There is no need to change any of the other settings here, as evaluation titles and due dates are configurable in the next stage.
  4. You can choose to make the link hidden from students until the peer evaluation opens by choosing the No option next to Availability.

*Please note: all peer evaluations for your current course will be accessible via the link you have just created, so you do not need to repeat the above steps for subsequent evaluations in the same course*

Creating a peer evaluation


  1. Click on the Buddycheck Peer Evaluation link you created in the step before.
  2. Click the +Create Evaluation button
  3. Choose a title for your evaluation. You can also reuse a peer evaluation from another course you teach on by selecting an Evaluation Template.
  4. Click Continue.

*Please note: you will have to click ‘Allow’ on the authorisation screen the first time you create an evaluation. You may also be prompted to Activate Buddycheck the first time it is added to a Blackboard course*

Adding and editing evaluation questions

You will be unable to distribute Buddycheck peer evaluations to students unless the relevant Groups have already been created within Blackboard.

  1. When selecting the groups to be included in the evaluation, select the correct Groups or Group Sets.
  2. Enter a start and end date (students can submit evaluations after the end date, but they will be marked ‘late’).
  3. Introduction: This free text box will be presented to students before starting the evaluation.
  4. Main questions: The 7 default questions are those based on the CATME Framework. You can add, remove and modify the questions as much as you wish, just note that this will deviate from the framework.
  5. You can use the question controls to:
    Edit (pencil icon)
    Preview (eye icon)
    Delete (trash icon)
    Reorder (= icon)
  6. Click + Create question to make a custom Team or Peer question.
  7. Click Preview Question or the eye icon when creating questions to check how they will appear to your students.
    • Open Question(s): answers are only seen by Instructors.
    • Peer Messages: answers are displayed to the relevant team member but can be made anonymous.
  8. Advanced options:
    • The Adjustment Factor (the contribution of a given student relative to the other members of the team) defaults should be sufficient for most evaluations, but you can change these if you wish.
    • Reminder date – an automated reminder will be emailed to students via Buddycheck
    • Auto lock date – Choose a date to lock the evaluation, this means no further submissions can be made.
  9. Click Open preview to check how your evaluation would appear to students.
  10. Click Create Evaluation to Save. Once the evaluation is live, students are emailed a link to complete the evaluation, they can access it via the link you created in your Blackboard space.

Once created, your evaluation will appear in a Peer Evaluations dashboard that lists all your current, previous and upcoming evaluations. Click the drop down chevron next to your evaluation title for a summary, including an alert if there are students on your course that have not been assigned to a group, start and end dates, the number of students and groups.

Resync the group enrolments in the software with your Blackboard course to check they are up to date (Open your Evaluation > More > Sync with course) – you can do this at any time during the evaluation.

*Please note: Buddycheck will email you before the evaluation start date to remind you to check group enrolments, as this may have changed since the evaluation was created*

Peer evaluation in progress

Once the evaluation has started, you will see a percentage bar showing current completions on the dashboard, and individual student evaluations will be immediately visible to you once submitted. All staff enrolled on the Blackboard course are able to monitor student progress.

Monitoring Results


 

  • Scores and reports: Click on the chevron next to any student’s name to view both their received and given scores, as well as their personal and group report (group reports are only visible to the instructor).
  • Labels: If group dynamic issues arise, this is usually reflected in the evaluation and the scores students award to each other. Buddycheck highlights this by using labels next to the student names – these act as an ‘early warning system’ for potential issues within a group that may affect the adjustment factor. To access more information as to what each label means, click on More info under the evaluation description, and then on the ? icon next to each label.
  • Average: The fourth column Avg shows the student’s average score as per their received scores.
  • Factor: Shows the adjustment factor for a given student.
  • Reliability: The reliability score is based on 3 criteria (for full details see the Buddycheck Calculations Guide).
    • The deviation of received scores should be taken into account when determining the reliability. When
      the received scores have a small deviation from the average, it is more reliable than when the received
      scores differ widely.
    • A minimum of two peers should fill in an evaluation. One peer is statistically meaningless.
    • Bigger groups are more reliable than smaller groups with the same percentage of handed in surveys
      because each peer received more scores, making the standard deviation more reliable.
  • Submitted: Shows the instructor when the individual submitted, if their evaluation was ‘reopened’, or if a student has not yet submitted.
  • Original Factor toggle: You can display adjustment factor in two ways: original or capped. Capped means the factor is a rounded number and is capped on a minimum of 0.5 and a maximum of 1.25. You can change the minimum and maximum in Edit evaluation> Advanced options.
  • Self scores toggle: Click to view the results with or without the student’s self scores. ‘With Self’ means that the student’s self-ratings are factored in. In some cases, exceptional conditions such as ‘Cliques’ may have an impact on the validity of the Adjustment Factor values, and in these cases the Adjustment Factor columns will be highlighted.

Managing Evaluations


You can select one or multiple students in the results table using the checkboxes on the left of the student’s name. By doing so, action buttons will appear on the top of the results page:

  • Email: Use this button to email students.
  • Reopen: Use this button if you want students to fill out the survey again.
  • Clear: Use this button to clear/remove the results entered by the student.
  • Remove: Use this button if students have left the course in the meantime.

These actions will become unavailable once the evaluation is locked or the results are released.

Sync with course: If there are any changes made to the Blackboard groups during the evaluation:

  1. In the top right corner of the Evaluation page, click on the More button.
  2. Click Sync with course to update the groups within the Buddycheck evaluation.

Send a reminder: Instructors can send reminders to those students who haven’t completed the evaluation yet:

  1. In the top right corner of the Evaluation page, click on the More button.
  2. Select Send a reminder from the drop-down menu. This will send an email, with a link to the evaluation, to each student who hasn’t completed the evaluation yet.

Late submissions: When the evaluation deadline has passed, the evaluation is formally closed. Students can still submit after the deadline. Their submissions will be marked red to indicate that they were late.

After the evaluation


 

Once the deadline has passed and the evaluation is formally closed, instructors can perform a few actions. In the top right corner of the Evaluation page, select the More button. The following actions will appear in the drop-down menu:

  1. Lock the evaluation: Students cannot submit anymore, even if you reminded them or reopened the evaluation for some students.
  2. Release the results: Select this action to release the results to the students. Once the results are released, the evaluation will automatically lock, and all students will receive an email with a link to their individual report. Students who haven’t completed the evaluation will also get the results. Students will not see individual rating from group members, just the total average of all group members.
  3. Download results: Select this action to download a complete summary of the evaluation as either Excel or CSV files. You will be prompted as to whether you export results excluding or including their self-score.
  4. Publish grades: Select this action to sync the evaluation results with the Grade Centre in your Blackboard course. There will need to be a column already created with the group scores that Buddycheck will then adjust using the adjustment factor results.

Leave a Reply