How to use the priority agreement

What's the priority agreement?

As you view the results of a Pairwise Ranking, you will come across the so-called agreement.

The agreement is a statistical number and tells you how much all people who responded to your Ranking agree on the priority of a choice option. (You could also say: how much they agree about the rank or importance of a choice option.) In ranking results, each choice option has an agreement.

The agreement is implicit because respondents to the ranking survey cannot submit their agreement about the priorities but it is a scientific number that is calculated automatically by our system based on their responses.

What's the agreement good for?

The agreement helps you to:

• make data-based decisions
• identify misunderstandings
• raise the right discussions
• align your team on the priorities
• understand confidence of priorities

Understanding high and low agreements

If the agreement percentage of a choice option is high, e.g. > 90%, it means that many people ranked the choice option similarly important within their individual Ranking. For instance, if in person Bob's responses the option A is rank #1, in Anne's responses the option A is rank #1, and in Peter's responses the option A is rank #1, then the agreement is 100%.

If however in person Bob's responses the option A is only rank #2, in Anne's responses the option A is rank #1, and in Peter's responses the option A is rank #1, then the agreement will be around between 90%-99% (the exact agreement depends on the total number of choice options, actual replies and the settings of the Ranking).

If the agreement percentage is low, it means that many people ranked the choice option very differently in their individual rankings. For instance, let's assume in a Ranking survey with 5 choice options, for Bob the option A is rank #1, for Anne the option A is rank #3, and for Peter the option A is rank #5. In this situation, the agreement would be a low percentage number, e.g. 2%.

Why is a choice option on rank #1 but has a low agreement?

Sometimes you might face the situation that the most important choice options or one of them has a low agreement. E.g. something is on rank #1 in the results of your ranking survey but the agreement is just 50%. What does that mean?

It means that no other choice option was as important as that choice option among all people who replied (leads to rank #1 in total) but there were still people who did not rank that particular choice option as the most important (leads to low agreement despite rank #1).

For better understanding, let's assume these two situations:

If a choice option is on rank #1 and the agreement is 100%, the choice option got the highest priority in everyone's individual ranking (or in other words: it is rank #1 in everyones individual ranking).

If a choice option is on rank #1 and the agreement is 50%, the choice option won the highest priority in many individual rankings, but not in all individual rankings. Nevertheless, the choice option ranked position #1 because no other choice option is more important after calculating all individual rankings into one combined ranking.

Why has a choice option 100% agreement but is not on rank #1?

A choice option can have 100% agreement on any rank. For example, if everyone ranks choice option A as the worst choice option, the choice option will land on the last position of the ranking with an agreement of 100%.

Similarly, if a choice option is ranked by everyone so that its position ends up being between #4 - #6 across all individual rankings, then the agreement will also be close to 100% (the exact agreement depends not only on the position but also on other factors as the number of ranking options.)