Prioneer

Free Online Stack Ranking Tool

Kostenloses Online Stack Ranking Tool

md:![Using the score of a pairwise ranking](https://prioneer.io/cloudassets/s1/prioneer/help-center/article-406887922441-pairwise-ranking-what-is-the-score.png)

## What is the score?

A choice options score in a pairwise ranking represents the relative preference of that choice option across all participants (or a group of participants if you filter results). The score is normalized on a value range from 0-100.
The most preferred choice option can have a score of 100. A choice option that was not selected once at all as a preference will have a score of 0.
 
The more often a choice option was preferred by participants in the pairwise comparison, the higher the choice options score.
 
**The choice option with the highest score is the most important option.**
This helps you to interpret the importance of a project, product characteristic, or other choice option to a single person (customer, user, ...) or a group of people.

## Answering scale and score range
Prioneer.io offers multiple answering scales for pairwise comparison.
Each answering scale gives a different way to choose the preferred choice option.
You can also say: it is the way "how you ask participants questions" about their preferences in the ranking survey.

Some answering scales allow choosing the preferred choice option while also expressing the *intensity* of the preference at the same time ("slider" or "preference scale").
Other answering scales just allow choosing the preferred option, without intensity.

When you use the “slider scale”, the score is influenced by the *intensity* of preference (for instance: strongly preferred vs. slightly preferred).
Using the “slider scale” helps you to receive more precise results because the score will be distributed according to the intensity of preference: e.g., in a Ranking where the results are Option A - 100, Option B - 20, Option C - 10, you know that A is much more preferred because the gap between the scores is so high (the score of Option A and Option B has a difference of 80 score points).

Learn how the score is calculated from survey answers and how it helps you to know the importance of a choice option.

What is the score in a pairwise ranking?

md:![Preview of a pairwise ranking survey and settings.](https://prioneer.io/cloudassets/s1/prioneer/help-center/article-406887922440-pairwise-ranking-survey-preview.png)

## Create a survey: step-by-step

1. Go to your [Rankings view](https://prioneer.io/home) in Prioneer.
2. Click on "Start a new Ranking"
3. You can now pick from several Ranking templates. Select "Pairwise Ranking" as your template.
4. In the next step, you can preconfigure the most important settings. You can adjust the settings or leave them as they are. You can change all settings later.
5. Your new Ranking will be opened. Follow the instructions on your screen, such as setting the Ranking name.
6. You are now in your Ranking. Inside the "Create" tab, you can set the Ranking question and add your choice options.
7. Set your Ranking question.

    Set the Ranking question as expressive as possible. The golden rule is that the better your question is, the better your results will be because participants will consider the question as guiding for their decision.
A good Ranking question should be related to a specific goal. Good examples are: "Which one is better to raise our number of signups?"
"Which one is better to increase brand awareness?". Think of what you want to achieve.
8. Add your choice options.

    Each choice option can have a name, a description, and image attachments. The name is the primary information a participant will view when answering your ranking survey, 
so make sure to write the choice option name as expressive as possible. Images will be displayed as an attachment on a choice option card during the pairwise comparison of the choice options (unless you change your Ranking to be an image ranking survey. In that case, the image will be shown "fullcard"-size as primary information).
9. Your Ranking survey is now ready to be shared with participants through a link (or other sharing options). You can either stay with the default settings (viable to be used immediately) or adjust some settings. However, you do not have to adjust any settings. To share your survey with participants, click on the Share button on the top right corner of the screen and select your preferred sharing method.

## Editing advanced settings, permissions, and design

To adjust settings, go to the "Configure" section inside of your Ranking. You can see a preview of your pairwise ranking survey and adjust settings for your ranking.

## How many pairs must a participant answer in a pairwise comparison survey?

The number of pairs that will be in the head-to-head comparison will depend on the number of choice options you have.
In theory, you (or a participant) need to compare each choice option with each other to select a preference.
However, this can make the process of answering very long.
When you create a ranking survey in Prioneer, you don't have to worry about the number of choice options: 
Prioneer uses a smart algorithm that selects only relevant pairs when a participant starts answering.
With each pair that a participant answers, Prioneer will calculate on-the-fly the next most relevant pair to compare
so that Prioneer can determine the unique preferences of the participant.
Because of that, a particpant does not have to answer all pairs (the Ranking survey will show the participant that he is finished).
This algorithm, which we also call speed mode, is unique to Prioneer.

## Frequent use cases

### Add images to a pairwise ranking survey
To attach an image to a choice option, follow these steps: go to the "Create" section of your Ranking. Click on the choice option that you want to attach an image to.
Next to the list of choice options, a details view will be shown in which you can edit fields related to your choice option.
Click on the attachment icon. You can now select an image from your file system that you want to upload.


### Bulk create options

You can create options in bulk by clicking the "Add multiple options" button beneath the list of options.
A textarea will appear on the screen in which you can add multiple options at once by adding them into the textarea, each item on its own row.

### Create a pairwise ranking survey from a CSV file or excel spreadsheet
To create a survey based on options you have saved inside a spreadsheet, go to your spreadsheet. Select the column with the names, and copy the names into your OS clipboard.
In Prioneer, go to the "Create" section inside a Ranking (your Ranking must have been created before) and click the "Add multiple options" button beneath the list of options.
You can now paste all names that you copied before from your spreadsheet into the textarea that is shown on the screen.
Usually, each row from your sheet will have its own row inside the textarea, thus each row will be recognized as a separate choice option and generated accordingly as soon as you confirm the action.

### Bulk upload images to a Ranking survey

You can upload multiple images at once. When you use the image bulk upload, our system will generate one choice option per image
and attach the image to each choice option.
If you want to attach an image to an existing option instead, please click on the choice option and attach an image via the details dialog that is shown next to your list of choice options. All images that you attach through the details dialog will be added to the particular choice option.

To perform bulk uploading images, click the "Add multiple options" button beneath the list of options.
You will see multiple possible actions. Click on the button "Images". You will be able to select images for upload now.

Learn how to create a pairwise ranking survey, how to write a good question, and how many choice options you should have.

How to create a pairwise ranking survey

md:![A gauge with the sections low, medium, and high, representing a high answer confidence.](https://prioneer.io/cloudassets/s1/prioneer/help-center/article-406887922439-confidence.jpg)

## What is the answering confidence?

The confidence of a participant is a value between 0-100%. Our system calculates the confidence of a participant while he/she gives answers. The more answers a participant gives, the higher is his/her Rankings confidence. It indicates how certain our system is of the participant's individual preferences,
and how high the certainty is for the unique ranking (rank 1,2,3...) of all choice options is.

The confidence of a participant is calculated in real-time while the participant answers the choice options of a pairwise comparison.
As long as the confidence is low, our ranking system will not be able to create a ranking of all options with a unique rank per choice options.
Only when the participant moves on through the pairs and keeps answering, the confidence increases and a clear ranking (such as Project A = Rank 1, Project B = Rank 5, Project C = Rank 3) will be established by our tool. 

The confidence is displayed in the progress bar to the participant during the ranking survey (for example: 50% progress bar filled represents 50% confidence).

## With Prioneers speed mode, Ranking surveys are answered faster
Based on the answering confidence, a participant can finish a pairwise comparison faster because the participant does not need to compare all possible pairs.
Prioneer can calculate transitive preferences based on a participants answers (without the participant having to answer all pairs) and create a unique list of a participant's priorities with high confidence.
The answering process will stop for the participant as soon as the confidence is high enough to be certain of his preferences.

### Finish threshold
With the default ranking settings, the participant can continue to answer if he likes, which will make his ranking quality finer. In the settings, you can modify the conditions that control when the ranking answering process should stop for the user.

Learn what the answering confidence is and how it speeds up the answering for a participant.

What is the answering confidence in a pairwise ranking survey

md:![Participants of a pairwise ranking, and the agreement about the priority. The team is aligned with the priority.](https://prioneer.io/cloudassets/s1/prioneer/help-center/article-406887922438-priority-agreement.png)

## What's the priority agreement?

As you view the results of a Pairwise Ranking, you will come across the so-called **agreement**.

The agreement is a statistical number and tells you how much all people who responded to your Ranking agree on the priority of a choice option.
(You could also say: how much they agree about the *rank* or *importance* of a choice option.)
In ranking results, each choice option has an agreement.

The agreement is implicit because respondents to the ranking survey cannot submit their agreement about the priorities but it is a scientific number that is calculated automatically by our system based on their responses.

## What's the agreement good for?

The agreement helps you to: 
* make data-based decisions
* identify misunderstandings
* raise the right discussions
* align your team on the priorities
* understand confidence of priorities


## Understanding high and low agreements

If the agreement percentage of a choice option is high, e.g. > 90%, it means that many people ranked the choice option similarly important within their individual Ranking.
For instance, if in person Bob's responses the option A is rank #1, in Anne's responses the option A is rank #1, and in Peter's responses the option A is rank #1,
then the agreement is 100%.

<div style="display: flex; justify-content: center;">
| Choice Option     | Bob's Rank    | Anne's Rank   | Peter's Rank    |   Agreement
| -----------       | -----------   | -----------   | -----------     | -----------   
| A                 | #1             | #1             | #1               |  100%
</div>
<div style="display: flex; justify-content: center;margin-bottom: 40px;">100% agreement because it is rank #1 for everyone.</div>

If however in person Bob's responses the option A is *only rank #2*, in Anne's responses the option A is rank #1, and in Peter's responses the option A is rank #1,
then the agreement will be around between 90%-99% (the exact agreement depends on the total number of choice options, actual replies and the settings of the Ranking).

<div className="alert-notice" style="margin-bottom: 50px;">
Please note: the agreement numbers in all following examples will differ for your own Rankings! The exact agreement always depends
on multiple factors such as the number of choice options, the exact replies your Ranking received, and the settings of the particular Ranking.
</div>


<div style="display: flex; justify-content: center;">
| Choice Option     | Bob's Rank    | Anne's Rank   | Peter's Rank    |   Agreement
| -----------       | -----------   | -----------   | -----------     | -----------   
| A                 | #2             | #1             | #1               |  95%
</div>
<div style="display: flex; justify-content: center;margin-bottom: 40px;">High agreement among respondents, but not 100%.</div>


If the agreement percentage is low, it means that many people ranked the choice option very differently in their individual rankings. For instance, let's assume in a Ranking survey with 5 choice options, for Bob the option A is rank #1,
for Anne the option A is rank #3, and for Peter the option A is rank #5.
In this situation, the agreement would be a low percentage number, e.g. 2%.  

<div style="display: flex; justify-content: center;">
| Choice Option     | Bob's Rank    | Anne's Rank   | Peter's Rank    |   Agreement
| -----------       | -----------   | -----------   | -----------     | -----------   
| A                 | #1             | #3             | #5               |  2%
</div>
<div style="display: flex; justify-content: center;margin-bottom: 40px;">Low/no agreement</div>

### Why is a choice option on rank #1 but has a low agreement?

Sometimes you might face the situation that the most important choice options or one of them has a low agreement.
E.g. something is on rank #1 in the results of your ranking survey but the agreement is just 50%.
What does that mean?

It means that no other choice option was as important as that choice option among all people who replied (leads to rank #1 in total)
but there were still people who did not rank that particular choice option as the most important (leads to low agreement despite rank #1).

For better understanding, let's assume these two situations:

If a choice option is on rank #1 and the agreement is 100%, the choice option got the highest priority in everyone's individual ranking (or in other words: it is rank #1 in everyones individual ranking).

If a choice option is on rank #1 and the agreement is 50%, the choice option won the highest priority in many individual rankings,
but not in all individual rankings.
Nevertheless, the choice option ranked position #1 because no other choice option is more important after calculating all individual rankings into one combined ranking.

<div style="display: flex; justify-content: center;">
| Choice Option     | Bob's Rank    | Anne's Rank   | Peter's Rank  | Total Rank  |   Agreement
| -----------       | -----------   | -----------   | -----------     | -----------     | -----------   
| A                 | #1             | #1            | #4           | #1   |  70%
</div>
<div style="display: flex; justify-content: center;margin-bottom: 40px;">A high position can have a low agreement</div>

### Why has a choice option 100% agreement but is not on rank #1?

A choice option can have 100% agreement on any rank.
For example, if everyone ranks choice option A as the worst choice option,
the choice option will land on the last position of the ranking with an agreement of 100%.

Similarly, if a choice option is ranked by everyone so that its position ends up being between #4 - #6 across all individual rankings,
then the agreement will also be close to 100% (the exact agreement depends not only on the position but also on other factors as the number of ranking options.)  

<div style="display: flex; justify-content: center;">
| Choice Option     | Bob's Rank    | Anne's Rank   | Peter's Rank  | Total Rank   |   Agreement
| -----------       | -----------   | -----------   | -----------   | -----------    | -----------   
| A                 | #10             | #10            | #10           | #10   |  100%
</div>
<div style="display: flex; justify-content: center;margin-bottom: 40px;">The last position can have a high agreement (assuming 10 choice options).</div>

<div style="display: flex; justify-content: center;">
| Choice Option     | Bob's Rank    | Anne's Rank   | Peter's Rank  | Total Rank   |   Agreement
| -----------       | -----------   | -----------   | -----------   | -----------    | -----------   
| A                 | #4             | #5            | #6           | #5   |  99%
</div>
<div style="display: flex; justify-content: center;margin-bottom: 40px;">A medium position can have a high agreement (assuming 10 choice options).</div>

Learn what the agreement is and how it helps you to make better data-driven decisions.

How to use the priority agreement

Rank ideas, customer problems, or choice options collectivly or by yourself.

Stack Ranking Tool

Rank a list of alternatives using pairwise comparison.

Pairwise Ranking

Gather insights about your audiences preferences with a ranking survey.

Ranking Surveys

Research the best picture, logos or product designs.

Image Ranking Surveys

Use group rankings to quickly rank items with a team of people.

Group Decision Tool

Prioritize Jira issues using RICE, ICE, AARRR, or custom criteria.

Jira Backlog Prioritization

Products

Blog

Advice on decision making and prioritization

Effective Decision Making

Get started - it's free

Decide the right things, with the right people.
Rank choice options from most important to least important
and achieve your goals 10x faster.

A simpler way for everyone to
make great decisions

Trusted by teams & individuals from all kinds of organizations for decision-making:

The decision-making tool <b700>that drives outcomes</b700>

Cesar Valdivia

Prioneer has been game changing for our work.

Use the prioritization power of leading decision-making techniques. Without fiddling in spreadsheets.

Want to decide together with others? No problem.
Add people to your Prioneer workspace or share a guest link. Everyone can submit their own opinion and Prioneer shows you the overall opinion of the group.

Invite anonymously
or with login

You control who can
see results

Asynchronous voting
without meeting

Don't let a winning team's time be used up by meaningless projects.

Earn applause for delivering
things that matter

Use Prioneer for:

Alexander Zakharov

Prioneer helped me establish an environment where stakeholder feedback plays a crucial part in product decisions. Clean and structured survey results make it very simple to separate 'must-haves' from 'nice-to-haves'.

Group Product Manager

We are using Prioneer for customer needs discovery […]. I can’t thank you enough for your help with this! This is <highlight>an incredible enhancement</highlight> that will save us a lot of time.

Project Management Consultant

I was looking for a way to make people do individual scorings to <highlight>avoid group thinking</highlight> and make their own assessments first, and then we'd make changes after that in discussion. Thank you!

Stephan Eyl

The team rating allows us to <highlight>quickly prioritize as a team</highlight>, without having to discuss each project one by one. We <highlight>save a lot of time</highlight>, and <highlight>the entire team has clarity</highlight> regarding which of the new projects we expect to have the greatest impact.

Joshua Land

For a startup team, it's important not to waste time on things that have no impact. With Prioneer everyone is on the same page about the most important things.

The tool met my immediate expectations providing a means to force rank multiple items.

Sign up now

Get started

Get started completely free.

Ready to prioritize the right things?

Article: Pairwise comparison method explained

Prioneer is a stack ranking tool. Get people's priorities ·  Research customer pain points · Make team decisions · Create beautiful ranking surveys in 1 minute!

2023-06-13

Start within minutes. Upgrade when you wish. Don't pay extra per user.

Take your decision-making to the next level.

Pricing - Prioneer

There is content that you provide to us through our services when using our services. This can be content such as workspace content or ranking content (e.g. ranking names, questions, item name, user name, guest name, your submitted rating value for an item/idee). Also, when you submit a rating value for an item (by rating an item/project/software feature through the rating feature), your rating value is stored together with your user ID or guest user ID or name. We collect and store this in order to provide this service. Please note that the ranking value that you submit can be visible to the creators/members of a ranking or the members of a workspace.

The data (including ideaspace names, items/ideas, guest user names, guest ranking and so on) is deleted when the ranking creators/members delete the particular workspace or ideaspace.

What can we help you with?

Articles

What is the score in a pairwise ranking?

How to create a pairwise ranking survey

What is the answering confidence in a pairwise ranking survey

How to use the priority agreement