Setting up ranking questions

6 min read · Updated 2026-11-20

Ranking pages enable you to create questions where participants are asked to order a set of options according to their preferences, in light of the context presented. This article shows you how to set up a ranking question, use the results between rounds and analyse them.

Ranking page example for round 2
Ranking page in round 2, as seen by participants

Overview

Plan: Delphi - Ranking pages are available on the Premium plan. See Which Welphi plan do I need?.

As with other types of questions in Welphi, rounds after the first can show the aggregated results as a chart, making it easier for participants to analyse the results and, if they wish, update their opinions.

Please keep in mind that if you have set up a ranking page in a previous round, you can only edit some details for that page in later rounds.

The article covers:

  1. Setting up ranking questions.
  2. Settings for round 2 and next.
  3. Ranking results between rounds.
  4. Viewing the ranking results.
  5. Analysing your ranking results.

Ranking questions

1. Add the page

  1. On the "Round details" screen, click "ADD NEW PAGE" and choose "Delphi - Ranking".
  2. Click the page name to open the "Ranking questions page".

2. Fill in the question

The page already contains a "Ranking Question", ready to be filled in:

  • QUESTION DETAILS – write the actual question being asked to participants.
  • RANKING OPTIONS – enter the options that you want participants to rank. Click "ADD NEW OPTION" to add a new line, as many times as you need. Use the ✕ next to an option to remove it.
  • MINIMUM RANKED OPTIONS – decide how many options each participant has to rank. Although it may seem obvious to have them rank all options, it may be useful to just ask people to choose and rank their top X options.

Note: check the MINIMUM RANKED OPTIONS value after adding your options, and set it to the number of options if you want everything to be ranked.

Setting up a ranking question
Ranking question set up with four options, top 3 to be ranked

3. Add texts, save and preview

  • Texts – in the "Texts" section of the left panel you can add an "INTRO TEXT", a "FOOTER TEXT" and an "INFO TEXT" for the page.
  • Save – click "SAVE AND PREVIEW" to save the page and see it as participants will.

You can also send participants to different pages depending on where they rank an option. See Sending participants to different pages with answer rules.

Settings – Round 2 and next

In later rounds, the "Settings" section of the left panel has three options:

  • Users can edit answers – leaving this box unchecked will prevent participants from changing their answer from the previous round. This can be helpful when you want to promote participant interaction during a round, for example by using a forum control, and let them change their opinions only at a later stage.
  • Show forum – the forum control is a live interactive chat control. Once you check this box, you can add forum topics ("ADD NEW TOPIC") you want your participants to focus on. You can also create new topics after the round has started.
  • Show chart – as long as the same page existed in a previous round, you can check this box to automatically include a chart with the aggregated answers of all participants from the previous round.

Ranking results between rounds

When "Show chart" is checked, Welphi automatically builds the statistics from the previous round and shows them to participants with each option.

What participants see

  • Bars – the group result of the previous round (all participants): the percentage of participants who placed the option in each position (1st, 2nd, 3rd…). Each bar adds up to 100%.
  • Markers – the "Round 1" square shows the participant's own ranking of that option in the previous round, and the "Now" square shows their current ranking (by default equal to their previous answer). A "–" means the option was not ranked.

Participants then drag and drop the options to confirm or change their ranking.

Which answers are counted

A few settings affect which answers are counted:

  • Include answers from uncompleted questionnaires – in Process settings, you decide whether answers from participants who did not complete the full questionnaire are included in the statistics. See Choosing your process settings.
  • Statistics filtering – on the Premium plan, participants can see the statistics of a specific group (for example, a region or area of expertise) alongside the global statistics. See Letting participants filter statistics by group.

Viewing the ranking results

  1. On the "Round details" screen, click the pie-chart icon ("View page analysis") next to the page.
  2. Click the question.

The analysis page shows:

  • the same chart, "ANSWERS GIVEN BY ALL PARTICIPANTS";
  • the answers given by each participant, in the "USER ID" list on the left: expand a participant to see their ranking. Participants who did not answer are marked "did not answer". The list shows Aliases unless you chose to reveal identities in Choosing your process settings;
  • the page's discussion forum.

You can do this while the round is still open: the results include the answers submitted so far.

Ranking results in the back office
Answers given by all participants

Analysing your ranking results

The statistics shown to participants are designed as feedback between rounds. Welphi does not compute a mean rank, a score or a consensus measure for ranking questions.

Exporting the data

To evaluate your Delphi results, for example to decide on consensus, stability or a final ranking:

  1. Click "EXPORT" on the "Round details" screen to download the round data.
  2. Apply the analysis defined in your study protocol.

Each participant is identified by an Alias in the export unless you chose to reveal identities (see Choosing your process settings).

Common measures

Measures commonly used for ranking questions include:

  • The mean or median rank of each option, and the final group order based on it.
  • The percentage of participants who placed an option in their top X.
  • Kendall's coefficient of concordance (W), to measure the level of agreement among participants, and how it changes between rounds.

Welphi does not impose a consensus definition: set your consensus criteria and stopping rules before the study starts and report them in your methods.

Methodological references

  • Schmidt, R. C. (1997). Managing Delphi surveys using nonparametric statistical techniques. Decision Sciences, 28(3), 763–774.
  • Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: an example, design considerations and applications. Information & Management, 42(1), 15–29.
  • von der Gracht, H. A. (2012). Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(8), 1525–1536.
  • Diamond, I. R., et al. (2014). Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401–409.
  • Beiderbeck, D., et al. (2021). Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. MethodsX, 8, 101401.