Comparison of techniques for eliciting views and judgements in decision‐making

Decision‐making is a complex process that typically includes a series of stages: identifying the issue, considering possible options, making judgements and then making a decision by combining information and values. The current status quo relies heavily on the informational aspect of decision‐making with little or no emphasis on the value positions that affect decisions. There is increasing realization of the importance of adopting rigorous methods for each stage such that the information, views and judgements of stakeholders and experts are used in a systematic and repeatable manner. Though there are several methodological textbooks which discuss a plethora of social science techniques, it is hard to judge the suitability of any given technique for a given decision problem. In decision‐making, the three critical aspects are “what” decision is to be made, “who” makes the decisions and “how” the decisions are made. The methods covered in this paper focus on “how” decisions can be made. We compare six techniques: Focus Group Discussion (FGD), Interviews, Q methodology, Multi‐criteria Decision Analysis (MCDA), Nominal Group Technique and the Delphi technique specifically in the context of biodiversity conservation. All of these techniques (with the exception of MCDA) help in understanding human values and the underlying perspectives which shape decisions. Based on structured reviews of 423 papers covering all six methods, we compare the conceptual and logistical characteristics of the methods, and map their suitability for the different stages of the decision‐making process. While interviews and FGD are well‐known, techniques such the Nominal Group technique and Q methodology are relatively under‐used. In situations where conflict is high, we recommend using the Q methodology and Delphi technique to elicit judgements. Where conflict is low, and a consensus is needed urgently, the Nominal Group technique may be more suitable. We present a nuanced synthesis of methods aimed at users. The comparison of the different techniques might be useful for project managers, academics or practitioners in the planning phases of their projects and help in making better informed methodological choices.


| INTRODUCTION
Effective decision-making is at the heart of successful biodiversity conservation and management.From addressing human-wildlife conflict (Redpath et al., 2013) to optimizing the use of resources, efficient management interventions are urgently required to address the loss of biodiversity.Research on decision-making in conservation has increased over the years (Figure 1) and it is increasingly accepted that there are benefits from using rigorous means of making assessments (Sutherland & Burgmann, 2015).
Decision-makers often have to make decisions in the absence of complete information or when the existing scientific information does not provide adequate evidence to solve the management problem at hand (Sutherland et al., 2004).For instance, the spatial and temporal scales at which ecologists conduct their research may be vastly different from those at which land-use decision-makers need to make decisions (Habel et al., 2013).Recent lawsuits filed by the Centre for Biodiversity in the US have forced several listing decisions by the US Fish and Wildlife Service based on inadequate information.Furthermore, for local-scale management issues, there might be a lack of site-specific studies or cost-effective solutions.
Decision-making in conservation (and elsewhere) is essentially a human enterprise.It is shaped by the same agencies, dynamics and biases that shape decisions in any context.Inevitably, values and verifiable facts shape conservation decision-making (Chan et al., 2016).
Since biodiversity conservation is also a social construct (Fischer & Young, 2007), its management requires an explicit recognition of the dynamics of shifting perceptions (framings) and goals of conservation (Brown & Everard, 2015;Mace, 2014).Failing to integrate the human dimension in conservation often leads to less effective interventions (Bennett et al., 2017).This is in contrast to the current paradigm of mechanistic underpinnings of solving decision-making problems (Gregory et al., 2012).Decision-making is too often approached as a structured problem that needs "fixing" without adequate emphasis on the value judgements that shape real world decisions.Researchers (often trained in natural sciences) typically tend to move rapidly into highly complex modelling exercises without any consideration of the value positions from which the different priorities arise.
Currently there seems to be growing recognition of the paradigm of post-normal science (Funtowicz & Ravetz, 1994).In the context of sustainability, knowledge creation consists of more than the rational, cognitive and technical procedures of science as previously understood.Instead, knowledge creation is perceived as a process or practice (Zanotti & Palomino-Schalscha, 2015) that needs to deal with unstructured problems.Unstructured problems are characterized by a lack of repeatability due to uncertainty over elements of components of the problem, such as norms and values (Hugé, Block, Waas, Wright, & Dahdouh-Guebas, 2016).Biodiversity and conservation decisions often incorporate unstructured problems (DeFries & Nagendra, 2017).
Where there is a need to incorporate different perspectives there is a need to involve multiple actors in these decisions.
conflict is low, and a consensus is needed urgently, the Nominal Group technique may be more suitable.
5. We present a nuanced synthesis of methods aimed at users.The comparison of the different techniques might be useful for project managers, academics or practitioners in the planning phases of their projects and help in making better informed methodological choices.
conservation, decision-making, Delphi technique, focus group discussion, interview, multicriteria decision, nominal group technique, Q methodology F I G U R E 1 Evolution of publications on decision-making in conservation.The figures for number of studies per year were searched and downloaded from the search engine Scopus in March 2017.The keywords used were 'decision-making', 'conservation' and 'decision-making AND conservation' respectively.To estimate the total number of publications in the Scopus database the following search string was used: (TITLE-ABS-KEY(a or e or i or o or u) AND (LIMIT-TO (DOCTYPE,"ar") OR LIMIT-TO (DOCTYPE,"re") OR LIMIT-TO (DOCTYPE,"ip"))).Number of total studies in all disciplines is approximate The rational choice for making decisions involving social constructs would be to emphasize the need to understand stakeholder's perceptions and values.For the purposes of this paper, we define decision-making as the process of identifying options and selecting a feasible solution, based on evidence combined with the decisionmaker's values and experience (DeFries & Nagendra, 2017).There is growing acknowledgement of the importance of producing a robust and comprehensive knowledge base in order to address the diverse challenges in biodiversity conservation (Adem Esmail, Geneletti, & Albert, 2017;Binder, Hinkel, Bots, & Pahl-Wostl, 2013).There are multiple sources of conservation-relevant knowledge, requiring crossing disciplinary boundaries (e.g.Adem Esmail & Geneletti, 2017).The call for inter-and trans-disciplinarity resonates in both the science (Bennett et al., 2017;Mace, 2014;Tallis et al., 2014) and policy communities (IPBES, 2016).Documenting the knowledge of practitioners is a challenge (let alone quantifying it), even though some studies suggest that practical knowledge and social learning play a crucial role in decision-making (Weiss, 1979).While the diversity of values and voices is to be welcomed as it can generate innovative and socially robust solutions, this diversity can also appear overwhelming.Translating the calls for coproduction of knowledge and inter-disciplinarity into practice is thus a daunting task (Sutherland, Gardner, Haider, & Dicks, 2013).This complexity of conservation decision-making motivates the need to find effective multidisciplinary tools for decision-making that can incorporate a range of sources of knowledge.We need a clear understanding of which methods are most suitable and in which context.For example, techniques such as the Nominal Group Technique or Delphi technique (Hugé & Mukherjee, 2017) improve decision-making compared to face-to-face meetings (Graefe & Armstrong, 2011).These methods originate in diverse and specific academic disciplines or "silos," making them generally relatively unknown and underused in ecology and conservation.Academic training of conservation scientists should ideally introduce them to the various options available for decision-making (Tallis et al., 2014).
To address the challenges above, we briefly introduce and compare six methods that can facilitate decision-making in biodiversity conservation.In decision-making, the three critical aspects are "what" decisions need to be made, "who" makes the decisions and "how" the decisions are made.The methods covered in this paper focus on "how" decisions can be made and on how perceptions can be elicited.For a review on "who" makes decisions, please see Reed et al. (2009) for a typology of stakeholder engagement methods.Brown and Everard (2015) provide a detailed typology of "what" type of decision responses are possible under an ecosystem approach.An in-depth overview of the range of social science techniques for conservation is provided in appendix A in Bennett et al. (2017).Succinct reviews on knowledge synthesis techniques can be found in Pullin et al. (2016) while integrative assessment methods are covered in Ness, Urbel-Piirsalu, Anderberg, and Olsson (2007).In terms of "how" decisions are made, three questions are key: (1) What drives decisions?(2) How can the process of decision-making be structured to reduce bias and inefficiencies?and (3) How best to evaluate past decisions so as to improve them?
There is no single optimal method for making decisions or for eliciting views and judgements leading to decisions.We compare six techniques that, to our knowledge, are best suited to the various stages of the decision-making process and for eliciting judgements in conservation.These techniques were selected based on consultation with experts as well journal editors.The techniques discussed here are Interviews, Focus Group Discussion (FGD), Nominal Group Technique, Q methodology (Q), Delphi technique, and Multi-criteria Decision Analysis (MCDA).Of these, five are reviewed for the first time in conservation in this special issue, and the Delphi technique was reviewed elsewhere (Mukherjee et al., 2015).This paper aims to: 1. describe and compare the six methods and their respective strengths and weaknesses; 2. review the current use/application of these methods in ecology and conservation; 3. provide guidance to assess the suitability and feasibility of using these methods for a given question.

| Review process
Narrative reviews (as opposed to systematic reviews or synopses) often suffer from cherry picking of evidence (confirmation bias) by the authors (Hagger-Johnson, 2014).To minimize this bias, we provide a comprehensive overview of the application of each of the six techniques, through a systematically conducted structured review in Scopus adapted from the guidelines by Collaboration for Environmental Evidence (CEE, 2013).We intended to understand how these techniques have been used as methodological tools in conservation in the last two decades in biodiversity, ecology and conservation research.For each technique, we used a combination of keywords such as 'Delphi*' AND 'conserv*', OR 'ecology', OR 'biodivers*' (where "*" denotes a wild card to search for alternative word endings), in a search query within the Scopus database (https://www.scopus.com),from 1996 to 2016.The database was accessed between 20 April and 21 April 2017.A detailed account of the search terms used can be found in the respective papers.The resulting primary articles (excluding reviews) were screened for relevance to conservation, biodiversity and ecology.We excluded articles that had focussed primarily on conservation of other natural resources (e.g.water or soil conservation) but did not have a direct bearing on biodiversity conservation.The relevant articles after the initial title and abstract screening were used for full text screening.We screened the full text of the articles based on two criteria (1) the technique was mentioned in the method section and used to answer a scientific question (as opposed to being alluded to in the discussion or introduction), (2) the research focus was unambiguously on some aspect of biodiversity.The resulting articles were included in the final review.
A total of 548 papers have been covered in this special issue most of which related to Interview (n = 227) and FGD (n = 170), while the rest were Q (51), NGT ( 14) and MCDA (86).In addition, in order to cover the most recent papers since the publication of the earlier review for the Delphi technique (Mukherjee et al., 2015), we updated the database as of 4 March 2017.The earlier paper (Mukherjee et al., 2015) had 36 studies, while the revised database has 49.To maintain parity across all techniques, we selected a subset of studies from each of the techniques for comparison (subset of papers for Interviews (n = 107) and FGD (n = 116), all of the rest (Q = 51, NGT = 14, MCDA = 86 and Delphi = 49).Thus, this paper is based on a subset of 423 of the total 548 papers.A common protocol was used for reviewing the literature for all the methods (Annex S1).This protocol included the decision context, geographic scale, sample size (group size), duration, rationale for choosing the method and its use with other methods.Advantages and disadvantages experienced by authors in their application of each method were also noted when mentioned.

| Brief description of the methods
The key questions that typically need considering when selecting a technique to aid decision-making or elicit perceptions span three aspects: Conceptual (Which methods can be used in which circumstance to achieve which objective?How to make the case for using a particular method as opposed to another?), Logistic (does the method require external guidance or expert assistance?What are the main requirements in terms of labour, skills and money?), and Complexity (Does the situation involve a high level of conflict?).
To answer each of these questions, we first divide decision-making into seven chronological steps based on the traditional policy cycle.These steps can be broadly categorized into pre-decision, decision and post-decision, analogous to ex ante and ex post evaluation in the policy cycle.Pre-decision includes understanding perspectives, which may be useful for making subsequent decisions.Decision (steps 2-5) refers to the actual process of engaging stakeholders to reach a consensus.Post-decision could include gathering feedback on a management intervention that is already in place or evaluating the impact of an existing policy.This feedback can be used to improve subsequent decisions or to draft new policies.The seven steps for decision-making are as follows: 1. Pre-decision: Gathering baseline information that could be relevant to defining or assessing the extent of a problem where decision-making is needed.
The chosen methods are briefly described below as they have been already described in detail in respective papers in this special issue (see references at the end of each technique).These articles provide the best practice guidelines for each of the techniques for conservation purposes or applications.For detailed descriptions of the methods, please refer to the original sources provided in these cited articles.
An Interview is an interchange between two or more people in which one of them attempts to elicit information or expressions of opinion or belief from the other person or persons.Interviews can be structured, semi-structured or unstructured in format (Young et al., 2017).
Focus Group Discussion is a technique where a researcher assembles a group of individuals to discuss a specific topic, aiming to draw from the complex personal experiences and personal actions, beliefs, perceptions and attitudes of the participants through a moderated interaction (Nyumba, Wilson, Derrick, & Mukherjee, 2017).
Nominal Group Technique is an interactive group decision-making technique primarily targeted at gathering consensus.Participants are requested to provide information silently and individually (hence nominal) to questions asked by a moderator.The moderator collates all the information and creates a list of unique items.Subsequently, the participants are asked to prioritize these items following a collective (hence group) discussion (Hugé & Mukherjee, 2017).
Q Methodology is a method to understand the main perspectives or opinions on a topic, within a group of key actors.Respondents are asked to rank a set of items that prompt a subjective opinion (such as from most agree to most disagree).It then uses multivariate data reduction techniques to synthesize all the rankings into a typology of perspectives about the issue under consideration (A.Zabala, C. Sandbrook, & N. Mukherjee, unpublished data).
The Delphi technique is a group-based, anonymous and iterative technique with controlled feedback.The Delphi technique is traditionally aimed at gathering consensus on a complex topic from a group of experts (Mukherjee et al., 2015).
Multi-criteria Decision Analysis is a method to support decisionmaking that explores the balance between the pros and cons of different alternatives to accomplish a specific goal.It assesses the performance of alternatives across criteria, and therefore assists in framing decision problems, exploring trade-offs, formulating a decision and testing its robustness (Adem Esmail & Geneletti, 2017).

| Comparison of the techniques
The suitability of a technique depends on a combination of its stage in the decision-making process, logistical issues and the degree of conflict.Three of the techniques (Interviews, Q method and FGD) are not used for making decisions as such but are highly useful for eliciting views.If the objective is to consult and understand the perceptions of stakeholders before a decision is made, then one might use Interviews or FGD.If the aim is to understand the interlinkages in opinions between topics or patterns of perspectives, the Q methodology may be more appropriate.Once the drivers are understood and the aim is to prioritize decision alternatives, MCDA or its variants (e.g.Analytic Hierarchy Process) may be more suited (see Anselin, Meire, & Anselin, 1989).These patterns of usage are also highlighted in the overview of studies (Figure 2).Multi-criteria methods may build upon baseline information gathered through other techniques (Interviews, FGD, Nominal Group Technique or Delphi).For instance, in the review we noted that workshops, Delphi technique, FGD, and face-to-face meetings have been used to understand the values and perceptions of stakeholders.However, if the aim is to evaluate a decision that has already been made (e.g. a management intervention or policy already in place), then one may use the FGD method (group-based) or Interview and Q methodology (both individual-based).We created a flowchart to guide users based on the decision context and the methods reviewed in this paper (Figure 3).
In terms of logistical constraints, both FGD and Interviews require relatively larger sample sizes compared to Q, Nominal Group Technique or the Delphi technique (Table 1).For understanding global scale issues the Delphi technique (iterative online and anonymous) is suitable, as geographic proximity is not required.However, organizing a Delphi technique as well as Q can take considerable pre-planning and preparation time.Our analysis of the application of five of the techniques that directly involve consulting people is presented in Table 1.MCDA is not included here as it builds upon the social data gathered by the other techniques.
The Delphi technique, Nominal Group Technique and FGD also require a highly skilled facilitator team (Table 1).Good communication and observation skills, ability to build a rapport and attention to detail are required for all the techniques.However, group-based techniques such as Nominal Group Technique or FGD require a keen eye for conflict management and facilitation skills to keep the process participatory and encourage silent stakeholders.The Q methodology requires ample patience on the part of the researcher to give respondents adequate time to sort all the items during the sorting phase.
Since the Delphi technique is primarily used for experts, it may need strong negotiation skills to arrive at a consensus when there are strong differences of opinion.
If the level of conflict is high and individuals might not be comfortable in a face to face setting, then one may use the Q methodology or anonymous Delphi technique.The Q methodology and Interviews are also free from biases such as group-think and dominance effect (see Table 2 below), which might affect group-based techniques such as Nominal Group Technique or FGD.Instead, if the objective is to rapidly reach a consensus (i.e.make a decision) in a face-to-face group setting where conflict is low, one may use the Nominal Group Technique.
While FGD s and Interviews are well-known, the Nominal Group Technique and Q methodology are still relatively unknown (Figure 4).Nominal Group Technique often suffers from being confused with the Delphi technique.We noted 11 cases where studies used a Nominal Group Technique but incorrectly reported it as the (modified) Delphi technique.Figure 4 also shows that FGD has been extensively used in the global South (particularly in Africa) for site-specific studies.This might be a reflection of the relative ease with which the technique can be applied and cultural aspects that underpin the preference of interactive group-based settings.

| Biases in decision-making
There are a range of psychological biases that can distort decisionmaking and reduce its effectiveness (Table 2).The precise manner in which decisions are made (e.g. in groups or individually) makes the decision process susceptible to certain types of bias.Groups are more rational decision-makers than individuals for a range of decisionmaking contexts.For instance, groups reduce the overconfidence T A B L E 2 List of potential biases affecting group-based decision-making.The ones with ( a ) have been described briefly in an earlier paper (Mukherjee et al., 2015).(FGD, Focus group discussion; NGT, Nominal Group Technique)

Methods not affected
Production blocking In a brainstorm session, one cannot think of new ideas while listening to others in the group at the same time.Individuals need to wait to verbalize an idea while someone else is talking and this leads to blocking of the thought process.One may even forget the initial idea due to limitations of the short term memory FGD NGT, Delphi Free riding or social loafing People reduce their effort when working in a group as opposed to working alone, expecting other group members to complete the task FGD NGT, Delphi Hindsight bias Individuals believe that they "knew it all along" i.e. an event is more predictable after it has already occurred than a priori Individuals in a group tend to seek concurrence among the group at the expense of independent critical thinking.Members tend to avoid creating disunity and support the decisions taken by the majority or the perceived leader of the group.The desire or pressure to be accepted as a good group member leads to acceptance of the majority solution that may not be logical or scientifically sound FGD Egocentrism a Individuals tend to preferentially rate their own opinion higher than that of others FGD effect by 24% in comparison to individuals (see review in Kugler, Kausel, & Kocher, 2012).Groups are also better than individuals in addressing information cascade bias, learning faster and engaging better in strategic play (Kugler et al., 2012).Using a group-based technique (Nominal Group Technique or Delphi technique) can thereby harness the collective power of the minds in the group.However, in interactive settings group-based techniques can be susceptible to biases such as production blocking, group think, dominance effect and halo effect.To illustrate this further, we outline a selection of key biases that can affect each of the methods (Table 2).These biases are selected based on a systematic review on group and individual decisionmaking (Mukherjee, Dicks, Shackelford, Vira, & Sutherland, 2016).
Though these biases cannot be overcome completely, it is important to be cognizant of them if and when they arise.Using an anonymous technique such as the Delphi technique can help in addressing some of the biases affecting interactive settings.

| DISCUSSION
Calls to integrate more social sciences in conservation are duly acknowledged (Bennett et al., 2017;Marshall et al., 2017), yet scientists, conservation practitioners and decision-makers need guidance on how to realize this integration.Methodological rigour is a prerequisite to achieve a fruitful integration of human perceptions and values, as well as expert opinions in support of effective conservation (Miller, Minteer, & Malan, 2011).As conservation decisions are based on the interaction between values, evidence, interests and biases (Levine, Chan, & Satterfield, 2015), there is a need for targeted and adequate methods that allow to address this web of decision-influencing factors.We provide a comparative analysis of a range of methods that researchers can use in support of inclusive decision-making.All the methods reviewed are increasingly being used in support of conservation decision-making, e.g.half the papers that have used the Q methodology in conservation were published in the last 3 years.However, there is a paucity of discussion about how they can specifically aid conservation management and of the specific challenges of using them in conservation.A consistent pattern was observed in the reporting of the studies reviewed.Only a handful of papers justified their choice of technique (see reviews in Nyumba et al., 2017;Young et al., 2017;A. Zabala, C. Sandbrook, N. Mukherjee, unpublished data), which may reflect the fact that the variety of available methods is relatively unknown, and/ or that many researchers applying these methods in the field of conservation are not adequately trained to report on the methodological aspects of social science methods with the required clarity and rigour.
This situation hampers a systematic selection, justification and uptake of a much-needed social science methodological toolkit in conservation decision-making.Most reviewed studies did not report the duration of the study, sample size or number of iterations or whether ethics clearance was obtained before the study.Rarely have the studies provided a critical reflection of the advantages and disadvantages of using the method selected.Consequently, to address these gaps, and to aid better reporting in the future, each paper in this issue provides a flowchart explaining the key steps for that technique.
A significant caveat in this exercise is the overarching bias on peer-reviewed literature and that published in the English language.
Language (both natural language and technical jargon) is a considerable barrier in the science-policy interface (Amano, Gonzalez-Varo, & Sutherland, 2016).Perhaps a reading of the literature in other languages would add to the nuances of decision-making in different settings.However, it was beyond the scope of this paper to investigate literature in other languages or grey literature sources.
This paper complements conceptual papers introducing the need for and the diversity of social science methods (such as Bennett et al., 2017) by providing a detailed comparison of the aims, practical steps, strengths and weaknesses of a range of methods.Bridging the jargon and concepts gap is a key step, in addition to the need to provide practical, science-based advice on which method to use when.This is what the present review aims to achieve.We hope that this review will lead to further insights and clarity regarding the suitability of the different methods, in order to integrate the social component into conservation research in a systematic and rigorous way.

F
I G U R E 2 Geographical scope and decision context of the studies for each of the techniques reviewed in this paper (n = 423).Sub-national = sites within one country, National = country level, Multi-country = two or more countries, including resources shared by a number of countries, e.g.transboundary areas, Global = worldwide in scope.The radii of the circles are proportional to the square root of the number of studies F I G U R E 3 Flowchart of suitable methods based on the seven decision-making steps.(Int = Interviews, FGD = Focus Group Discussion, NGT = Nominal Group Technique, MCDA = Multi-criteria Decision Analysis).The suggestion of the methods for each of the seven steps is based on the reviewed applications of the techniques in conservation as well the consideration of three parameters (conceptual, logistic and complexity) discussed in this paper.The columns on the right represent the different stakeholder groups for which the techniques might be most suitable for through there is considerable room for flexibility T A B L E 1 Comparison of five techniques used for decision-making in conservation This structured review of six key methods to aid decision-making allowed us to identify the strengths and limitations of each.The list of techniques is certainly not exhaustive.Two commonly used methods that are not covered here are questionnaires (surveys) and choice modelling.The survey technique is also used to understand perceptions of stakeholders.However, it has already been reviewed elsewhere(White, Jennings, Renwick, & Barker, 2005).Choice modelling F I G U R E 4 Geographical distribution of each of the six techniques applied in conservation decision-making.Studies that were conceptual or had a global reach have not been included in the map.For studies covering more than one country, all the countries have been included in the map is useful for assigning value to environmental goods and services or for gathering baseline information on illegal behaviours or controversial issues, which are difficult to discern otherwise (e.g. in the case of orchid tradeHinsley, Verissimo, & Roberts, 2015).These could be combined within the scope of an Interview or survey (if individually administered) or a FGD (if group-based) and are thereby complementary to the techniques already covered in this paper.