COST

There are three key aims within this theme.  Below we provide an overview of approaches to elicitation of subjective probabilities and a list of key references.

 

Theme Aims:

• To identify state-of-the-art approaches to expert elicitation which aim to reduce the elicitation burden on domain experts.
• To identify examples of good practice relating to labour-saving elicitation approaches.
• To investigate a subset of promising, labour-saving approaches to elicitation with a view to their testing and further development.

  

Elicitation of Probabilities:

Direct Methods

Use of a joint numerical and verbal scale was employed by van der Gaag et al (2002) in the elicitation of many probabilities which were required to build a Bayesian network model. The authors reported on the successful use of such an approach in the medical domain. In particular the clinical experts involved in the elicitation were very comfortable with it.

 

Indirect Methods

Bayesian networks offer a powerful approach to probabilistic modelling and have been applied widely. It is generally agreed, however, that the biggest obstacle to the development of a BN is the elicitation burden when hard data are lacking and domain expert elicitation is required to populate the conditional probability tables (CPTs).

The most popular way of overcoming this obstacle to date has been the use of so-called canonical models which, given certain usually plausible assumptions, allow large CPTs to be filled based on the elicitation of a small number of parameters. The Noisy OR model is the best known example of this in which several parent nodes, often corresponding to different potential causes of some common effect represented by a child node, all converge on the child node within the BN representation. The main assumption is the independence of causal influence – in particular the elicited parameters usually correspond to the probabilities that each separate parent cause will be inhibited, resulting in the effect not occurring when just that parent cause is acting, and so it is assumed that the inhibition mechanisms are independent. Zagorecki and Druzdzel (2004) undertook an empirical study which showed that BN models developed using the Noisy OR model could be just as accurate as equivalent BN models which were developed requiring the full elicitation of all probabilities. While the Noisy OR model is suitable for Boolean variables, a generalization of it exists for ordinal variables called Noisy Max.

Fenton et al (2007) outline an approach to the problem of eliciting conditional probability tables for large-scale Bayesian networks, applicable to a class of nodes called ‘ranked nodes’. A ranked node is defined as a discrete node having an ordinal scale which is an abstraction of some underlying continuous quantity. The discrete states are associated with continuous intervals of equal size on a normalized [0,1] scale, e.g. with five discrete states, the lowest would correspond to the interval [0,0.2]. This approach is implemented in their AgenaRisk software. It can be used in many common situations, for example, where a child node has many parent causes and the child is essentially a weighted average of the parent values. In the paper, they discuss practical aspects of BN model building and estimate the savings in elicitation effort over a conventional BN model for two case studies which they have been involved in.

 

Elicitation of Weights for Decision Analytic Models:

A recent survey considering the elicitation of criteria weights for prescriptive decision analytic models is provided by Riabacke et al (2012). While emphasising the crucial role which the elicitation process plays in the development of prescriptive decision analytic models, the authors point to trends making greater use of visual aids, verbal expressions, intervals and ranking. Rank Order Centroid (ROC) weights are a particular example of an approach to eliciting criteria weights which reduces the elicitation workload as it only requires a ranking of the criteria. They also highlight the role which sensitivity analysis could play in making the elicitation process more efficient.

 

Elicitation of Dependencies:

An alternative way of constructing Bayesian networks, requiring the elicitation of conditional and unconditional rank correlation coefficients rather than the elicitation of conditional and marginal probability tables, is described by Kurowicka and Cooke (2005) and Morales et al (2007). The considerable reduction in the required elicitation effort is demonstrated by a case  study in the air safety domain. The approach is supported by the UNINET software (Cooke et al, 2007).

 

Other Efficiency Improvements:

These are compatible with any of the above approaches to parameter elicitation.

Grouping of Parameters

Identifying several parameters which the expert judges to be sufficiently close that they can be represented with the same parameter value leads to a two-step elicitation process. In the first step, parameters are sorted into groups within which the values are judged to be close. In the second step, a value is elicited for each group, potentially using any suitable approach.

 

Parameter Ranking by Sensitivity

It is well known that some parameters have greater influence than others in both analytic and simulation models. Model results exhibit greater sensitivity to some parameters than others and this can be measured in various ways. This suggests that a more efficient use of limited elicitation time with domain experts is to focus the available effort in line with the sensitivity of the parameter. Nonetheless, some initial estimate of all model parameters will be required in order to obtain the required sensitivities. The idea is that this initial elicitation would be much cruder than that then afforded to the most sensitive parameters in the second phase. Such an approach is discussed in O’Hagan (2012), for example.

Bibliography:

Cooke RM, Kurowicka D, Hanea AM, Morales  O and Ababei DA, Ale B and Roelen A (2007) Continuous/Discrete Non-Parametric Bayesian Belief Nets with UNICORN and UNINET. In: Bedford T, Quigley J, Walls L and Babakalli A (eds.), Proceedings of the Mathematical Methods in Reliability Conference, Glasgow, 2007.

Fenton NE, Neil M and Caballero JG (2007), Using ranked nodes to model qualitative judgments in Bayesian networks. IEEE Transactions on Knowledge and Data Engineering 19(10), 1420-1432.

van der Gaag LC, Renooij S, Witteveen CLM, Aleman BMP and Taal BG (2002), Probabilities for a probabilistic network: a case study in oesophageal cancer. Artificial Intelligence in Medicine 25(2), 123-148.

Hora SC (2007), Eliciting probabilities from experts. In R. Edwards, R. Miles, & D. von Winterfeldt (eds.), Advances in Decision Analysis. Cambridge University Press.

Hora SC (2010), Expert Probability Elicitation Tools. Research Paper Summaries, Paper 17.

(http://research.create.usc.edu/project_summaries/17)

Kurowicka D and Cooke RM (2005), Distribution-free continuous Bayesian belief nets. In: Wilson A, Limnios N, Keller-McNulty S and Armijo Y (eds.), Modern Statistical and Mathematical Methods in Reliability, World Scientific Publishing Co., 309-323.

Meyer MA and Booker JM (2001), Eliciting and Analyzing Expert Judgment: A Practical Guide, ASA-SIAM.

Morales O, Kurowicka D and Roelen A (2007), Eliciting conditional and unconditional rank correlations from conditional probabilities. Reliability Engineering and System Safety 93, 699-710.

O’Hagan A (2012), Probabilistic uncertainty specification: overview, elaboration techniques and their application to a mechanistic model of carbon flux. Environmental Modelling and Software 36, 35-48.

O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE and Rakow T (2006), Uncertain Judgements: Eliciting Experts’ Probabilities. Wiley, Chichester.

Riabacke M, Danielson M and Ekenberg L (2012), State-of-the-art prescriptive criteria weight elicitation. Advances in Decision Sciences. Article ID 276584. DOI:10.1155/2012/276584

Zagorecki A and Druzdzel M (2004), An empirical study of probability elicitation under Noisy-OR assumption. In: Barr V and Markov Z (eds.), Proceedings of the 17^th International Florida AI Research Society Conference (FLAIRS 2004), 880-885. AAAI Press, Menlo Park, CA, USA.