COST

This theme is concerned with graphical methods which are used to support elicitation of expert judgment.  Below is a summary of available software with associated references.

 

The Sheffield Elicitation Framework (SHELF) is a package of documents, templates and software to carry out elicitation of probability distributions for uncertain quantities from a group of experts. The software functions come with a set of word templates that might be useful in the elicitation process. In order to make  the package function, one has to install R and ‘source’ the functions. Subsequently the functions have to be addressed correctly. Visual, graphical feedback about the elicited distributions is provided.

The software also includes a web-based tool for eliciting probability distributions. Users can log in from different sites and they can all see and interact with the same graphics. SHELF carry out elicitation of probability distributions aiming particularly at quantifying uncertainty from a group of experts.

The univariate distributions use to model opinion are: Normal, Student t, scaled beta, gamma, log-normal and log Student-t. An extension quantifies opinion about a multinomial distribution by first eliciting marginal (beta) probabilities for each category, and then reconciling them to form a Dirichlet distribution. Offers a choice of assessment tasks for quantifying probabilities: quartiles, tertiles, fixed interval, roulette.

A positive aspect is that the academic basis of the package is solid and that several acknowledged methods for expert elicitation have been implemented. A drawback is the steep learning curve before one can successfully use the framework.

 

SHELF References:

O' Hagan, A., C.E. Buck, A. Daneshkhah, J.R. Eiser, P.H. Garthwaite, D.J. Jenkinson, et al. (2006). Uncertain judgements: eliciting experts' probabilities. Chichester: John Wiley & Sons Ltd.

SHELF and associated R routines are available http://www.tonyohagan.co.uk/shelf/index.html

Recently a website for eliciting model parameters based on the SHELF framework has been launched. See http://elicitator.uncertweb.org/

In order to support the practical application of expert analysis, two relevant software programs have been developed by the Department of Mathematics at Delft University of Technology:

EXCALIBUR (Expert CALIBRation) is a Windows program for expert judgment analysis. It allows the user to input experts’ quantile assessments and parameters, and combine their assessments based on equal weights, user weights, and expert performance-based weights (Cook 1991). EXCALIBUR implements Cooke’s classical method for elicitation and opinion pooling.

The performance of experts as probability assessors is measured by the experts’ ability to correctly and precisely provide estimates for a set of seed variables. Subsequently, different weighting schemes are applied in order to obtain combined PDFs as a weighted linear combination of the expert's individual PDFs. Robustness analysis shows how sensitive the results are to the choice of experts and the choice of calibration variables. Discrepancy analysis shows how the assessment made by individual experts differ from a specific aggregated PDF. The output is compatible with modern text processors and spreadsheets.

UNICORN (Uncertainty Analysis with Correlations) is a software package for uncertainty analysis. It is designed for dependence modeling with high dimensional distributions, including graphic features, such as cobwebs. Kurowicka and Cooke (2006) have used this package in examples mentioned in their book ‘Uncertainty Analysis with High Dimensional Dependence Modelling’. They state that Unicorn is a package for sensitivity analysis like @Risk and Crystal Ball, but that those packages do not support features such as multiple copula, vine modelling, cobweb plots, iterated and conditional sampling and probabilistic inversion.

Both programs are designed to facilitate the formal procedure of expert analysis, which is based on the principles of rational consensus. 

 

EXCALIBUR AND UNICORN References:

Cooke, R.M. (1991). Experts in Uncertainty; Opinion and Subjective Probability  in Science. New York & Oxford: Oxford University Press.

Cooke, R.M. & L.H. Goossens. (1999). Procedure guide for a structured Expert  Elicitation. Nuclear Science and Technology, EUR 18820: TU Delft.

Cooke, R.M. & D. Solomatine. (1992). EXCALIBUR - Integrated System for Processing Expert Judgements. Delft, the Netherlands: Delft University of Technology and SoLogic Delft.

Kurowicka, D. and R.M. Cooke (2006). Uncertainty Analysis with High Dimensional Dependence Modelling. Wiley

The Elicitator software allows to define an elicitation problem and invite experts to participate, who subsequently login to the website and complete a list of questions that make up the elicitation process. The Elicitator is built for applications in ecological modelling and uses maps to give experts instantaneous feedback about the consequences of their judgements on the model in question.

ELICITATOR References:

https://wiki.aston.ac.uk/foswiki/bin/view/UncertWeb/TheElicitator

http://www.uncertweb.org/documents

PEGS (Probability Elicitation Graphical Software) software is designed to implement a methodology for producing a subjective (personal opinion) distribution for the parameters of a generalised linear model (GLM) in a Bayesian format. The output of the software, the values of the hyper parameters of the subjective distribution, comes in R/S-plus format.

Separate programs (and a single combined program) elicit: Dirichlet and Connor-Mossiman priors,  Dirichlet and Gaussian copula priors and MVN prior for multinomial logistic model. Program that elicits an MVN prior also quantifies opinion about: the error variance in a normal linear model and the scale parameter in a gamma GLM. These are also available in separate stand-alone programs.

 

PEGS References:

Al-Awadhi, S A and Garthwaite, P H. (2006). Quantifying expert opinion for modelling fauna habitat distributions. Computational Statist., 21, 121-140.

Garthwaite, P H, Chilcott, J B, Jenkinson, D J and Tappenden, P. (2008). Use of expert knowledge in evaluating costs and benefits of alternative service provision: A case study. Int. J. Technol. Assess. Health Care, 24, 350-357.

Garthwaite, P H, Alawadhi, A S, Elfadaly, F and Jenkinson, D J. (2013). Quantifying subjective opinion about generalized linear and piecewise-linear models. J. Applied Statist., 40, 59-75.

http://statistics.open.ac.uk/elicitation