COST

Where we are:

In mathematical aggregation a mathematical rule is used to combine the judgements of the experts. There are two main ways ofdoing this: opinion pools and Bayesian aggregation. In opinion pools weights are given to each expert and then judgements are combined linearly or logarithmically using these weights. The weights can be specified based on performance of the experts on seed variables or the judgements of the decision maker. In Bayesian aggregation the expert judgements are regarded as data and are combined using Bayes theorem. 

There could be dependencies inherent in this process; the judgements of experts are dependent with other judgements made by that expert and of judgements made by other experts conditional on the true value of the unknown quantity. That is, individual experts may be subject to the same biases consistently, different experts may be subject to the same biases and different experts will typically have similar backgrounds and experience. Thus it seems likely that dependencies will exist within expert judgement studies and these need to be included in models. The former dependence is the basis for the use of seed questions in mathematical approaches. The latter is generally not captured in mathematical aggregation models. 

There are some models in the literature which consider correlations with multiple experts. [4] proposed a copula approach to model the dependencies between the judgements. In this approach the marginals were specified separately and the correlation was expressed in form of bivariate Frank copula. Another Bayesian approach to aggregation that incorporated correlations was the Normal model specified by [6]. In Winkler’s approach the marginals were uniform and the correlations were expressed in terms of a multivariate Normal distribution. [1] adopted a similar approach to Winkler. [3] developed a method of aggregation of judgements where the experts specified certain moments of a distribution rather than specifying the whole probability distribution. [8] and [7] studied the aggregation problem with analytical hierarchy process. [2] proposed an Empirical Bayes approach towards aggregating correlated expert judgements and also a non-parametric approach based on Lagrange multipliers with constrained optimisation for aggregation. 

There has been little work on evaluating whether these dependencies are exhibited in practice. An exception is [5] which looked at the [4] approach and implemented it for expert judgement data gathered at the T.U. Delft. Experts showed less dependence than might have been supposed. Clemen’s copula for aggregation was implemented and performance was compared with performancebased combinations for two illustrative cases. The Classical method out-performed this approach. 

Another issue is when panelss of experts working on different small sections of the same larger scientific problem. Suppose that a user and the panels now plan to combine these probabilistic judgments into a single comprehensive coherent model to inform policy. The different panels share as common knowledge a description of the qualitative relationships between the inputs and output random vectors of different probabilistic components. In this case a statistical methodology to inform the design of a Decision Support System (DSS) to address policy analyses within this environment or similar would be useful.

Targets for this theme:

• To evaluate whether dependency between experts is simply an interesting mathematical problem or is practically relevant by empirically assessing the dependency between experts in real expert judgement studies and controlled experiments.
• To assess the best way of evaluating the dependency between experts conditional on the true value of the variable when variables are on possibly very different scales.
• To consider possible methods based on seed variables for mathematical aggregation explicitly incorporating dependency between experts.
• To operationalise Bayesian approaches to mathematical aggregation incorporating dependency both between and within experts by considering elicitation issues for the parameters of such models.
• To investigate graphical models as tools to draw together probabilistic judgements incorporating dependency.

References:

[1] S. Chibber, G. Apostolakis, and D. Oaktrent. A taxonomy of issues related to the use of expert judgments in probabilistic safety studies. Reliability Engineering and System Safety, 38:27–45, 1992.
[2] T. Ganguly. Mathematical aggregation of probabilistic expert judgements (pending). PhD thesis, 2014.
[3] C. Genest, K. McConway, and M. Schervish. Characterization of externally bayesian pooling operators. the Annals of Statistics, 14:487–501, 1986.
[4] M. Jouini and R. Clemen. Copula models for aggregating expert opinions. Operations Research, 44:444–457, 1996.
[5] M. J. Kallen and R. M. Cooke. Expert aggregation with dependence.
[6] R. Winkler. Combining probability distributions from dependent information sources. Management Science, 27:479–488, 1981.
[7] Z. Xu. On consistency of the weighted geometric mean complex judgement matrix in ahp. European Journal of Operational Research, 126:683–687, 2000.
[8] E. Zio. On the use of the analytic hierarchy process in the aggregation of expert judgments. Reliability Engineering and System Safety, 53:127–138, 1996.