Home  ⁄  Publications & case studies  ⁄  Publications



No events listed.



STSMs 2016/2017

The action has funded 6 STSMs over the coming months! 

Successful meeting on Project Risk & Asset Management Uncertainty Assessment

October 2016: The Action held a workshop on Project Risk & Asset Management Uncertainty hosted by colleagues at TU Delft

Expert Judgement Workshop, 26th August 2016

An expert judgement workshop is being held at the University of Strathclyde on Friday 26th August!



Elicitation: The Science and Art of Structuring Judgement (Dias, Morton, Quigley, 2018)

Papers by Workgroup

Workgroup 1 - Processes & Procedures (Methodology)

Colson, A.  and Cooke, R.M., (2017) Cross Validation for the Classical Model of Structured Expert Judgment, Reliability Engineering and System Safety, Volume 163, July 2017, Pages 109–120 

Flandoli, F., E. Giorgi, W. P. Aspinall and A. Neri (2011) Comparison of a new expert elicitation model with the Classical Model, equal weights and single experts, using a cross-validation technique. Reliability Engineering and System Safety, 96, 1292-1310 doi:10.1016/j.ress.2011.05.012

Hanea, A.M., Burgman, M.A., Hemming, V., in press (accepted in Jan. 2017), IDEA for Uncertainty Quantification. In  Dias L. C., Morton A., Quigley J. (Eds), Elicitation: The science and art of structuring judgement,  Springer, New York, 2018 (Chapter 5)

J. Gonzalez, D. Ríos Insua, V. Radovic Modelling preferences and adversarial preferences, to appear in  (Dias, Morton, Quigley, eds). Handbook of Elicitation, Springer

Quigley J., Colson A., Aspinall W. and Cooke R. (2018) Elicitation in the Classical Model.  Ch. 2 in: Elicitation: The science and art of structuring judgement (eds. Dias L.C., Morton A., Quigley J.). Springer, New York, (in press, 31 May 2017).

Colson, Abigail R., and Roger M. Cooke. "Cross validation for the classical model of structured expert judgment." Reliability Engineering & System Safety 163 (2017): 109-120.

Werner, C., Bedford, T. and Quigley, J., The sequential refined partitioning method: addressing under- and overspecification of detailed expert judgement in probabilistic dependence modelling, under review (Risk Analysis)

Werner, C., Bedford, T. and Quigley, J., Mapping Conditional Scenarios for Knowledge Structuring in
(Tail) Dependence Elicitation, under review (Journal of the Operational Research Society)

Expert judgement for dependence in probabilistic modelling: A systematic literature review and future research directions (Werner, et al., 2017)

An investigation of dependence in expert judgement studies with multiple experts (Wilson, 2017)


Workgroup 2 - Dependence Elicitation (Methodology)

These papers are closely linked to the topic of the workshop held in Warsaw, Poland

Nogal, M.; Morales-Napoles O. and O'Connor, A. Understanding the vulnerability of traffic networks by means of structured expert judgment elicitation. Proceedings of the Irish Transport Research Network, Sept. 2016, Dublin (Ireland).


Workgroup 3 - Uncertainty Modelling: Adversarial Risk Assessment

These papers are closely linked to the topic of the workshop held in Madrid, Spain - workshop presentations can be viewed here

Banks, D, Ríos, J., Rios Insua, D. Adversarial Risk Analysis. Francis Taylor  (2015)

Cano, J., Rios Insua, D., Tedeschi, A., Turhan, U. (2016) Security Economics: An adversarial risk analysis approach to airport security, Annals of Operations Research (2016), 1, 359-378.

Rios Insua, F. Ruggeri, J. Gómez, C. Alfaro Robustness in adversarial risk analysis,   in Zopounidis, Doumpos (eds) Robustness Analysis in Decision Aiding, Optimization and Analytics (2016), 39-58,Springer

Ortega, J., Rios Insua, D., Ruggeri, F., Soyer, R. Hypothesis testing in presence of adversaries, submitted

Couce, A.,  Houmbd, S., RIos Insua, D. GIRA: A general model for incident risk analysis, with A. Couce, S. Houmbd, to appear in Journal of Risk Research.

Rios Insua, D, Ruggeri, F., Soyer, R., Rasines, D. Adversarial Reliability Analysis, with D. Rasines, F. Ruggeri, R. Soyer, to appear in European Journal of Operational Research

Rios Insua, D., Ruggeri, F., Soyer, R., Wilson, S. Advances in Bayesian games and decisions in reliability, to appear in European Journal of Operational Research.

P. Esteban, D. Ríos Insua An affective model for a non-expensive utility-based decision agent, to appear in IEEE Transactions in Affective Decision Making


Workgroup 4 - Applying Expert Judgement: technical, scientific and commercial

Natural Hazards

These publications are closely linked to the topic of the workshop held in Rome, Italy (presentations) & Reykjavic, Iceland (presentations)

Aspinall, W.P., Cooke, R.M., (2013) Expert Elicitation and Judgement. In: Risk and Uncertainty Assessment in Natural Hazards. (Rougier, J.C., Sparks R.S.J., Hill, L., eds).  Cambridge University Press, Chapter 4, 64-99.

Bamber, J.L., Aspinall, W.J. and Cooke, R.M. (2016) “A commentary on ‘How to interpret expert judgment assessments of 21st century sea-level rise’ by Hylke de Vries and Roderik SW van de Wal”, Climatic Change DOI 10.1007/s10584-016-1672-7 

Bevilacqua, A., Isaia, R., Neri, A., Vitale, S., Aspinall, W.P. and eight others (2015) Quantifying volcanic hazard at Campi Flegrei caldera (Italy) with uncertainty assessment: I. Vent opening maps. Journal of Geophysical Research - Solid Earth; AGU. doi:10.1002/2014JB011775

Hincks, T.K., Komorowski, J.-C., Sparks, R.S.J. and Aspinall, W.P. (2014) Retrospective analysis of uncertain eruption precursors at La Soufrière volcano, Guadeloupe, 1975-77: volcanic hazard assessment using a Bayesian Belief Network approach. Journal of Applied Volcanology 3:3 doi: 10.1186/2191-

Koch, Benjamin J., Febria, Catherine M. , Cooke, Roger M.  Hosen, Jacob D. , Baker, Matthew E. , Colson, Abigail R.  Filoso, Solange, Hayhoe, Katharine, Loperfido, J.V. , Stoner, Anne M.K. ,  Palmer, Margaret A. , (2015) Suburban watershed nitrogen retention: Estimating the effectiveness of storm water management structures, Elementa, DOI 10.12952/journal.elementa.000063

Koch, Benjamin J., Filoso, S., Cooke, R. M.  Hosen, J. D., Colson, A.R. Febria, Catherine M., Palmer, M. A. , (2015) Nitrogen in stormwater runoff from Coastal Plain watersheds: The need for empirical data, reply to Walsh , Elementa DOI 10.12952/journal.elementa.000079. 

Komorowski, J.-C., T. Hincks, R. S. J. Sparks, W. Aspinall and CASAVA ANR project consortium. (2015) Improving crisis decision-making at times of uncertain volcanic unrest (Guadeloupe, 1976). Ch. 8 in: Global Volcanic Hazards and Risk (eds: Loughlin, S.C., Sparks, S., Brown, S.K., Jenkins, S.F. & Vye-Brown, C.); Cambridge University Press, pp. 255-262 ISBN: 9781107111752

Neri, A., Bevilacqua, A., Esposti Ongaro, T., Isaia, R., Aspinall, W.P. and nine others (2015) Quantifying volcanic hazard at Campi Flegrei caldera (Italy) with uncertainty assessment: II. Pyroclastic density current invasion maps. Journal of Geophysical Research - Solid Earth; AGU.  doi:10.1002/2014JB011776

Scourse, E., Aspinall, W., Chapman, N. and Sparks, S. (2017) Expert judgement elicitation: application to science issues in siting facilities for geological disposal of nuclear waste. Ch. 18 in: Geological Repository Systems for Safe Disposal of Spent Nuclear Fuels and Radioactive Waste (eds. Apted, M. & Ahn, J.), Elsevier, 529-550.

Scourse, E., Aspinall, W.P. and Chapman, N. (2014) Using expert elicitation to characterise long-term tectonic risks to radioactive waste repositories in Japan. Journal of Risk Research doi: 10.1080/13669877.2014.971334

Sparks, R.S.J., Aspinall, W.P., Chapman, N.A., Hill, B.E., Kerridge, D.J., Pooley, J. and Taylor, C.A. (2013) Catastrophic impacts of natural hazards on technological facilities and infrastructure.  In Risk and Uncertainty Assessment in Natural Hazards. Rougier, J.C., Sparks R.S.J., Hill, L. (eds).  Cambridge University Press, Chapter 13, 445-480

Tadini A., Bevilacqua A., Neri A., Cioni R., Aspinall W.P., Bisson M., Isaia R., Mazzarini F., Valentine G.A., Vitale S., Baxter P.J., Bertagnini A., Cerminara M., De’ Michieli Vitturi M., Di Roberto A., Engwell S., Esposti Ongaro T., Flandoli F., Pistolesi M. (2017), Assessing future vent opening locations at the Somma-Vesuvio volcanic complex: 2. Probability maps of the caldera for a future Plinian/sub-Plinian event with uncertainty quantification, Journal of Geophysical Research : Solid Earth, 122(6): 4357-4376, doi:10.1002/2016JB013860

Tadini A., Bisson M., Neri A., Cioni R., Bevilacqua A., Aspinall W. P. (2017), Assessing future vent opening locations at the Somma-Vesuvio volcanic complex: 1. A new information geo-database with uncertainty characterizations, Journal of Geophysical Research : Solid Earth, 122(6): 4336-4356, doi:10.1002/2016JB013858

Ceppi, C., Menendez, K., Martin, J., Rios Insua, D. Risk analysis for climate change adaptation: the case of Jiquilisco, submitted

A two-dimension dynamic Bayesian network for large-scale degradation modelling with an application to a bridges network (Kosgodagan-Dalla Torre et al., 2017)

Characterisation of precipitation through copulas and expert judgement for risk assessment of infrastructure (Morales-Napoles, et al., 2017)

The accountability imperative for quantifying the uncertainty of emission forecasts: evidence from Mexico (Puig, et al., 2017)

Food Safety

These publications are closely linked to the topic of the workshop held in Dubrovnik, Croatia - workshop information can be viewed here

Aspinall, W.P., Cooke, R.M. Havelaar, A.H., Hoffmann, S. and Hald. T. (2015) Evaluation of a Performance-Based Expert Elicitation: WHO Global Attribution of Foodborne Diseases, PLoS One. 2016 Mar 1;11(3):e0149817. doi: 10.1371/journal.pone.0149817. eCollection 2016

Hald T, Aspinall W, Devleesschauwer B, Cooke R, Corrigan T, Havelaar AH, et al. (2016) World Health Organization Estimates of the Relative Contributions of Food to the Burden of Disease Due to Selected Foodborne Hazards: A Structured Expert Elicitation. PLoS ONE 11(1): e0145839. doi:10.1371/journal.pone.0145839

Hald, T.  Aspinall,W.  Devleesschauwer, B.  Cooke, R.M., Corrigan,T., Havelaar, A.H., Gibb, H., Torgerson, P., Kirk, M., Angulo, F., Lake, R., Speybroeck, N., and Hoffmann, S.  (2015) World Health Organization estimates of the relative contributions of food to the burden of disease due to selected foodborne hazards: a structured expert elicitation. PLOS ONE, : January 19, 2016 DOI: 10.1371/journal.pone.0145839,

Havelaar AH, Kirk MD, Torgerson PR, Gibb HJ, Hald T, Lake RJ, et al. (2015) World Health Organization Global Estimates and Regional Comparisons of the Burden of Foodborne Disease in 2010. PLoS Med 12(12): e1001923. doi:10.1371/journal.pmed.1001923

Hoffmann, S., Aspinall, W., Cooke, R.M. Cawthorne, A., Corrigan, T. Havelaar,A.H., Gibb, H., Torgerson, P., Kirk, M., Angulo, F.J. , Lake, R., Speyboeck, N., Devleesschauwer, B., Hald, T. (2016), World Health Organization, Foodborne Epidemiology Reference Group, Source Attribution Task Force.  Research Synthesis Methods in an Age of Globalized Risks:  Lessons from the Global Burden of Foodborne Disease Expert Elicitation.  Risk Analysis 36(2): 191-202. 

Hoffmann, S., Devleesschauwer, B., Aspinall, W., Cooke, R., Corrigan, T., Havelaar, A. et al. (2017). Attribution of global foodborne disease to specific foods: Findings from a World Health Organization structured expert elicitation.  PLoS ONE 12(9): e0183641. doi: 10.1371/journal.pone.0183641

Sandra Hoffmann, Brecht Devleesschauwer, Willy Aspinall, Roger Cooke, Tim Corrigan, Arie H. Havelaar, Frederick J. Angulo, Herman J. Gibb, Martyn D. Kirk, Robin J. Lake, Niko Speybroeck, Paul R. Torgerson,Tine Hald (2017)  Estimates of the relative role of specific food categories in causing major foodborne enteric diseases around the world: Findings from a World Health Organization structured expert elicitation,  PLoS ONE, Published: September 14, 2017


These publications are closely linked to the topic of the workshop held in Valletta, Malta - workshop presentations can be viewed here

Daniell, K., Morton, A., Rios Insua, D.  Policy analysis and policy analytics,  Annals of Operations Research, (2016), 236, 1, 1-13. 

pubmed/27733828"></wbr"="">" target="_self">Ermisch, M., Bucsics, A., Vella Bonanno, P., Arickx, F., Bybau, A., Bochenek, T., et. al. 2016. Payers' views of the changes arising through the possible adoption of Adaptive Pathways. Frontiers in Pharmacology. DOI: 10.3389/fphar.2016.00305

Schang, L, Hynninen, Y, Morton, A & Salo, A 2016, 'Developing robust composite measures of healthcare quality: ranking intervals and dominance relations for Scottish Health Boards' Social Science and Medicine, vol 162, pp. 59-67. DOI: 10.1016/j.socscimed.2016.06.026

Tyshenko, M.G., ElSaadany, S., Oraby, T., Darshan, S., Catford, A., Aspinall, W., Cooke, R. and Krewski, D. (2012) Expert judgement and re-elicitation for prion disease risk uncertainties. International Journal of Risk Assessment and Management, 16(1-3), 48-77. doi:10.1504/IJRAM.2012.047552

Vella Bonanno, P., Ermisch, M., Godman, B., Martin, A.P., Van Den Bergh, J., Bezmelnitsyna, L., Bucsics, A., Arickx, F., Bybau, A., Bochenek, T., van de Casteele, M., Diogene, E., Eriksson, I., Furst, J., Gad, M., Greiciute-Kuprijanov, I., van der Graaf, M., Gulbinovic, J., Jones, J., Joppi, R., Kalaba, M., Laius, O., Langner, I., Mardare, I., Markovic-Pekovic, V., Magnusson, E., Melien, O., Mashkov, D.O., Petrova, G.I., Selke, G., Sermet, C., Simoens, S., Schuurman, A., Ramos, R., Rodrigues, J., Zara, C., Zebedin-Brandl, E., Haycox, A. 2017. Adaptive Pathways: Possible Next Steps for Payers in Preparation for their Potential Implementation. Frontiers in Pharmacology  August 2017, Volume 8, Article 497. doi: 10.3389/fphar.2017.00497. 

Kozlova M., Morreale A., Lo Nigro G., Collan M. (2017) Modeling bio-pharmaceutical alliance games: Studying the effect of information availability on decision-making. Proceedings of the ROW17 - Real Options Workshop. Lappeenranta, Finland, pp. 4-5.


These publications are closely linked to the topic of the workshop held in Helsinki, Finland - workshop presentations can be viewed here

Porthin M, Rosqvist T, and Kunttu S. Risk Assessment using Group Elicitation: Case Study on Start-up of a New Logistics System, in: Dias LC, Morton A, Quigley J (eds), Elicitation: The science and art of structuring judgement.  Springer, New York, 2018 (Chapter 20)

Zdiarski, M., Nane, T., and Kinga, K. (2017). STRUCTURED EXPERT JUDGMENT SUPPORT FOR THE INITIAL INTERNATIONAL EXPANSION DECISION OF AN INNOVATIVE START UP The AIB-CEE Chapter Conference on the Changing Global Landscape and Regional Cooperation, Ljubljana, Slovenia, September 2017

Case Studies & Teaching Material

Below you can find some relevant case studies & teaching materials related to the work of Action IS1304:

Hemophilia Case
Summary: Cost-effectiveness of treatment strategies for Severe Haemophilia

Related Papers:

Summary: Kuwait Fires Case

Related Papers:
i) Mortality in Kuwait due to PM from oil fires after the Gulf War: Combining expert elicitation assessments
ii) What risk assessment can tell us about the mortality impacts of the Kuwaiti oil fires
iii) A probabilistic Characterization of the Relationship Between Fine Particulate Matter and Mortality: Elicitation of European Experts

The expert data from the case study in Excalibur format can be found here: PM25

Summary: Asian Carp Invasion

Related Papers:
i) Out-of-Sample Validation for Structured Expert Judgement of Asian Carp Establishment in Lake Erie
ii) Structured expert judgement to forecast species invasions: Bighead and silver carp in Lake Erie
iii) Using Structured Expert Judgement to Assess Invasive Species Prevention: Asian Carp and he Mississippi Great Lakes Hydrologic Connection

The expert data from the case study in Excalibur format can be found here: 


Related Papers:

 The expert data from the case study in Excalibur format can be found here: EunrcDD

Other case study summaries 
 Ice Sheet Dynamics
 Nitrogen runoff
 WHO Study Global

Teaching Materials
Training course on steering an expert judgement elicitation

Useful Links

Here you can find some useful links that will bring you to some external sources with useful content in the area of Structured Expert Judgement.

Common Resources, the Resources for the Future blog: Deep and Shallow Uncertainty in Messaging Climate Change (Roger Cooke) - see more at: