COST

Kick Off Conference - Experts in Uncertainty

The First meeting of the Action will be at the University of Strathclyde, Glasgow on 2-4 April 2014. Speakers include Roger Cooke, Willy Aspinall, Abby Colson and Mark Burgman.

Programme:

Expert Judgment Network: Bridging the Gap Between Scientific Uncertainty and Evidence-Based Decision Making

Open meeting  - University of Strathclyde, Glasgow, 2-4 April 2014

The programme for 2 April is designed for decision makers, policy makers and advisors, and will give an overview of Structured Expert Judgement methods and applications with leading international figures.

Professor Tim Bedford (University of Strathclyde, Glasgow):

Structured Expert Judgement

Governments and Businesses have to make decisions in the face of risks and uncertainties, but time pressures too often hamper the collection of large-scale datasets that might help reduce those uncertainties. Structured Expert Judgement aims to provide a proper evidence base that enables decision makers to obtain reasonable uncertainty assessments in a timely and defensible way. The best methods of Structured Export Judgement ensure that expert data is collected in a way that minimizes potential biases and, crucially, tests expert performance.

Decision makers can and should demand that expert judgement procedures provide for the most unbiased possible forms of judgement. Unfortunately events such as the prosecution of experts in Italy in relation to the Aquilla earthquakes illustrate that there are many conflicting priorities which can negatively affect the potential for future unbiased judgement. In this talk we will take a broad look at the challenges and demands for uncertainty assessment that make this COST Action so important. 

Professor Mark Burgman (Australian Centre of Excellence in Risk Analysis, University of Melbourne):

The intelligence game

This presentation will outline the rules of the game designed by IARPA in the United States to evaluate expert predictions of geopolitical events. It will describe the full results from the first two years of operation. It will outline preliminary results from 2014, the final year of the game, including the performance of alternative approaches to weighting. In 2010, the US Intelligence Advanced Research Projects Activity (IARPA) announced a 4-year forecasting “tournament”. Five collaborative research teams attempted to outperform a baseline opinion pool in predicting hundreds of geopolitical events. Biosecurity specialist Mark Burgman led one of the teams, eliciting forecasts from Delphi-style groups in the US and Australia. Probabilities for events like "Will Australia formally transfer uranium to India by 1 June 2012?" are elicited and combined using measures of expert performance.  The Delphi teams perform well compared to a host of alternative approaches, and have the advantage of operational simplicity. More generally, the project has emphasized the need to develop and test tools for elicitation that disregard conventional views on an expert’s status. Even this ambitious project explored a tiny part of the expert judgment space, namely, binary predictions for geopolitics assessed using a Brier score. It underscores how much work there is to do, and how urgent and critically important it is.

Professor Willy Aspinall (University of Bristol, Aspinall & Associates):

Uncertainty Erupting

When a volcano becomes restless, public officials must take life and death decisions in the face of massive uncertainties. Geophysicist Willy Aspinall is internationally renowned for corralling the, often conflicting, opinions of individual volcanologists in order to provide to decision-makers in the volcanic hot seat real time assessments of the evolving probabilities for catastrophic eruption.  Over the years his approach has yielded probabilistic forecasting returns of which stock brokers can only dream, and Willy will share some of his more colourful experiences and show how his results stack up over the long term Montserrat Volcano Observatory.  Also, he will share practice insights from facilitating elicitations for a multitudinous range of diverse scientific, medical, engineering, security and safety problems.

Professor Roger Cooke (Resources for the Future, Washington DC and University of Strathclyde, Glasgow):

Quantifying Uncertainty on thin Ice

Risk analysis is about improbable and  unwelcome events. Data is always in short supply and uncertainty quantification is the order of the day. Mathematician risk analyst Roger Cooke has a distinguished career of taking big uncertainties to the mat.  He has recently been engaged in quantifying uncertainty in ice sheet dynamics in the face of global warming. This problem is not only the major wild card in global climate change, it forces analysts to go beyond the off-the-shelf methods.   Roger was lead author in the 5th assessment of the Intergovernmental Panel on Climate Change for the chapter on Risk and Uncertainty.

Abigail Colson (Center for Disease Dynamics, Economics and Policy, Washington DC):

Bang for Benevolent Bucks

Worthy goals are easy to formulate but hard to verify.  Even philanthropic policy makers want to know the bang for their bucks. How much good have we accomplished? What would have happened if we hadn't been there? The Robert Wood Johnson Foundation recently contracted for the development of innovative measures of policy effectiveness, and Abby Colson was able to apply the principles of structured expert judgment to score policy effectiveness. There are many pointers for wider application.  

Professor David Rios Insua (Rey Juan Carlos University, Madrid):

Some problems in uncertainty modelling and foundational issues 

I shall outline some possible problems of interest concerning uncertainty modelling and foundational issues in relation with expert judgement. These might include: elicitation for main distributions, risk matrices, proper discretisations, multivariate extreme models, scoring rules, expert dependencies, preference modelling, adversarial uncertainty, sensitivity analysis and technology. My objective is to stimulate discussion and brainstorm about problems of interest for WG3 in coordination with other WGs.

Dr Anca Hanea (Delft University of Technology):

Eliciting dependence: The why, what and how

Most elicitation methods are focused on making assessments of the uncertainty of single quantities.  However, risk models usually have more than one uncertain input, and we must consider how to represent uncertainty about multiple quantities jointly. The simplest option is to assume independence, but this is often a faulty assumption, especially for tail events. Nevertheless, to formulate a structured process for eliciting dependence, one has to first answer questions of the following sort:

-- What should we ask experts about, when we want to represent dependence?
-- How should we formulate the questions for experts in an optimal way?
-- How can we evaluate experts'  performance as dependence assessors?

to only mention a few. 

Other questions, concerns and research themes emerge along the way. The dependence modelling working group offers a common platform for formulating and answering questions, tackling problems and formulating "best" solutions.

Professor John Quigley (University of Strathclyde, Glasgow):

Multiple Experts Making Multiple Assessments: A Case Concerning Infrastructure Assessment

Testing engineering structures to assess reliability can be challenging and expensive, hence motivating the design of novel test and sampling strategies.  For example, restricted access may limit the possible number of samples; safety considerations may constrain the location from which samples can be gathered and the types of engineering tests that can be undertaken.  While test and statistical sampling strategies can be devised, a mechanism is required to select between alternative proposals to ensure that representative and informative test samples will be obtained. However it is challenging to support decisions between alternative testing strategies due to the lack of objective data and the reliance on expert judgement, which may be vague and potentially in conflict.

This research involves developing a process to elicit and structure expert engineering judgement to be represented as a Bayesian Network (BN), which provides a model to support comparison of alternative test and sampling strategies.  A BN is capable of modelling the interaction between multiple factors and, where the influences of these factors are uncertain, measuring these uncertainties probabilistically through a prior distribution. The decision support provided by the BN is achieved by comparing the anticipated test results obtained through the elicitation of the prior distribution with pre-posterior assessments. 

The case study motivating our research will be presented. We discuss how our method has been used to support a comparison of alternative test and sampling strategies being considered to assess the state of the anchorages of the Forth Road Bridge, a suspension bridge near Edinburgh.  The Forth Road Bridge has more than 11,600 steel wires grouped into 37 strands within each main cable.  The cables are secured to the north and south shores of the river by pairs of anchorages, the key components of which are buried deep in rock with no direct means of access.  Five groups of test were considered to assess the state of the anchorages.  The BN constructed is based on the judgements of multiple engineering experts, and captures the nine variables representing the state of the strands and the inter-relationships between them.

This talk will be used to highlight several challenges in the process of eliciting expert judgment and be used to motivate the key objective for Working Group 1 of the COST project.

For more information regarding our programme please contact: Dr. Kevin Wilson, University of Strathclyde- kevin.j.wilson@strath.ac.uk

Information about Arriving, Accommodation:

For information about getting to Strathclyde, please see below:

Getting to Strathclyde University

If you are looking for a possible accommodation during this time, there are many options for hotels within a short walk from the conference venue and bookings can be made directly on the hotels’ own websites. A few options are listed below:

Premier Inn Glasgow City Centre

Millenium Hotel Glasgow

Mercure Glasgow City