Ensemble Modeling in Fisheries and Wildlife: Providing Scientific Advice to Natural Resource Managers Can Be a Risky Business!

•  Providing scientific advice to natural resource managers can be a risky business! In many resource systems, identifying a single best ecological model will fail to capture scientific uncertainty about the true state of nature. In such cases, using a single analytical model will likely result in underestimation or mischaracterization of the actual uncertainty around the scientific advice, which may compromise risk-based decision tools used in natural resource management. However, the understanding and prediction of system dynamics, and therefore resulting scientific advice, can often be improved by analyzing the range of plausible working hypotheses where the scientific advice represents a synthesis across these hypotheses. Ensemble modeling accounts for multiple working hypotheses by employing multiple candidate models to explain and predict outcomes of the target system. Each candidate in the model ensemble makes a distinct claim about system dynamics and process outcomes and represents a unique hypothesis about how the system works. Synthesizing the results of the model ensemble can provide a more accurate characterization of the target system, especially when individual model predictions are unbiased and the covariance between model predictions is low. This symposium includes presentations and discussion of scientific advancements in ensemble modeling approaches, as well as perspectives and case studies where ensemble modeling has been applied in fisheries and wildlife ecology. This includes model selection, model weighting, multi-model inference (including averaging and other techniques), machine learning for ecological predictions, and communicating results of ensemble modeling for robust resource management. We encourage presentations from a wide range of participants who can share successful methods, challenges, and issues related to applying ensemble modeling techniques in fisheries and wildlife and effectively communicate the results of those applications to natural resource managers.

3:20PM Multimodel Inference, Not so Fast! Acknowledging Sources of Uncertainty and Potential Limitations to Protect Against Unjustified Conclusions or Management Decisions
  Katharine Banner, Megan Higgs
Statistical models approximate natural phenomena by summarizing signals in data and, beyond non-stochastic mathematical models, they also attempt to quantify known sources of uncertainty using probability. Competing hypotheses about how a system works can be reflected through a set of carefully specified statistical models, representing uncertainty in model choice. When results are conditioned on a single model, model uncertainty is left unquantified, which some believe leads to naïve inference. Model averaging (MA) is a popular technique accounting for model uncertainty, but MA can lead to unintended complications in interpretations of results and potentially an oversold faith in the uncertainty quantified. Particularly, when interest lies in explaining relationships, complicated interpretations can affect the utility of model-averaged results for making management decisions. We present tools to help researchers make informed decisions about when to use MA; demonstrate cases where the “protection against naïve inference” is not realized; and encourage more critical thought about what sources of uncertainty are actually quantified through the process of MA, and multimodel inference in general. Discussions about the advantages of multimodel inference are common, but it is important to also discuss potential limitations.
3:40PM Exploring Ensemble Modeling with Different Model Complexities
  Elizabeth Brooks, Jon Brodziak
Model uncertainty and model complexity are important considerations when fitting data for stock assessment and providing management advice. Model uncertainty includes structural decisions and alternative parameterizations, while model complexity refers to the degree of data aggregation and its statistical treatment. In practice, management advice typically comes from one best-fitting model, where alternative models provide reassurance that the main trend has been captured. We summarize simulations that explored ensemble modeling across multiple assessment and projection models, and make comparisons between models of varying complexity. We also explore the issues of selecting a set of candidate models, how different candidate models should be from one another, whether or not the true model is bounded by the model set, and sensitivity of the model weights used to average results. For each assessment model in the ensemble, current estimates of stock status and reference points are weighted to produce ensemble stock status. Similarly, ensemble catch advice is produced from the set of projection models. We compare the performance of this ensemble approach with the status quo “single best model” approach, and discuss important issues related to performance consistency and data weighting.
4:00PM Ensemble Random Forest As a Machine Learning Tool to Model Rare Events
  Zach Siders, Nicholas Ducharme-Barth, Felipe Carvalho, Donald Kobayashi, Summer Martin, Jennifer Raynor, T. Todd Jones, Robert Ahrens
Rare events are difficult to model. Generalized linear or additive models require high sample sizes of rare events. Maximum entropy models tend to drastically overfit. Decision tree machine learning ensemble models, such as Random Forest, tend to be weak learners and poor predictors of at least one of the data classes. Complicating such matters, modeling the spatial distribution of rare events can rely on underlying environmental responses making the choice between linear or nonlinear responses particularly important. As remotely sensed covariates become increasingly utilized in the distribution models, the ability to rationalize linear responses often fails due to the proxy nature of the data product. We developed Ensemble Random Forest as a methodological tool to overcome the challenges imparted by rare events. We demonstrate the ability of Ensemble Random Forest to model rare events using random fields spatial simulations as well as application to protected species bycatch in the Hawaiian pelagic longline fisheries. In particular, we focus on showing the evolution of individual weak learning Random Forest to the strong learning ensemble. Additionally, we demonstrate the impact of sampling spatial covariation and sample size on the performance of Ensemble Random Forests, an ensemble of ensembles.
4:20PM Multi-Species Surplus Production Modeling: A Data-Limited Approach for Evaluating Species Interactions
  Howard Townsend, Ronald Klasky, Robert J. Gamble, Andy Beet, Jason Link
Single species surplus production models (SSSPM) are the least complex and least data intensive among the commonly used stock assessment models (e.g., Virtual population Analysis (VPA), Statistical Catch at Age (SSCA)). Multi-species surplus production models (MSSPM) extend SSSPM by incorporating species interactions, such as predation and competition, and even technical interactions and environmental indices. MSSPM are most appropriate when there are challenges to data inputs but well-known interactions, such as many forage fishes. We have developed software that provides a flexible framework to allow users to build, fit, and evaluate multi-species surplus production models. Users can build models selecting from a variety of population growth forms, species interactions/functional response curves, and harvesting strategies. These models can then be fit to data by optimizing from a range of objective functions and optimization algorithms. Diagnostic tools are then available to evaluate the model fit and model robustness. We demonstrate the tool with data used in a more complicated (age-structured) multi-species model and compare the MSSPM results with that model. Our results show that the MSSPM provides robust, readily solvable solutions to increasingly common fisheries challenges. This flexible, modular software approach facilitates rapid development of multiple models to be included in an ensemble.
4:40PM Operationalizing Model Ensembles to Provide Scientific Advice for Fisheries Management
  Ernesto Jardim, Jon Brodziak, Manuela Azevedo, Elizabeth Brooks, Kelli Johnson, Nikolai Klibansky, Coilin Minto, Colin Millar, Iago Mosqueira, Richard Nash, Paris Vasilakopoulus, Brian Wells
Providing scientific advice to fisheries managers can be a risky activity! It’s not uncommon that a model which was working perfectly fails to properly fit an additional year of data, or to find that projections made in the past did not materialize when new information was made available. Scientists deal with very complex systems, with many unknown or poorly understood processes and limited information, which make advisory tools sensitive to alternative system representations, model assumptions or new data. Our approach to mitigate the potential lack of robustness and instability of fisheries advice is to expand its basis to integrate structural uncertainty using model ensembles. Two main reasons to use model ensembles are: to include structural uncertainty captured by differences across models of the same system, and to integrate across initial conditions and process errors in projections. This paper discusses and speculates about the utility and implementation of model ensembles for scientific advice to fisheries management. We discuss ensemble utilization, ensemble types, weighting metrics, model space and model expansion. We make the case for using ensembles in three main situations: to estimate stock status, to set future fishing opportunities, and to build operating models for management strategy evaluation.
5:00PM Conclusions and Recommendations on Ensemble Modeling from NOAA Fisheries’ National Stock Assessment Workshop
  Kristan Blackhart, Melissa A. Karp, Patrick Lynch
NOAA Fisheries’ National Stock Assessment Workshops (NSAWs) provide opportunities for agency scientists from across the nation to come together and address stock assessment-related issues of common importance. At the 13th NSAW in 2018, ensemble modeling was one of two major themes of the workshop. Ensemble modeling is increasingly being explored by NOAA stock assessment scientists as an approach to better characterize uncertainty, but these modeling techniques remain relatively untested in the context of providing advice to fishery managers. The ensemble modeling session at NSAW 13 provided an opportunity for NOAA’s stock assessment scientists from different regions to discuss challenges and opportunities as they further develop and apply ensemble models in a fishery management context. The objectives of the session were to 1) identify benefits and challenges to using ensemble modeling in fisheries stock assessments, and 2) identify ‘best’ practices for conducting ensemble modeling in fisheries stock assessment and using results for management decisions. During this presentation, I will summarize discussions during the NSAW ensemble modeling session, including key conclusions and recommendations for future agency action on this topic.

Organizers: Jon Brodziak, Patrick Lynch
Supported by: Marine Fisheries Section

Location: Reno-Sparks CC Date: October 1, 2019 Time: 3:20 pm - 5:20 pm