Is your ad hoc model selection strategy affecting your multimodel inference?
Abstract Ecologists routinely fit complex models with multiple parameters of interest, where hundreds or more competing models are plausible. To limit the number of fitted models, ecologists often define a model selection strategy composed of a series of stages in which certain features of a model a...
Main Authors: | , , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Wiley
2020-01-01
|
Series: | Ecosphere |
Subjects: | |
Online Access: | https://doi.org/10.1002/ecs2.2997 |
id |
doaj-1aee6bf67af64c75b717695135a71888 |
---|---|
record_format |
Article |
spelling |
doaj-1aee6bf67af64c75b717695135a718882020-11-25T03:04:09ZengWileyEcosphere2150-89252020-01-01111n/an/a10.1002/ecs2.2997Is your ad hoc model selection strategy affecting your multimodel inference?Dana J. Morin0Charles B. Yackulic1Jay E. Diffendorfer2Damon B. Lesmeister3Clayton K. Nielsen4Janice Reid5Eric M. Schauber6Department of Wildlife, Fisheries and Aquaculture Mississippi State University Box 9680 Mississippi State Mississippi 39762 USASouthwest Biological Science Center U.S. Geological Survey 2255 N. Gemini Drive Flagstaff Arizona 86001 USADenver Federal Center U.S. Geological Survey, Geosciences and Environmental Change Science Center Denver Colorado 80225 USAPacific Northwest Research Station U.S. Forest Service and Department of Fisheries and Wildlife Oregon State University 3200 SW Jefferson Way Corvallis Oregon 97331 USACooperative Wildlife Research Laboratory and Department of Forestry Southern Illinois University 251 Life Science II, Mail Code 6504 Carbondale Illinois 62901 USAPacific Northwest Research Station U.S. Forest Service 777 NW Garden Valley Blvd Roseburg Oregon 97471 USAIllinois Natural History Survey Prairie Research Institute University of Illinois Urbana‐Champaign 1816 S. Oak Street Champaign Illinois 61820 USAAbstract Ecologists routinely fit complex models with multiple parameters of interest, where hundreds or more competing models are plausible. To limit the number of fitted models, ecologists often define a model selection strategy composed of a series of stages in which certain features of a model are compared while other features are held constant. Defining these multi‐stage strategies requires making a series of decisions, which may potentially impact inferences, but have not been critically evaluated. We begin by identifying key features of strategies, introducing descriptive terms when they did not already exist in the literature. Strategies differ in how they define and order model building stages. Sequential‐by‐sub‐model strategies focus on one sub‐model (parameter) at a time with modeling of subsequent sub‐models dependent on the selected sub‐model structures from the previous stages. Secondary candidate set strategies model sub‐models independently and combine the top set of models from each sub‐model for selection in a final stage. Build‐up approaches define stages across sub‐models and increase in complexity at each stage. Strategies also differ in how the top set of models is selected in each stage and whether they use null or more complex sub‐model structures for non‐target sub‐models. We tested the performance of different model selection strategies using four data sets and three model types. For each data set, we determined the "true" distribution of AIC weights by fitting all plausible models. Then, we calculated the number of models that would have been fitted and the portion of "true" AIC weight we recovered under different model selection strategies. Sequential‐by‐sub‐model strategies often performed poorly. Based on our results, we recommend using a build‐up or secondary candidate sets, which were more reliable and carrying all models within 5–10 AIC of the top model forward to subsequent stages. The structure of non‐target sub‐models was less important. Multi‐stage approaches cannot compensate for a lack of critical thought in selecting covariates and building models to represent competing a priori hypotheses. However, even when competing hypotheses for different sub‐models are limited, thousands or more models may be possible so strategies to explore candidate model space reliably and efficiently will be necessary.https://doi.org/10.1002/ecs2.2997AICinformation criterionmodel selectionmultimodel inferenceoccupancy modelsparameter estimation |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Dana J. Morin Charles B. Yackulic Jay E. Diffendorfer Damon B. Lesmeister Clayton K. Nielsen Janice Reid Eric M. Schauber |
spellingShingle |
Dana J. Morin Charles B. Yackulic Jay E. Diffendorfer Damon B. Lesmeister Clayton K. Nielsen Janice Reid Eric M. Schauber Is your ad hoc model selection strategy affecting your multimodel inference? Ecosphere AIC information criterion model selection multimodel inference occupancy models parameter estimation |
author_facet |
Dana J. Morin Charles B. Yackulic Jay E. Diffendorfer Damon B. Lesmeister Clayton K. Nielsen Janice Reid Eric M. Schauber |
author_sort |
Dana J. Morin |
title |
Is your ad hoc model selection strategy affecting your multimodel inference? |
title_short |
Is your ad hoc model selection strategy affecting your multimodel inference? |
title_full |
Is your ad hoc model selection strategy affecting your multimodel inference? |
title_fullStr |
Is your ad hoc model selection strategy affecting your multimodel inference? |
title_full_unstemmed |
Is your ad hoc model selection strategy affecting your multimodel inference? |
title_sort |
is your ad hoc model selection strategy affecting your multimodel inference? |
publisher |
Wiley |
series |
Ecosphere |
issn |
2150-8925 |
publishDate |
2020-01-01 |
description |
Abstract Ecologists routinely fit complex models with multiple parameters of interest, where hundreds or more competing models are plausible. To limit the number of fitted models, ecologists often define a model selection strategy composed of a series of stages in which certain features of a model are compared while other features are held constant. Defining these multi‐stage strategies requires making a series of decisions, which may potentially impact inferences, but have not been critically evaluated. We begin by identifying key features of strategies, introducing descriptive terms when they did not already exist in the literature. Strategies differ in how they define and order model building stages. Sequential‐by‐sub‐model strategies focus on one sub‐model (parameter) at a time with modeling of subsequent sub‐models dependent on the selected sub‐model structures from the previous stages. Secondary candidate set strategies model sub‐models independently and combine the top set of models from each sub‐model for selection in a final stage. Build‐up approaches define stages across sub‐models and increase in complexity at each stage. Strategies also differ in how the top set of models is selected in each stage and whether they use null or more complex sub‐model structures for non‐target sub‐models. We tested the performance of different model selection strategies using four data sets and three model types. For each data set, we determined the "true" distribution of AIC weights by fitting all plausible models. Then, we calculated the number of models that would have been fitted and the portion of "true" AIC weight we recovered under different model selection strategies. Sequential‐by‐sub‐model strategies often performed poorly. Based on our results, we recommend using a build‐up or secondary candidate sets, which were more reliable and carrying all models within 5–10 AIC of the top model forward to subsequent stages. The structure of non‐target sub‐models was less important. Multi‐stage approaches cannot compensate for a lack of critical thought in selecting covariates and building models to represent competing a priori hypotheses. However, even when competing hypotheses for different sub‐models are limited, thousands or more models may be possible so strategies to explore candidate model space reliably and efficiently will be necessary. |
topic |
AIC information criterion model selection multimodel inference occupancy models parameter estimation |
url |
https://doi.org/10.1002/ecs2.2997 |
work_keys_str_mv |
AT danajmorin isyouradhocmodelselectionstrategyaffectingyourmultimodelinference AT charlesbyackulic isyouradhocmodelselectionstrategyaffectingyourmultimodelinference AT jayediffendorfer isyouradhocmodelselectionstrategyaffectingyourmultimodelinference AT damonblesmeister isyouradhocmodelselectionstrategyaffectingyourmultimodelinference AT claytonknielsen isyouradhocmodelselectionstrategyaffectingyourmultimodelinference AT janicereid isyouradhocmodelselectionstrategyaffectingyourmultimodelinference AT ericmschauber isyouradhocmodelselectionstrategyaffectingyourmultimodelinference |
_version_ |
1724682649149112320 |