Developing an automated iterative near-term forecasting system for an ecological study

Most forecasts for the future state of ecological systems are conducted once and never updated or assessed. As a result, many available ecological forecasts are not based on the most up-to-date data, and the scientific progress of ecological forecasting models is slowed by a lack of feedback on how well the forecasts perform. Iterative near-term ecological forecasting involves repeated daily to annual scale forecasts of an ecological system as new data becomes available and regular assessment of the resulting forecasts. We demonstrate how automated iterative near-term forecasting systems for ecology can be constructed by building one to conduct monthly forecasts of rodent abundances at the Portal Project, a long-term study with over 40 years of monthly data. This system automates most aspects of the six stages of converting raw data into new forecasts: data collection, data sharing, data manipulation, modeling and forecasting, archiving, and presentation of the forecasts. The forecasting system uses R code for working with data, fitting models, making forecasts, and archiving and presenting these forecasts. The resulting pipeline is automated using continuous integration (a software development tool) to run the entire pipeline once a week. The cyberinfrastructure is designed for long-term maintainability and to allow the easy addition of new models. Constructing this forecasting system required a team with expertise ranging from field site experience to software development. Automated near-term iterative forecasting systems will allow the science of ecological forecasting to advance more rapidly and provide the most up-to-date forecasts possible for conservation and management. These forecasting systems will also accelerate basic science by allowing new models of natural systems to be quickly implemented and compared to existing models. Using existing technology, and teams with diverse skill sets, it is possible for ecologists to build automated forecasting systems and use them to advance our understanding of natural systems.

To go from raw data to forecast presentation involves a number of stages, each of which requires unique tasks, tools and infrastructure. The stages are interdependent, with outputs from one stage forming the inputs for the subsequent stage. Tasks in all stages are run using code written in R. B) Continuous integration system. Each box denotes the core infrastructure used for each stage of the forecasting pipeline. Continuous integration (denoted by the Travis icon, a woman wearing safety glasses and hardhat) triggers the code involved in events that link the stages of the pipeline, such as using the output from the forecasting stage (purple box) to create an updated website (rose box). Travis also runs tasks within a stage, such as testing code and adding weather data (icons on arrows originating and ending on the same box). The code for driving different stages of this pipeline is stored on GitHub (denoted by the GitHub icon, an "octocat").
individual components. Continuous analysis uses a set of tools originally designed for 175 software development called "continuous integration" (CI). CI combines computing 176 environments for running code with monitoring systems to identify changes in data or 177 code. Essentially, CI is a computer helper who watches the pipeline and, when it sees a 178 change in the code or data, runs all the computer scripts needed to ensure that the 179 forecasting pipeline runs from beginning to end. This is useful for iterative near-term 180 forecasting because it does not rely on humans to create new forecasts whenever new 181 models or data are added. These tools are common in the area of software development, 182 where they are used to automate software testing and integrate work by multiple 183 developers working on the same code base. However, these tools can be used for any 184 computational task that needs to be regularly repeated or run after changes to code or

192
We use CI to quality check data, test code using "unit tests" (Wilson et al., 2014), build 193 models, make forecasts, and publicly present and archive the results (Figure 1b).

194
To ensure that software pipelines continue to run automatically as software 195 dependencies change, a key component of "continuous analysis" is the use of a  Software containers are standalone packages that contain copies of everything needed to 199 run a given piece of software, including the operating system (Boettiger, 2015). Once 200 created, a software container is basically a time capsule, containing all the software dependencies in the exact state used to develop and run the software (Boettiger, 2015).

202
If those dependencies change (or disappear) in the wider world, they still exist, 203 unchanged, in the container. We use an existing platform, Docker (Merkel, 2014), to 204 store an exact image of our complete software environment by adding our project 205 specific code to a container created by the Rocker project, which is a Docker image with 206 many important R packages (i.e., the tidyverse packages; Wickham, 2017) pre-installed 207 (Boettiger & Eddelbuettel, 2017). We implemented this system because we experienced 208 issues with external dependencies breaking our pipeline (e.g., when the tscount 209 package (Liboschik et al., 2015), was temporarily removed from CRAN and could not 210 be installed in the usual way). In combination, the automated running of the pipeline 211 (continuous integration) and the guarantee it will not stop working unexpectedly due to 212 software dependencies (via a software container) allows continuous analysis to serve as 213 the glue that connects all stages of the forecasting pipeline.

214
Data Collection, Entry, and Processing 215 Iterative forecasting benefits from frequently updated data so that state changes can be 216 quickly incorporated into new forecasts (Dietze et al., 2018). Both frequent data 217 collection and rapid processing are important for providing timely forecasts. Since we 218 collect data monthly, ensuring that the models have access to the newest data requires a 219 data latency period of less than 1 month from collection to availability for modeling. To New data is double-entered into Microsoft Excel using the "data validation" feature.

224
The two versions are then compared using an R script to control for errors in data entry.

225
Quality control (QC) checks using the testthat R package (Wickham, 2011) are run 226 on the data to test for validity and consistency both within the new data and between the 227 new and archived data. The local use of the QC scripts to flag problematic data greatly 228 reduces the time spent error-checking and ensures that the quality of data is consistent.

229
The cleaned data is then uploaded to the GitHub-based PortalData repository   The Portal Project has a long history of making its data publicly available so that anyone 249 can use it for forecasting or other projects. Historically, the publication of the data was 250 conducted through data papers (Ernest et al., 2009(Ernest et al., , 2016, the most common approach 251 in ecology; this approach, however, caused years of data latency. With the recent switch 252 to posting data directly to a public GitHub repository ( Figure 1) with a CC0 waiver 253 (i.e. no restrictions on data use; https://creativecommons.org/publicdomain/zero/1.0/), 254 data latency for everyone has been reduced to less than one month, making meaningful  Second, this kind of code is rarely robust to changes in data structure and location. we hope to implement in the future. As long as a model script can load the provided 299 data and produce the appropriate output, it will be run and its results incorporated into 300 the rest of the forecasting system. This means that anyone can add a new model to the  Figure 2: Demonstration of plugin infrastructure. All model scripts (represented here by the example AR1.R) are housed in a single folder. Each model script uses data provided by the core forecasting code (represented here by rodent.csv) and returns its forecast outputs in a predefined structure that is consistent across models (represented here by the example 2017_12_08forecasts.csv). Outputs from all models run on a particular date are combined into the same file (i.e. 2017_12_08forecasts.csv) to allow cross-model evaluations. Model output files are housed in a folder containing all forecast outputs from all previous dates to facilitate archiving and forecast assessment.
support flexibility in what the models predict. Allowing models to make forecasts for 306 system properties ranging from individual species' population abundances to total 307 community biomass facilitates exploration of differences in forecastability across 308 different aspects of ecological systems. We designed a forecast output format to support 309 this. Each forecast output file contains the date being forecast, the collection date of the 310 data used for fitting the models, the model name, the date the forecast was made, the 311 state variable being forecast (e.g., rodent biomass, the abundance of a species), and the 312 forecast value and associated uncertainty of that forecast (Figure 2). This allows us to 313 store a variety of different forecasts in a common format and may serve as a useful 314 starting point for developing a standard for storing ecological forecasts more generally.

315
Forecasts are currently evaluated using root mean square error (RMSE) to evaluate 316 point forecasts and coverage to evaluate uncertainty. We plan to add additional metrics, 317 like deviance, that incorporate both accuracy and uncertainty and better match the  Since hindcasting is conducted using data that has already been collected, it allows 325 model comparisons to be conducted on large numbers of hindcasts and provides insight 326 into which models make the best forecasts without needing to wait for new data to be 327 collected (Harris et al., 2018). It can also be used to quickly evaluate new models 328 instead of waiting for an adequate amount of data to accumulate. As the performance of 329 different models is understood through evaluation of forecasts and hindcasts, models 330 can be refined or removed from the system or ensemble to iteratively improve the 331 resulting forecasts.
Publicly archiving forecasts before new data is collected allows the field to assess,   community-level variables such as total abundance. To create this iterative near-term 394 forecasting system, we used R to process data and conduct analyses and leveraged 395 existing tools and services (i.e. GitHub, Travis, Docker) for more complicated 396 cyberinfrastructure tasks. Thus, our approach to developing iterative near-term 397 forecasting infrastructure provides an example for how short-term ecological 398 forecasting systems can be developed. 399 We designed this forecasting system with the goal of making it relatively easy to build, 400 maintain, and extend. We used existing technology for both running the pipeline and 401 building individual components, which allowed us to build the system relatively cheaply when changes are made to data, models, or associated code, we have reduced the time 408 required by scientists to run and maintain the forecasting pipeline. To make the system 409 extensible so that new models could be easily incorporated, we used a plugin-based infrastructure so that adding a new model to the system is as easy as adding a single file 411 to the 'models' folder in our repository (Figure 2). This should substantially lower the 412 barriers to other scientists contributing models to this forecasting effort. We also 413 automatically archive the resulting forecasts publicly so that the performance of these interdisciplinary teams or additional training will often be required for creating these 491 pipelines until tool development improves. To improve the success of these diverse 492 groups, we believe efforts at providing 'team science' training to scientists interested in 493 forecasting will be beneficial for the success of iterative forecasting attempts for the 494 foreseeable future (Read et al., 2016). 495 We developed infrastructure for automatically making iterative forecasts with the goals

Author Contributions
All authors conceived the ideas and designed methodology; All authors developed the 539 automated forecasting system; EPW and SKME led the writing of the manuscript. All Frequent data collection allows models to be regularly updated and forecasts to be 549 frequently evaluated (Dietze et al., 2018). Depending on the system being studied, this 550 frequency could range from sub-daily to annual, but typically the more frequently the 551 data is collected the better.   To quickly learn about the best approaches to forecasting different aspects of ecology, 574 multiple modeling approaches should be compared (Harris et al., 2018). Different 575 modeling approaches should also be combined into ensemble models, which often 576 outperform single models for prediction (Weigel et al., 2008).    Docker. An open-source Linux program for containerization (see software container).

609
git. An open-source version control system. GitHub. A web-based host for git projects.