The nlrx r package: A next‐generation framework for reproducible NetLogo model analyses

Abstract

1. Agent‐based models find wide application in all fields of science where large‐scale patterns emerge from properties of individuals. Due to increasing capacities of computing resources it was possible to improve the level of detail and structural realism of next‐generation models in recent years. However, this is at the expense of increased model complexity, which requires more efficient tools for model exploration, analysis and documentation that enable reproducibility, repeatability and parallelization. NetLogo is a widely used environment for agent‐based model development, but it does not provide sufficient built‐in tools for extensive model exploration, such as sensitivity analyses. One tool for controlling NetLogo externally is the r‐package RNetLogo. However, this package is not suited for efficient, reproducible research as it has stability and resource allocation issues, is not straightforward to be setup and used on high performance computing clusters and does not provide utilities, such as storing and exchanging metadata, in an easy way.
2. We present the r‐package nlrx, which overcomes stability and resource allocation issues by running NetLogo simulations via dynamically created XML experiment files. Class objects make setting up experiments more convenient and helper functions provide many parameter exploration approaches, such as Latin Hypercube designs, Sobol sensitivity analyses or optimization approaches. Output is automatically collected in user‐friendly formats and can be post‐processed with provided utility functions. nlrx enables reproducibility by storing all relevant information and simulation output of experiments in one r object which can conveniently be archived and shared.
3. We provide a detailed description of the nlrx package functions and the overall workflow. We also present a use case scenario using a NetLogo model, for which we performed a sensitivity analysis and a genetic algorithm optimization.
4. The nlrx package is the first framework for documentation and application of reproducible NetLogo simulation model analysis.

1 INTRODUCTION

Agent‐based models are an increasingly popular method for understanding complex systems (DeAngelis & Grimm, 2014). They are developed and applied across many different research disciplines from natural sciences over archaeology to socio‐economic sciences (e.g. Dislich et al., 2018; Vincenot, 2018). Agent‐based models incorporate the heterogeneity of entities at the individual level in order to observe patterns emerging on broader scales (Grimm & Railsback, 2005). Due to the release from computing power constraints in recent years, next‐generation agent‐based models have evolved that are structurally realistic, powerful and detailed enough to address real‐world problems (Cabral, Valente, & Hartig, 2017; Grimm & Berger, 2016). However, next‐generation agent‐based models require reproducability, repeatability and parallelization of model analyses. Access to scripts, runs and results of statistical analyses is a key criterion for reproducible research (Peng, 2011; Sandve, Nekrutenko, Taylor, & Hovig, 2013). Yet, incentives for sharing code of agent‐based models and model analyses alongside publications are still lacking (Janssen, 2017).

A widely used programming language to develop agent‐based models in ecological and socio‐economic sciences is NetLogo (Abar, Theodoropoulos, Lemarinier, & O'Hare, 2017; Vincenot, 2018; Wilensky, 1999). NetLogo is a Java‐based modelling environment that features a very comprehensible syntax, which allows for fast prototyping of agent‐based models but also offers capabilities to formulate and implement complex agent‐based models efficiently (Railsback et al., 2017). With rising complexity of NetLogo models, increasing efforts need to be spent on model analyses and documentation of such analyses. Model analysis is a crucial part of the modelling cycle, not only for understanding model processes but also during model refinement and development (Grimm & Railsback, 2005). Often sensitivity analyses are the central part of such refinements, because they give access to detailed information on the relative importance of model parameters for model outputs (Saltelli et al., 2008). Sensitivity analyses need to simulate thousands of different parameter value combinations to gain a better understanding of the modelled systems. Unfortunately, the capabilities of the built‐in tool of NetLogo to run such analyses, the NetLogo BehaviorSpace, are limited. If more than one parameter is changed within one experiment, BehaviorSpace automatically creates a full‐factorial parameter matrix including all possible combinations of parameter values. This might be generally suitable for simple models, but with rising model complexity more efficient ways of scanning the parameter space and a better control of the design of parameter matrices are required. Additionally, post‐processing of NetLogo model output is mostly done within statistical software, such as r (the most prominent programming language for ecologists; Lai, Lortie, Muenchen, Yang, & Ma, 2019). Transfer of the output data from NetLogo into r may not be straightforward, especially when dealing with spatially‐explicit model output. This is because spatial NetLogo output is mostly reported in nested lists which need to be parsed and split to convert the data into a usable format.

The r package RNetLogo currently is the only r package available to set up NetLogo model simulations directly from r (Thiele, Kurth, & Grimm, 2012, 2014). However, the package establishes an interactive connection between the r session and the Java virtual machine that runs the NetLogo model application, which has two main drawbacks: (a) RNetLogo establishes the interactive connection via the rJava package, which may be cumbersome to set up, especially on remote systems, and requires access to system‐wide environmental variables. (b) Resource allocation of rJava and RNetLogo may be problematic when running large jobs with many simulations. Memory may not be cleared efficiently between runs and Java virtual machines may freeze due to memory constraints. Furthermore, RNetLogo does not provide a convenient workflow to set up experiments with minimal coding, utility functions to generate simulations, or features to enable reproducible research (see a full comparison in Table 1).

Table 1. Main differences between NetLogo's BehaviorSpace, RNetLogo and nlrx

Feature description	BehaviorSpace	RNetLogo	nlrx
High‐performance computing (HPC)
Straightforward setup of HPC machines to submit and run large NetLogo simulation jobs.	–	–	✓
Controllable Java environment
Possibility to open a new Java virtual machine for each model simulation.	–	–	✓
Interactive Java environment
Possibility to control model parameters during model run.	–	✓	–
Reproducible research tools
Possibility to perform model analyses in a non‐interactive, reproducible way. Means to control for random number generator seeds and exchange of model files.	–	–	✓
Spatial integration of agent metrics
Coerce NetLogo agent metrics to geospatial data objects (vector and raster data).	–	–	✓
Tidy data format
Data format in which each observation is a unique cell and each variable a unique column (Wickham, 2014).	–	–	✓
Utility functionality
Functionality to perform pre‐ and post‐processing of simulation data.	–	–	✓

We have, therefore, developed the r package nlrx to provide a flexible framework for self‐contained and reproducible analysis of NetLogo models from r, while also delivering performance gains (see Figure 1 and Supplementary Material S1). The nlrx framework serves four main needs for next‐generation NetLogo modellers: (a) it provides an interface between r and NetLogo. Simulations are completely defined and stored within r objects. This allows the application of a huge toolbox of statistical methods to create simulation experiments and corresponding parameter input matrices. The nlrx package utilizes the controlling API of NetLogo's BehaviorSpace to set up and run experiments. Thus, in contrast to RNetLogo, the connection between r and NetLogo is not interactive and does not rely on the rJava package. (b) it enables reproducible research (Sandve et al., 2013) by storing a simulation experiment in one single r object (including parameter input matrix, applied simulation method and simulation results) and allows easy sharing of code and results among colleagues, with students and for publication. (c) nlrx enables utilization of high performance computing clusters (HPCs) in a very convenient way (see Supplementary Material S2). (d) nlrx provides post‐processing analysis functions of NetLogo model output for several applications, such as sensitivity analyses and calculation of descriptive statistics. nlrx also enables gathering spatial output from models, such as coordinates and properties of individuals and patches and provides functions to convert the output into spatial r objects (raster and vector data).

Figure 1

Comparison of execution time (top panel) and memory usage (bottom panel) of nlrx (blue) and RNetLogo (red). Benchmarks were calculated on different machines with various processor types, available memory and operating systems (details see Supplementary Material S1, boxplots show variation between systems). Both packages were used to execute NetLogo model simulations with the Wolf Sheep Model from the NetLogo Models Library. With each package, we simulated eight replications of a Latin‐Hypercube Sampling design with 100 samples and a run time of 500 ticks of each simulation. Available memory was measured before starting simulations (Pre‐simulation) and after simulations were completed (Post‐simulation). A third measurement of available memory was taken after manually executing the r garbage collection function gc (Post‐gc)

2 THE r PACKAGE nlrx

The nlrx package utilizes the NetLogo BehaviorSpace interface by calling NetLogo from the command line via a bash script (*.sh, linux) or a batch file (*.bat, windows). By default, NetLogo BehaviorSpace experiments are stored in XML format within the *.nlogo model file. However, when NetLogo is started in headless mode from command line, it is also possible to commit XML BehaviorSpace experiment files to setup and run experiments. The nlrx package creates such XML BehaviorSpace experiment files dynamically in order to run simulation experiments from r.

2.1 nlrx workflow

The typical workflow of the nlrx package starts with creation of an nl object, which is an S4 class object that stores basic information on NetLogo (see Figure 2, and Table 2). The nl object contains information on the NetLogo version (nlversion), the path to the NetLogo directory (nlpath), the path to the NetLogo model file (modelpath) and the amount of reserved memory for each Java virtual machine (jvmmem).

Figure 2

Workflow of the nlrx package. The nl object stores all the information needed to run NetLogo models. The experiment object, which contains all model‐specific information, such as names of setup and go procedures, runtime, variable range and metrics and the simdesign object, which contains all simulation‐related information, such as input parameter matrix, simulation method and simulation output, are nested within this nl object. Abbreviations: bat = batch file; JVM = java virtual machine; sh = bash script; XML = Extensible Markup Language

Table 2. List of slots of an nl S4 class object

Slot	Example	Description
nlversion	"6.0.4"	Version number of the NetLogo version
nlpath	"/home/NetLogo~6.0.4"	Path to a NetLogo installation
modelpath	"/home/NetLogo~6.0.4/app/models/Sample Models/Biology/Wolf~Sheep~Predation.nlogo"	Path to a *.nlogo model file
jvmmem	1024	Java virtual machine memory in megabytes
experiment		Stores an experiment S4 class object
simdesign		Stores a simdesign S4 class object

Next, an experiment needs to be created and attached to the nl object (see Figure 2, and Table 3). The experiment object stores all information that would be entered into a typical BehaviorSpace experiment (see Table 3). The experiment object contains the name of the experiment (expname), the directory where output is written to (outpath), the number of repeated runs within BehaviorSpace (repetition), a logical variable whether output should be measured on each tick or on the final tick only (tickmetrics), names of setup and go procedures (idsetup, idgo), the maximum number of ticks that should be simulated (runtime, 0 = infinite), a vector of ticks for which output is reported (evalticks), a stop condition (stopcond), output metrics (metrics, metrics.turtles, metrics.patches, metrics.links), variable parameters and corresponding value ranges (variables), and constant parameters (constants).

Table 3. List of slots of an experiment S4 class object

Slot	Example	Description
expname	"nlrx‐example"	Name of the BehaviorSpace experiment
outpath	"/home/nlrxout"	Path to an output directory
repetition	1	Number of repetitions
tickmetrics	"true"	If true, metrics are collected on each model tick, otherwise only on the last tick
idsetup	"setup"	Name or vector of names containing NetLogo procedures that are called to setup the model
idgo	"go"	Name or vector of names containing NetLogo procedures that are called to run the model
idfinal	"final"	Name or vector of names containing NetLogo procedures that are called at the end of one run
idrunnum	"id‐input"	Name of a NetLogo numeric widget to transfer the current nlrx run number to NetLogo
runtime	100	Maximum runtime of model simulations (number of ticks)
evalticks	seq(90, 100)	Number or vector of ticks indicating when metrics should be collected
stopcond	"not any? turtles"	A NetLogo reporter that reports TRUE/FALSE. If it reports TRUE the current simulation run is stopped
metrics	c("count turtles")	Valid NetLogo reporters that are used to collect output data
metrics.turtles	list("turtles"=c("who","pxcor","pycor","color"))	A list with named vectors of strings defining valid turtles‐own variables that are taken as output measurements
metrics.patches	c("pxcor","pycor","pcolor")	A vector of strings defining valid patches‐own variables that are taken as output measurements
metrics.links	list("links"=c("end1","end2"))	A list with named vectors of strings defining valid links‐own variables that are taken as output measurements
variables	list("variable.parameter.1"=list(min=50, max=150, step=10, qfun="qunif"))	A list with NetLogo globals that should be varied in a simdesign
constants	list("constant.parameter.1"=4, "constant.parameter.2"=0.1)	A list with NetLogo globals that should stay constant in a simdesign

Finally, a simdesign needs to be attached to the nl object (see Figure 2, and Table 4). The nlrx package provides many helper functions to create pre‐defined simdesigns for simple parameter screenings (simple, distinct, full‐factorial), sensitivity analyses [Morris (Morris, 1991), Sobol (Sobol, 1990), eFast (Saltelli, Tarantola, & Chan, 1999)] or parameter optimizations [simulated annealing (Kirkpatrick, Gelatt, & Vecchi, 1983), genetic algorithm (Holland, 1992)]. These helper functions take information from the experiment (variables, constants) to create a simdesign object that contains the name of the simulation design method (simmethod), a vector of random seeds (simseeds), the generated parameter matrix (siminput) and an empty tibble slot to attach output results after simulations have been finished (simoutput). The siminput and simoutput slots store tibble data, a modern representation of rectangular data in r (Müller & Wickham, 2018). The sensitivity analyses and optimization simdesign helper functions also create a simulation object (simobject) that contains further information needed for post‐processing, such as calculation of sensitivity indices. The optimization algorithms do not generate a parameter matrix in advance, because the parameterization of later runs depends on the results of previous runs. It is also possible to write user‐defined helper functions to create simdesign objects.

Table 4. List of slots of a simdesign S4 class object

Slot	Description
simmethod	Name of the simdesign method (e.g. "ff"=full factorial, "lhs"=latin hypercube sampling)
siminput	tibble providing input parameterizations for the NetLogo model (cols = parameter, rows = runs)
simobject	List containing a simdesign object with further details on the simmethod that may be needed for output analysis
simseeds	Vector containing model seeds
simoutput	Slot to attach simulation results to the nl object

After initialization of the nl object, simulations can be run by calling one of the run_nl_*() functions that come with nlrx. For simdesigns that generate a parameter matrix, run_nl_one() can be used to execute one specific row of the simdesign parameter matrix siminput by specifying the row ID and a model seed. This can be useful for testing specific parameterizations. The function run_nl_all() executes all simulations from the parameter matrix siminput across all seeds. run_nl_all() uses the future_map_dfr() function (furrr package) to loop over random seeds and rows of the siminput parameter matrix (see Figure 2). This allows users to run the function sequentially or in parallel, on local machines and remote HPC machines. Parallelization options can be easily adjusted by using different future plans before calling the function (for more details see documentation of r package future). For optimization simdesigns that do not come with a pre‐generated parameter input matrix, run_nl_dyn() can be used to execute model simulations with dynamically generated parameter settings.

The connection between r and NetLogo is realized within these three run_nl_*() functions. First, a BehaviorSpace experiment XML file is created from the information that is stored within the provided nl object. This XML file contains the random seed, parameter settings and experiment settings. Next, depending on the operating system, a bash script or batch file is generated that is used to execute NetLogo via the command line in headless mode. The file contains information that is stored within the nl object (such as NetLogo path, model path, Java virtual machine memory and the path to the previously generated XML file). Afterwards, the script is executed to run the NetLogo simulations. When the simulation is finished, the NetLogo model output data is read into r. In the case of multiple simulations, results are bound together and reported as one large tibble.

After simulations are finished, the resulting output table can be attached to the nl object by using the provided setter function setsim. This stores a complete experiment setup, including output results, in one r object. After outputs have been attached to the nl object, the output results can also be written to a *.csv file by calling write_simoutput(). The output can also be further analysed by calling analyze_nl(). For example, for an nl object with a Morris simdesign and attached simulation output results running analyze_nl(), lists Morris sensitivity indices for each parameter and each measured model output. For reproducible research, it is important to attach the output to the nl object even if one does not want to conduct further analyses. By attaching the output to the nl object, all information of the simulation experiment, including parameter inputs and simulation output, is stored within an nl object, which can be easily stored as *.rds file for documentation purposes.

2.2 Further functionality

nlrx offers a function unnest_simoutput() to turn agent‐specific metrics that were collected during the simulation into a wide‐format table that splits every type of agent, patch and link in a unique column and nests all measured variables in this column. To be able to derive this information, the nl object offers slots to specify agent, patch and link metrics. The unnested spatial data can easily be visualized to illustrate model behaviour (Figure 3). Furthermore, the functions nl_to_points(), nl_to_raster() and nl_to_graph() coerce the nl object directly into a spatial data type (e.g. spatial points and raster data for agents and patches, as well as undirected network graphs for links). To be able to use nlrx as a fully reproducible framework, the functions export_nl() and import_nl() store r and corresponding NetLogo model scripts in a zip file, which uses relative paths and thus enables easy collaboration. Furthermore, nlrx provides functions to generate documentation files as *.html, *.pdf and *.docx files from NetLogo model files containing specific documentation comment sections. nlrx also provides a function to generate a network of NetLogo model procedures (for details see Supplementary Material S3). nlrx will thus enable ecologists to make use of agent‐based NetLogo models in their research while permitting a workflow that follows modern scientific standards.

Figure 3

Visualization of spatial output gathered from the Ants model (see Section 3 for a description of the model) via the nlrx package. Ants positions (brown yen signs), position of food cells (black), position of the nest cell (brown circle), and amount of chemicals on all cells (heat shading, with darker colours indicating higher amount of chemicals) were measured using the metrics.turtles and metrics.patches slots of the nlrx experiment. Panels show output for every 100th tick of a model simulation

3 USE CASE: ANTS MODEL

This section describes two example use cases of nlrx using the Ants simulation model from the NetLogo Models Library: (a) a Sobol sensitivity analysis based on r's advanced statistical packages, (b) a genetic algorithm optimization approach to optimize foraging speed.

The Ants model is a spatially‐explicit model of an ant colony foraging for food. The landscape is set up as a patch lattice. The centre cell of the landscape is the nest of the ant colony and all ants (number depending on parameter population) are created within this nest cell on initialization. Additionally, there are three clusters of food source cells created within the landscape. Food source clusters are created at different distances from the ant nest, with food source 1 being closest to the nest and food source 3 being furthest away from the nest. During a model run, ants move randomly in order to find food. If any ant finds a food source cell, a certain amount of food is transferred from that cell to the ant, which carries the food back to the nest. Each food cell only supplies a certain amount of food and is reset to a non‐food cell once all food from this cell has been gathered. Any ant carrying food releases chemicals at its current position. Chemicals also diffuse to neighbouring cells, depending on the parameter difussion‐rate and evaporate depending on the parameter evaporation‐rate. If ants sense chemicals on their current or neighbouring patch, they change their movement pattern from random walk to directed walk following the trail of chemicals by moving into the direction which contains the most chemicals. The main output of the model simulation is the time needed to gather all food from all food sources. We also measure the time that is needed to completely deplete specific food source clusters. Furthermore, we are interested in the spatial distribution of ants under different parameterizations of the model (for details see Supplementary Material S4).

3.1 Sensitivity analysis with nlrx

In order to demonstrate the usability and workflow of the nlrx package, we performed a global sensitivity analysis of the Ants model (Sobol, 1990). We varied the three model parameters population, diffusion‐rate and evaporation‐rate by applying a Sobol parameter variation approach. As the main output, we measured the simulation time (ticks) until all food sources were completely depleted by the ants.

To set up the sensitivity analysis with nlrx, we first defined an nl object (Listing 1).

In the next step, we attached an experiment to the previously generated nl object (Listing 2). We defined that output should be measured only on the final tick (tickmetrics = "false"). We set the runtime to infinite (runtime, 0 = infinite) and defined a stop condition to stop model execution once no food cell is left in the model landscape (stopcond). As output metrics we measured the number of ticks simulated when the model run finishes (metrics).

To construct an input parameter matrix from the defined experiment and variable ranges, a simdesign needs to be attached to the nl object (Listing 3). For our Sobol sensitivity analysis, we used the simdesign helper function simdesign_sobol(). We defined the number of samples (samples), the order of interaction effects (sobolorder), the number of bootstrapping replicates (sobolnboot), the confidence interval (sobolconf), the number of Sobol analysis repetitions (nseeds) and the precision level of values within the parameter matrix (number of decimal digits, precision).

This nl object stores all information needed to run the sensitivity analysis. We can execute all simulations by calling run_nl_all() (Listing 4). Because in our experiment we set tickmetrics to "false", the output will only be reported for the final simulation step.

In order to execute the analyze_nl() function, the results tibble first needs to be attached to the nl object (Listing 5).

The Sobol sensitivity analysis revealed a very large main effect of the population parameter on the effectiveness of food collection (measured as number of ticks until all food sources are depleted) by the ant colony (see Figure 4). The more ants were present, the faster the colony depleted all food sources. The two chemical‐related parameters evaporation rate and diffusion rate showed only small main effects (see Figure 4). However, we found some interaction effects between population and evaporation rate, albeit at a very low level. There were no interactions between population and diffusion rate and between diffusion rate and evaporation rate.

Figure 4

Main effects (upper panel) and interaction effects (lower panel) of each parameter of the Ants model on the total number of simulation steps that were simulated until all food sources were depleted by the ant individuals. Parameter effects were derived by applying a Sobol sensitivity analysis approach. Points indicate parameter mean sensitivity (Sobol effect) of six model simulation replicates, linebars indicate the corresponding standard deviation

After running all simulations with nlrx, we store the simulation output as a *.csv file using the write_simoutput() function (see Listing 6). Additionally, we store the nl object, together with underlying model files as a zip folder using the export_nl() function.

3.2 Genetic algorithm with nlrx

In order to demonstrate the optimization functions of nlrx, we applied a genetic algorithm to the Ants model. The genetic algorithm varies parameters of the model within defined ranges, in order to minimize a defined fitness criterion. In our case, we want to minimize the number of time steps needed by the ant colony to deplete all food sources completely.

For this analysis, we do not need to make any changes to the nl object and experiment that we used in the Sobol sensitivity analysis example, thus we can reuse the nl object and just add a different simdesign. In order to set up a genetic algorithm optimization simdesign, we used the simdesign helper function simdesign_GenAlg() (Listing 7). We defined the number of initial parameterizations (popSize = 200), the number of iterations (iters = 100), the index of a defined output metric (see metrics slot in experiment) that is used as evaluation criterion (evalcrit), the elitism rate (elitism), the mutation probability (mutationChance) and the number of random seeds for replications of the algorithm (nseeds).

In contrast to sensitivity analysis simdesigns, for optimization simdesigns there is no parameter input matrix set up in advance. Instead, random starting values are chosen within the defined parameter ranges. The parameterization of the next simulation run is dynamically created, depending on the output of the previous simulation. Thus, for optimization simdesigns, the nlrx function run_nl_dyn() needs to be used to execute the simulations (instead of run_nl_all() which iterates over a pre‐generated parameter matrix) (Listing 8).

The results from the genetic algorithm simdesign are reported as a list which contains information of parameterizations and the evaluation criterion of each population of the genetic algorithm. We can also identify the best found parameterization, i.e. the parameterization that resulted in the smallest fitness value.

The genetic algorithm was able to decrease the time to deplete all food sources from more than 3,000 ticks to below 500 ticks over 100 iterations (see Figure 5, upper left). The parameterization of the best found solution had a moderately large population (166), a medium diffusion rate (18.28), and a low evaporation rate (8.41) (see Figure 5, upper right).

Figure 5

A genetic algorithm was applied to minimize simulation time until all food sources were depleted. The figure illustrates the minimization process over iterations of the algorithm (upper left) and the best found parameterization in terms of minimum simulation time (upper right). This best found parameterization was used to run the Ants model with 10 replicates and measuring the percentage of food gathered at each tick (lower left, colours indicate individual simulations). We also aggregated these 10 runs to measure at which time steps the three individual food source clusters were completely depleted (lower right). Dashed lines illustrate final time step of each curve

To evaluate the stability of the optimization result under different random seeds, we simulated an additional 10 replicates (each with a different random seed) of the best parameterization. In order to run a specific parameterization, the nlrx package provides the simdesign helper function simdesign_simple(), which creates a parameter table for one single simulation that uses only defined values of constant parameters (constants slot of experiment). We removed all variable parameters from the experiment and defined those parameters as constants instead (Listing 9). We also added counts of food sources as output metrics to the metrics slot of the experiment. Finally, we used the simdesign_simple() function to attach a simdesign containing a siminput matrix with only one parameterization and a simseeds vector with 10 random seeds (nseeds=10).

Again, we used the run_nl_all() function to execute the simulations and calculated the percentage of food gathered at each tick and identified the tick at which specific food source clusters were depleted completely. We observed some variation in total simulation time between these replicates, but all runs lay between 500 and 800 ticks (see Figure 5, lower left). While the first two food sources were depleted very fast in all replicates, depletion times were more variable for the third food source, which had the largest distance from the ant nest (see Figure 5, lower right).

3.3 Conclusion and outlook

Next‐generation agent‐based models must be reproducible and repeatable. The new nlrx r‐package fulfils the need for an efficient tool for running reproducible and repeatable analyses for the popular programming language NetLogo. nlrx has several advantages in comparison to previous approaches that connect r and NetLogo: (a) the package does not rely on the rJava package which is known to be unstable and causes many problems especially when working on machines with different configurations; (b) setup of model experiments is very similar to NetLogo BehaviorSpace. Thus, NetLogo users who do not have much experience with r will feel familiar with the workflow of nlrx. (c) multiprocessing and execution of large model simulations on remote HPC machines can be easily done with nlrx by adjusting the future plan of the r session. (d) all information on the model version, simulation experiment definitions and simulation design definitions are stored within one single r object. By using the nlrx export and import functions, these objects can be archived and shared, together with all required NetLogo model files. This functionality enables a reproducible workflow and easy collaboration.

In contrast to RNetLogo, NetLogo instances of nlrx are not interactive. There is no direct connection from r to NetLogo and thus, it is not possible to manipulate the NetLogo model from within r while the model is running. However, for most use cases it is sufficient to set up and run model simulations in a convenient way because manual exploration of NetLogo models can also be done directly in NetLogo.

The presented use cases of the Ants model highlight the flexibility and convenient workflow of the nlrx framework. With the nested class layout the same nl object and experiment can be reused for several simulation design approaches (as demonstrated in the genetic algorithm use case). Thus, research questions can be tackled from very different angles with minimal code adjustments. Because reported output from model simulations is reported in a tidy data format (see Data Availability Statement section for additional r code examples), post‐processing, data analysis and result visualization can benefit from the well‐established tool set provided by the tidyverse framework (e.g. dplyr, ggplot2 packages). Collecting agent metrics and coercing them to spatial objects is now possible with nlrx in two lines of code, thus enabling ecologists to use advanced statistical methods on the individual level of their simulation models (see Supplementary Material S4).

Increasing model complexity across many disciplines increases the need of standardized tools and open standards for model documentation, application and analysis. For example, the ODD (Overview, Design concepts, and Details) protocol and the TRACE (TRAnsparent and Comprehensive Ecological modelling documentation) framework are such standards for documentation of agent‐based models that are widely accepted and applied within the field of agent‐based modelling (Grimm et al., 2014, 2006, 2010; Müller et al., 2013). However, given the rising importance of reproducible research as a standard in academia, standards for documentation and reproducibility of applied model analysis and corresponding code are still lacking. nlrx is a first step to provide a framework for documentation and application of reproducible NetLogo simulation model analysis. While nlrx mainly provides tools that enable reproducible research, future work might also give recommendations on a best practice approach that also incorporates data management, collaboration and version control.

nlrx has been officially released on CRAN (current version 0.2.0) and peer‐reviewed by rOpenSci (https://github.com/ropensci/software-review/issues/262).

ACKNOWLEDGEMENTS

J.S. and K.W. were funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number 192626868 – SFB 990 in the framework of the collaborative German – Indonesian research project CRC 990. M.S. and K.W. were supported by the DFG through grant number WI 1816/18‐1 (FOR2432) and DFG‐GRK 1644/3. We thank rOpenSci for their extensive code review and three reviewers for their helpful and constructive comments on an earlier version of this manuscript.

AUTHORS' CONTRIBUTION

J.S. and M.S. developed the r package. The case study was designed by J.S. J.S. and M.S. drafted the manuscript and all authors contributed critically to the manuscript and gave final approval for publication.

Open Research

DATA AVAILABILITY STATEMENT

The nlrx package is hosted at https://doi.org/10.5281/zenodo.3367095 All the code for the figures and use cases in this manuscript can be found at https://doi.org/10.5281/zenodo.2650853.