# Nestedness interacts with subnetwork structures and interconnection patterns to affect community dynamics in ecological multilayer networks

**Handling Editor: ** A. Bradley Duthie

## Abstract

- Ecological networks describe ecological interactions among species in ecosystems. In natural ecosystems, plant–mutualist (PM) and plant–herbivore (PH) networks are two of the most documented bipartite ecological networks, which are often interconnected through shared plants to form multilayer networks (here referred to as ecological networks with multiple interaction types). Recent developments in multilayer networks have challenged the effects of topological properties on biodiversity and stability once found in ecological networks with a single interaction type.
- In this study, my goal was to theoretically test the effects of the nested topology of subnetworks (i.e. plant–mutualist and plant–herbivore networks) on the local stability and persistence of the entire community and determine how their effect sizes were dependent on subnetwork structures and interconnection patterns.
- I used a simple algorithm to construct plant–mutualist or plant–herbivore networks with different levels of nestedness while fixing connectance and network size. By artificially interconnecting plant–mutualist and plant–herbivore networks through shared plants, I also manipulated the inter-subnetwork connection patterns as positive, negative and no correlations between the number of interacting partners of shared plants of two subnetworks. Community dynamics were simulated to show how subnetwork nestedness interacted with other network properties to affect local stability and persistence of multilayer networks.
- I found that low nestedness of both plant–mutualist and plant–herbivore subnetworks promoted stability and persistence. Effect sizes of the focal PM- or PH-subnetwork nestedness were positively associated with the nestedness levels of the interconnected subnetworks. A positive correlation between the mutualistic and herbivory generalism of plants also led to higher (signed) effect sizes of subnetwork nestedness. Further analyses showed that the effect sizes of the subnetwork nestedness also depended on subnetwork complexity and intraguild competition intensity. Finally, the modularity of interconnected subnetworks had little association with the effect sizes of subnetwork nestedness irrespective of interconnection patterns.
- The results demonstrate that the effects of topological structures (such as nestedness) on community dynamics in single-interaction networks may be altered by the architectures of multilayer networks, which highlights the need to study the interactions between the architectures of within- and inter-subnetworks in affecting ecosystem stability and biodiversity.

## 1 INTRODUCTION

Early ecologists believed that more diverse communities enhanced ecosystem stability (Elton, 1958; MacArthur, 1955). This was later challenged by Robert May showing mathematically that increased complexity destabilized community dynamics in randomly constructed ecological networks (May, 1972), which are representations of ecological communities accounting for both species and their interactions. From then on, the network approach has greatly advanced our understanding of the diversity–stability relationship (McCann, 2000), and many theoretical and empirical studies have been devoted to disentangling the topological structures and their implications on community dynamics of ecological networks in natural and human-disturbed ecosystems (Landi et al., 2018). In the past two decades, various types of empirical ecological networks have accumulated, such as food webs and bipartite mutualistic or herbivory networks (Bascompte & Jordano, 2013; Delmas et al., 2019; Kéfi et al., 2015). However, most empirical data on ecological networks include only a single type of interaction (e.g. single-interaction networks with only antagonistic or mutualistic interactions), and those with multiple types of interactions have been relatively less reported (García-Callejas et al., 2018a). Nonetheless, empirical multi-interaction networks have started to increase recently, suggesting trade-offs among interaction types and/or topological structures (Hackett et al., 2019; Pocock et al., 2012). For example, the multilayer networks from agricultural ecosystems showed that mutualistic networks were more nested than antagonistic networks, and antagonistic networks were consistently more modular (Morrison et al., 2020; Morrison & Dirzo, 2020). These empirical cases highlight the importance of understanding the relationships between network structures and community dynamics from the perspective of multilayer networks, which is still a challenging problem. The multilayer nature of empirical ecological networks might add at least two challenges to the mechanisms of community dynamics. First, although the topological structures such as nestedness and modularity within each layer (also called a subnetwork) have shown significant effects on stability and persistence in single-interaction networks, their interactive effects among multiple subnetworks on the dynamics of a multilayer network are unknown yet. Second, subnetworks might not be randomly interconnected, but have different interconnection patterns. It is also largely unknown whether the interconnection patterns interact with the subnetwork topologies to affect the dynamics of a multilayer network.

Theoretical exploration of ecological networks with diverse interaction types has thrived in the last decade. Non-random mixture of different types of interactions could significantly affect the simulated dynamics of ecological networks (García-Callejas et al., 2018b; Kéfi et al., 2012; Kéfi et al., 2015; Kéfi et al., 2016; Lurgi et al., 2016; Melián et al., 2009; Mougi & Kondoh, 2012; Suweis et al., 2014). In more recent years, the multilayer network models have been frequently applied to represent the networks that contain two or more layers defined as interaction types, habitats, time slices, etc. (Astegiano et al., 2017; Costa et al., 2016; García-Callejas et al., 2018a; Genrich et al., 2017; Mello et al., 2019; Mourier et al., 2019; Pilosof et al., 2017; Timóteo et al., 2018). The multilayer approach has focused on how inter-layer structural properties affect community dynamics (Dáttilo et al., 2016; Melián et al., 2009; Sauve et al., 2014; Sauve, Thébault, et al., 2016). For example, the non-random way that plants connect pollination and herbivory networks may promote simulated community stability (Sauve, Thébault, et al., 2016) and Gracia-Lázaro et al. (2018) showed also that competitive and mutualistic subnetworks traded off in affecting species persistence. However, there are still gaps in understanding how the intra-layer topologies and inter-layer interconnections interact with each other in affecting the community dynamics of multilayer networks.

In the past two decades, nestedness has become one of the most documented topological features in empirical ecological networks. It describes a pattern where species that interact with specialists belong to a subset of the species that interact with generalists (Bascompte et al., 2003), and mutualistic networks tend to be more nested than antagonistic ones (Thébault & Fontaine, 2010). An intuitive explanation of this pattern is trait-matching. For instance, if a pollinator has a long proboscis, it can obtain nectar from plants with both deep and shallow pollen tubes, but a pollinator with short proboscis can only access the plants with shallow pollen tubes (Staniczenko et al., 2013). However, since its introduction into the study of ecological networks, there have been intensive debates on the fundamental mechanisms behind its emergence. Some studies stressed that nestedness was a consequence of optimizing species/community-level fitness (Minoarivelo & Hui, 2016; Pinheiro et al., 2019; Suweis et al., 2013; Zhang et al., 2011), a by-product of trait coevolution and community assembly (Krishna et al., 2008; Maynard et al., 2018; Valverde et al., 2018), or an emergent property from degree sequences and assortativity (Jonhson et al., 2013; Payrató-Borràs et al., 2019).

Regardless of the debated reasons behind its emergence, the prevalence of nestedness in empirical networks might have important implications for ecosystem stability (Mariani et al., 2019; Payrato-Borras et al., 2020), which have been intensively studied (Baumgartner, 2020; Rohr et al., 2014; Staniczenko et al., 2013; Thébault & Fontaine, 2010) in single-interaction networks, yet attaining a consensus on this role has been elusive. The conclusions from studies on single-interaction networks depend critically on the choices of stability measures and the ways of constructing networks and model parameterization (Mariani et al., 2019). If interspecific competition is not considered, nestedness increases the vulnerability of a community to small disturbance based on random matrix theory (i.e. locally unstable; Allesina & Tang, 2012; Staniczenko et al., 2013), but with interspecific competition included, nestedness may reduce effective interspecific competition and enhance biodiversity and structural stability (Bastolla et al., 2009; Rohr et al., 2014). However, their effects on community dynamics were much less tested in ecological multilayer networks (Mariani et al., 2019). Sauve et al. (2014) found that the effect of nestedness was weakened in empirical networks combining mutualistic and antagonistic interactions. Morrison et al. (2020) found that nestedness showed positive and modularity showed negative effects on the robustness of empirical multilayer networks if least abundant species were lost first. In multilayer networks combining two bipartite ecological networks, it is not hard to notice the potential interactions between subnetwork nestedness and inter-subnetwork connections. On the one hand, nestedness represents an orderly and nested pattern of the interaction partners of the species of one trophic level in a subnetwork, forming a gradient from generalists to specialists; on the other hand, the way of the shared species connecting the interaction partners from two subnetworks determines whether a shared generalist or specialist species in one subnetwork is a generalist or specialist in the whole network. Currently, it is largely unknown how the interactions among subnetwork nestedness, inter-subnetwork connection patterns and other network topologies affect community dynamics of a whole multilayer network.

The present study aims to theoretically explore how the nested structures of subnetworks interact with other subnetwork structures and with inter-subnetwork connection patterns in affecting the community dynamics of multilayer networks. In reported empirical ecological networks, bipartite mutualistic and herbivory networks are among the most described (Bascompte & Jordano, 2013; Dáttilo & Rico-Gray, 2018), and a few available data have shown that plants could connect them to form a multilayer network (Morrison et al., 2020; Pocock et al., 2012). Therefore, I simulated theoretical multilayer networks including plant–herbivore, plant–mutualist and intraguild competition networks (Figure 1) and asked specifically the following three questions: (a) How do the nestedness of plant–herbivore and plant–mutualist subnetworks interact with each other and with interconnection patterns in affecting local stability and persistence of the multilayer networks? (b) How do the subnetwork complexity and intraguild competition intensity affect the effect sizes of subnetwork nestedness? (c) Does subnetwork nestedness interact with modularity in affecting local stability and persistence?

## 2 MATERIALS AND METHODS

### 2.1 Construction of multilayer networks

In the present study, I assumed that each multilayer network had three guilds, including plants (S_{p}), mutualists (S_{m}) and herbivores (S_{h}), and thus contained a plant–mutualist network (PM-subnetwork), a plant–herbivore network (PH-subnetwork) and three intraguild interspecific competition networks including plant–plant (PP), herbivore–herbivore (HH) and mutualist–mutualist (MM) competition subnetworks (Figure 1a,b). Here the intraguild competition refers to interspecific competition for various resources (such as suitable habitat, space, water, sunshine, etc.) within a guild other than interaction partners. For simplification, I also assumed the construction of networks suffered no sampling bias, and the simulated multilayer networks captured the main intra- and inter-layer topologies in this study. To focus on the effects of nestedness, I first determined the species number, connectance and interaction strength in the subnetworks independently for a multilayer network. Three random species numbers of plants, mutualists and herbivores were generated independently from a uniform distribution U[10, 60]. The uniform distribution was chosen for its equal probability of values within predetermined intervals, which ensured that the simulated ecological networks had a similar probability of species number and connectance for later statistical analyses; meanwhile, drawing values independently from a uniform distribution made the parameters unlikely to be equal across subnetworks in a multilayer network. Connectance is the proportion of realized interactions out of all possible interactions within a network. As connectance was usually scaled with network size in empirical plant–mutualist and plant–herbivore networks, I determined the connectance (C_{pm} and C_{ph}) of PM- or PH-subnetwork by following empirical relationships (on log scales, connectance = 0.964–0.699 × richness for PM-subnetwork; connectance = 0.683–0.765 × richness for PH-subnetwork) provided by Fontaine and Thébault (2015). I independently drew three connectance values from U[0.1, 0.8] for PP-, MM- and HH-competition subnetworks in each multilayer network to test how their competition intensity affected the effect sizes of subnetwork nestedness. The sampling ranges of connectance in competition networks were chosen for establishing a wide gradient of competition intensity. I assumed symmetric interaction links but with non-symmetric interaction strengths. Interaction strength values between species *i* and *j* (*α*_{i,j}) of all subnetworks were independently drawn from half-normal distributions ±|N[0, *σ*^{2}]| (‘| |’ indicates absolute value) with a mean = 0 and a standard deviation (*σ*) drawn from U[0.05, 0.2], assuming right-skewed distributions of interaction magnitudes (i.e. many weak interactions and few strong ones) that are commonly found in empirical networks (Berlow et al., 2004; Wootton & Emmerson, 2005). Specifically, for the realized interactions, I drew values from |N[0, *σ*^{2}]| for *α*_{i,j} and *α*_{j,i} separately in PM-subnetworks, |N[0, *σ*^{2}]| and -|N[0, *σ*^{2}]| for *α*_{i,j} and *α*_{j,i}, respectively, in PH-subnetworks, and -|N[0, *σ*^{2}]| for *α*_{i,j} and *α*_{j,i} separately in three intra-guild competition subnetworks. I obtained 50,000 combinations of species number, connectance and interaction strength levels of subnetworks. This work did not require ethical or fieldwork approval because it was a modelling study.

### 2.2 Nestedness of subnetworks

Because nestedness is related to network size and connectance, previous studies have typically used standardized measures of nestedness and path analysis to test the effects of nestedness (Bascompte et al., 2003; Thébault & Fontaine, 2010). Unlike those studies, I directly varied the relative nestedness of subnetworks under the condition of fixing the network size, connectance and interaction strength, which could exclude other disturbing factors and enabled me to derive the ‘pure’ effect sizes of subnetwork nestedness. I used presence–absence-based nestedness rather than weighted nestedness (often by interaction frequency) in this study. Although weighted nestedness could reveal another aspect of nested topology, it had unclear information in my simulations because interaction strengths of plant–herbivore subnetworks had both negative and positive values that were not suitable for its calculation. In addition, the presence–absence-based nestedness index had a closer relationship with the examined interconnection pattern (as shown in details below), calculated using the sum of presences of plant interaction partners from two subnetworks. Here, the process for PM-subnetworks was described, and PH-subnetworks were similarly constructed. Following the greedy algorithm by Song et al. (2017), I constructed the maximumly nested bipartite network (left panel in Figure 1c) with defined network size, connectance and interaction strengths, implemented by the r package maxnodf (Hoeppke & Simmons, 2021). It should be noted that this algorithm required a minimum connectance value to generate the maximumly nested topology, so the connectance would be increased to the lower limit if the original connectance was too low. Because the maximally nested matrix was divided into two parts (the cells with links and the cells without links, termed L_{in} and L_{out}, respectively), I then had S_{p}S_{m}C_{pm} = L_{in}C_{in} + L_{out}C_{out}, where C_{in} and C_{out} are the connectances of the two parts, respectively. By setting C_{in} with a sequential gradient from C_{pm} (randomly connected, i.e. minimally nested) to 1 (maximally nested), five sequential values of C_{out} were obtained to construct mutualistic networks with expected nestedness levels (see Figure S1 for an example). The nestedness value was calculated as the NODFc metric, which proved to be an effective measure of nestedness comparable across networks (Song et al., 2017). The gradient from nearly random to maximumly nested subnetwork topology allowed me to specifically calculate the effect sizes of subnetwork nestedness within a multilayer network (see below).

### 2.3 Interconnection patterns of plants linking two subnetworks

A previous study suggested a metric of inter-subnetwork patterns: the correlation between the mutualistic and antagonistic interaction partners of plants (Sauve, Thébault, et al., 2016). I opted for this metric because the ranks of interaction partners of plants in subnetworks were closely associated with the nested structure in two subnetworks. The three patterns of plants connecting mutualists and herbivores were set as positive, random and negative associations between the mutualistic and herbivory interaction partners, which were attained by rearranging the ranks of plants in the PM- and PH-subnetworks (as shown in Figure 1d–f). With five levels of nested structures in PM- and PH-subnetworks and three inter-subnetwork connection patterns to form a multilayer network (Figure S2), a total of 5 × 5 × 3 × 50,000 = 3.75 × 10^{6} multilayer networks were generated.

### 2.4 Subnetwork complexity and intraguild competition intensity

The complexities of both PM- and PH-subnetworks were calculated as $\sigma \sqrt{\mathit{SC}}$, following May (1972), where *σ*, *S* and *C* are the interaction strength, network size and connectance of the respective subnetworks, respectively. The intraguild competition intensity was calculated as *C* × *σ* from the PP-, HH- and MM-competition networks.

### 2.5 Modularity of subnetworks

To test how subnetwork nestedness interacted with the modularity of the interconnected subnetwork in affecting community dynamics, I tested the effects of nestedness of the focal (PM- or PH-) subnetwork when the other interconnected subnetwork had a random or modular topology. For simplification, a maximally modular subnetwork with three modules was constructed to compare with a random subnetwork given the same connectance and network size. Briefly, the species of the focal subnetwork were randomly assigned to three modules and species were linked randomly within and between modules. It might lose some generality to compare random subnetworks with three-module subnetworks, but nestedness is the focus of this work and a scenario of random vs. modular subnetwork would maximize the interactive effects between nestedness and modularity on community dynamics. In addition, I also tested whether the modularity of both PM- and PH-subnetworks interacted with interconnection patterns to affect community dynamics.

### 2.6 Stability and persistence measures

For simplification and comparability with previous studies (May, 1972; Sauve, Thébault, et al., 2016), I made several assumptions about the modelling system: (a) all species interactions influenced population dynamics on the same and immediate timescale, and no phenological and delayed effects were included; (b) species affected each other with linear functional responses and (c) no environmental perturbation to the system. I focused on the local stability and persistence of community dynamics.

Local stability is ecologically interpreted as the tendency of community dynamics returning to equilibrium after small disturbances (May, 1972), and is most useful at a steady state, which requires the evaluation of the Jacobian matrix by linearizing a system with species abundances at equilibrium (Novak et al., 2016). Following previous studies (Sauve, Thébault, et al., 2016), the stability measure was evaluated using the minimum intraspecific negative feedback necessary for community local stability, that is, the level of diagonal elements needed for all eigenvalues in a Jacobian matrix to have negative real parts, by assuming that all diagonal elements were the same. I opted to use this stability measure because it is closely related to local stability and comparable to previous studies (Neutel et al., 2007; Sauve, Thébault, et al., 2016). The stability measure is negative and a value closer to zero indicates higher local stability, and it was highly correlated to resilience (Pearson correlation = 0.88, *p* < 0.001) that mainly indicates how fast a system returns to equilibrium and is often measured by the leading eigenvalue of Jacobian matrix (Thébault & Fontaine, 2010). I focused on the results of local stability because the results of the two measures were similar.

*p*

_{i},

*m*

_{i}and

*h*

_{i}, respectively. The intrinsic growth rate was represented by

*r*

^{p},

*r*

^{m}or

*r*

^{h}, the self-negative density dependence was represented by

*d*

^{p},

*d*

^{m}or

*d*

^{h}, the intraguild competition was represented by

*α*

^{pp},

*α*

^{mm}or

*α*

^{hh}, and the plant–mutualist and plant–herbivore interactions were represented by

*α*

^{pm}and

*α*

^{ph}:

*AutoTsit5*solver with the call-back function

*TerminateSteadyState*, performed by the packages DifferentialEquations and JuliaCall in the Julia and R environments (Li, 2019; Rackauckas & Nie, 2017). Initial population densities were randomly drawn from a uniform distribution U[0.1, 10].

*r*

^{p},

*r*

^{m}and

*r*

^{h}were randomly drawn from U[0.5, 1.5], U[−0.5, 0.5] and U[−1, 0], respectively, assuming that herbivores relied on plants to survive, and mutualism could be facultative or obligate.

*d*

^{p},

*d*

^{m}or

*d*

^{h}were randomly drawn from U[0.5, 1.5] for introducing variation to negative density-dependent effects. The persistence was calculated as the proportion of surviving species at the end of the simulation (density > 1 × 10

^{−6}). The persistence of the whole multilayer network and different guilds could be calculated. However, I focused on the persistence of the whole multilayer network because it was positively correlated with the persistence of different guilds in this study (Table S1).

### 2.7 Effect size of subnetwork nestedness on stability or persistence

The (signed) effect size of subnetwork nestedness represented the sign and magnitude of the change in the stability or persistence per unit of subnetwork nestedness. It was evaluated as the regression coefficient (*β*) of a simple linear regression, *Y* = intercept + *βX* + stochasticity. Because I focused on the factors affecting β in a large number of networks, the *p* value of *β* in each linear regression was not checked. The response variable (*Y*) was the stability or persistence of the entire multilayer network and the single exploratory variable (*X*) was the PH- or PM-subnetwork NODFc values of five nestedness levels. Associations of effect sizes of the focal subnetwork nestedness with nestedness of the other subnetwork and interconnection patterns were examined by showing mean values at different levels of nestedness and interconnection patterns. I examined the associations of PM- and PH-subnetwork complexities with effect sizes of nestedness using tensor-product smooth function that is widely used for showing interactive effects of exploratory variables. The ‘knot’ parameter, which controlled the upper limit of the degrees of freedom (i.e. the degree of curvature of smoothing splines) in tensor-product smooths, was set as knot = 4 after model selection by lowest AIC (Akaike information criterion) values. Nonlinear regressions were applied to show the associations of intraguild competition intensity with effect sizes of nestedness for each competition subnetwork. The tensor-product smooths and nonlinear regressions were performed by mgcv package in r environment (Wood, 2006). I did not report *p* values from the statistical analyses on the associations of various factors with effect sizes of nestedness because an increase in simulation number always led to smaller *p* values and the vast sample size (3.75 × 10^{6} multilayer networks) made it unnecessary.

### 2.8 Sensitivity analyses

Sensitivity analyses were performed to test whether the variation of model parameters might change the results. (a) I applied stronger variability on the parameters of initial population densities, intrinsic growth rates and self-density-dependent effects (Appendix S1). (b) As interaction strength was found to affect network persistence (Bunin, 2017), I tested the results at different levels of interaction strength by varying the standard deviations (*σ*) for half-normal distributions of interaction strengths from U[0.01, 0.05], U[0.05, 0.1], U[0.1, 0.15], U[0.15, 0.2] to U[0.2, 0.3].

## 3 RESULTS

### 3.1 Interactive effects of subnetwork nestedness and interconnection patterns on stability and persistence

Overall, the stability and persistence of the whole multilayer network were highest when the relative nestedness of the PH- and PM-subnetworks were low, regardless of interconnection patterns (Figure 2). The effect sizes of the relative nestedness of a focal PM-/PH-subnetwork depended on the nestedness of the interconnected PH-/PM-subnetwork and the interconnection patterns (Figure 3). As the PM-subnetwork nestedness increased, the effect sizes of the relative PH-subnetwork nestedness on stability shifted from negative to positive, whereas its negative effect sizes on persistence became weaker (Figure 3a,c). As the PH-subnetwork nestedness increased, the negative effect sizes of the relative PM-subnetwork nestedness on stability and persistence became weaker (Figure 3b,d). Compared with random and negative interconnections, the positive interconnection pattern caused the effect sizes of the relative PH- and PM-subnetwork nestedness to increase, either more positive or less negative. The negative interconnection resulted in more negative effect sizes of subnetwork nestedness than did the random interconnection, except for that of PH-subnetwork nestedness. The results on resilience were similar to those on local stability (Figure S3).

### 3.2 Association of subnetwork complexity with the effect sizes of subnetwork nestedness on stability and persistence

PM- and PH-subnetwork complexities showed interactive effects on the effect sizes of subnetwork nestedness (Figure 4). As the PM-subnetwork complexity increased, the effect sizes of PH-subnetwork nestedness on stability and persistence shifted from negative to positive (Figure 4a,c). The effect sizes of PM-subnetwork nestedness on stability and persistence shifted from neutral to negative in response to the increase of PM-subnetwork complexity (Figure 4b,d). The PH-subnetwork complexity showed weak associations with the effect sizes of subnetwork nestedness.

### 3.3 Association of intraguild competition with the effect sizes of subnetwork nestedness on stability and persistence

Intraguild competition intensity showed different associations with the effect sizes of subnetwork nestedness. For the effect sizes of PM-subnetwork nestedness on stability and persistence, PP- and MM-competition intensity reduced the negative effects, whereas HH-competition intensity showed little association (Figure 5). For the effect sizes of PH-subnetwork nestedness, the associations of intraguild competition intensity were negligible (Figure S4).

### 3.4 Association of subnetwork modularity with the effect sizes of subnetwork nestedness on stability and persistence

The nestedness of PM- or PH-subnetwork on stability and persistence showed similar effect sizes no matter the interconnected subnetwork had a modular or random topology (Figure 6). A positive interconnection pattern always led to higher effect sizes of subnetwork nestedness. Additional analyses showed that a modular PM-subnetwork led to lower stability and persistence than a random one, but the effects of random and modular PH-subnetworks showed no difference (Figure S5). These results suggested subnetwork modularity did not interact with nestedness and interconnection patterns in affecting the stability and persistence of the multilayer network.

### 3.5 Sensitivity of model parameters

Applying a larger variability on the parameters of initial population densities, intrinsic growth rates and self-density-dependent effects did not affect the results qualitatively (Appendix S1). With increasing interaction strength levels, the means of stability and persistence both decreased, but their variations increased (Figure S6). Meanwhile, the effect sizes of subnetwork nestedness showed little change in the mean values but increased the variation as interaction strength levels increased (Figure S7). Notably, the variations were very small for the effect sizes on stability at the level of 0.01–0.05 interaction strength and the effect sizes of the PH-subnetwork nestedness on persistence at the level of 0.2–0.3 interaction strength, that is, too low or too high levels of interaction strengths were inappropriate for the detection of effect sizes of subnetwork nestedness. In most cases, the increase in interaction strength levels did not qualitatively affect the results of the interactive effects between subnetwork nestedness and interconnection patterns on stability (Figures S8a and S9a). However, at the low levels of interaction strengths, the nestedness of interconnected subnetwork showed little association with the effect sizes of the focal subnetwork nestedness on persistence when interconnection patterns were random and negative (Figures S8b and S9b). These analyses suggested the broad range of interaction strength (0.05–0.2) was appropriate for the detection of effect sizes of subnetwork nestedness in this study and the results were generally robust with respect to local stability and persistence.

## 4 DISCUSSION

Nestedness is one of the most frequently studied topological structures in ecological networks (Bascompte et al., 2003; James et al., 2012; Staniczenko et al., 2013), and its relationship with community dynamics has been debated in the past two decades. Nonetheless, previous studies focused mainly on its impacts on community dynamics in single-interaction networks, while those in multilayer networks are relatively less known. I found that the effects of subnetwork nestedness on local stability and persistence (of the entire multilayer networks) were highly dependent on interconnection pattern, subnetwork complexity and intraguild competition, which implies the necessity to re-evaluate the effects of the topological structures of single-interaction networks in the context of ecological multilayer networks.

Using empirical single-interaction networks, previous studies have shown that nestedness decreased local stability and persistence without considering interaction trade-offs (Staniczenko et al., 2013; Thébault & Fontaine, 2010), which were also verified by my simulated networks (see Appendix S2). Based on random matrix theory, the negative impact of nestedness on local stability was caused by larger eigenvalues in nested mutualistic networks than that in random ones (Allesina & Tang, 2012; Staniczenko et al., 2013), which made it more vulnerable to small perturbations. Thébault and Fontaine (2010) found that nestedness showed a strong negative effect on persistence in mutualistic networks but a weak one in antagonistic networks. In this study, the overall negative effects of nestedness of PM-subnetworks on local stability and persistence are consistent with those of single-interaction networks, but the effects of nestedness of PH-subnetworks can be negative or positive, depending on the properties of other subnetworks. In simulated multilayer networks, Sauve et al. (2014) found that the nestedness of PH-subnetworks showed a weak and significant negative effect on persistence through path analyses, but no effect for the nestedness of PM-subnetworks was detected. Inconsistent with their results, my analyses have a high sensitivity in detecting effects of nested topology, probably because I specifically simulated relative nestedness gradients under the condition of fixing network size, connectance and interaction strength.

The results showed that there were interactive effects among the nestedness levels of two subnetworks and interconnection patterns on stability and persistence. The effect sizes of the nestedness of a focal subnetwork were positively correlated with the nestedness of the interconnected subnetwork, which indicated the reduction of negative effects or a shift from negative to positive effects. Previous studies also suggested that a positive correlation of plant generalism in PM- and PH-subnetworks facilitated stability because it cancelled the indirect positive and negative effects between plants (Sauve, Fontaine, & Thébault, 2016). Consistent with this, my results showed that the positive interconnection pattern not only directly benefited local stability and persistence but also reduced or reversed the negative effects of subnetwork nestedness. Therefore, the stabilizing effect of having a positive interconnection between the nestedness patterns of the two subnetworks counteracts the destabilizing effect of subnetwork nestedness. This is probably because the positive interconnection can balance the positive and negative feedback from two subnetworks, which is especially crucial for some plants that receive extensive negative and positive interactions simultaneously. However, deciphering the mechanism underlying the interdependence between subnetwork nestedness and interconnection patterns is challenging and warrants further exploration in empirical multilayer networks.

Because a multilayer network is composed of multiple subnetworks, I expected that the relative complexity of subnetworks should affect the effect sizes of subnetwork topologies on community dynamics. My results confirmed this expectation, but the associations between subnetwork complexities and effect sizes of nestedness depended on the focal subnetwork and dynamics measure. For instance, PH-subnetwork nestedness was found to destabilize in multilayer networks with less complex PM-subnetworks, but became stabilizing as the complexity of PM-subnetworks increased. Previous studies suggested that moderate proportions of mutualistic interactions benefit community stability (Mougi & Kondoh, 2012), and my study adds to that by showing that subnetworks of mutualistic interactions might interact with nestedness of antagonistic interactions to promote community stability. These complexity-dependent effects of subnetwork nestedness provide a new angle for understanding the interactive stabilizing effects of network architectures.

Although intraguild competition interactions have often been considered in bipartite trophic or non-trophic networks, their interactions with other topological properties on community dynamics were less investigated. Recent studies have shown that intraguild competition networks might affect species coexistence together with other subnetworks (Bastolla et al., 2009; Gracia-Lázaro et al., 2018; Pascual-García & Bastolla, 2017). My results have extended these studies by showing that competition intensity could also interact with nestedness to affect both local stability and persistence; however, different guilds showed distinct interactions with the effect sizes of subnetwork nestedness. The attenuation of negative effects of PM-subnetwork nestedness by intraguild competition of plants and mutualists indicated that there might be trade-offs between intraguild competition and nestedness in affecting community dynamics.

Irrespective of interconnection patterns, I found that modular topology of interconnected subnetworks did not change the effect sizes of nestedness of focal subnetworks on community dynamics, suggesting that modularity of one subnetwork might not affect the effects of nestedness of the interconnected subnetwork in multilayer networks. This is different from the interactive effects between nestedness of two interconnected subnetworks, which might be caused by the way of interconnecting subnetworks. The interconnection patterns examined in my simulations were mainly related to nested ranks of species in subnetworks, but not related to species roles of modular subnetworks. Consistent with previous studies in which modularity showed strong negative effects on persistence in empirical mutualistic rather than antagonistic networks (Thébault & Fontaine, 2010), my analysis confirmed that the modularity of PM-subnetworks, rather than PH-subnetworks, showed a decreasing effect on the stability and persistence irrespective of interconnection patterns and modularity of interconnected subnetworks (Figure S5). However, the simulation of random vs modular subnetworks represented only a simplified scenario. Modular topology is determined by the size and number of modules and links among and within modules. Therefore, the details of modular topology, which are out of the scope of the current study, should be modelled to obtain robust conclusions on its effects on community dynamics in future studies.

In general, the absolute values of effect sizes were small in the simulations. The reasons are at least threefold. First, the effect sizes in previous studies are usually standardized path coefficients (Sauve et al., 2014; Thébault & Fontaine, 2010), but the effect sizes in this study directly reflected the change of stability or persistence per NODFc. Because the measures of stability and persistence had a limited range of variation and small values, that is, from approximately −2 to 0 for stability and from 0 to1 for persistence (Figure S6), the values of effect sizes were expected to be small. The small effect sizes are actually not contrasting with previous studies in which the effects of nestedness were found to be weak or only mediate the effects of connectance and network size (Sauve et al., 2014; Thébault & Fontaine, 2010). Second, a subnetwork is only one part of the whole multilayer network. The small effect sizes suggest that the topological structure of a subnetwork could only have a relatively weak impact on the dynamics of the whole multilayer network. The higher proportion of species and links one subnetwork accounts for in a multilayer network, the more important role the subnetwork topology might play. For example, the negative effect sizes of PM-subnetwork nestedness became stronger as PM-subnetwork complexity increased (Figure 4b,d). Third, the effect sizes were shown as mean values (Figures 1-4). A large number of simulations incorporated many variations of basic complexity (i.e. species richness, connectance and interaction strength) and interconnection patterns, which all affected the effects of nestedness and caused the mean effect sizes to be close to zeros. Despite the small effect sizes, the important thing here is that the effect sizes of subnetwork structures can change with subnetwork complexity and interconnection patterns, which highlights the need to evaluate the effects of subnetwork nestedness in the context of multilayer networks.

Due to the assumptions involved in the simulations, the results in this work have several limitations. First, some studies showed nestedness and modularity were negatively correlated (Fortuna et al., 2010; Thébault & Fontaine, 2010) while they might appear simultaneously to form compound structures in empirical single-interaction networks, especially with larger spatial or taxonomic sampling scales (Flores et al., 2011; Genrich et al., 2017; Lewinsohn et al., 2006; Mello et al., 2019). My theoretical framework might represent an undersampling case (Bluthgen, 2010); thus, caution should be taken to interpret the results when a compound topological structure exists in subnetworks. Second, I adopted a phenomenological approach that connected nestedness directly with community dynamics (Ings et al., 2009), that is, nestedness was the sole property considered in subnetworks. The destabilizing effects of subnetwork nestedness suggest community stability may not be the reason for its prevalence in nature (Staniczenko et al., 2013). Recently, there is a boost in developing ecological and/or evolutionary models for nestedness emergence (Minoarivelo & Hui, 2016; Pinheiro et al., 2019). Considering the underlying processes for its emergence might facilitate the understanding of the stabilizing effects of subnetwork nestedness (and other topologies) in multilayer networks. Third, I used limited parameter values and specific models in the simulations, so the results must be cautiously interpreted. These are weaknesses common to most simulation studies (Peck, 2004). As empirical data of multilayer networks accumulate (Liao et al., 2020; Mello et al., 2019), it is worthwhile to consider more realistic parameters and modelling frameworks in future studies. Finally, my results are limited to the choices of dynamics measures. It is known that the dynamics of ecological communities have multiple dimensions and different metrics give different insights (Domínguez-García et al., 2019). Local stability and persistence used in this study are only two aspects of the dynamics of multilayer networks. Subnetwork nestedness may exhibit completely different effects if other metrics (such as tolerance to species and interaction loss or the so-called robustness) are considered. For example, contrary to the general negative effects on local stability and persistence in this study, nestedness was found to increase robustness in single-interaction and multilayer networks (Memmott et al., 2004; Morrison et al., 2020). Therefore, the patterns related to the interactive effects of subnetwork nestedness and interconnection patterns are worth being explored for other stability metrics.

In conclusion, this work adds to the fast-growing field that advocates for the description of multilayer features of ecological networks and their consequences on natural community dynamics. The results demonstrate that the effects of nestedness on simulated community dynamics depend on other structural properties of the whole network, which suggests that the nestedness cannot be studied in isolation, but should be investigated in the context of multilayer networks. This might be true for other topological structures that used to be mostly investigated in single-interaction ecological networks. The findings of this study also argue for more empirical studies to uncover the within- and inter-layer structures in ecological networks, which require more sampling efforts in terms of taxa, interactions, time and space.

## ACKNOWLEDGEMENTS

I am thankful to Dr Xinqiang Xi for the discussion. This work was supported by National Natural Science Foundation of China (32071542) and the Fundamental Research Funds for the Central Universities (lzujbky-2020-ct02).

## CONFLICT OF INTEREST

I declare no conflict of interest.

## Open Research

# DATA AVAILABILITY STATEMENT

The R codes for simulations can be accessed via https://doi.org/10.6084/m9.figshare.16909411 (Yan, 2022).