An RGB-D framework for capturing soft-sediment microtopography

Abstract

1. Capturing sediment topography is one component in understanding how large animals in soft sediment ecosystems affect ecosystem processing. Traditional approaches for sampling soft-sediment systems are too invasive and destructive for the fragile sediment structures. In contrast, non-invasive approaches, such as LIDAR and monocular structure-from-motion (SfM) systems, can be expensive and cumbersome for data processing.
2. We developed a low cost and practical framework for measuring morphological characteristics of soft sediment topography at the millimetre scale. Using an RGB-D (red, green, blue and depth) device and a semi-opaque container, the sediment surface can be imaged rapidly while operating in outdoor environments.
3. The RGB-D device imaged 0.3 m² intertidal sediment surfaces, creating depth imagery over seven sites and 82 surfaces. First, using simulated surface data, we formulated and tested four variations of a geometrical detrending model to extract and measure the undulating surface microtopography. We then computed nine morphological characteristics, including arithmetic mean roughness, mean sediment peaks and rugosity, for the detrending models. We then used the two best performing models, linear and quadratic detrending models, to compare extracted morphological characteristics from the field data. We found a strong correlation between the models for extracted surface measures and similar extracted sediment trends.
4. The results showed that RGB-D real-time imaging is a promising rapid scanning tool for collecting field data in intertidal regions and can be expanded to other fragile sediment surfaces outside the marine environment. The low costs and real-time feedback makes it an attractive data collection tool for environments where data collection is challenging.

1 INTRODUCTION

There is a well-established link between geometrical complexity of a habitat and its functions within an ecosystem. Examples of this such as species abundance and distribution (Ferrari et al., 2018) and habitat biodiversity (Graham et al., 2006) are often studied to understand ecological gradients. Small ($\mathit{mm}-m$) surface structures, known as microtopography, are often overlooked although they underpin key ecological processes such as facilitating plant recruitment (Beatty, 1984), and affecting species distribution (Wang et al., 2019) and soil properties (Smith & Warburton, 2018). In marine soft sediments, microtopography influences near-surface hydrology (Bolam & Fernandes, 2002; Thrush et al., 1992) and are deeply connected to biodiversity and marine ecosystem functions (Thrush et al., 2001). While these relations have been demonstrated in specific experimental studies (Hope et al., 2020), broader application is constrained by accessible approaches to characterise sediment microtopography in situ, which is highlighted by a lack of publications and solutions in this field of study. As recognised by the experts, there is a need to reliably extract high quality structural information.

Soft sediment structures are fragile, rendering traditional approaches for measuring the geometrical complexity of entities destructive, for example, chain-tape measures (Luckhurst & Luckhurst, 1978; Risk, 1972) and moulds (Sanson et al., 1995). Another approach to microtopography measurement includes needle-based devices (Leatherman, 1987) with acceptable spatial frequency, for example 6 mm. However, this may not be usable for soft-sediments as the needles could disrupt the sediment surface. Furthermore, the device would prove difficult to transport to the sampling sites due to their remoteness. Less invasive instruments such as laser based mechanical devices1 exist and provide good measurement accuracy (6 microns), although their cost and cumbersome nature makes them highly impractical for field experiments. Aerial and terrestrial LIDAR have been used for spatial analysis across a range of scales (Brock et al., 2004; Sankey et al., 2011) but are prohibitively costly and may lack fine Z-resolution (Bianco et al., 2013). Consumer-grade cameras are an affordable alternative with pixels in camera stills used for characterisation (Zollhöfer, 2019). Progress in 3D reconstruction algorithms presented ecologists with new tools, such as stereophotogrammetry (Hirschmuller, 2005) and structure-from-motion (SfM) (Schonberger & Frahm, 2016; Westoby et al., 2012) to extract both accurate and precise 3D geometric information (Leon et al., 2015; Marre et al., 2019) from video and stereo-image pairs, providing features such as slope, orientation, rugosity and other topographic features at varying scales (Du Preez, 2015; Friedman et al., 2012).

These algorithms require unique visual features (visual variability) to extract 3D geometric information using feature matching. Visually homogenous surfaces result in more ambiguous feature matches and, therefore, less reliable 3D information extraction. Furthermore, there is additional cost and processing time when using high quality 3D photogrammetry software.

The introduction of RGB-D (red, green, blue and depth) devices such as the Microsoft Kinect,² Asus Xtion³ and the Intel RealSense⁴ offers new opportunities for non-invasive and low cost spatial analysis at the centimetre and millimetre scale (Mankoff & Russo, 2013). The devices use a projected infrared (IR) patterns in combination with an IR camera and RGB cameras to overcome ambiguous feature matching and create 3D depth images. Open-source software libraries allow interfacing with the devices, removing the complexity around 3D data acquisition and provide real-time feedback on the acquisition quality, important for narrow acquisition windows. Though primarily developed for indoor applications, the sensors have seen use outdoors to characterise 3D structures provided appropriate shielding from direct sunlight (Azzari et al., 2013; Nicholson et al., 2016; Paulus et al., 2014; Thomsen et al., 2015), shallow water bathymetry (Klopfer et al., 2017; Mankoff & Russo, 2013), surface roughness (Marinello et al., 2015) and habitat complexity (Kamal et al., 2014).

To the best of our knowledge, there are currently no example applications in the literature regarding the use of these devices to capture soft sediment structure at centimetre to millimetre scale in marine environments.

In this study, we present the suite of tools and methods to answer the following two research questions:

• Can we utilise a highly portable, affordable and self/USB-powered, commercial grade off-the-shelf RGB-D device to gather fine-scale topographic variability of soft sediment surfaces on intertidal sandflats?
• Can we apply this methodology in a case-study investigating the effects of changes in the macrofaunal communities that modify sediment topography through bioturbation?

2 MATERIALS AND STUDY AREA

2.1 Study areas and plot treatment

The system was tested at seven intertidal flats across Mahurangi Harbour, New Zealand (36°26′S 174°42′E). The sites were named with abbreviations based on the location, see Figure 2. The seven sites were PSE, PW, PN, BB, PE, LBS and PS. At each site, six control and six exclusion plots were distributed along a shore-parallel line. Exclusion plots (1 m², n = 6) were covered with a sheet of fibreglass net (1 m², 1.34 mm mesh size) buried horizontally 3 cm deep in the sediment to prevent large macrofauna from accessing the sediment surface (Kamal et al., 2014). The plot was depleted of large animals as the macrofauna migrate out or die. Adult cockles Austrovenus stutchburyi found in the surface sediment were removed. Fences made of 5 mm mesh were buried along the perimeter (3 cm buried, 3 cm above ground) to prevent recolonization. No fences, netting or bivalves were removed from the control plots (1 m², n = 6), allowing freedom of movement into and out of the plot. Control plots were spaced approximately 1 m away from exclusion plots and each pair of plots separated by 5 m. Plots were left for 30 weeks before experiments were performed, allowing for microbial and chemical gradients to re-establish (Hillman et al., 2020, 2021). There was no evidence from sediment ripples that the fences affected flow and no samples were collected within 25 cm of the plot perimeter to avoid edge effects and effects of the fences on the measurements. As no animals were collected from the estuary, no field permit was required to carry out the plot treatment. The changes in the size of the macrofauna present in the experimental plots—and thus shifts in surface topography—provided an excellent opportunity to test the depth acquisition process.

2.2 Data acquisition

Data were acquired using an Asus Xtion Pro Live 3D RGB-D device. The Asus Xtion utilises a structured light sensor where 3D information is obtained through the deformation of a projected IR speckle pattern. Objects within the field-of-view (FOV) of the camera deform the pattern, allowing depth information to be extracted. The Asus Xtion was chosen over comparable structured light devices such as the Microsoft Kinect V1 and Intel® RealSense™ D435 due to portability and image quality. The Asus Xtion has no moving parts unlike the Microsoft Kinect which has motorised parts that make it heavier. In the marine environment, motorised parts can be corroded by salt in the air. Furthermore, the Microsoft Kinect requires a proprietary cable connection and power supply which reduces portability and device longevity, while the Asus Xtion is powered through a standard USB port. Compared to the D435, the Asus Xtion depth images were less noisy in initial testing in shaded areas outdoors.

The device was placed into an opaque plastic container to reduce sunlight IR interference during field operation (Figure 1). The Asus Xtion was fixed by a bolt to the inside top of the container. The wire ran through the pipe outside where it was connected to the portable computing device (e.g. tablet or laptop with a USB port). The white opaque plastic on the bottom allowed diffuse light to illuminate the surface while maintaining the minimal required imaging height of 0.8 m. The Asus Xtion and similar devices have a known poor performance in outdoor conditions, that is direct sunlight illumination. Initial experiments confirmed that shaded areas are imaged with the same accuracy and resolution as indoor conditions. Impact at outdoor illumination (intensity) was minimal at the distance range (<1.5 m) with the root mean square (RMS) on best fit of plane on planar surfaces at 1.5 mm at 3 mm (Kuan et al., 2019). The device was powered and operated by a Microsoft Windows tablet. Captured images were 320 × 240 pixels resolution but due to the set height of acquisition (0.8 m) and the device FOV, the inner sides of the container were visible in the raw image. With the sides excluded, the surfaces were represented by 240 × 200 pixels with a ground sampling resolution of 2.5 mm per pixel resulting in 0.3 m² sampled of the 1 m² experimental sediment plots. Table 1 details the manufacturer specifications for the device.

FIGURE 1

Left: The surface imaging tool with the Asus Xtion visible; right: The measurement box in situ with its perspex—white—extender to reach Asus Xtion's minimal measurable height.

TABLE 1. Asus Xtion Pro manufacturer specifications and depth resolution measurement from Khoshelham and Elberink (2012)

Asus xtion pro
Field of view	58° H; 45° V; 70° D
Depth image size	($320\times 240$ pixels) 60 FPS
Colour image size	($320\times 240$ pixels)
Distance of use	$0.8\mathrm{m}\le x<3.5\mathrm{m}$
Operating temperature	5–40°C
Dimensions	$18\times 2.5\times 3.5\mathrm{cm}$
Depth resolution	1 mm at 0.8 m
Power consumption	Below 2.5 W
Weight	700 g

FIGURE 2

Location of the seven sites used for examining the efficacy of the RGB-D sensor in Mahurangi Bay, New Zealand. Modified from Hillman et al. (2020).

Data were gathered in diffuse lighting during the early morning or overcast days at low tide to minimise device interference by strong sunlight. The device was oriented parallel to the sediment surface and fastened tightly to a slider that moved smoothly within the box. For each control and treatment plot, the box containing the device was placed over the central plot area. The height was then measured with a laser interferometer, and a depth and RGB image were captured simultaneously with the application. Real-time feedback from the device's video feed to the application ensured each snapshot was of acceptable quality. Images were stored locally on the tablet to be processed later. A typical scan would take less than a minute for each plot.

2.3 Depth resolution

Depth resolution is the imaging device's precision in resolving the distance between objects in a scene and itself and needs to be determined first. For this investigation, the aim is to characterise the topography of soft sediment surfaces affected by animal activity. The expected animal activity will create sediment features in the millimetre and centimetre scales, such as worm tubes or crab burrows. Thus, at least 1 mm depth resolution is required.

The Asus Xtion makes use of a PrimeSense Carmine 1.08 SOC (system on a chip) for imaging and is calibrated after device assembly to a depth resolution of $\le 1\mathrm{mm}$ at operating distance of 0.8 m (Halmetschlager-Funek et al., 2018).

2.4 Device output

The depth information extracted is encoded into a 16-bit greyscale image known as a depth image (Figure 3). Each pixel in the image encodes the distance of surfaces from the device in millimetres. The darker the pixel, the closer to the device the surface is, and because they cover a finite spatial resolution determined by the operational height and device specifications, the continuous sediment surface is discretised.

2.5 Software

To operate the device, the tablet was loaded with custom software for image acquisition and storage. The application was developed in C# using the free OpenNI (now OpenNI2⁵) libraries to interact with the device and present the user the depth data stream. The user captures the depth image when the desired quality is achieved.

Routines were written with Matlab 2017a®. As no Matlab®–specific packages were used, the presented algorithm can be replicated in any programming language using image processing functions. The numerical surface characteristics from the processing were written to comma-separated value (CSV) files for processing.

3 METHODS

3.1 Calibration

RGB-D devices were distributed with a factory calibration for depth extraction, allowing them to be used immediately. However, in order to remove lens distortion from colour images as well as project depth maps into 3D space, the intrinsic calibration parameters of the device were needed. Each camera on the device needed to be calibrated individually. Calibration for both RGB colour and IR depth cameras were performed using a black and white checkerboard and a variant of Tsai calibration (Gee et al., 2015; Tsai, 1987). While the RGB camera was readily calibrated, the IR camera calibration required the IR projector to first be obfuscated (covered with a plastic block to remove the speckle pattern) to remove interference while detecting the checkerboard corners (see Figure 4). Second, to improve contrast of the checkerboard pattern, an IR light source, such as an incandescent lamp, was used (calibrating outdoors in the sun was a viable alternative). Table 2 provides the intrinsic parameters of the device where $f_x$ and $f_y$ are the focal lengths, $c_x$ and $c_y$ are the camera's centre coordinates in the x and y direction, $k_1$ and $k_2$ are the first and second order radial distortion, and $p_1$ and $p_1$ are the first and second order tangential distortion parameters.

FIGURE 3

Device output: The colour (RGB) image (a) and corresponding depth image (b) of a sediment surface captured by the Asus Xtion showing fine sediment banding from tidal action. The 2D profile plot (c) shows the trended surface along the yellow dashed line in (b).

FIGURE 4

Comparison between images captured by the IR camera without the projector covered (a) and with the IR projector covered (b). The IR speckle pattern produced by the IR projector can interfere with the calibration process.

FIGURE 5

Example of a plane detrending a trended sediment surface captured by the RGB-D device. All points are denoted by their pixel position in the RGB-D depth image and their pixel value which stores the distance from the camera. Points below the detrended surface are coloured in red while those above are coloured cyan.

FIGURE 6

Simulated surfaces were used to test the trend fitting mode. For example surface (a) is a generated non-trended surface. This is then combined with randomised trends to produce a trended surface. Surfaces (b)–(e) are surface (a) combined with randomised trends of plane, quadratic, cubic and quartic, respectively.

TABLE 2. Calibration intrinsics and distortion parameters

	Intrinsic parameters
	$f_x$	$f_y$	$c_x$	$c_y$	$k_1$	$k_2$	$p_1$	$p_2$
			pixels	pixels	${\mathrm{mm}}^{-2}$	${\mathrm{mm}}^{-4}$	${\mathrm{mm}}^{-2}$	${\mathrm{mm}}^{-2}$
Depth	285.23	286.35	155.31	122.02	−0.038	0.058	−0.0016	−0.0024
RGB	267.17	268.24	159.13	121.32	0.053	−0.18	−0.000056	−0.0015

3.2 Feature extraction

Animal activities, such as burrowing or filter feeding, create sediment features much smaller than that of the regular natural sediment structural gradients (e.g. sand waves, slope or drainage channels). These can be lost in the underlying sediment trends when scanned but can be highlighted with detrending.

In order to perform surface detrending, depth image pixels were treated as points in 3D Cartesian space where a pixel's location along the width and height of the image lies on the X and Y axes, and the pixel depth values lie on the Z-axis of the coordinate system. Any ensuing (plane) slant from the camera fitting was corrected, with respect to the horizontal plane, using the Euler-Rodriguez formula for rotation by an angle $\theta$ according to the right-hand rule around the axis of rotation of direction $k$ (Rodrigues, 1840):

$$v_{\mathit{rot}}=v\cos \theta +\left(k\times v\right)\sin \theta +k\left(k.v\right)\left(1-\cos \theta \right).$$(1)

A polynomial model was then fitted to the 3D points to extract the underlying trend. Here, four models were tested against simulated data: these included a plane, a quadratic, a cubic and a quartic model (Equations 2–5).

$$z=a_0+a_{1}x+a_{2}y,$$(2)

$$z=a_0+a_{1}x+a_{2}y+a_3{x}^2+a_4\mathit{xy}+a_5{y}^2,$$(3)

$$z=a_0+a_{1}x+a_{2}y+a_3{x}^2+a_4\mathit{xy}+a_5{y}^2+a_6{x}^3+a_7{x}^{2}y+a_8{\mathit{xy}}^2+a_9{y}^3,$$(4)

$$\begin{array}{c}z=a_0+a_{1}x+a_{2}y+a_3{x}^2+a_4\mathit{xy}+a_5{y}^2+a_6{x}^3+a_7{x}^{2}y+a_8{\mathit{xy}}^2+a_9{y}^3\\ +a_{10}{x}^4+a_{11}{x}^{3}y+a_{12}{x}^2{y}^2+a_{13}{\mathit{xy}}^3+a_{14}{y}^4,\end{array}$$(5)

where $a_n$ are the polynomial coefficients.

A system of linear equations was set up with polynomial coefficients and the 3D points of the surface, and solved using least squares. An example of an over-determined system of linear equations for a fourth order polynomial fitted against surface points $\left(x_p,y_p\right)$, for $\left\{p=1,\dots ,l\right\}$, is shown in Equation 6, where $z$ is the depth of the surface point in millimetres and $\left\{a_0,\dots ,a_{15}\right\}$ are the coefficients of the polynomial function. The resulting model of best fit was then removed from the original depth image to produce a detrended depth image (now referred as $Z_d$), where the sediment surface is represented by positive and negative values about the fitted model (Figure 5).

$$\begin{array}{c}\left[\begin{array}{cccccccc}1\hfill & x_1& y_1& \dots & x_1^2{y}_1^2& \dots & x_1{y}_1^3& y_1^4\\ 1& x_2& y_2& \dots & x_2^2{y}_2^2& \dots & x_2{y}_2^3& y_2^2\\ ⋮& ⋮& ⋮& \ddots & ⋮& \ddots & ⋮& ⋮\\ 1& x_p& y_p& \dots & x_p^2{y}_p^2& \dots & x_p{y}_p^3& y_p^4\\ ⋮& ⋮& ⋮& \ddots & ⋮& \ddots & ⋮& ⋮\\ 1& x_l& y_l& \dots & x_l^2{y}_l^2& \dots & x_l{y}_l^3& y_l^4\end{array}\right]\left[\begin{array}{c}{a}_0\\ a_1\\ a_2\\ ⋮\\ a_{12}\\ ⋮\\ a_{14}\\ a_{15}\end{array}\right]=\left[\begin{array}{c}{z}_0\\ z_1\\ z_2\\ ⋮\\ z_p\\ ⋮\\ z_{l-1}\\ z_l\end{array}\right]\hfill \end{array}$$(6)

3.3 Extracting surface characteristics

The detrended depth images of each plot were imported into Matlab® and a set of programming instructions was used to process them. Holes or imaging artefacts, which possess pixel values of zero, located within the imaged plot area were filled using morphological reconstruction approach (Soille, 2013) (implemented in Matlab® as imfill⁶). Holes that could not be filled were excluded from analysis. Table 3 shows the surface characteristics extracted by the Matlab® routine. Peaks and troughs refer to positive and negative pixel values respectively. As the pixels' values are in millimetres, the surface characteristics are as well.

TABLE 3. Surface characteristics calculated from detrended depth images. $N_u$ and $N_v$ are the image horizontal, respectively vertical, dimensions in pixels, $Z_d\left(u,v\right)$ is the depth value at pixel location $\left(u,v\right)$, $\overline{Z_d}$ is the detrended surface mean.

Arithmetical mean roughness ($S_a$)	$S_a=\frac{1}{N_u{N}_v}\sum _{u=1}^{N_u}\sum _{v=1}^{N_v}\mid Z_d\left(u,v\right)\mid$	Mean of absolute deviations from the detrended surface
Root mean square Deviation ($S_q$)	$S_q={\left[\frac{1}{N_u{N}_v}\sum _{u=1}^{N_u}\sum _{v=1}^{N_v}{Z}_d^2\left(u,v\right)\right]}^{\frac{1}{2}}$	Root-mean-squared value of vertical deviations from the detrended surface
Mean peak	$M_p=\frac{1}{N_u{N}_v}\sum _{u=1}^{N_u}\sum _{v=1}^{N_v}{Z}_d\left(u,v\right)$	Mean height of positive surface pixel values where $Z_d\left(u,v\right)>0$
Peak standard deviation	${\sigma }_p={\left[\frac{1}{N_u{N}_v-1}\sum _{u=1}^{N_u}\sum _{v=1}^{N_v}{\left(Z_d\left(u,v\right)-\overline{Z_d}\right)}^2\right]}^{\frac{1}{2}}$	Standard deviation of positive surface pixel values where $Z_d\left(u,v\right)>0$
Mean trough	$M_t=\frac{1}{N_u{N}_v}\sum _{u=1}^{N_u}\sum _{v=1}^{N_v}{Z}_d\left(u,v\right)$	Mean height of negative surface pixel values where $Z_d\left(u,v\right)<0$
Trough tandard deviation	${\sigma }_t={\left[\frac{1}{N_u{N}_v-1}\sum _{u=1}^{N_u}\sum _{v=1}^{N_v}{\left(Z_d\left(u,v\right)-\overline{Z_d}\right)}^2\right]}^{\frac{1}{2}}$	Standard deviation of negative surface pixel values where $Z_d\left(u,v\right)<0$

Rugosity, a popular habitat complexity measure for SfM-derived 3D reconstructions (Leon et al., 2015), was additionally extracted. It is defined as the ratio of the surface area to the area of the surface projected onto a plane producing a value greater than 1 $f_r>1$. A comparably more geometrically complex surface will produce a larger $f_r$,

$$f_r=\frac{A_r}{A_g},$$(7)

where $A_r$ is the true surface area and $A_g$ is the projected surface area.

Depth maps can be converted into 3D meshes using the focal length and optical centres of the device derived earlier during calibration. To convert the depth maps into 3D point clouds, we first converted the row and column positioning of the pixels into 3D coordinates using the following equations:

$$X=\frac{Z\left(u,v\right)}{f_x}\left(u-c_x\right),$$(8)

$$Y=\frac{Z\left(u,v\right)}{f_y}\left(v-c_y\right),$$(9)

where $X$ and $Y$ are the pixels point in 3D Cartesian space, $Z\left(u,v\right)$ is the value representing the distance between the device and an object in the scene (also known as depth) at the location $u$ and $v$ of the depth image, $f_x$ and $f_y$ are the focal lengths, and $c_x$ and $c_y$ are the pixel coordinates of the device's optical centre.

We then compiled the newly calculated $X$, $Y$ and the pixel value (heights) $Z$ for every pixel into a list of vertices. Using Friedman et al. (2012) formulation for rugosity, we converted the vertices into a triangulated irregular network (TIN) mesh using Matlab®'s “tri” function. The TIN was then passed into Friedman et al.'s “trisurfeat”⁷ Matlab® function (Friedman, 2012) to produce a rugosity value.

Finally, the reduced peak height, $S_{\mathit{pk}}$, and reduced valley depth, $S_{\mathit{vk}}$, were calculated from an Abbott-Firestone curve constructed from the detrended depth values which formed the topography as per ISO 25178-2:2012.⁸

3.4 Simulated data

The performance of the detrending model to remove an underlying trend was first tested against simulated data created to mimic the undulating topography of the sediment surfaces observed using the approach in Garcia and Stoll (1984). A simulated Gaussian surface was obtained as a set of random numbers $G$ generated from an uncorrelated normal Gaussian distribution convolved with a Gaussian autocovariance function $C$ into function $D$ in the spectral domain:

$$C=\exp \left[-\frac{\parallel p_i-p_j{\parallel }^2}{2{\theta }^2}\right],$$(10)

$$D=\frac{2L}{\theta \sqrt{\pi }\sqrt{N_s}}{F}^{-1}\left[F\left(G\right)F\left(C\right)\right],$$(11)

where $p_i$ is the pixel of image coordinates $\left(x_i,y_i\right)$ and $p_i^j$ its $j\mathrm{th}$ neighbour pixel of image coordinates $\left(x_j,y_j\right)$, $\theta$ is the correlation length, $N_s$ is the number of points along one side of the surface and $F$ is the Fast Fourier transform. Simulated surfaces of $100$ by $100$ points (see Figure 6) were created with a known mean and root-mean-square (RMS) height and trend. The RMS height for each surface was within a three standard deviation band ($-\sigma :2\sigma$) of the real dataset's mean arithmetical mean roughness (see Table 3). Planar, quadratic, cubic and quartic surface trends were generated and added to each of the simulated surfaces. The detrending algorithm was then run against each of the simulated trended surfaces.

4 RESULTS

4.1 Sensor noise

To quantify the sensor noise in the RGB-D depth image output, the RGB-D device was used to image and optically flat surface (topographical changes of <1 mm) at a distance of 0.8 m within the opaque box. Ten images were captured of the same surface and surface features were extracted from this surface after detrending with a planar model, see Table 4. Sensor depth measurements possessed an uncertainty of $\pm 3\mathrm{mm}$ but were consistent with a standard deviation of <1 mm.

FIGURE 7

Graphs comparing surface measures extracted from field plots using a quadratic detrending model against a plane model. Plots (a)–(h) are the $S_a$, $S_q$, Trough Mean, Trough Standard Deviation, Peak Mean, Peak Standard Deviation, $S_{\mathit{vk}}$ and $S_{\mathit{pk}}$ respectively. Roughness values below 1 mm were removed due to uncertainty.

FIGURE 8

Plane (a) and quadratic (b) detrended depth map differences for an outlier plot. In this outlier plot, points at the top-left and bottom-right edges are higher than centre points. The quadratic detrending model can compensate for this while the plane detrending model cannot, leading to displacement differences of greater than 5 mm_.

FIGURE 9

A selection of field plots with colour (RGB), original depth images (contrast adjusted), and detrended depth image, respectively.

FIGURE 10

Comparison of surface measures extracted from field sites using a plane detrending model. Graphs (a)–(e) show the $S_a$ (arithmetical mean roughness), $S_q$ (root-mean-square deviation), Peak Mean, Trough Mean, Rugosity, $S_{\mathit{pk}}$ (reduced peak height) and $S_{\mathit{vk}}$ (reduced valley depth) respectively. The dark and light grey bars represent control and experimental plots, respectively

TABLE 4. Surface measurements obtained when using the RGB-D device to image an optically flat surface (<1 mm topographical changes)

	$S_a$ (mm)	$S_q$ (mm)	$M_p$ (mm)	$M_t$ (mm)	Rugosity
Mean	1.30	1.60	1.20	1.30	1.10
SD	0.012	0.016	0.009	0.018	0.002

4.2 Simulated data

Table 5 shows the RMSE of the surface attributes between the detrended ($Z_d$) and original non-trended simulated surfaces ($S_d$). For all the simulated surfaces, the detrending algorithm successfully extracted trends with a RMSE below 1 mm. The plane detrending model extracted the original surface with the smallest set of RMSE for all attributes, while the quartic detrending model produced the largest RMSE values.

TABLE 5. The RMSE of the surface measures, in millimetres, between the original non-trended simulated surface $S_d$ and the detrended simulated surface $Z_d$ for different detrending models

RMSE of $\parallel S_d-Z_d\parallel$	Mean	SD	Trough	Trough	Peak	Peak	$S_a$	$S_q$
			Mean	SD	Mean	SD
Plane	0.36	0.05	0.06	0.05	0.05	0.05	0.05	0.05
Quad	0.36	0.13	0.11	0.09	0.11	0.09	0.10	0.13
Cubic	0.36	0.19	0.17	0.13	0.17	0.13	0.16	0.19
Quartic	0.36	0.27	0.23	0.17	0.23	0.18	0.22	0.27

4.3 Field data

Before examining any existing microtopography variations, the underlying surface trends must be removed following the detrending method exemplified in Section 3.3. Considering the results from the simulated surfaces in Section 4.2, detrending was performed on the plots surveyed using the two models with the lowest RMSE, the linear plane and quadratic model and the surface features were extracted. Figure 7 shows the values for different surface measurements for a quadratic model plotted against a linear plane model. Across all measurements, there is a strong correlation between the models with $R^2$ values of $0.73$ or greater. Despite this, larger differences between the fitting models can occur when sediment change is greatest near the edges. Figure 8 is an example of an outlier plot where the differences in surface measurements Peak Mean, $S_a$ and $S_q$ were between 2 and 3 mm. The differences here are caused by peaks near the edges of the imaged areas. In Figure 8b, the quadratic trend fitting is pulled upwards by these edge points, lowering the displacement from the fitted trend. This manifests as a >5 mm difference when comparing the same points fitted with a planar trend as in Figure 8a. To avoid boundary effects with detrending, it may be appropriate to use a plane detrending model.

While interesting topographical formations, such as holes, and elevation changes can be seen in the acquired depth images e.g. in Figure 9, any measurements requires a correction of the inherent image slant (see Figure 9, second column) due to the experimental conditions and ground nature (e.g. muddy flats).

Figure 10 presents the surface features of the field plots separating the control (non-netted) and experimental (netted) plots for a plane detrending model. The measurement medians between the control and experimental plots differed across sites, with control plots at sites PSE, PW, and BB exhibiting higher medians of 2.5, 2.4 and 1.8 mm for trough mean measurements and 2.4, 2.1 and 1.8 mm for peak mean measurements respectively when compared to experimental plots. PSE, PW and BB control plots also exhibited larger $S_a$ and $S_q$ medians of 2.5, 2.3 and 1.8 mm for $S_a$ measurements and 3.3, 2.9 and 2.4 mm for $S_q$ measurements compared to their experimental counterparts. Conversely, experimental sites at PE and LBS exhibit higher medians for trough mean measurements with 1.6 and 2.7 mm and 1.6 and 2.4 mm for peak mean measurements, respectively.

Over the seven sites, control plots typically exhibited more variability within a similar range of values, with an $S_{\mathit{vk}}+S_k+S_{\mathit{pk}}$ (where $S_k$ is the mean core roughness) mean at 25 mm and standard deviation at 7.2 mm, compared to the experimental plots, with a similar mean but lower standard deviation at 7.15 mm, following expectations that control plots allowed large animals freedom of movement. Experimental plots at sites PN and BB however, presented larger variance than their control counterparts which may be related to the high number of cockle remains related to the site being a possible feeding ground. This was further reinforced by the variations of $S_{\mathit{pk}}$ and $S_{\mathit{vk}}$ values (see plots (f) and (g) in Figure 10) between control and experiment plots.

5 DISCUSSION

In this investigation, we introduced an affordable and portable microtopography solution using an RGB-D device and polynomial detrending to extract surface features at millimetric scale. In a use case for marine sediment, the solution proved useful in highlighting the effects on sediment microtopography by restricting large animals in sediment mudflats. Surface roughness characteristics such as $S_a$ and $S_q$ deviation extracted by the solution were key in highlighting differences between control and experimental plot surfaces and correlated strongly with other rugosity measures. This was in line with larger animals affecting control plots’ topography (larger peaks and valleys) while smaller animals dominated in experimental plots thus affecting the topography accordingly (smaller peaks and valley and larger rugosity values). Fine scale microtopography influences the hydro- and sediment-dynamics, and the number and types of species found in the habitat (Le Minor et al., 2021).

An affordable tool to measure microtopography provides the spatial representation of surface heterogeneity within the test area, is capable of operating across sandy to muddy habitats and therefore provides valuable information about sediment health.

While the microtopographic mapping of sandflats is as yet poorly studied, it offers the potential for new insights into biodiversity, ecosystem function and sediment stability. On flats dominated by biogenic microtopographic features, many different organisms can contribute to the ‘map’ and this makes physical validation with the introduction of artificial structures difficult to do and interpret. The newly developed technique was validated through the use of modelled surfaces, a common modelling approach. Moreover, the application of this technique is not simply to precisely map microtopographic features on the mm–m scale, rather the utility in using statistics derived from these maps as a surrogate variable for biodiversity and ecosystem function.

5.1 Fitting and roughness

5.1.1 Simulated results

Throughout the plots ($n=82$), we observed the surface trends to be relatively flat, and any large deviations in the surface was safely mitigated using a polynomial fitting.

While randomised Gaussian surfaces are a common approach for simulating surfaces when testing new methods (Pawlus et al., 2020), intertidal mudflats may be occupied by features more irregularly shaped than a Gaussian mound. The simulated surfaces proved to be useful in determining how sensitive the linear least squares fitting of polynomials were to noise (i.e. animal features on the surface). The results showed that a plane detrending model was the least sensitive to surface variation when compared to the quartic polynomial model. This was expected due to the increase in the degrees of freedom that higher order polynomials possess. Polynomial functions are additionally oscillatory in nature and this can contribute to poorer fitting outcomes for rougher surfaces.

5.1.2 Field data

Given the underlying trends in the field data, both linear and quadratic models showed strong correlation in the extracted surface measures and similar planar trends. The quadratic detrending model however was sensitive to large elevations on the edges of the imaged area, resulting in larger differences between the two detrending models. This can be seen in an outlier plot in Figure 8. The plane detrending model is likely more appropriate in this instance where the study area sediment trends were mostly planar.

From Figure 10, comparing the roughness features—$S_a$ and $S_q$—against rugosity, five of the six sites were comparable in that plots with large animals excluded (netted) plots were less rough compared to control (non-netted) plots. The outlying site, site PS, had higher rugosity values but lower $S_a$ and $S_q$ values between their control and experimental plots. When examined, the control plots had larger data artefacts that could not be filled in completely. When constructing the TIN surface, larger triangles stand to contribute more to rugosity measurements as a rougher surface. The use of Friedman's implementation of rugosity is probably better suited for SfM meshes rather than patchy imperfect data. $S_a$ and $S_q$ examine different aspects of roughness compared to rugosity, seeking to quantify the deviations from the mean elevation of the underlying trend rather than a normalised roughness value. $S_a$ and $S_q$ are more complementary to rugosity in capturing surface microtopography than an alternative measure. Additionally the mean crest and mean trough values (see Table 3) add further detail to the surface by characterising the features found in the sediment with the potential to attribute the activity to certain animals in the area, similar to the work done by Brunier et al. (2020) where species activity is characterised by the surface features extracted from a SfM mesh.

5.2 RGB-D device considerations and extensions

The device successfully imaged fine scale details across the seven sites of different sediment qualities, from sandy to muddy. This shows promise in applicability to other sediment types. However, some caveats need to be made when using RGB-D devices in the field. Interference from strong sunlight with the device's projected pattern is an important aspect to be controlled as it can create large areas without measurement. While this investigation successfully sheltered the device from direct sunlight by making use of opaque containers to house the device, this is not practical for all applications. For investigations that require a greater field of view, timing acquisition for low ambient light during dawn, dusk, overcast, or in shaded regions will minimise the interference. Consideration must be made when dealing with pools of water, which are typical features of intertidal areas, and can create holes in data (Khoshelham & Elberink, 2012). Klopfer et al. (2017) showed that the RGB-D device's projected pattern (a Kinect v1⁹ in their investigation) was able to penetrate water up to 30 cm depth in outdoor conditions. In this investigation, plots were mostly absent of features that accumulated large amounts of water at low tide and this was not an issue.

Direct extensions for use with an RGB-D device is to increase device coverage as well as extending capabilities to more complex geomorphological intertidal surfaces. Imaging in this investigation was limited to a local plot scale of 0.3 m². Understanding the impact of macrofauna and microfauna on sediment topography at larger scales would require greater coverage. The device is able to operate with low power computers such as the Raspberry Pi,¹⁰ which operates at around 5 Watts, and attaching it to a mobile platform such as a UAV (Tripicchio et al., 2015) or vehicle would increase the potential imaging area. However, work by Livingston et al. (2012) and Halmetschlager-Funek et al. (2018) shows that distortion in RGB-D devices increase at distances greater than 1.5 m limiting the altitude at which the UAVs may operate and the maximum FOV.

The device was arranged with a nadir orientation to the surface; features (shells and mounds) are small enough that any occlusion caused by the geometry would be minimal relative to the overall change of the surface. For surfaces with more complex geometry, a nadir depth image may not be sufficient to accurately characterise the surface. Approaches similar to Kinect Fusion (Newcombe et al., 2011) or SfM can be performed with the device allowing for more complex 3D surface reconstructions. For example, Kamal et al. (2014) used a similar styled RGB-D sensor in a SfM approach to obtain 3D geometrical complexity of mangrove structures. Underwater applications may use a similar approach but would require a more powerful laser for the structured light pattern. Du Preez and Tunnicliffe (2012) showed the usefulness of similar technologies to capture underwater surface microtopography with an underwater vehicle operating green lasers at a wavelength of 532 nm.

6 CONCLUSIONS

Soft sediment systems and other equally fragile systems lack accessible and cost-effective tools for quantifying habitat structure and complexity, which have potential ecological significance. Alternative non-invasive imaging devices such as terrestrial LIDAR and monocular SfM systems can be inaccessible due to hardware costs and require further processing to extract 3D geometry. The presented approach provides a low cost and practical framework for measuring fragile, intertidal and soft sediment topography at a millimetre-scale suitable for capturing microfauna sediment interaction and extracting sediment surface morphological characteristics. Output depth maps and 3D models/point clouds of the sediment surface allow ecologists to add geometric dimensions to their models or investigate other avenues of soft sediment topography, for example implications in near surface hydrological effects, sedimentation and microfauna spatial distribution. RGB-D real-time imaging is promising as a rapid scanning tool for collecting field data in intertidal regions. We are currently validating the application of this new method through extensive field data gathering to help resolve ecosystem patterns and processes.

AUTHORS' CONTRIBUTIONS

M.A., T.G., W.v.d.M., S.F.T. and P.D. conceived the ideas and designed methodology; P.D., M.A., S.S. and J.R.H. collected the data; M.A. and P.D. analysed the data; M.A., S.F.T., and P.D. led the writing of the manuscript. All authors contributed critically to the drafts and gave final approval for publication.

ACKNOWLEDGEMENTS

This project has been supported by the Oceans of Change initiative from the University of Auckland, Institute of Marine Science. The authors' team have no conflict of interest to declare. The authors thank the reviewers for their valuable comments, which improved the quality of the paper. Open access publishing facilitated by The University of Auckland, as part of the Wiley - The University of Auckland agreement via the Council of Australian University Librarians. [Correction added on 07 July, 2022, after first online publication: CAUL funding statement has been added.]

Endnotes

1 https://www.microptik.eu/product/lt-profile-measurement#concept

² https://web.archive.org/web/20130903010223/http://www.microsoft.com/en-us/kinectforwindows/

³ https://web.archive.org/web/20161108005652/http://www.asus.com/us/3D-Sensor/Xtion_PRO_LIVE/overview/

⁴ https://www.intelrealsense.com/depth-camera-d455

⁵ https://structure.io/openni

⁶ https://mathworks.com/help/images/ref/imfill.html

⁷ https://au.mathworks.com/matlabcentral/fileexchange/38632-trisurf-multi-scale-rugosity-slope-and-aspect

⁸ https://www.iso.org/obp/ui/#iso:std:iso:25178:-2:ed-1:v1:en

⁹ https://web.archive.org/web/20120821173126/http://www.microsoft.com/en-us/kinectforwindows/discover/features.aspx

¹⁰ https://www.raspberrypi.org/products/raspberry-pi-3-model-b/

Open Research

DATA AVAILABILITY STATEMENT

Depth images captures as part of this study can be found at Figshare https://doi.org/10.17608/k6.auckland.19680933 (Azhar et al., 2022).