EasyDCP: An affordable, high-throughput tool to measure plant phenotypic traits in 3D

Abstract

en

1. High-throughput 3D phenotyping is a rapidly emerging field that has widespread application for measurement of individual plants. Despite this, high-throughput plant phenotyping is rarely used in ecological studies due to financial and logistical limitations.
2. We introduce EasyDCP, a Python package for 3D phenotyping, which uses photogrammetry to automatically reconstruct 3D point clouds of individuals within populations of container plants and output phenotypic trait data. Here we give instructions for the imaging setup and the required hardware, which is minimal and do-it-yourself, and introduce the functionality and workflow of EasyDCP.
3. We compared the performance of EasyDCP against a high-end commercial laser scanner for the acquisition of plant height and projected leaf area. Both tools had strong correlations with ground truth measurement, and plant height measurements were more accurate using EasyDCP (plant height: EasyDCP r² = 0.96, Laser r² = 0.86; projected leaf area: EasyDCP r² = 0.96, Laser r² = 0.96).
4. EasyDCP is an open-source software tool to measure phenotypic traits of container plants with high-throughput and low labour and financial costs.

概要

ja

1. ハイスループットな3次元の表現型測定技術は急速に発展しており、植物の測定に広く応用されている。にもかかわらず、経済的なコストと手間が大きいため、ハイスループットの植物表現型解析が生態学研究に用いられることはほとんどなかった。
2. われわれは、画像解析技術を用いて大量の鉢植え植物個体の3次元点群を自動的に再構成し、形質データを出力するPythonパッケージ (EasyDCP) を開発・公開した。この論文では、撮影のためのセットアップと必要なハードウェアを最小限に抑えてDIYで行う方法を説明し、EasyDCPを行う一連の作業の流れと、有用性を紹介する。
3. 本手法の性能を検証するために、手作業での測定結果、市販の高価格なレーザースキャナーでの測定結果と比較した。EasyDCPで推定された投影葉面積は、手作業での測定値と非常に高い相関があり (EasyDCP: r² = 0.96, laser scanner: r² = 0.96) 、植物の草丈に関してはレーザースキャナーよりも高い精度で形質を測定できた (Easy DCP: r² = 0.96, laser scanner: r² = 0.86)。
4. EasyDCPは, 鉢植え植物の表現型形質を, 高い処理能力と低い労力・金銭的コストで測定できるオープンソースのツールである。

摘要

zh

1. 高通量植物三维表型是一个迅速崛起的领域, 在测量单株植物方面有着广泛的应用。尽管如此, 受到购买和使用成本的限制, 高通量植物三维表型很少用于生态研究领域。
2. 我们开发了一个用于三维表型的Python工具EasyDCP, 其使用摄影测量技术自动重建盆栽实验植物的三维点云并输出表型参数。文中也指导了成像设置和所需硬件, 非常轻量且可DIY, 并且介绍了EasyDCP的功能和工作流程。
3. 我们比较了EasyDCP与基于三维激光扫描的工业级植物表型设备在获取植物高度和植被覆盖度方面的表现。结果表明, 对于植被覆盖度, EasyDCP和三维激光扫描设备的测量和实测值都有很强的统计相关性 (EasyDCP: r² = 0.96, laser scanner: r² = 0.96) 。而对于高度, EasyDCP展示出更强的统计相关性 (Easy DCP: r² = 0.96, laser scanner: r² = 0.86)。
4. EasyDCP是一个测量盆栽实验植物三维表型参数的高通量、低劳动力和低成本的开源工具。

1 INTRODUCTION

Plant ecology and evolution emerge from interactions between the phenome, genome and environment, requiring scholars to develop a deep knowledge of all three components. However, there currently exists a mismatch between phenomics and genomics, as modern genotyping techniques enable low-cost generation of large amounts of data, whereas phenotyping throughput is still limited (Cobb et al., 2013; Furbank & Tester, 2011; Houle et al., 2010; Minervini et al., 2015; Pieruschka & Schurr, 2019). High-throughput phenotyping aims to alleviate this bottleneck by introducing tools capable of quickly generating large morphological datasets (Araus & Cairns, 2014; Houle et al., 2010; Pieruschka & Schurr, 2019; Tardieu et al., 2017; Walter et al., 2015).

Measurement of phenotypic traits can be conducted in multiple dimensions, including two-dimensional (e.g. shoot growth (Li et al., 2020), projected leaf area (Guo et al., 2017), herbivory (Machado et al., 2016)) using digital image processing, and three-dimensional (e.g. morphological structure: Paproki et al., 2012; shoot biomass: Golzarian et al., 2011; total leaf area: Xiao et al., 2020; plant posture: Wu et al., 2019) using mainly laser scanning (Kjaer & Ottosen, 2015; Paulus et al., 2014) and photogrammetry (Agapito et al., 2015; Duan et al., 2016) techniques.

3D phenotyping is a promising new toolkit to study plant ecology and evolutionary biology. First, 3D phenotyping at the individual plant level will enable detailed quantification of morphological trait differences among individuals, populations and related species that could not be measured manually and can therefore advance our understanding of local adaptation and plasticity of these traits. Second, non-destructive 3D phenotyping measurement has the advantage of being able to measure individual morphological changes over time. Third, 3D morphological data can be useful for modelling and simulation in genetics studies (Chen et al., 2019).

Despite its advantages, 3D phenotyping has not been used extensively in ecology studies, primarily due to the large population sizes in ecology research requiring high-throughput measurement. Existing high-throughput 3D phenotyping systems are costly, large-scale and require special equipment or dedicated facilities, such as the PHENOARCH (Cabrera Bosquet et al., 2015) and The Plant Accelerator (Honsdorf et al., 2014). Many of these high-throughput phenotyping platforms employ an imaging chamber, limiting the maximum plant size suitable for measurement (Czedik-Eysenberg et al., 2018; Rahaman et al., 2015, table 2). Some lower-cost phenotyping tools are in development but they have either limited throughput capacity (Tovar et al., 2018), significant hardware requirements (An et al., 2016; Paulus et al., 2014) or only support 2D traits (Gehan & Kellogg, 2017).

The purpose of this work was to develop and evaluate a tool to calculate phenotypic traits from a set of captured images of container plants at different size scales with minimal labour time and financial cost. Additionally, support for multiple image sets (e.g. time-series data, large populations) was required so that datasets could be processed with high throughput.

Here we introduce EasyDCP (Easy Dense Cloud Phenotyping), a software tool which operates a photogrammetry pipeline to extract 3D phenotypic traits from container plants using a regular digital camera and a combination of commercially available and open-source software. EasyDCP has the following advantages: (a) populations of container plants can be measured outdoors or in a controlled environment with little to no relocation, (b) large populations can be measured quickly and (c) low financial and labour costs. Additionally, although the scope of this paper is limited to the measurement of container plants, EasyDCP may be used to measure any group of appropriately set up objects. We provide detailed instructions to operate EasyDCP from image acquisition to data output. We evaluate the performance and accuracy of EasyDCP, compare with a commercial plant phenotyping tool and provide a case study demonstrating the high-throughput capability of EasyDCP.

2 EasyDCP OVERVIEW

The EasyDCP workflow (Figure 1) consists of an image acquisition component (Section 2.1) and two data processing components: EasyDCP_Creation (Section 2.2), which creates a 3D point cloud from 2D images; and EasyDCP_Analysis (Section 2.3), which analyses that point cloud and performs trait calculation. EasyDCP source code and documentation are available on GitHub (https://github.com/UTokyo-FieldPhenomics-Lab/EasyDCP).

FIGURE 1

The overall workflow of EasyDCP. (a) Each group of plants must be imaged, and images and configuration files must be prepared for EasyDCP. (b) EasyDCP_Creation performs sequential processing on all image sets, creating a 3D point cloud for each group of plants. (c) EasyDCP_Analysis performs classification, segmentation and calculation on each 3D point cloud to measure plant traits including plant height and projected leaf area

2.1 Image acquisition

Plants must be imaged prior to EasyDCP measurement, and the image acquisition area can be set up according to the user's needs (Figure 2a,b). The image acquisition area should have as little inclination as possible. One printed target page (.pdf provided with the software) must be placed in a corner of the image acquisition area, to set the orientation and scale. A second target page may be installed in the opposite corner, to define the region of interest within EasyDCP and reduce processing time. Additional target pages may be installed at intervals throughout the image acquisition area if its length exceeds 3 m. By default, EasyDCP expects the first target page to be located at the bottom-left corner of the scene, and the second target page at the top-right corner of the scene (Figure 2). Plants must be arranged in a single row (Figure 2a) or a staggered double row (Figure 2b). Plant ID (i.e. measurement order) will increase starting from the side containing the first target page. A gap of at least 10 cm between plant canopy perimeters is required. Using the same number of plants for all measurement groups and maintaining the same plant spacing is recommended. Covering the ground with a non-reflective cloth (Figure 2c) is recommended especially if weeds are present in the image acquisition area.

FIGURE 2

Image acquisition. (a) and (b) show the basic image acquisition protocol for EasyDCP. Target pages are placed in the corners of the image acquisition area. Plants can be arranged in either (a) a single row or (b) a staggered double row. Numbers next to plants indicate plant ID, which always increases from the first target page to the second. (c) shows oblique view of a 3D point cloud of a group of three container plants (Amaranthuspatulus) in a 1 m × 2 m image acquisition area. Four parallel rows of images indicated by blue rectangles were captured, totalling 92 images and averaging 11.5 images per meter for each angle

Images may be captured using any digital camera with recommended resolution of 8 megapixels or higher. Imaging angle, distance, capture interval (i.e. distance between capture locations), resolution and sharpness all affect the quality of the resulting 3D point cloud. The 3D point cloud is reconstructed using the structure from motion photogrammetry technique, which relies on overlapping area between neighbouring images (Ullman, 1979). To avoid biases and to maximize overall image sharpness, we suggest designing an imaging protocol that allows 90% of overlap between individual images (Andújar et al., 2018; Kawamura et al., 2020; Madec et al., 2017) and capturing as close to nadir (directly downward) as possible. For example, 90% image overlap will be achieved if images are captured at a distance of 1 m from the plants with a capture interval of 10–15 images per meter and a horizontal field of view of 70°. Capturing at least two rows of images will create side overlap, reducing the likelihood of measurement bias due to spherical distortion (An et al., 2016, 2017). Additional rows of images can be captured to increase the image overlap and measurement accuracy (e.g. 5–15° off-nadir; Figure 2c). Higher image resolution will ensure accurate measurement at longer imaging distances.

2.2 Point cloud creation

EasyDCP_Creation is an automatic pipeline that creates a 3D point cloud from a set of 2D images (Figure 1b). This pipeline controls the commercial photogrammetry software Metashape Professional 1.6.6 (Agisoft LLC) via its Python API (Agisoft, 2020b; Van Rossum & Drake, 2009) to perform 3D reconstruction via structure-from-motion (Hartley & Zisserman, 2003; Paulus, 2019; Seitz et al., 2006; Ullman, 1979). EasyDCP_Creation supports sequential processing for multiple image sets (i.e. groups of plants).

First, images are imported into a Metashape project. The image quality (IQ) is estimated by Metashape's built-in function and all images with an IQ value below the iq_threshold parameter are disabled. Next, coded targets are detected in the images and Metashape's Align Cameras function is executed to create a tie point cloud containing key matching points. The locations and known distances between the coded targets are used to scale and orient the point cloud and define the edges of the image acquisition area. Finally, a dense point cloud is created (Figure 1b) and exported to .ply format. Additionally, a Metashape report is exported to .pdf format and the Metashape project is saved to .psx format. The user may check an output .ply file with CloudCompare (https://www.cloudcompare.org) or check a .pdf report to view a top-down image of the point cloud and to see if any images were excluded from the process.

2.3 Point cloud analysis

EasyDCP_Analysis (Figure 1c) includes three steps, which aim to distinguish plants from the background (classification); individualize plants (segmentation); and measure 2D and 3D traits (calculation). EasyDCP_Analysis is implemented in the open-source Python (Van Rossum & Drake, 2009) programming language and based on several popular packages, including Open3D (Zhou et al., 2018), SciPy (Virtanen et al., 2020), scikit-learn (Pedregosa et al., 2011), scikit-image (Van Der Walt et al., 2014), Matplotlib (Hunter, 2007), NumPy (Van Der Walt et al., 2011) and Pandas (McKinney, 2010). Sequential processing of multiple point clouds previously generated by EasyDCP_Creation is supported, providing high-throughput phenotyping functionality.

The classification step identifies each point of the input point cloud as vegetation (Figure 1c.1) or background using the classification and regression tree algorithm (Breiman et al., 1984; Guo et al., 2013) in Scikit-learn. The user must provide training data for the classification algorithm in the form of .png image files containing samples of vegetation and background. The classification result strongly depends on the quality of the training data, and example training data are provided with the software for reference. Noise points are removed from the resulting vegetation and background point clouds using the remove statistical outlier and remove radius outlier functions in Open3D.

The segmentation step separates individual plants within the vegetation point cloud (Figure 1c.2) using the voxelization function in Open3D and the clustering algorithm DBSCAN (Ester et al., 1996) in Scikit-learn. In some cases, groups of noise points can be incorrectly segmented and considered as plants. To correct this, the user may enable the K-means clustering algorithm (Arthur & Vassilvitskii, 2007) in Scikit-learn to remove non-plant segments based on point count. Segmentation works best when the number of measured plants remains constant so that the user may inform EasyDCP_Analysis of the expected number of plant segments.

The calculation step performs trait measurement on each individual plant point cloud (Figure 1c.3). Traits can be calculated in one, two and three dimensions, and the scope of this paper is limited to plant height and projected leaf area (PLA). Plant height is calculated by finding the mean distance from the ground of all plant points above the percentile parameter, which is user-adjustable and set to 98 by default. The ground height parameter can be automatically detected or manually specified. The container height parameter is used to offset ground height and must be entered by the user if nonzero. PLA is calculated by finding the area occupied by the projection of the voxelized point cloud onto the ground plane. The voxel size parameter may be adjusted to improve PLA measurement accuracy. Other traits supported by EasyDCP include the lengths of the long and short axis by ellipse regression and convex hull volume (Figure 1c.3).

3 PERFORMANCE TEST

We tested EasyDCP on 24 container plants (Amaranthus patulus (n = 6), Commelina communis (n = 6), Eleusine indica (n = 6) and Galinsoga quadriradiata (n = 6)) in a greenhouse at Institute for Sustainable Agro-ecosystem Services, the University of Tokyo, Tokyo, Japan. We selected two phenotypic traits for comparison: plant height and projected leaf area (PLA) due to their wide interest among biologists (Andújar et al., 2018; Christian Rose et al., 2015; Fahlgren et al., 2015; Kjaer & Ottosen, 2015; Machado et al., 2016; McCormick et al., 2016; Paulus et al., 2014; Tovar et al., 2018; Xiao et al., 2020; Zhou et al., 2019) and being representative of all three dimensions. To evaluate EasyDCP performance, one investigator measured the same traits both manually and with a commercial laser scanner (PlantEye F500 DualScan; Phenospex LLC). We captured images for EasyDCP and concurrently scanned the plants on the PlantEye platform in groups of three due to the size limitations of the PlantEye. Additional details on the performance test methodology are provided in Supporting Information Section 1.

We evaluated the scaling accuracy of the point clouds generated by EasyDCP and the PlantEye using three precision scale bars (Cultural Heritage Institute, San Francisco, USA) installed at different angles (0, 6 and 12°) throughout the image acquisition area (see Figures 1b and 2a,b). The precision scale bar lengths were not used as input for the scaling step of EasyDCP_Creation. We measured the scale bar lengths within the point clouds and calculated the differences between the actual and measured values. We used a simple linear regression model to compare the measured trait data from both EasyDCP and the PlantEye to ground truth, using the coefficient of determination (r²) to evaluate the correlation between the independent (ground truth) and dependent (measured) variables.

Additionally, to evaluate high-throughput efficacy, we applied EasyDCP on a population of 217 Digitaria ciliaris individuals (Supporting Information Section 2) from an ecology study (Fukano et al., 2020).

4 RESULTS AND DISCUSSION

4.1 Performance test

All measured scale bar lengths (n = 24) were found to be within 0.7% of the actual values (Figure 3a), indicating that the point cloud generated by EasyDCP_Creation was scaled with acceptable accuracy.

FIGURE 3

(a) The differences between the actual and measured scale bar lengths using EasyDCP and the PlantEye laser scanner. (b) Calculation of plant height and projected leaf area plotted on the y-axis against ground truth. The black line represents a 1-to-1 measurement with ground truth

When comparing the measurements of plant height and projected leaf area (PLA) with the ground truth data (n = 24), the correlation coefficient (r²) equalled or exceeded that of a high-end commercial laser scanner for phenotyping (Figure 3b; plant height: EasyDCP r² = 0.96, PlantEye r² = 0.87; PLA: EasyDCP r² = 0.96; PlantEye r² = 0.96).

4.2 Case study: High-throughput phenotyping application for ecology

Our high-throughput test was very efficient. Images were acquired in under 3 hr and the EasyDCP pipeline completed processing in approximately 3 hr on our high-end desktop PC. We measured the same plants using the PlantEye laser scanner the following day, completing the process in under 6 hr. We did not directly compare the results from EasyDCP and the PlantEye because the plants were measured on different days. The successful measurement by EasyDCP in a similar time period as the PlantEye demonstrates that EasyDCP is a viable alternative to a commercial laser scanner for high-throughput measurement.

4.3 Using the system

We created EasyDCP to be suitable for a wide variety of measurement conditions. The user may decide the dimensions of the image acquisition area and the number of plants per measurement group. If plants and targets are appropriately set up, plants may be imaged in place (e.g. on cultivation tables), significantly reducing imaging time. Advanced photogrammetry techniques, such as using a multi-camera system (as in An et al., 2016, Figure 3) or extracting frames from video, may further expedite imaging. EasyDCP has several parameters described in the documentation that can be modified to ensure compatibility with the user's image acquisition protocol. EasyDCP can run on any Windows, Mac or Linux computer that meets Metashape's minimum requirements (notably 16GB RAM) (Agisoft, 2020a). A graphics processing unit (GPU) is recommended but not required.

One challenge is that EasyDCP tends to overestimate PLA, likely due to the voxelization technique and the unpredictable density of point clouds produced by photogrammetry. The bias may be reduced by adjusting the voxel size parameter or by calibrating with plants of known PLA.

4.4 Future priorities

We intend to optimize the image acquisition technique to minimize the required time and materials without compromising the measurement accuracy. Extraction of frames from video files is currently in development. We may add a color calibration function to EasyDCP_Creation to improve the 3D point cloud quality (Berry et al., 2018). We acknowledge that there are relevant licensing costs associated with EasyDCP_Creation and hope to replace Metashape with a free and open-source photogrammetry tool. Lastly, we plan to continue developing EasyDCP_Analysis to improve its measurement accuracy and accuracy, and support more traits including total leaf area and leaf count.

5 CONCLUSION

We presented EasyDCP, a software tool to extract plant phenotypic traits in 3D from images of container plants using a digital camera. We have shown EasyDCP to be a viable low-cost tool for phenotyping populations of plants, measuring traits with comparable or slightly greater accuracy than a commercial laser scanner. Computer hardware and commercial 3D reconstruction software are financial costs, but the total cost is affordable compared to other 3D plant measurement tools and high-throughput phenotyping solutions. EasyDCP has the advantages of greater ease of use and fewer manual steps than other 3D-based approaches, enabling high-throughput operation with minimal training.

ACKNOWLEDGEMENTS

This study was partially funded by the JST CREST Program (grant numbers JPMJCR1512, JPMJCR1601 and JPMJCR1602), SICORP Program JPMJSC16H2, and JSPS KAKENHI grant number 18K14464 and 18KT0087. The authors declare that we have no conflicts of interest.

AUTHORS' CONTRIBUTIONS

A.F. and W.G. conceived the ideas and designed the methodology; A.F. and H.W. collected and analysed the data; A.F. and H.W. led the writing of the manuscript with the input of all authors; Y.F. provided all plant materials; W.G., S.N. and Y.K. supervised the whole work. All authors contributed critically to the drafts and gave final approval for publication.

Open Research

PEER REVIEW

The peer review history for this article is available at https://publons.com/publon/10.1111/2041-210X.13645.

DATA AVAILABILITY STATEMENT

Data from the performance test (source images, point clouds, trait data and R files), EasyDCP source code, example scripts and detailed documentation are archived using Zenodo https://doi.org/10.5281/zenodo.4756537 (Feldman et al., 2021).