Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants

Dutagaci, Helin

doi:10.55730/1300-0632.4018

Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants

Helin DUTAĞACI (Osmangazi Üniversitesi, Mühendislik Mimarlık Fakültesi, Elektrik-Elektronik Mühendisliği Bölümü, Eskisehir, Türkiye)

Turkish Journal of Electrical Engineering and Computer Sciences

7 0

Yıl: 2023 Cilt: 31 Sayı: 5 Sayfa Aralığı: 792 - 813 Metin Dili: İngilizce DOI: 10.55730/1300-0632.4018 İndeks Tarihi: 22-11-2023

Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants

Öz:

In this work, the use of t-SNE is proposed to embed 3D point clouds of plants into 2D space for plant characterization. It is demonstrated that t-SNE operates as a practical tool to flatten and visualize a complete 3D plant model in 2D space. The perplexity parameter of t-SNE allows 2D rendering of plant structures at various organizational levels. Aside from the promise of serving as a visualization tool for plant scientists, t-SNE also provides a gateway for processing 3D point clouds of plants using their embedded counterparts in 2D. In this paper, simple methods were proposed to perform semantic segmentation and instance segmentation via grouping the embedded 2D points. The evaluation of these methods on a public 3D plant data set conveys the potential of t-SNE for enabling 2D implementation of various steps involved in automatic 3D phenotyping pipelines.

Anahtar Kelime: Point clouds plants visualization superpoint segmentation t-distributed stochastic neighbor embedding

Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık

[1] Yang G, Liu J, Zhao C, Li Z, Huang Y et al. Unmanned aerial vehicle remote sensing for field-based crop phenotyping: Current status and perspectives. Frontiers in Plant Science 2017; 8:1111. https://doi.org/10.3389/fpls.2017.01111
[2] Garbez M, Symoneaux R, Belin E, Caraglio Y, Chéné Y et al. Ornamental plants architectural characteristics in relation to visual sensory attributes: A new approach on the rose bush for objective evaluation of the visual quality. European Journal of Horticultural Science 2018; 83 (3): 187–201. https://doi.org/10.17660/eJHS.2018/83.3.8
[3] Roldán JJ, del Cerro J, Garzón-Ramos D, Garcia-Aunon P, Garzón M et al. Robots in agriculture: State of art and practical experiences. In: Neves AJR (editor). Service Robots. InTech, 2018, pp. 67–90. https://doi.org/10.5772/intechopen.69874
[4] Minervini M, Scharr H, Tsaftaris SA. Image analysis: The new bottleneck in plant phenotyping [applications corner]. IEEE Signal Prrocessing Magazine 2015; 32 (4): 126–131. https://doi.org/10.1109/MSP.2015.2405111
[5] James KMF, Sargent DJ, Whitehouse A, Cielniak G. High-throughput phenotyping for breeding targets— Current status and future directions of strawberry trait automation. Plants People Planet 2022. https://doi.org/10.1002/ppp3.10275
[6] Paulus S. Measuring crops in 3D: Using geometry for plant phenotyping. Plant Methods 2019; 15 (1): 1–13. https://doi.org/10.1186/s13007-019-0490-0
[7] Okura F. 3D modeling and reconstruction of plants and trees: A cross-cutting review across computer graphics vision and plant phenotyping. Breeding Science 2022; 72 (1): 31–47. https://doi.org/10.1270/jsbbs.21074
[8] Schunck D, Magistri F, Rosu RA, Cornelißen A, Chebrolu N et al. Pheno4D: A spatio-temporal dataset of maize and tomato plant point clouds for phenotyping and advanced plant analysis. Plos One 2021; 16 (8): e0256340. https://doi.org/10.1371/journal.pone.0256340
[9] Mirande K, Godin C, Tisserand M, Charlaix J, Besnard F et al. A graph-based approach for simul- taneous semantic and instance segmentation of plant 3D point clouds. Frontiers in Plant Science 2022. https://doi.org/10.3389/fpls.2022.1012669
[10] Van der Maaten L, Hinton G. Visualizing data using t-SNE. Journal of Machine Learning Research 2008; 9 (11).
[11] Abdelmoula WM, Balluff B, Englert S, Dijkstra J, Reinders MJ et al. Data-driven identification of prognostic tumor subpopulations using spatially mapped t-SNE of mass spectrometry imaging data. Proceedings of the National Academy of Sciences 2016; 113 (43):12244-9. https://doi.org/10.1073/pnas.1510227113
[12] Devassy BM, George S. Dimensionality reduction and visualisation of hyperspectral ink data using t-SNE. Forensic science international 2020; 311:110194. https://doi.org/10.1016/j.forsciint.2020.110194
[13] Song W, Wang L, Liu P, Choo KK. Improved t-SNE based manifold dimensional reduction for remote sensing data processing. Multimedia Tools and Applications 2019;78:4311-26. https://doi.org/10.1007/s11042-018-5715-0
[14] Elnashef B, Filin S, Lati RN. Tensor-based classification and segmentation of three-dimensional point clouds for organ-level plant phenotyping and growth analysis. Computers and Electronics in Agriculture 2019; 156:51–61. https://doi.org/10.1016/j.compag.2018.10.036
[15] Liu X, Hu C, Li P. Automatic segmentation of overlapped poplar seedling leaves combining mask R-CNN and DB- SCAN. Computers and Electronics in Agriculture 2020; 178:105753. https://doi.org/10.1016/j.compag.2020.105753
[16] Sun S, Li C, Chee PW, Paterson AH, Jiang Y et al. Three-dimensional photogrammetric mapping of cotton bolls in situ based on point cloud segmentation and clustering. ISPRS Journal of Photogrammetry and Remote Sensing 2020; 160:195–207. https://doi.org/10.1016/j.isprsjprs.2019.12.011
[17] Golbach F, Kootstra G, Damjanovic S, Otten G, van de Zedde R. Validation of plant part measurements using a 3D reconstruction method suitable for high-throughput seedling phenotyping. Machine Vision and Applications 2016; 27:663–680. https://doi.org/10.1007/s00138-015-0727-5
[18] Liu J, Liu Y, Doonan J. Point cloud based iterative segmentation technique for 3D plant phenotyping. In: IEEE 2018 International Conference on Information and Automation (ICIA); 2018. pp. 1072–1077. IEEE 2018. https://doi.org/10.1109/ICInfA.2018.8812589
[19] Hétroy-Wheeler F, Casella E, Boltcheva D. Segmentation of tree seedling point clouds into elementary units. International Journal of Remote Sensing 2016; 37 (13): 2881–2907. https://doi.org/10.1080/01431161.2016.1190988
[20] Li D, Cao Y, Tang X-s, Yan S, Cai X. Leaf segmentation on dense plant point clouds with facet region growing. Sensors 2018; 18 (11): 3625. https://doi.org/10.3390/s18113625
[21] Duan T, Chapman S, Holland E, Rebetzke G, Guo Y et al. Dynamic quantification of canopy structure to characterize early plant vigour in wheat genotypes. Journal of Experimental Botany 2016; 67 (15): 4523–4534. https://doi.org/10.1093/jxb/erw227
[22] Wang D. Unsupervised semantic and instance segmentation of forest point clouds. ISPRS Journal of Photogram- metry and Remote Sensing 2020; 165:86–97. https://doi.org/10.1016/j.isprsjprs.2020.04.020
[23] Liu Y, Guo J, Benes B, Deussen O, Zhang X et al. TreePartNet: Neural decomposition of point clouds for 3D tree reconstruction. ACM Transactions on Graphics 2021; 40 (6). https://doi.org/10.1145/3478513.3480486
[24] Dey D, Mummert L, Sukthankar R. Classification of plant structures from uncalibrated image se- quences. In: IEEE 2012 Workshop on the Applications of Computer Vision (WACV); 2012. pp. 329–336. https://doi.org/10.1109/WACV.2012.6163017
[25] Li Y, Fan X, Mitra NJ, Chamovitz D, Cohen-Or D et al. Analyzing growing plants from 4D point cloud data. ACM Transactions on Graphics (TOG) 2013; 32 (6): 1–10. https://doi.org/10.1145/2508363.2508368
[26] Ziamtsov I, Navlakha S. Machine learning approaches to improve three basic plant phenotyping tasks using three- dimensional point clouds. Plant Physiology 2019; 181 (4): 1425–1440. https://doi.org/10.1104/pp.19.00524
[27] Shi W, van de Zedde R, Jiang H, Kootstra G. Plant-part segmentation using deep learning and multi-view vision. Biosystems Engineering 2019; 187:81 – 95. https://doi.org/10.1016/j.biosystemseng.2019.08.014
[28] Majeed Y, Zhang J, Zhang X, Fu L, Karkee M et al. Deep learning based segmentation for auto- mated training of apple trees on trellis wires. Computers and Electronics in Agriculture; 2020 170:105277. https://doi.org/10.1016/j.compag.2020.105277
[29] Japes B, Mack J, Rist F, Herzog K, Töpfer R et al. Multi-view semantic labeling of 3D point clouds for automated plant phenotyping. arXiv preprint arXiv:1805.03994 2018.
[30] Itakura K, Hosoi F. Automatic leaf segmentation for estimating leaf area and leaf inclination angle in 3D plant images. Sensors 2018; 18 (10): 3576. https://doi.org/10.3390/s18103576
[31] Jin S, Su Y, Gao S, Wu F, Hu T et al. Deep learning: Individual maize segmentation from terres- trial lidar data using faster R-CNN and regional growth algorithms. Frontiers in Plant Science 2018; 9:866. https://doi.org/10.3389/fpls.2018.00866
[32] Wang J, Chen X, Cao L, An F, Chen B et al. Individual rubber tree segmentation based on ground-based LiDAR data and faster R-CNN of deep learning. Forests 2019; 10 (9): 793. https://doi.org/10.3390/f10090793
[33] Chaudhury A, Boudon F, Godin, C. 3D plant phenotyping: All you need is labelled point cloud data. In: European Conference on Computer Vision 2020; 244–260. Springer, Cham. https://doi.org/10.1007/978-3-030-65414-6_18
[34] Conn A, Pedmale UV, Chory J, Navlakha S. High-resolution laser scanning reveals plant architectures that reflect universal network design principles. Cell Systems 2017; 5 (1): 53–62. https://doi.org/10.1016/j.cels.2017.06.017
[35] Conn A, Pedmale, UV, Chory J, Stevens CF, Navlakha S. A statistical description of plant shoot architecture. Current biology 2017; 27 (14):2078–2088. https://doi.org/10.1016/j.cub.2017.06.009
[36] Dutagaci H, Rasti P, Galopin G, Rousseau D. ROSE-X: an annotated data set for evaluation of 3D plant organ segmentation methods. Plant methods 2020;16 (1):1-4. https://doi.org/10.1186/s13007-020-00573-w
[37] Miao T, Wen W, Li Y, Wu S, Zhu C, Guo, X. Label3DMaize: Toolkit for 3D point cloud data annotation of maize shoots. GigaScience 2021; 10 (5):giab031. https://doi.org/10.1093/gigascience/giab031
[38] Saeed F, Sun S, Snider J, Liu T, Li C. 3D annotation and deep learning for cotton plant part segmentatione and architectural trait extraction. 2022. PREPRINT available at Research Square. https://doi.org/10.21203/rs.3.rs- 2179960/v1
[39] Turgut K, Dutagaci H, Galopin G, Rousseau D. Segmentation of structural parts of rosebush plants with 3D point-based deep learning methods. Plant Methods 2022; 18 (1):1–23. https://doi.org/10.1186/s13007-022-00857-3
[40] Qi CR, Su H, Mo K, Guibas LJ. PointNet: Deep learning on point sets for 3D classification and seg- mentation. In: IEEE 2017 Conference on Computer Vision and Pattern Recognition; 2017. pp. 652–660. https://doi.org/10.1109/CVPR.2017.16
[41] Qi CR, Yi L, Su H, Guibas LJ. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems 30 (NIPS); 2017; 30.
[42] Morel J, Bac A, Kanai T. Segmentation of unbalanced and in-homogeneous point clouds and its application to 3D scanned trees. The Visual Computer 2020; 36 (10):2419–2431. https://doi.org/10.1007/s00371-020-01966-7
[43] Chaudhury A, Hanappe P, Azaïs R, Godin C, Colliaux D. Transferring PointNet++ segmentation from virtual to real plants. In CVPPA-ICCV, 2021.
[44] Boogaard FP, van Henten EJ, Kootstra G. Boosting plant-part segmentation of cucumber plants by enriching incomplete 3D point clouds with spectral data. Biosystems Engineering 2021; 211:167–182. https://doi.org/10.1016/j.biosystemseng.2021.09.004
[45] Boogaard FP, van Henten EJ, Kootstra G. Improved point-cloud segmentation for plant phenotyping through class- dependent sampling of training data to battle class imbalance. Frontiers in Plant Science 2022; 13:838190–838190. https://doi.org/10.3389/fpls.2022.838190
[46] Turgut K, Dutagaci H, Rousseau D. RoseSegNet: An attention-based deep learning architecture for organ segmen- tation of plants. Biosystems Engineering 2022; 221:138–153. https://doi.org/10.1016/j.biosystemseng.2022.06.016
[47] Du R, Ma Z, Xie P, He Y, Cen H. PST: Plant segmentation transformer for 3D point clouds of rape- seed plants at the podding stage. ISPRS Journal of Photogrammetry and Remote Sensing 2023; 195:380-392. https://doi.org/10.1016/j.isprsjprs.2022.11.022
[48] Ghahremani M, Williams K, Corke FM, Tiddeman B, Liu Y, Doonan JH. Deep segmentation of point clouds of wheat. Frontiers in Plant Science 2021; 12:608732. https://doi.org/10.3389/fpls.2021.608732
[49] Li D, Shi G, Li J, Chen Y, Zhang S, Xiang S, Jin S. PlantNet: A dual-function point cloud segmentation network for multiple plant species. ISPRS Journal of Photogrammetry and Remote Sensing 2022; 184:243–263. https://doi.org/10.1016/j.isprsjprs.2022.01.007
[50] Li D, Li J, Xiang S, Pan A. PSegNet: Simultaneous semantic and instance segmentation for point clouds of plants. Plant Phenomics 2022. https://doi.org/10.34133/2022/9787643
[51] Li Y, Wen W, Miao T, Wu S, Yu Z et al. Automatic organ-level point cloud segmentation of maize shoots by integrating high-throughput data acquisition and deep learning. Computers and Electronics in Agriculture 2022; 193:106702. https://doi.org/10.1016/j.compag.2022.106702
[52] Wang L, Zheng L, Wang M. 3D point cloud instance segmentation of lettuce based on PartNet. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2022. pp. 1647–1655. https://doi.org/10.1109/CVPRW56347.2022.00171
[53] Landrieu L, Simonovsky M. Large-scale point cloud semantic segmentation with superpoint graphs. In: IEEE 2018 Conference on Computer Vision and Pattern Recognition; 2018. pp. 4558–4567. https://doi.org/10.1109/CVPR.2018.00479
[54] Rusu RB. Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments. PhD, Technische Universitaet Muenchen, Munich, Germany, 2009.
[55] Shi J, Malik J. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence 2000; 22 (8): 888–905. https://doi.org/10.1109/34.868688
[56] Ng A, Jordan M, Weiss Y. On spectral clustering: Analysis and an algorithm. In: Advances in Neural Information Processing Systems 14 (NIPS); 2001; 14.
[57] Thomas H, Goulette F, Deschaud J-E, Marcotegui B, LeGall Y. Semantic classification of 3D point clouds with multiscale spherical neighborhoods. In: IEEE 2018 International Conference on 3D Vision (3DV); 2018. pp. 390–398. https://doi.org/10.1109/3DV.2018.00052
[58] Scharr H, Minervini M, French AP, Klukas C, Kramer DM et al. Leaf segmentation in plant phenotyping: a collation study. Machine Vision and Applications 2016; 27:585–606. https://doi.org/10.1007/s00138-015-0737-3
[59] Gu W, Bai S, Kong L. A review on 2D instance segmentation based on deep neural networks. Image and Vision Computing 2022; page 104401. https://doi.org/10.1016/j.imavis.2022.104401
[60] Rosu RA, Schütt P, Quenzel J, Behnke S. LatticeNet: Fast spatio-temporal point cloud segmentation using permutohedral lattices. Autonomous Robots 2022; 46 (1): 45–60. https://doi.org/10.1007/s10514-021-09998-1

APA	Dutagaci H (2023). Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. , 792 - 813. 10.55730/1300-0632.4018
Chicago	Dutagaci Helin Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. (2023): 792 - 813. 10.55730/1300-0632.4018
MLA	Dutagaci Helin Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. , 2023, ss.792 - 813. 10.55730/1300-0632.4018
AMA	Dutagaci H Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. . 2023; 792 - 813. 10.55730/1300-0632.4018
Vancouver	Dutagaci H Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. . 2023; 792 - 813. 10.55730/1300-0632.4018
IEEE	Dutagaci H "Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants." , ss.792 - 813, 2023. 10.55730/1300-0632.4018
ISNAD	Dutagaci, Helin. "Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants". (2023), 792-813. https://doi.org/10.55730/1300-0632.4018

APA	Dutagaci H (2023). Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. Turkish Journal of Electrical Engineering and Computer Sciences, 31(5), 792 - 813. 10.55730/1300-0632.4018
Chicago	Dutagaci Helin Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. Turkish Journal of Electrical Engineering and Computer Sciences 31, no.5 (2023): 792 - 813. 10.55730/1300-0632.4018
MLA	Dutagaci Helin Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. Turkish Journal of Electrical Engineering and Computer Sciences, vol.31, no.5, 2023, ss.792 - 813. 10.55730/1300-0632.4018
AMA	Dutagaci H Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. Turkish Journal of Electrical Engineering and Computer Sciences. 2023; 31(5): 792 - 813. 10.55730/1300-0632.4018
Vancouver	Dutagaci H Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants. Turkish Journal of Electrical Engineering and Computer Sciences. 2023; 31(5): 792 - 813. 10.55730/1300-0632.4018
IEEE	Dutagaci H "Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants." Turkish Journal of Electrical Engineering and Computer Sciences, 31, ss.792 - 813, 2023. 10.55730/1300-0632.4018
ISNAD	Dutagaci, Helin. "Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants". Turkish Journal of Electrical Engineering and Computer Sciences 31/5 (2023), 792-813. https://doi.org/10.55730/1300-0632.4018