Houseplant leaf classification system based on deep learning algorithms

Abstract

Botanical experts are typically relied upon to classify houseplants since even subtle differences in characteristics such as leaves can distinguish one species from another. Therefore, an automated system for recognizing houseplant leaves with accuracy and reliability becomes a valuable asset for the identification of indoor plant species. In this paper, a houseplant leaf classification system utilizing deep learning algorithms is proposed, which has been improved to effectively classify and identify a variety of houseplant leaf types. The system uses the ResNet-50 architecture based on convolutional neural network to analyze features of the leaf images and extract relevant information for classification. In addition, this work presents a newly constructed local dataset consisting of 2500 images to classify species of houseplant leaves. The dataset includes ten types of houseplant leaves that are suitable for cultivation in various climates at home. The dataset was augmented using data augmentation algorithms to increase its size and reduce overfitting. The developed system was training and testing using a local dataset. To evaluate the improved model, comparative experiments were conducted utilizing pre-trained models (original ResNet-50 and MobileNet_v2). The improved model revealed recognition accuracy of 99% with the augmented dataset and 98.60% without the augmentation, affirming its effectiveness. The improved model could potentially be used in various fields, including horticulture, plant pathology, and environmental monitoring to identify plant species.

Introduction

With the development of civilization and the transformation of lifestyles, people nowadays have considered cultivating an attractive and delightful environment close to their homes by residing with flowers and other kinds of plants. In the growth of civilization, including in the areas of food, medicine, research, industry, and environmental protection, among others, plants are crucial and essential to humans. In agriculture science, plants have to be classified and categorized via these many plant types. Currently, there are several ways for plant classification including the traditional method that utilizes the human eye or base identification and the automated method that utilizes computer-based identification. Therefore, being able to identify and classify each kind of existing plant is significant.

Artificial intelligence (AI) and computer vision, in particular, have become common due to the development of computer technology, and they now play a significant role in numerous aspects of human life. Through the development of AI, image detection and recognition tasks are becoming more fully developed, and they have also been effectively utilized in numerous domains, including object detection, facial recognition, and various other areas of application. In the field of agriculture, AI has a significant role in terms of identifying different plant types. Currently, houseplants are widely used in homes and workplaces. Hence, computer applications have an important role in assisting experts and non-experts to classify and arrange plants according to the patterns and shapes of their leaves. Accordingly, houseplant recognition is significant, especially based on their leaves. For this reason, an automated houseplant leaf classification and detection system is deemed vital based on deep learning algorithms since deep learning has reached major developments in image recognition and identification.

The identification process of plants necessitates more information about the biochemical and physiological appearances of plants, such as their color, shape, texture, and other attributes, especially the shape of leaves, which is more important for the identification procedure. Therefore, the leaf of a houseplant was utilized in this investigation. Traditional methods of plant leaf identification, as estimated by expert's experience, may require more effort for identification. In contrast, the incorporation of computer technology into the identification process renders it markedly more dependable, accurate, and less time-consuming.

Nowadays, deep learning and machine learning, when used together, have brought great improvements and developments in image classification and object detection systems, especially in the agricultural sector [1,2,3,4,5,6,7,8,9]. Currently, the lack of data is a big issue for all researchers. For this purpose, the study contributions are summarized as follows:

• A new dataset has been constructed that contains ten classes such as Dieffenbachia, Tradescantia zebrine, Callisia, Ficus elastica, Croton plant, Epipremnum aureum, Gena Garchak, Zambia, and Silver Lining Kiwi.

• Based on deep learning models, the ResNet-50 model has been improved and modified specifically for the purpose of houseplant leaf classification.

• Additionally, the improved model incorporates hyperparameter tuning, including the freezing of model layers. This practice results in a reduction of both the number of layers and parameters within the model, consequently leading to a less time-consuming practice.

The research has a defined organization that starts with an introduction. Section two provides a theoretical background by reviewing variety of methods used to classify houseplant leaves. The methods used in this paper are described in section three. The performance evaluation and experiment results are comprehensively presented in section four. The research is concluded in section five.

Related work

The automatic recognition of plants is an essential and actively researched field within the domains of machine learning and computer vision. The ability to automatically classify plant species through images has many potential applications, particularly in agriculture. Researchers have developed various approaches for automatic plant recognition, including traditional computer vision techniques and deep learning-based methods. These approaches often rely on image processing techniques, extracting features, and machine learning algorithms to classify the plant types.

In agricultural domain, plants could be divided into several parts such as roots, stem, flowers, fruits, and leaves. Leaves play an important role in the growth of plants, and leaves are used for many different purposes such as plant species recognition, plant health assessment, genetic research, and disease diagnosis.

Most plant identification tools rely on image processing and recognition techniques as their basis. For instance, the authors of [10] used a segmentation method for leaf frame based on the combination of Gaussian interpolation and wavelet transform (WT). Machine learning algorithms are extensively used in plant leaf recognition. For example, the authors of [11] introduced an automated plant species recognition image system known as LeafSnap. Furthermore, the study included the development of a mobile application designed to aid botanists in identifying trees by capturing leaf images. The identification process in the study employed the nearest neighbors (NN) technique. In addition to the utilization of classical machine learning methods, the author of [12] employed local binary patterns (LBPs) for the recognition of plant leaves. Also, support vector machine (SVM) was used in [13] as a classifier to identify plant types based on their leaves. The model obtained an average precision of 84%, a recall of 83%, and an accuracy result of 82.67%. To identify and classify tomato and lemon leaf, [14] proposed a decision tree. The experimental findings demonstrated that, in comparison with K-Mean and SVM algorithms, decision trees achieved good accuracy while taking less time. And, linear discriminant analysis and Naive Bayesian classification model were used in [15] to recognize the leaf. The results for the accuracy of leaf recognition were 91.56 and 98.44%, respectively.

Furthermore, deep learning algorithms have been applied in the field of agriculture, particularly for plant recognition based on their leaves. For example, Carlos et al. employed models such as VGG16, Inception V3, and Xception to address the challenges associated with plant species classification. They also noted the potential for grouping various plant species by utilizing a publicly available dataset and various pre-training techniques. The experimental findings showed that Xception model exhibited superior performance compared to the other models, which achieved an accuracy result of 86.21% [6]. Additionally, in [16] a model based on deep learning algorithms was proposed to observe the botanist’s behavior with leaf identification. The pretrained architecture MobileNetV2 was employed along with the transfer-learning technique. Moreover, [17] demonstrated that a hybrid deep learning architecture (CNN and SVM) could efficiently extract leaf image features and classify them through the use of a CNN for feature extraction and an SVM for classification. In addition, to develop a LeafNet CNN-based plant identification system, in [18], a deep learning system was suggested to learn discriminative characteristics from leaf pictures using a classifier for species identification of plants.

The CNN model was also employed in [19] to perform and achieve plant identification. Instead of utilizing CNN, they provided a technique to identify the learnt features that were extracted using deconvolution networks (DN). They achieved a 99.6% accuracy rate. Multi-scale fusion convolutional neural network (MSF-CNN) was additionally proposed in [20] to identify input image plants based on their leaves; the highest accuracy of 99.6% was achieved. There are also several alternative neural network architectures used for large-scale image classification, including ResNet, GoogleNet, and similar models [21]. In [22,23,24,25], a CNN-based leaf identification system was proposed and stated that the model is appropriate and reliable for the identification procedure.

Moreover, in [26], a classification method was developed to classify four potato leaves using VGG16 and VGG19 based on CNN architectural models. The achieved classification rate of 91% demonstrated the viability of the deep neural network approach. Based on CNN, in [27], tea leaves were classified. According to the experiment's findings, the CNN beat the SVM and BP neural networks, which had accuracy rates of 89.36 and 87.69%, respectively. As a result, the CNN-based classification system outperformed the other models [27]. In addition, an automatic method was employed, including learning vector quantization (LVQ) and CNN model for the classification of diseases affecting tomato plant leaves [28]. According to the reviewed studies, deep learning models generally outperformed machine learning approaches [29]. During the literature review, it was revealed that numerous techniques have been employed for identifying plant leaves.

Based on the reviewed literature, one of the limitations of previous studies on houseplant leaf recognition using deep learning is the lack of availability of publicly accessible datasets. It would pose a limitation for practical implementation in real-world scenarios where acquiring such comprehensive datasets might be challenging. This could impact the generalizability and performance of the models. This study endeavored to construct a new dataset with diverse houseplant species and variations. In addition, data augmentation techniques were used in this study that helps the model learn invariant features and improves its ability to recognize patterns in diverse conditions. However, this is not mentioned in most of the reviewed studies. Another limitation is time complexity for training models. Furthermore, classification accuracy result and less error rate yet pose another challenge.

Materials and methods

Dataset description

The essence of such a proposed system in machine learning and deep learning techniques lies on the collection of data. A new dataset with a structured format was created, comprising various species of leaves from indoor plants. We captured images of plant leaves in natural surroundings from rural areas and public gardens located in Kurdistan region. During this process, we made a conscious effort to select plant species that were not already included in the available datasets.

The dataset contains images of ten different species of indoor plants, with 250 images available for each species class. The dataset consists of a total of 2500 images. All the images were captured under natural conditions in the public gardens, with a total of five different gardens being utilized to gather and create the dataset. Some of these species have distinct and recognizable visual features, such as Geranium, while others may have very similar appearances. The names of indoor plant leaf types are assigned based on the expertise of agricultural specialists, farmers' experiences, and internet guidelines. The scientific names of the dataset samples are Dieffenbachia, Tradescantia zebrine, Callisia, Ficus elastica, Croton plant, Epipremnum aureum, Gena Garchak, Zambia, and Silver Lining Kiwi. Figure 1 shows sample dataset images.

Fig. 1

Sample species of the houseplant dataset

Images were captured in a public garden under a variety of weather conditions, including midday and evening. These images were taken from various angles, including top and level perspectives, and exhibited variations in aspect ratio, orientation, and size (500 × 375 and 375 × 500) pixel spatial resolution. As a pre-processing step, the input images were resized into 224 × 224 × 3 to reduce the computational processing time of the model before being fed into it.

The hardware devices that have been used for collecting all sample images were (Redmi Note 10, full HD, 48 MP), (iPhone 8 plus, full HD, 12MP), and (Huawei 13 MP). All samples were collected in five months (on November 1, 2021–March 10, 2022). In the experiment, the dataset was divided into three sets: the training set, validation set, and test set, with proportions of 80, 10, and 10%, respectively.

Dataset augmentation

A large amount of data that is used in the neural networks training has a significant impact on their accuracy rate in deep learning [30]. Due to a variety of factors, it is difficult to combine sufficient data, especially avoiding duplication. The size of the dataset can be increased through various techniques like cropping, rotation, translation, scaling, etc. These actions performed on raw images to expand the training dataset are referred to as data augmentation procedures. Augmenting data in this way helps enhance dataset diversity.

In this paper, we utilized data augmentation techniques as a pre-processing step to enhance the model's performance, enlarge the constructed dataset, and mitigate overfitting. In our dataset, we collected ten different classes of houseplant leaves, with each class containing 250 leaf images. The training set consists of 200 images for each class. We applied five different data augmentation techniques to each training image sample for every class individually. All operations of the augmentation on a sample image are shown in Fig. 2. These augmentation methods include rotating the image by 45°, random zooming inside the images, vertical flipping, horizontal flipping, and shifting the images by 20% of their total height. As a result of the data augmentation process, the number of image samples for each class in the training set increased by 1000 images, totaling 1200 images per class, including the original images. Consequently, the total number of training images in the dataset, including both original and augmented images, increased from 2000 to 12,000 images. In contrast, data augmentation was not applied to the testing set, which consists of a total of 500 images. Table 1 provides an overview of the images in the training and testing sets.

Fig. 2

Demonstrates the effects of each data augmentation approach on one sample of the dataset

Table 1 Sample image numbers of the dataset used for model performance

CNN models

There are numerous hyperparameters in convolutional neural network models. Consequently, determining an optimal hyperparameter combination necessitates trial and error and requires time when creating an effective CNN architecture. Therefore, many advanced CNN architectures have been designed with natural imagery in mind and were trained on large, publicly available datasets like ImageNet. Inception, VGG, DenseNet, MobileNet, and ResNet are the most widely used pre-trained models. However, every field of study cannot benefit from the basic design of these networks. Fortunately, we can adjust these CNNs to address the shortcoming because of the manner they are created. A variety of features of the pre-trained CNNs can be fine-tuned, including learning rate, dense layers, layer freezing, and optimization techniques. The most common technique is freezing layers so that the model can be adjusted to fit a different field of study. Considering that one of the most effective pre-trained CNN architectures is ResNet-50, it was chosen to conduct this study. The next two sections detail ResNet-50 and MobileNet.

• (a) ResNet-50 model

  Deep residual network or ResNet is an innovative pre-trained CNN architecture that was developed by He et al. [31]. The architecture enables the training of deep networks with hundreds or thousands of layers while maintaining high performance. The ResNet model offers the advantage of maintaining performance even with increasing depth in its architecture. Furthermore, it achieves lighter computational calculations, enhancing network training capabilities [31, 32] Figure 3 shows the ResNet-50 model architecture.

• (b) MobileNet

  The architecture introduced by Howard et al. [33], known as MobileNet, is crafted on depth wise separable convolutions with the aim of constructing a compact deep CNN. This design results in a lightweight model, minimizing computational requirements. As a consequence of its efficiency, MobileNet finds applicability in various recognition tasks, such as object detection, face attributes, fine-grain classification, and landmark recognition.

Fig. 3

ResNet-50 Model Architecture

An improved ResNet-50 architecture

This paper focuses on improving the performance of the pre-trained ResNet-50 model, a variant of the ResNet architecture with 50 layers. The overall framework of the improved system is illustrated in Fig. 4, which involves the process of collecting image data, feature extraction and classification. To achieve the highest level of accuracy, various hyperparameters were adjusted. Particularly, the improved model managed to reduce the number of trainable parameters from 23,651,146 in the original ResNet-50 to 23,633,994, thereby eliminating 17,152 parameters. Additionally, features extracted using the ResNet-50 convolutional layers and the fully connected layers with the 'SoftMax' function were used to classify ten classes of houseplant leaves. The enhanced ResNet-50 hyperparameters are shown in Table 2.

Fig. 4

Framework of the improved system

Table 2 The improved ResNet-50 Hyperparameters

A series of experiments were conducted to evaluate the performance of the pre-trained ResNet-50 model in the context of houseplant leaf classification, to produce the highest level of accuracy. To fulfill this, a method involving the selective freezing and unfreezing of particular internal layers within the ResNet-50 model was employed. In other words, freezing prevents the weights of a neural network layer from being modified during the training phase.

In this manner, two different scenarios have been presented to investigate the potential of improving the ResNet-50 model. In the first scenario, the training of ResNet-50 involved the utilization of all layers without any layer freezing, and the network's previously learned weights were used during the process.

In the second scenario, we attempted to find the ideal number of freeze layers using the constructed dataset to train the ResNet-50 model. As a result, this strategy was started by the initial freezing of the first five layers within the architectural framework, followed by the subsequent freezing of the next five layers, and this progression was repeated until the final layer of the model was reached. At each stage of the progression in the architecture, a set of five frozen layers was changed. Figure 5 shows the improved ResNet-50 model.

Fig. 5

The framework of the proposed technique for houseplant leaves

Model evaluation metrics criteria

The model's performance was assessed using various evaluation metrics, including accuracy, recall, precision, and F1-score, based on the analysis of the confusion matrix. In the studies that relate to plant leaf classification based on deep learning, the most common evaluation metric is accuracy [34, 35]. The accuracy equation can be defined in the following formula:

$${\text{accuracy}} = \frac{{{\text{TP}} + {\text{TN}}}}{{{\text{TP}} + {\text{TN}} + {\text{FN}} + {\text{FP}}}}$$

(1)

where TP = true positive, TN = true negative, FP = false positive, and FN = false negative.

Recall measures the proportion of true positives correctly predicted among all actual positive instances. As defined in the following:

$${\text{Recall}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}}}}$$

(2)

Precision quantifies the proportion of accurate predictions among all positive instances predicted by the model. As showed in the following equation:

$${\text{Precision}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FP}}}}$$

(3)

The F1-Score value considers recall and precision rates. As defined in the following:

$$F1 - {\text{Score}} = 2* \frac{{{\text{Recall}} \times {\text{Precision}} }}{{{\text{Recall}} + {\text{Precision}} }}$$

(4)

Experimental environment

In this work, the experiments utilized a computer system consisting of an Intel Core i7-7700HG processor, an NVIDIA GeForce GTX 1060Ti (12 GB) graphics card, and 32 GB of RAM. These experiments were performed on a desktop computer running the Windows 10 operating system. The implementation of the program was processed in Python 3.8 using the Anaconda3 environment with CUDA support.

Results and discussion

Houseplant leaf classification system

Numerous experiments were carried out for this investigation. A selection and testing of ResNet-50 with different structures were done. Accuracy rates are shown for each experiment scenario in Tables 3 and 4. The experimental results were based on a ResNet-50 model structure. Five layers of the ResNet-50 architecture were frozen. The testing began by freezing the non-overlapping layers of ResNet-50 one by one, from the beginning to the end.

Table 3 Test results of the improved model with freezing 5-Layers consequently with data augmentation

Table 4 Test results of the improved model with freezing 5-layers consequently without data augmentation

Two different scenarios were applied to the dataset in order to examine the effect of layer freezing on ResNet-50 performance. In the first scenario, augmentation techniques were utilized as a pre-processing step to improve the model's performance. The results of the experiment for this particular scenario are shown in Table 3. As can be observed, the test trial that involved freezing layers from the 41st to the 45th had the lowest accuracy result of 97.6%. On the other hand, the experiment that froze layers from the 21st to the 25th layer obtained the highest accuracy result of 99%.

In the second scenario, no augmentation techniques were used; thus, the experiments were applied to the created dataset. Table 4 presents the outcomes of this scenario's experiment as well as the experiment's greatest accuracy result, which freezes layers 21 through 25. It achieved a 98.60% accuracy rate. Also, the experiment that freezes layers from the 45th to the 50th achieved the worst accuracy result, reaching 97.6%, while the other results are very similar to one another.

Our experiments presented in Tables 3 and 4 showed the influence of freezing layers and implementing data augmentation techniques on achieving optimal performance with the ResNet-50 architecture, as applied to our constructed dataset. Table 3 revealed that the worst result was achieved with ResNet-50 (41–45), whereas the others exhibited superior performance. The highest outcomes were attained with ResNet-50 (21–25), which has an accuracy of 99%.

The confusion matrix for the best-performing model's outcomes is shown in Fig. 6. The improved ResNet-50 model correctly classifies all test samples into six of ten categories, with an error rate of about 1% for the other three classes and 2% for only one class. The improved technique’s training and testing accuracy rate is displayed in Fig. 7a. The loss function is represented visually in Fig. 7b, demonstrating the importance of the training and testing iterations. It demonstrates that the loss is being reduced during the learning process.

Fig. 6

Confusion Matrix of the improved model with data augmentation

Fig. 7

a Model performance Accuracy b Loss Function of the improved model

Improved model comparison with other models

To evaluate the performance of the improved ResNet-50, pre-trained models (original ResNet-50 and MobileNet_v2) have been tested on the constructed houseplant leaf dataset. Accuracy, precision, recall, and F1-score are employed as evaluation criteria based on the confusion matrix. During the investigation of the impact of layer freezing and data augmentation on houseplant leaf classification, it is evident that the utilization of the ResNet-50 architecture proves to be an effective approach for improving recognition capabilities in such tasks.

Finally, as can be seen from the findings in Table 5, the improved ResNet-50 approach outperformed original ResNet-50 and MobileNet_v2. The highest accuracy result was 99%, whereas the highest precision, recall, and F1-score were 99.03, 99.00, and 99.01%, respectively. Additionally, the effectiveness of our approach is evaluated and compared with that of MobileNet_v2 and original ResNet-50 using a graph, as shown in Fig. 7.

Table 5 Test results of different models with augmented dataset

Based on the findings presented in Table 5 and Fig. 8, the improved technique outperformed the original ResNet-50 and MobileNet_v2 on the augmented dataset. The test accuracy results obtained are 99.00, 97.80, and 97.20%, respectively. The modified model demonstrated superior performance when compared to both the original ResNet-50 and MobileNet_v2.

Fig. 8

Performance evaluation of different models using data augmentation

In addition, the improved ResNet-50 demonstrated better performance compared to the original ResNet-50 and MobileNet_v2 when tested on the built dataset without applying augmentation techniques. These results are presented in Table 6.

Table 6 Test results of different models without data augmentation

The findings showed that the improved ResNet-50 model outperformed the other models (original ResNet-50 and MobileNet_v2) in terms of accuracy with and without data augmentation. Furthermore, the enhanced model achieved higher accuracy than other techniques while concurrently reducing training weight parameters. Additionally, it reduced the time needed to attain optimal performance during training, completing the task in 3 min and 42 s for 20 epochs. In contrast, the original ResNet-50 necessitated 3 min and 47 s to accomplish the same workload.

Conclusion

Categorizing houseplants based on their leaf characteristics is a challenging task, typically demanding the expertise of botanical specialists. However, the development of an accurate and a reliable automated houseplant leaf classification system could provide valuable assistance to individuals engaged in the agricultural domain. This paper proposed a houseplant leaf classification system that used deep learning algorithms and the state-of-the-art ResNet-50 model to analyze leaf images and extract relevant information for classification. The paper also introduced and established a new dataset of ten types of houseplant leaves. To enlarge the dataset, five different data augmentation techniques (rotating, randomly zooming, vertical flip, horizontal flip, and height shifting by 20%) were applied through which the total images of the dataset increased into 12,500 images. The quantitative experiments confirmed that the improved ResNet-50 model based on freezing layers and data augmentation achieved an outstanding recognition outcome of 99%. In order to assess the performance of the improved ResNet-50 model, the original ResNet-50 and MobileNet_v2 were analyzed as pre-trained models. The comparative test results clearly indicated that the improved model performed significantly superior performance in comparison with the other models. The proposed model achieved the best result by freezing select layers and reducing the parameter count. For future work, this study aims to optimize its performance in real-time field conditions. Moreover, there is a plan to enlarge the dataset by integrating supplementary class categories including diverse houseplant leaf types.