Vision-based Size Estimation and Centroid Positioning of Partially Occluded Fruits

The objective of this study was to propose a simple and efficient image processing algorithm for estimating the size and centroid of partially occluded round fruits. In the proposed method, the size and centroid of partially occluded fruit were estimated based on the mathematical properties of the arc-radius. The experimental tests were conducted in a laboratory with orange, Sunkist, apple, and tomato fruits by setting different occlusion conditions. The occlusion percentage was varied between 0% and 90%. The accuracy and processing time of the proposed method were compared with that of the widely-used conic-section circle fitting method. The results showed that the proposed method resulted in an overall accuracy of 95.1% and processing time of 0.66 s, as opposed to 60.2% and 0.68 s, respectively, using the conic-section equation. Compared with the conic-section equation, the proposed method was able to give a more robust prediction, even for low resolution images.


INTRODUCTION
With a constantly growing world population, fruit production is increasingly in demand today. In horticulture, approximately round-shaped fruits (such as Sunkist, orange, apple, brown pears, and tomato) are popularly produced and highly consumed all over the world, including in Myanmar. At the time of harvesting in every season, thousands of fruits need to be manually picked in a conventional way, namely handpicking. Thus, harvesting is the most timeconsuming and laborious work among the processes of fruit and vegetable production. A large farm needs a large number of human labourers in harvesting. Sometimes, handpicking can even degrade the quality of fruit (Młotek et al. 2015). Therefore, automatic harvesting systems have become an attractive option to be practiced in today's horticulture, enabling a reduction in time requirements, manpower requirements, and degradation of fruit quality.
Before automatic harvesting, mechanical harvesting methods were investigated and practiced in the early 1960s (Li et al. 2011). Mechanical harvesting systems work by shaking the tree mechanically to remove fruits from the canopy without destroying the whole. However, mechanical harvesting cannot maintain the size and the quality of the fruit. Moreover, it can even cause physical damage to the tree. Then, the concept of an automatic harvester was proposed and investigated in 1968 in the research of (Schertz and Brown 1968). An automatic harvester uses a robotic arm to position a gripper toward the fruit, which overcomes the problems in mechanical harvesters. From this day, automatic fruit picking robots become attractive to be practiced in horticulture. The main functions of fruit picking robots are to identify and locate the matured fruits precisely and quickly and then pick the fruit. However, knowing the actual size of the fruit and locating the centroid of fruit precisely so that it fits in the gripper (picker) are still great challenges for fruit picking robots, especially when on-tree fruits are partially covered with leaves and/or canopy, as shown in Figure 1.
Concerning detection, recognition, and positioning of especially clustered fruits, overlapping fruits and occluded fruits, there have been a number of studies on these aspects (e.g., Luo et al. 2016;Mehta et al. 2017;Le et al. 2019;Lv et al. 2019b;Yu et al. 2019;Zapotezny-Anderson and Lehnert 2019;Ge et al. 2019;Zhang and Gao 2020;Gené-Mola et al. 2020;Kang and Chen 2020). From these studies, some relevant works have been surveyed focusing on the circle fitting method used as follows. Meng and Wang (2015) proposed a method for recognising overlapping apple fruits based on boundary curvature estimation. The curvature at every point on the boundary of the overlapping fruits was calculated. Then, the complete boundaries of overlapping fruits were achieved by applying least square circle fitting to the pixels of each valid boundary segment. According to their results, their proposed method can separate and recognise fruits with an overall accuracy rate of about 90.6%.
In the works of Lu and Sang (2015), the cluster barycentre method, edge barycentre method, circular Hough transform method, and least square circle fitting method were applied to reconstruct the complete size of occluded fruits. The authors ultimately recommended the least square circle fitting method for designing the fruit-picking robot in a natural scene when the occlusion area ratio was less than 52% or the occlusion angle was less than 216°.
Using binocular stereo vision technology, a recognition method for clustered tomatoes was presented by Xiang et al. (2014). The least square circle regression method was used for completing the boundary of occluded fruits. A total of 189 pairs of stereo images were tested, and the recognition accuracy was 87.9% when the leaf or branch occlusion area ratio was less than 25%. Jidong et al. (2016) developed a recognition method for non-occluded, overlapping and occluded apples in a natural environment. Then, the improved random Hough transform method was applied to recognise the missing parts of apple fruits. The authors tested the proposed method with 60 images containing 113 apple fruits with different visual characteristics. The accuracy was 100% for both non-occluded and overlapping fruits, and 86% for fruits occluded by leaves and branches.
A method for detecting citrus fruits and occlusion recovery under natural illumination conditions can be found in previous research by Lu and Sang (2015). The least square circle fitting was used for recovering the occluded boundaries of fruit. Their experimental results showed that the method was effective, and had a relative error of occlusion of 5.27%.
Differently, Gongal et al. (2015) proposed a new regression model for the prediction of apple sizes by considering the distance between the camera and fruit in the model. An accuracy of 84.8% was achieved in size estimation.
From this survey on previous studies, it is obvious that most existing methods for round fruit recognition are mostly based on a conic-section equation solved by least square circle fitting. However, the conic-section equation does not show good fit when the occlusion percentage on the fruit is more than 70%. This motivated us to develop a new, simple and effective method to estimate the occluded part of the occluded fruits.
Therefore, the objectives of this study were to develop a new, efficient image processing algorithm for the size estimation and centroid positioning of occluded fruits based on the arc-radius equation, and to experimentally verify the performance of the proposed method by comparing it with the performance of the conic-section equation.

Conic-section equation
As aforementioned, most previous studies have made use of the conic-section equation for fruit size and centroid estimation. Thus, a brief overview of this method is provided below.
In nature, there are a number of agricultural products, including fruits, whose cross-section falls in one category of conic sections (circle, ellipse, parabola, and hyperbola). As such, the conic-section equation is widely used to estimate the complete shape of partially visible objects. A general conic-section equation is described in Equation 1.
where, A, B, C, D, E, and F are five constants of coordinates pairs (x, y) on conic-section boundaries. A circle's coordinate pairs (x, y) and centroid (k, h) can be represented by a circle equation, as in Equation 2.
Thus, for a given set of coordinate pairs (x, y), a general conic-section equation can be solved using the least square method. Then, from the known coefficients (A, B, C, D, E), the required coefficients for the circle equation (A, C, k, h) can be obtained. A detailed explanation of the least square method for an over-determined system of linear equations can be found in (Selesnick 2013).

PROPOSED METHOD FOR FRUIT SIZE AND CEN-TROID ESTIMATION
For estimating the size and the centroid of occluded fruits, the proposed method is based on the arc-radius equation. Then, its performance is compared with the performance of the conic-section equation. The flowchart of the proposed method is shown in Figure 2. The detail of each process is explained in the following subsections.

Image segmentation
From the acquired RGB colour image, the visible part of the fruit was segmented based on colour-based segmentation. For segmentation, the average colour thresholds were achieved from pre-learning 300 colour data of a number of fruit images. For the segmenting of the fruit colour of Sunkist, apples, tomatoes, and oranges, the following colour thresholds were used (Equation 8).
For the segmenting of the green colour of the leaf, the following colour thresholds were used (Equation 9).
where, R, G, B are red, green, and blue colour intensities at i th row and j th column of the image and I bw is the binary image. After segmenting the fruit region and leaf region, the boundaries were detected as shown in Figures 3b and 3c. From two regions, the common boundary pixels were searched and they were removed from the fruit boundary pixels. This resulted in it showing only the arc boundary of the fruit (Figure 3d). After searching the observable arc boundary of the fruit (Figure 3d), the width and height were calculated. First, the width of the arc (P 1 P 2 ) was calculated using Pythagoras' theorem (Equation 10). Then, the midpoint (P m ) was calculated using Equations 11 and 12. After achieving the coordinate of midpoint (P m ), the angle (θ) between the two vectors can be checked using Equation 13. As shown in Figure 4, from all coordinate points on the boundary, the peak point (P p ) was defined as the point that gives a right angle (θ = 90°) between → U and → V .
where, x 1 , y 1 , x 2 , y 2 , x m , y m , x p , and y p are coordinates of the arc end points (P 1 , P 2 ), midpoint (P 3 ), and peak point (P 3 ), respectively, n is the total number of arc boundary pixels, and x i and y i are coordinate pairs of the arc boundary points. Then, the width and height of the arc boundary can be calculated using Equations 14, 15, and 16.

Finding fruit radius
When the arc-height and arc-width have been identified, its radius can be found using the arc-radius equation. If two chords in a circle perpendicularly intersect each other (as shown in Figure 5), then the products of the segments of the two chords are equal. According to this mathematical property, the following relationships between width, height, and radius can be derived (Equations 17,18,and 19).
where H, W, and R a are the height, width, and radius of the arc, respectively, and L is the rest portion of the circle diameter.
FIGURE 3. Image segmentation and arc boundary searching. (a) True colour image, (b) fruit region and its boundary pixels, (c) leaf region and its boundary pixels, (d) arc boundary pixels.

Searching fruit centroid
Although the fruit radius could be found as described in the previous section, its centroid has not been found out, yet. As shown in Figure 6, the centroid (P c ) of the arc is the coordinate point that gives θ = 90°(or −→ Pm P 1 ⊥ −→ Pm Pc ) and P c P 1 ∼ = P c P 2 ∼ = P c P p ∼ = R a . Thus, the point (P c ) can be searched from all of the coordinates in the whole image using Equations 13, 14, 15, and 16.
Here, one thing to explain is how to use the arc-radius equation when there is no occlusion on the fruit. When the full boundary of the fruit is detected, only approximately three-quarters of the boundary pixels are taken for calculation using the arc-radius equation.

Experimental setup and procedure
To verify the performance of the proposed method, it was experimentally tested to estimate the missing region of occluded spherical fruits, namely Sunkist, orange, apple, and tomato. Figure 7a shows the experimental setup and apparatus. A Canon camera (Power Shot SX110 IS) was used to capture the images. It was held on a fixed stand to make it stable at 18 cm above the fruit. When the images of the fruits were captured, room lamp lights were used. However, no special lighting was used. Table 1 shows the specifications of the camera used. Its available resolution was 10 mega pixels. The dimensions of the images captured by the    Image format JPEG camera were 640 × 480 pixels and the images were saved in JPEG format. Then, each image was resized to one-third of original size and stored as a second version in order to use it for testing the performance of the proposed method at a lower resolution. A vernier caliper was used to measure the actual size (diameter) of the fruits (Figure 7b). The predicted size from each image was in a pixel unit. Thus, to obtain each unit in cm, these predicted dimensions were converted by multiplying them with a constant value (cm/pixel), which was found out in advance. For any object that is captured from a distance of 18 cm, the cm to pixel ratio is 5/148 ∼ = 0.0338.
The proposed method was implemented in the MAT-LAB software. The microprocessor used was an Intel(R) Core TM i5-3210M, with a processing speed of 2.5 GHz. In the experimental tests, different fruit samples with different sizes were selected. An orange, Sunkist, round apple, and tomato were used as test samples. The smallest sized fruit was 4.1 cm while the largest was 8.6 cm. In this study, the ratio of the occlusion area to the total area ratio in 2D was termed as the occlusion percentage, with this percentage ranging from 0% to 90%. In the tests, the occlusion conditions (collusion direction and occlusion percentage) were manually and randomly created by covering leaves on the fruit in different directions (Figure 7a). Thus, the occlusion percentage and direction mentioned here were approximated values. The performance of the proposed algorithm was evaluated in terms of its accuracy in shape and size recognition and the processing time for each piece. The resulted shapes were visually judged. The accuracy was calculated using Equation 20 and the processing time was recorded by the program itself.

RESULTS AND DISCUSSION
3.1 Non-occlusion conditions Figure 8 shows the actual and estimated sizes and shapes of the different types of fruits (TM: tomato, OR: orange, SU: Sunkist, and AP: apple) in a non-occluded condition.
The estimated boundaries are superimposed on the original colour images in order to visualise the estimated size/shape and actual size/shape in comparison. The resulting arc boundaries after segmentation are arranged in the first column. Here, as it has been discussed, only 3 /4 of the whole boundary is taken for non-occlusion conditions. The estimated boundaries using the conic-section equation are shown in the second column while the estimated boundaries using our proposed method are shown in the last column, from top to bottom. The yellow drop in each result represents the estimated centroid. In the results of the proposed method, the arc-width and centroid-to-peak point (radius) are also superimposed in addition to a boundary line to be different from the results of the conic-section equation.
One can see very good agreement between the estimated boundaries and actual sizes/shapes of the fruits, with the estimated centroid and estimated boundaries al- most coinciding with each fruit's actual structures. This means that the conic-section equation and our proposed method have the same performance during a nonocclusion condition. The results also showed that both methods can provide this same performance regardless of fruit size. For the cases of a non-occluded condition, the whole boundary of the fruit was visible and three-quarters of boundary pixels were taken. Thus, the coordinates of the different parts of the fruit boundary were included in the solving equations. This makes both methods stable in terms of producing a perfect boundary estimation.

Occlusion conditions
The estimated centroids and boundaries for tomatoes under different occlusion conditions are shown in Figure 9. The approximate occlusion percentages and occlusion di- rections for different cases are shown in the first column. For example, in the of case TM-1, the occlusion percentage was 40% and occlusion direction was from 45°to 75°. Since the tomato was a bit small, the maximum occlusion percentage was 70% and the minimum occlusion percentage was 40%.
One should note that the occlusion direction can be different even if the occlusion percentage is the same (see, for example, TM-2 and TM-3). Sometimes, the total occlusion area is somewhat large, but a big portion of fruit boundary is still observable (for example, in the case of TM-1).
Thus, in the cases of TM-1 and TM-2, the fitted boundaries and centroids using the conic-section equation and the proposed method are almost exactly the same with the actual boundary. In TM-3's case, the observable arc boundary was in a vertical position. For this case, the fitting with the conic-section equation gave a small elliptical shape wrongly, while the proposed method gave robust results. As shown with TM-4, the conic-section equation gave a hyperbolic shape rather than a circle shape when the arc boundary was very short. The proposed method was found to still have a good performance even with the very short arc boundary. The reason can be explained as follows. Indeed, multiple coordinates in different directions are required in solving the conic equation with the least square method. When the observable boundary is shorter, the number of coordinate points is small. Moreover, if only the flat boundary of the fruit is visible, it produces coordinates in a single direction. The coordinates in other directions are missing. That is the reason for the conic-section equation making unstable and wrong shapes.
The results for the orange fruit are shown in Figure 10, wherein the minimum occlusion percentage was 50% and increased up to 80%. In OR-1 and OR-2's cases, the visible arc boundaries mainly existed in the first quadrant and second quadrant, respectively. In these conditions, fitting with the conic-section equation showed acceptable results. However, when the detected arc boundary becomes short or it becomes in a vertical position (see , the conic-section equation fails in its prediction of the fruit's boundary and centroid. In contrast, the proposed method was found to show consistent and reliable predictions for all cases.
According to the results of cases OR-3 and OR-4, the conic-section equation was unstable even if the observable boundary was large enough because the direction of coordinates was a bit flat. Meanwhile, the proposed method still showed better performance since it did not depend on segment direction. Rather, it only needed the measurable width and height of the observable boundary.
The performances of the conic-section equation and proposed method were additionally confirmed in the cases of Sunkist and apple fruits (Figures 11 and 12). One can observe in cases of SU-1, SU-2, AP-1, AP-2, and AP-3 that the visible arc boundaries are in vertical positions. The coordinate points along such boundaries produced a large range of y and a small range of x. In the same way, for the arc boundaries which laid in a horizontal position also resulted in a large range of x and a small range of y. This is the main reason why fitting with the conic-section equation  produces elliptical shapes rather than a circle shape for arc boundaries in vertical and horizontal positions. Meanwhile, perfect circle boundaries were achieved by means of the proposed method regardless of arc boundary orientation. In contrast, the proposed method (namely the arcradius equation) can work even with a few coordinate points as long as it forms an arc with an identifiable width and height. The performance of the proposed method was verified even with lower resolution images. As the higher resolution (OR-HR) and lower resolution (OR-LR) images show (Figure 13), the proposed method can still provide acceptable results even when the image resolution is reduced to one-third. Conversely, fitting with the conic-section equation results in an unfavorable prediction when the image resolution becomes lower because only a fewer number of coordinates are available from the visible boundary.
Comprising a total of 19 cases, the accuracy comparisons of the proposed method are listed in Table 2. From the experimental tests, the maximum accuracy of our proposed method was 99.77% and the minimum accuracy was 86.66%, whereas the maximum accuracy in prediction using the conic-section equation was 98.76% and the minimum 0%, the latter because it failed in some cases.
There were a total of 113 tests with different occlusion conditions, although the visualised results are not il- lustrated here. Thus, the average accuracy and processing time associated with each method is compared in Table  3. The processing time was not significantly different between the two methods, since most processes are the same in both methods. It should also be noted that processing time was also dependent on the microprocessor used and the image size.

CONCLUSIONS
In this study, a new method was proposed for estimating boundaries and centroids of partially occluded round fruits. The proposed method was based on the arc-radius equation. In performance verification, tomato, orange, round apple, and Sunkist fruits with different occlusion conditions were used as samples. The occlusion percentage was changed from 0% up to a maximum of 90%. For a non-occlusion condition, our proposed method and the conic-section equation gave reliable predictions of fruit sizes, shapes, and centroids. In different occlusion conditions, the conic-section equation was reliable only when the occlusion percentage was less than 60%. In particular, when the observable arc boundary was in a vertical or horizontal position, fitting with the conic-section produced an elliptical shape rather than a circle shape. When the visible arc boundary was very short (about 10% of the whole fruit boundary), the conic-section equation even failed to predict the unknown boundary. In contrast, the proposed method worked well for circle fitting in any occlusion direction and occlusion percentage of up to 90%. In the experimental performance tests, the proposed method showed a maximum accuracy of 99.77% and minimum of 86.66%. Compared with fitting using the conic-section equation, the proposed method can give a better prediction of an occluded boundary and centroid. From a total of 113 tests, the overall accuracy of the proposed method was 95.1% and the processing time taken by this method was 0.66 s, which is 0.02 s faster than fitting with the conicsection equation.