REVIEW ARTICLE


Recent Advances in Intelligent Automated Fruit Harvesting Robots



Yanbin Hua1, *, , Nairu Zhang1, , Xin Yuan1, , Lichun Quan1, Jiangang Yang1, Ken Nagasaka2, *, Xin-Gen Zhou3, *
1Fruit Industry Service Center of Fufeng County, Fufeng 722299, Shaanxi, P.R. China
2Department of Electrical Engineering and Computer Science, Tokyo University of Agriculture and Technology, 2-24-16 Nakamachi, Koganei-shi, Tokyo 184-8588, Japan
3Texas A&M AgriLife Research Center, Texas A&M University System, 1509 Aggie Drive, Beaumont, TX 77713, USA


Article Metrics

CrossRef Citations:
0
Total Statistics:

Full-Text HTML Views: 352
Abstract HTML Views: 202
PDF Downloads: 150
ePub Downloads: 95
Total Views/Downloads: 799
Unique Statistics:

Full-Text HTML Views: 211
Abstract HTML Views: 119
PDF Downloads: 105
ePub Downloads: 69
Total Views/Downloads: 504



© 2019 Yanbin Hua et al.

open-access license: This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0), a copy of which is available at: (https://creativecommons.org/licenses/by/4.0/legalcode). This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

* Address correspondence to these authors at the Fruit Industry Service Center of Fufeng County, Fufeng 722299, Shaanxi, P.R. China; Tel: +86 917 5211454; E-mail: tsiaoybhua@163.com (Y. Hua)
Department of Electrical Engineering and Computer Science, Tokyo University of Agriculture and Technology, 2-24-16 Nakamachi, Koganei-shi, Tokyo 184-8588, Japan; Tel: +81-42-388-7481;
E-mail: bahman@cc.tuat.ac.jp (K. Nagasaka)
Texas A&M AgriLife Research Center, Texas A&M University System, 1509 Aggie Drive, Beaumont, TX 77713, USA; Tel: +1 409 752 2741; E-mail: xzhou@aesrg.tamu.edu (X.-G. Zhou) These authors contributed equally to this work.


Abstract

Robotic harvesting offers a solution to reducing labor costs, optimizing harvest scheduling, enabling selective harvesting, and increasing operation efficiency. These attributes allow the users of robotic harvesters to maximize production efficiency and profits. This article reviews automated fruit harvesting systems of sweet pepper, tomato, apple and kiwifruit as an example to demonstrate the recent advances in intelligent automatic harvesting robots in horticulture.

Keywords: Intelligent, Efficiency, Automated fruit harvesting robot, Kiwifruit, Horticulture, Selective harvesting.



1. INTRODUCTION

Automation of agriculture is an innovation that can reduce the utilization of resources and help meet with increasing demand for more productivity and higher quality of food production [1]. Human has long strived for mechanization in crop management and harvest to reduce labor costs. The operation of harvest usually occurs when a crop reaches its mature stage. Harvesting a large hectarage of crop within a short period of time, however, is difficult, labor-some, and time-consuming. A solution to these problems is to develop and use mechanical systems that are able to complete the harvesting operations. Mechanized fruit harvesters are one of these systems that have a conventional electro-hydraulic control mechanism to shake nut and fruit trees to help matured nuts or fruit drop off the trees. Such harvesters generally need to be driven to a location proximate to a tree to be harvested, with a shaker head that can extend towards the tree. The shaker head includes movable jaws with pads that clamp the tree and a motor that powers the shaking process [2]. The next development towards advanced, automated mechanization is to integrate with artificial intelligence to increase selectivity, precision, and robustness of farming. This is especially true for harvesting of high value crops, such as capsicums, since harvest operation must occur multiple times during a growing season. Automation of the harvesting task can result in a significant labor saving and provide gentle handling of the fruit as well [3].

Undesirable performance of computer sensing technology is one of the major obstacles in the development of intelligent automated crop management systems. For the past decades, object localization through computer-assisted vision has been developed, resulting in a success of its application in many industry areas [4]. However, the application of these technologies in agriculture is still in its infancy stage [5, 6]. This is due to the difficulties and infeasibilities to collect a large amount of detailed and annotated agricultural data that are required to develop these automated crop management systems [4].

Autonomous robotic harvesting is a challenging but exciting technology for modern agriculture. Developing such systems requires the integration of multiple subsystems including crop detection, motion planning, and dexterous manipulation [7]. An important part of any robotic fruit picking system is the end effector. The end effector is used by the robot to touch and interact with the crop. Thus, its design is critical to reliable handling and detachment of the crop [3]. This article reviews automated fruit harvesting systems of sweet pepper, tomato, apple and kiwifruit as an example to demonstrate the recent advances in intelligent automatic harvesting robots in horticulture.

2. INTELLIGENT AUTOMATED FRUIT HARVESTING ROBOTS

2.1. Robotic Harvesting of Sweet Pepper

An automated sweet pepper harvesting robot, named ‘Harvey’, has been developed recently [7]. This robot successfully addresses three key challenges: detection, grasp selection and manipulation. They developed a simple and effective vision-based algorithm for fruit detection, a three-dimensional localization and grasp selection method, and a novel end-effector design for fruit harvesting. To reduce the complexity of motion planning and minimize occlusions, Lehnert et al. 2017 [7] focused on developing a system picking sweet peppers in a protected cropping environment where plants were grown on planar trellis structures (Fig. 1).

In Fig. (1), the left photo (A) is the custom mobile platform of the harvesting robot. Harvesting is performed with a custom harvesting tool and 7 Degree of Freedom (hereafter, DOF) manipulator (6-DOF articulated arm + lift joint) integrated into a custom differential drive mobile base. The right photo (B) is the custom harvesting tool attached to the robot end effector [7].

In the field evaluations, this sweet pepper harvesting system resulted in a 58% harvesting success rate, 81% grasping rate, and 90% detachment rate. Such high success rates represent a significant improvement over the previous state-of-the-art technologies, demonstrating an encouraging progress towards the possibility of developing a commercially viable autonomous sweet pepper harvester. Moreover, Fig. (1A) shows the custom differential drive platform that is designed and powered by an internal 3 kWh lead-acid battery to work between crop rows for up to 8 hours. The platform has a 6-DOF revolute arm (Universal Robotics UR5) mounted on a prismatic lift joint (Thomson LM80). The differential drive mobile base houses the batteries, drive motors, gearboxes, computer hardware, robot controller and forward-facing laser scanner for mobile navigation and obstacle detection. Fig. (1B) shows the custom harvesting tool that can grip sweet peppers with a suction cup and cut them free from the plant using an oscillating cutting blade. Due to variations in crop size, shape and orientation, it is challenging and unreliable to use a single end-effector to grasp and cut each sweet pepper simultaneously.

To overcome this difficulty, this harvesting tool is redesigned with a key feature of having a passive decoupling mechanism that allows the gripping and cutting operations to occur sequentially at independently chosen locations. The decoupling mechanism is a flexible strip that tethers the suction cup to the body of the end effector. The suction cup is also magnetically attached to the underside of the cutting blade, allowing the robot arm to guide the suction cup during the attachment phase. After attachment, the cutting blade is lifted to decouple the suction cup from the cutting blade. The suction cup is then only attached to the end effector via the flexible tether, allowing the cutting blade to move independently of the suction cup through the cutting operation. After detachment, the sweet pepper falls off the plant and hangs freely from the flexible tether. The suction cup and cutting blade can be magnetically re-coupled ready for the next harvesting cycle using gravity by simply pointing the harvesting tool downwards. The sweet pepper is released into a collection crate by releasing the vacuum. This simple and passive decoupling method requires no additional actuators, allowing for a greater harvesting success rate. The harvesting tool also contains an RGB-D camera (Intel R Realsense SR300 RGB-D) sensor to perceive the crop and a micro-switch to check whether the suction cup is coupled with the cutting blade. The body of the end effector contains a modified oscillating multi-tool that is used to cut fruit stalks. A pressure sensor on the vacuum line helps detect the success or failure of attachment of the suction cup.

2.2. Robotic Harvesting of Tomato

The robotic harvesting system of tomato is presented in Fig. (2).

Fig. (2A) shows the system configuration of the tomato harvesting robot and its external appearance. In Fig. (2), the left photo (A) is the tomato harvesting robot system. And the right photo (B) is the 3-dimentional position measurement of the tomato fruit: A proposed detection and discrimination method [8].

The robot assumes a large-scale tomato production facility as the house environment. In the house, there are tomato seedlings with multiple bunches, with shelves aligned in a line. And along the shelf rails for the bogies, the workers move and harvest tomatoes. This system consists of a Kinect v.2, a USB camera, a 6-axis serial link manipulator, an end-effector, a computer, and a moving mechanism that carries and moves them along the rails. To move around, there are three operation steps:

Step 1 - Move to the edge of the shelf with a bogie and generate an environmental map of the entire shelf from the acquired image;

Step 2 - Extract information such as tomato condition to be harvested, its fruit position and total harvest time through analyzing the environmental map; and

Step 3 - Move the dolly to the front of the harvested bunches and harvest the fruits in order.

Fig. (2B) shows a flow of 3-dimensional position measurement of the fruit. A Kinect sensor captures the RGB-D image and IR image of tomato at 30 fps. Next, the IR image is processed to detect the position of the fruit. From the pixel information of the RGB-D image corresponding to the detection position, the fruit color determination and the 3-dimensional position measurement are performed. In this paper, a fruit detection and discrimination method in the room is proposed. The fruit on the IR image shows a concentric reaction with a strong center and a weak perimeter. The gradient orientation (8 azimuth angle) of the IR image is calculated and showed to be not dependent on the absolute value of the response intensity, and the gradient orientation image (each orientation is represented by eight colors) fruit pattern template matching. In comparison [8], they only extract the region with high coincidence rate by the binarization process and label it to detect the position of the fruit.

Fig. (1). The mobile harvesting robot ‘Harvey’ of sweet pepper operating in a protected cropping environment.

Finally, the correspondence between the red region of image extracted from the RGB image and the pixel position of the IR image is performed, and the coloring judgment is carried out.

In Fig. (3), a kinematically redundant picking manipulator with 8-DOF for apple fruit collection is presented. Fig. (3A) shows the 8-DOF picking manipulator includes all revolute arm with 6-DOF fastened to the base actuated by two stepper motors. On the redesigned catching robot, each Dynamixel Pro actuator is placed at the joint location. Fig. (3B) shows the experimental set-up used during the pick-and-place harvesting motion. The picking manipulator deposits the artificial apple in the container after it is detached from the replica tree. Fig. (3C) shows the picking manipulator grasping fruit from the tree canopy. The machine vision system is located behind the harvesting robot [9].

The picking manipulator has 8-DOF and a 6-DOF all-revolute arm with Dynamixel Pro actuators (Robotis Inc., Irvine, CA) fastened to the base that can displace in the x-y plane. Both prismatic joints on the base are actuated by NEMA 23 stepper motors and consist of steel rails with linear bearings and timing belts. The picking end-effector design is similar to that described by Silwal et al. [10]. The catching manipulator is a planar design with two links. Geometric parameters are selected such that the catching manipulator could reach every possible drop position in the workspace of picking manipulator. Each joint is actuated by a Dynamixel Pro model L54-50-S500-R (Robotis Inc., Irvine, CA). The stepper motors and mechanical transmissions used in the preliminary design [11] are replaced with Dynamixel Pro actuators in order to increase torque output, increase velocity, and reduce backlash.

It should be noted that although this design modification increases the overall system cost, it significantly improves its maximum end-effector velocity as well as accuracy and repeatability from reduced backlash. The manipulator’s links are fabricated from aluminum plate. The catching end-effector is a plastic funnel lined with flexible baffles manufactured with a three-dimensional printer. The fruit collection system is gravity fed. A flexible hose attached to an opening at the bottom of the catching end-effector funnels fruit into a storage container.

2.3. Robotic Harvesting of Apple

The robotic harvesting system of apple is presented in Fig. (3).

Fig. (3A) shows the robotic harvesting system of apple that consists of picking manipulator, picking end-effector, catching manipulator and catching end-effector.

Fig. (3B) shows that a wooden storage crate is fastened to the table adjacent to the picking manipulator during harvesting cycles using the pick-and-place method. The location of the storage crate is not optimized according to the criteria of performance. Rather, a convenient location for the described setup is chosen such that the crate does not obstruct robot hardware during the picking motion. After detaching an apple from the tree, the picking manipulator deposits the fruit in the container. For each harvesting cycle, three-dimensional coordinates of all fruit positions are generated using Matlab’s (Mathworks, Natick, MA) random number generator.

Both harvesting methods are then used per cycle so that total displacement between apples remains constant when comparing cycle times. Sequential fruit selection in a harvesting cycle is considered as the Traveling Salesman Problem (TSP). The TSP is an optimization problem predicated on finding the shortest path through a set of points that passes through each point once and only once [12]. Matlab’s k-nearest neighbor algorithm [13] is used for fruit prioritization.

The algorithm’s starting point for the search is the end-effector coordinates at the picking manipulator’s home configuration. The algorithm selects the closest fruit as the first apple for harvesting and then, sequentially, the nearest neighbors for the remaining fruit in a cycle. Prioritization planning is completed offline before the start of each cycle. Trapezoidal velocity profiles are used for manipulator movements planned in the joint space with the maximum velocity of all revolute joints set at 60o per second. Also, the approach distance d is set at 15 cm. Fig. (3C) shows that the integrated system is mounted on the back of an electric utility vehicle for harvesting the apple variety Envy in a V-trellis orchard system. Optimization of the collection system below the level of the catching end-effector, such as implementation of a bin filling device, requires additional works. Initial observations from preliminary testing indicate that the current design requires substantial modifications prior to more extensive field evaluations.

Fig. (2). Tomato harvesting robot.

Fig. (3). The dual robot coordination for apple harvesting.

2.4. Robotic Harvesting of Kiwifruit

The robotic harvesting system of kiwifruit is presented in Fig. (4).

In Fig. (4) the kiwifruit harvesting robot demonstrates its functions of separating kiwifruit, nondestructive picking and unloading of clustered kiwifruit in scaffolding cultivation by the integrated grabbing-picking-sliding harvesting method. Fig. (4A) is the image of the kiwifruit harvesting robot and Fig. (4B) shows its simulation test in the laboratory [14].

Fig. (4A) shows the kiwifruit picking robot consisting of five main parts: Machine vision, end-effector, coordinate manipulator, vehicle system, and control system. The picking robot can automatically undertake the tasks of fruit information recognition, positioning, picking order planning, movement to kiwifruit picking area, nondestructive picking by bionic fingers, and unloading fruit to the basket.

Fig. (4B) shows a kiwifruit picking test being conducted with emulated fruit in an artificial shelf in farm machinery laboratory of Northwest Agriculture and Forestry University of China. Among the 30-fruit set (divided into 6 groups, 5 per group) to be harvested, 27 were successfully harvested with a success rate of 90%. The 10% failure to harvest the fruit was due to the fact that the parts with IR sensors affected the grabbing movement of the end-effector when it enveloped the adjacent fruits. The average picking time was 4 seconds, which was similar to that of manual picking but remarkably more efficient than the formerly developed end-effectors. The lab simulation test validates that the end-effector can completely fulfill all given functions in harvesting kiwifruit.

Fig. (4). Kiwifruit harvesting robot.

CONCLUSION

In this article, we have reviewed four intelligent automated fruit harvesting robots that are designed to harvest sweet pepper, tomato, apple and kiwifruit. Each harvesting robot has its unique features to complete its operation. The sweet pepper harvesting robot has a novel end-effector that facilitates effective vision system for fruit detection, 3-dimensional localization and grasp selection, resulting in a great success rate of grasping, detachment and harvesting.

The main feature of the tomato harvesting robot is to use a Kinect sensor, particularly use 3-dimensional position detection and color discrimination together with a fruit pattern template matching, to capture the RGB and IR images, resulting in detecting tomatoes more efficiently.

The apple harvesting robot has a very effective catching manipulator that can reach every possible drop position in the workspace of picking manipulator. Such design increases the end-effector speed and accuracy and reduces the backlash for repeatability.

The kiwifruit harvesting robot has the features of being able to separate kiwifruit, nondestructive picking, and unloading of clustered kiwifruit in scaffolding cultivation by the integrated grabbing-picking-sliding harvesting method. Such a system increases the success rate of harvesting, ensuring its accomplishment of all given functions in kiwifruit harvest.

With the reduction of workforce and the increase of production costs, more research efforts and resources will be expectedly placed on the development of more advanced automatic harvesting robots in agriculture, especially horticulture. However, such development will require more close multidisciplinary collaboration involved in many areas, including agricultural engineering, mechatronics, computer science, sensors and instrumentations, deep learning and intelligent systems, software development, system integration, agronomy, and crop management. The development and application of these innovative tools will significantly increase the productivity and sustainability of modern agriculture.

CONSENT FOR PUBLICATION

Not applicable.

FUNDING

None.

CONFLICT OF INTEREST

The authors declare no conflict of interest, financial or otherwise.

ACKNOWLEDGEMENTS

Declared none.

REFERENCES

[1] Bac CW, Van Henten EJ, Hemming J, Edan Y. Harvesting robots for highvalue crops: State-of-the-art review and challenges ahead. J Field Robot 2014; 31(6): 888-911.
[2] Needham DL, Andersen BM, Niday TC, Mayo DP. Tree location sensing system and process for agricultural tree harvesting. US010178830B2, 2019.
[3] Russel R, Beaver PA, Lehnert C. A robotic harvester. US20190029178A1, 2019.
[4] Barth R, IJsselmuiden J, Hemming J, Van Henten EJ. Data synthesis methods for semantic segmentation in agriculture: A Capsicum annuum dataset. Comput Electron Agric 2018; 144: 284-96.
[5] Gongal A, Amatya S, Karkee M, Zhang Q, Lewis K. Sensors and systems for fruit detection and localization: A review. Comput Electron Agric 2015; 116: 8-19.
[6] Nasir A, Rahman M, Mamat A. A study of image processing in agriculture application under high performance computing environment. Int J Comput Sci Telecommun 2012; 3(8): 16-24.
[7] Lehnert C, English A, McCool C, Tow AW, Perez T. Autonomous sweet pepper harvesting for protected cropping systems. IEEE Robot Autom Lett 2017; 2(2): 1-8.
[8] Yasukawa S, Li BH, Sonoda T, Ishii K. Development of a tomato harvesting robot. The 2017 international conference on artificial life and robotics (ICAROB 2017), seagaia convention center, miyazaki, Japan, January 19-22, 2017; 408-11.
[9] Davidson JR, Hohimer CJ, Mo CK, Karkee M. Dual robot coordination for apple harvesting. 2017 ASABE Annual International Meeting Spokane, Washington. July 16-19, 2017; 1-9.
[10] Silwal A, Davidson JR, Karkee M, Mo CK, Zhang Q, Lewis K. Design, integration, and field evaluation of a robotic apple harvester. J Field Robot 2017; 34(6): 1140-59.
[11] Davidson JR, Hohimer CJ, Mo CK. Preliminary design of a robotic system for catching and storing fresh market apples. Int Federation Auto Control 2016; 49(16): 149-54.
[12] Arkin EM, Hassin R. Approximation algorithms for the geometric covering traveling salesman problem. Discrete Appl Math 1994; 55(3): 197-218.
[13] Friedman JH, Bentley JL, Finkel RA. An algorithm for finding best matches in logarithmic expected time. ACM Trans Math Softw 1977; 3(3): 209-26.
[14] Mu LT, Liu YD, Cui YJ, et al. Design of end-effector for kiwifruit harvesting robot experiment. 2017 ASABE Annual International Meeting Spokane, Washington. July 16-19, 2017; 1-8.