Recent Advances in Intelligent Automated Fruit Harvesting Robots

2.1. Robotic Harvesting of Sweet Pepper

An automated sweet pepper harvesting robot, named ‘Harvey’, has been developed recently [7]. This robot successfully addresses three key challenges: detection, grasp selection and manipulation. They developed a simple and effective vision-based algorithm for fruit detection, a three-dimensional localization and grasp selection method, and a novel end-effector design for fruit harvesting. To reduce the complexity of motion planning and minimize occlusions, Lehnert et al. 2017 [7] focused on developing a system picking sweet peppers in a protected cropping environment where plants were grown on planar trellis structures (Fig. 1).

In Fig. (1), the left photo (A) is the custom mobile platform of the harvesting robot. Harvesting is performed with a custom harvesting tool and 7 Degree of Freedom (hereafter, DOF) manipulator (6-DOF articulated arm + lift joint) integrated into a custom differential drive mobile base. The right photo (B) is the custom harvesting tool attached to the robot end effector [7].

In the field evaluations, this sweet pepper harvesting system resulted in a 58% harvesting success rate, 81% grasping rate, and 90% detachment rate. Such high success rates represent a significant improvement over the previous state-of-the-art technologies, demonstrating an encouraging progress towards the possibility of developing a commercially viable autonomous sweet pepper harvester. Moreover, Fig. (1A) shows the custom differential drive platform that is designed and powered by an internal 3 kWh lead-acid battery to work between crop rows for up to 8 hours. The platform has a 6-DOF revolute arm (Universal Robotics UR5) mounted on a prismatic lift joint (Thomson LM80). The differential drive mobile base houses the batteries, drive motors, gearboxes, computer hardware, robot controller and forward-facing laser scanner for mobile navigation and obstacle detection. Fig. (1B) shows the custom harvesting tool that can grip sweet peppers with a suction cup and cut them free from the plant using an oscillating cutting blade. Due to variations in crop size, shape and orientation, it is challenging and unreliable to use a single end-effector to grasp and cut each sweet pepper simultaneously.

To overcome this difficulty, this harvesting tool is redesigned with a key feature of having a passive decoupling mechanism that allows the gripping and cutting operations to occur sequentially at independently chosen locations. The decoupling mechanism is a flexible strip that tethers the suction cup to the body of the end effector. The suction cup is also magnetically attached to the underside of the cutting blade, allowing the robot arm to guide the suction cup during the attachment phase. After attachment, the cutting blade is lifted to decouple the suction cup from the cutting blade. The suction cup is then only attached to the end effector via the flexible tether, allowing the cutting blade to move independently of the suction cup through the cutting operation. After detachment, the sweet pepper falls off the plant and hangs freely from the flexible tether. The suction cup and cutting blade can be magnetically re-coupled ready for the next harvesting cycle using gravity by simply pointing the harvesting tool downwards. The sweet pepper is released into a collection crate by releasing the vacuum. This simple and passive decoupling method requires no additional actuators, allowing for a greater harvesting success rate. The harvesting tool also contains an RGB-D camera (Intel R Realsense SR300 RGB-D) sensor to perceive the crop and a micro-switch to check whether the suction cup is coupled with the cutting blade. The body of the end effector contains a modified oscillating multi-tool that is used to cut fruit stalks. A pressure sensor on the vacuum line helps detect the success or failure of attachment of the suction cup.

2.2. Robotic Harvesting of Tomato

The robotic harvesting system of tomato is presented in Fig. (2).

Fig. (2A) shows the system configuration of the tomato harvesting robot and its external appearance. In Fig. (2), the left photo (A) is the tomato harvesting robot system. And the right photo (B) is the 3-dimentional position measurement of the tomato fruit: A proposed detection and discrimination method [8].

The robot assumes a large-scale tomato production facility as the house environment. In the house, there are tomato seedlings with multiple bunches, with shelves aligned in a line. And along the shelf rails for the bogies, the workers move and harvest tomatoes. This system consists of a Kinect v.2, a USB camera, a 6-axis serial link manipulator, an end-effector, a computer, and a moving mechanism that carries and moves them along the rails. To move around, there are three operation steps:

Step 1 - Move to the edge of the shelf with a bogie and generate an environmental map of the entire shelf from the acquired image;

Step 2 - Extract information such as tomato condition to be harvested, its fruit position and total harvest time through analyzing the environmental map; and

Step 3 - Move the dolly to the front of the harvested bunches and harvest the fruits in order.

Fig. (2B) shows a flow of 3-dimensional position measurement of the fruit. A Kinect sensor captures the RGB-D image and IR image of tomato at 30 fps. Next, the IR image is processed to detect the position of the fruit. From the pixel information of the RGB-D image corresponding to the detection position, the fruit color determination and the 3-dimensional position measurement are performed. In this paper, a fruit detection and discrimination method in the room is proposed. The fruit on the IR image shows a concentric reaction with a strong center and a weak perimeter. The gradient orientation (8 azimuth angle) of the IR image is calculated and showed to be not dependent on the absolute value of the response intensity, and the gradient orientation image (each orientation is represented by eight colors) fruit pattern template matching. In comparison [8], they only extract the region with high coincidence rate by the binarization process and label it to detect the position of the fruit.

Finally, the correspondence between the red region of image extracted from the RGB image and the pixel position of the IR image is performed, and the coloring judgment is carried out.

In Fig. (3), a kinematically redundant picking manipulator with 8-DOF for apple fruit collection is presented. Fig. (3A) shows the 8-DOF picking manipulator includes all revolute arm with 6-DOF fastened to the base actuated by two stepper motors. On the redesigned catching robot, each Dynamixel Pro actuator is placed at the joint location. Fig. (3B) shows the experimental set-up used during the pick-and-place harvesting motion. The picking manipulator deposits the artificial apple in the container after it is detached from the replica tree. Fig. (3C) shows the picking manipulator grasping fruit from the tree canopy. The machine vision system is located behind the harvesting robot [9].

The picking manipulator has 8-DOF and a 6-DOF all-revolute arm with Dynamixel Pro actuators (Robotis Inc., Irvine, CA) fastened to the base that can displace in the x-y plane. Both prismatic joints on the base are actuated by NEMA 23 stepper motors and consist of steel rails with linear bearings and timing belts. The picking end-effector design is similar to that described by Silwal et al. [10]. The catching manipulator is a planar design with two links. Geometric parameters are selected such that the catching manipulator could reach every possible drop position in the workspace of picking manipulator. Each joint is actuated by a Dynamixel Pro model L54-50-S500-R (Robotis Inc., Irvine, CA). The stepper motors and mechanical transmissions used in the preliminary design [11] are replaced with Dynamixel Pro actuators in order to increase torque output, increase velocity, and reduce backlash.

It should be noted that although this design modification increases the overall system cost, it significantly improves its maximum end-effector velocity as well as accuracy and repeatability from reduced backlash. The manipulator’s links are fabricated from aluminum plate. The catching end-effector is a plastic funnel lined with flexible baffles manufactured with a three-dimensional printer. The fruit collection system is gravity fed. A flexible hose attached to an opening at the bottom of the catching end-effector funnels fruit into a storage container.

2.3. Robotic Harvesting of Apple

The robotic harvesting system of apple is presented in Fig. (3).

Fig. (3A) shows the robotic harvesting system of apple that consists of picking manipulator, picking end-effector, catching manipulator and catching end-effector.

Fig. (3B) shows that a wooden storage crate is fastened to the table adjacent to the picking manipulator during harvesting cycles using the pick-and-place method. The location of the storage crate is not optimized according to the criteria of performance. Rather, a convenient location for the described setup is chosen such that the crate does not obstruct robot hardware during the picking motion. After detaching an apple from the tree, the picking manipulator deposits the fruit in the container. For each harvesting cycle, three-dimensional coordinates of all fruit positions are generated using Matlab’s (Mathworks, Natick, MA) random number generator.

Both harvesting methods are then used per cycle so that total displacement between apples remains constant when comparing cycle times. Sequential fruit selection in a harvesting cycle is considered as the Traveling Salesman Problem (TSP). The TSP is an optimization problem predicated on finding the shortest path through a set of points that passes through each point once and only once [12]. Matlab’s k-nearest neighbor algorithm [13] is used for fruit prioritization.

The algorithm’s starting point for the search is the end-effector coordinates at the picking manipulator’s home configuration. The algorithm selects the closest fruit as the first apple for harvesting and then, sequentially, the nearest neighbors for the remaining fruit in a cycle. Prioritization planning is completed offline before the start of each cycle. Trapezoidal velocity profiles are used for manipulator movements planned in the joint space with the maximum velocity of all revolute joints set at 60^o per second. Also, the approach distance d is set at 15 cm. Fig. (3C) shows that the integrated system is mounted on the back of an electric utility vehicle for harvesting the apple variety Envy in a V-trellis orchard system. Optimization of the collection system below the level of the catching end-effector, such as implementation of a bin filling device, requires additional works. Initial observations from preliminary testing indicate that the current design requires substantial modifications prior to more extensive field evaluations.

2.4. Robotic Harvesting of Kiwifruit

The robotic harvesting system of kiwifruit is presented in Fig. (4).

In Fig. (4) the kiwifruit harvesting robot demonstrates its functions of separating kiwifruit, nondestructive picking and unloading of clustered kiwifruit in scaffolding cultivation by the integrated grabbing-picking-sliding harvesting method. Fig. (4A) is the image of the kiwifruit harvesting robot and Fig. (4B) shows its simulation test in the laboratory [14].

Fig. (4A) shows the kiwifruit picking robot consisting of five main parts: Machine vision, end-effector, coordinate manipulator, vehicle system, and control system. The picking robot can automatically undertake the tasks of fruit information recognition, positioning, picking order planning, movement to kiwifruit picking area, nondestructive picking by bionic fingers, and unloading fruit to the basket.

Fig. (4B) shows a kiwifruit picking test being conducted with emulated fruit in an artificial shelf in farm machinery laboratory of Northwest Agriculture and Forestry University of China. Among the 30-fruit set (divided into 6 groups, 5 per group) to be harvested, 27 were successfully harvested with a success rate of 90%. The 10% failure to harvest the fruit was due to the fact that the parts with IR sensors affected the grabbing movement of the end-effector when it enveloped the adjacent fruits. The average picking time was 4 seconds, which was similar to that of manual picking but remarkably more efficient than the formerly developed end-effectors. The lab simulation test validates that the end-effector can completely fulfill all given functions in harvesting kiwifruit.