Sort The Pick Points (within and between layers)

For a complex situation where a variety of objects are scattered, it’s necessary to consider the height of the object and the probability of successful grasp. To solve this problem, this procedure stratifies the spatial pick points by their height and obtain the corresponding planar pick points. Then sort the pick points of each layer according to the specified rules. After that the index list that meet the requirements is obtained. Finally the procedure sort the spatial pick points by the index list. The program of this procedure is shown in Figure 1.

Figure 1 The program of sorting the pick points in Mech-Vision

The introduction of each step in the program is as follows :

1. Layer Point Clouds : Input the point clouds in the region of interest, stratity it by their height, project it to the plane and obtain the mask of the objects in different heights.

2. Sort Pick Points Within The Layer : Input the result of the previous step, the spatial pick points and their labels and confidence level to obtain the planar pick points which will be put into different groups and sorted inside the group by specified rules. Output the list of the pick points after sorting with their labels and their confidence level.

3. Sort Pick Points Between Layers : Input the result of the previous step, extract the Z value of the first pick point in each layer, use the index to sort the different layers by the specified rules. Output the list of the pick points after sorting with their labels and their confidence level.

4. Unpack and Merge Data : Input the pick points list (label list) obtained in the previous step, unpack it and merge it into a new pick points list (label list) then output it.

Layer Point Clouds

The program of this procedure is shown in Figure 2.

Figure 2 The program of laying point clouds

The introduction of each step in the program is as follows :

1. Point Cloud Clustering : Input the point clouds in the region of interest, classify the point clouds which meet the rules into the same category and output the list of the point clouds.

2. Merge Point Clouds with similar height : input the result of the previous step, merge multiple point clouds within a certain height range in the specified direction into one point cloud, then output the list of point cloud after merging.

3. Project 3D Point Cloud to 2D Image : Input the result of the previous step, calculate the projection of each 3D point and obtain the corresponding 2D image (the mask) then output it.

Sort Pick Points Within The Layer

The program of this procedure is shown in Figure 3.

Figure 3 The program of sorting the pick points in the same layer in Mech-Vision

The introduction of each step in the program is as follows :

1. Allocator : Allocate the input to the next few steps.

2. Transform Poses : Input the pick points list, transform it from the robot coordinate system to the camera coordinate system and output it. This step is for the calculation of the planar pick points.

3. From 3D Poses to 2D Poses : Input the result of the previous step, calculate the corresponding 2D pick points and output it.

4. Group 2D Poses : Input the 2D pick points list and the mask of the objects at different heights, put the pick points which are in the same mask into the same group and output the corresponding index list.

5. Group Data : According to the index list from the previous step, make different groups for the pick points in the robot coordinate system, their confidence level and their labels then output it.

6. Sort 3D Poses : Input the result of the previous step, sort the pick points, their confidence level and their labels according to the specified rules and then output it.

Sort Pick Points Between Layers

The program of this procedure is shown in Figure 4.

Figure 4 The program of sorting the pick points in different layers in Mech-Vision

The introduction of each step in the program is as follows :

1. Allocator : Allocate the input to the next few steps.

2. Trim Input List : Input the pick points list after layering and sorting, output the first few elements in each layer according to the settings.

3. Unpack and Merge Data : Input the pick points list obtained in the previous step, unpack it and merge it into a new pick points list then output it.

4. Decompose Object Dimensions : Input the result of the previous step, put the X, Y, Z values of each pick point into three different lists and output them.

5. Sort and Output Index List : Input the list corresponding to the Z value of each pick point, sort it by the specified rules and then output the corresponding index list.

6. Reorder by Index List : Sort the ordered pick points in the same layer, their labels and their confidence level according to the index list and then output them.