2D Rectangle Matching(Deep Learning)

2D Rectangle Matching(Deep Learning) is capable of calculating locations and pick points of jointed and homogeneous cubes with complex textures. Inputs：original colored image, original depth image, masks from instance segmentation, the mask of the higest layer, original point cloud and actual size of the object Prior information of deep learning can provide the data of approximate location of objects. The main process includes 4 parts: generating rectangle edge templates, detecting lines in the image corresponding to the mask, 2D matching and calculating sizes and poses by correct rectangle information. as Figure 1. shown.

Figure 1. Procedure of 2D rectangle matching(Deep Learning)

Generating Rectangle Edge Templates

The procedure of generating rectangle edge templates is as shown in Figure 2..

Figure 2. Procedure of generating rectangle edge templates

The introduction of each steps is as follows:

1. Apply Masks to Image Get the mask of the object after instance segmentation by calculating overlaps between the mask of the highest layer and the mask after instance segmentation.

2. Apply Masks to Point Cloud Get the point cloud of the object after instance segmentation by processing the object masks after instance segmentation and the original point cloud.

3. From Actual Dimensions to Dimensions in Pixels Calculate the size of the object(in pixel) with the point cloud after instance segmentation and the actual size of the object.

4. Generate Rectangular Edge Templates of Specified Sizes Generate the object surface(rectangle) edge template with the size of object(in pixel).

Detecting Lines In the Image

Detecting lines in the image corresponding to the mask is as Figure 3. shown.

Figure 3. Procedure of detecting lines in the image corresponding to the mask

The introduction of each steps is as follows:

1. Apply Masks to Image Obtain the object mask of the higest layer by calculating overlaps between the mask of the higest layer and the original colored image.

2. Get First Image from Image List Get the image of the result corresponding to the first mask image.

3. Detect Line Segments Detect lines in segmented objects by processing the colored image corresponding to the mask and the mask generated by instance segmentation.

2D Matching

By matching detected lines with generated rectangle edge templates, Mech-Vision would output polygons consisted of intact lines and also polygon matching loss.

Calculating sizes and Poses

Calculating sizes and poses by correct rectangle information is as Figure 4. shows.

Figure 4. Calculating sizes and poses by correct rectangle information

The introduction of each steps is as follows:

1. Remove Polygons outside Mask Based on the 2D matching result and the mask of the higest layer, the polygons whose overlapping areas with the mask are smaller than the threshold would be removed. This step is to make sure that the object corresponding to the polygon is on the top layer.

2. Remove Overlapped Polygons The inputs are the result of remove_polygons_outside_mask, the original colored image and the mask of the highest object. This step is to remove overlapped polygons or polygons outside the mask. Make sure the object corresponding to the polygon is not overlapped.

3. Calc Poses and Dimensions of Rectangles This step needs the original depth image, the rectangle information after filtering and the mask of the higest layer. Pick points will be calculated with the depth information and rectangle vertexes.

Typical Application: 2D Matching