Acquire RGB Images Based on Depth Images

The principle behind this combination of steps is to get the RGB image based on the highest layer within the depth images ROI to extract the valid regions and to remove other regions from interfering with the deep learning recognition.

Figure 1 Combination of Steps to Acquire RGB Images Based on Depth Images

As shown in Figure 1, Acquire RGB Images Based on Depth Images procedure consists of three parts: Acquiring The Highest Layer of The Mask , Morphological Transformations and Acquiring The Colour Image Corresponding to The Mask .

Acquiring The Highest Layer of The Mask

Figure 2 Combination of Steps to Acquiring The Highest Layer of The Mask

As shown in Figure 2, the process for Acquiring The Highest Layer of The Mask combination of steps is as follows.

1. Invalidate Depth Pixels outside 3D ROI step extracts the depth data of the ROI area.

2. Segment Depth Map step to segment the depth image and filter small pieces of clutter.

3. Get Highest Layer Regions in Depth Map Step to generate a mask of valid regions.

4. Merge Mask Images Step to merge multiple mask images into one image.

Tip

There are two ways to set the ROI region, directly editing the json file and using Mech-Viz software. The essential difference between the two ways lies in the way the json file is manipulated, the second method is recommended.

Morphological Transformations

This step expands the masked image, which serves to expand the edge areas and preserve the edge information of the ROI in the image.

Acquiring The Colour Image Corresponding to The Mask

Figure 3 Combination of Steps to Acquiring The Colour Image Corresponding to The Mask

1. Apply Masks to Image step to extract the 2D image of the corresponding region of the mask image.

2. Get First Image from Image List step to get the result image corresponding to the first mask image.

Tip

If there is no Get First Image from Image List step in the list of steps on the left, the user can uncheck Show only frequently used steps in the settings.

Example Implementation

Input

Figure 4 RGB Original Image

Figure 5 Depth Original Image

Output

Figure 6 Result Image

Attention

When it is necessary to use Mech-Viz software to perform the ROI setting for the Step Invalidate Depth Pixels outside 3D ROI. The project must contain a extract point cloud within ROI step. Otherwise Mech-Viz will not complete the ROI setting and a message will pop up: ROI setting failed RPC failed.