3D Fine Matching

Function

Accurately match the model point cloud with the original point cloud, and outputs more accurate poses of target objects.

Hint

Please refer to Generate Point Cloud Model to create a point cloud model.

../../../../_images/function_description3.png

Sample Scenario

This Step performs fine matching based on the matching results of 3D Coarse Matching, and outputs more accurate object poses, which can be used as pick points.

Input and Output

../../../../_images/input_output_20.png

Parameters

The Parameters Tuning Level, found under Execution Flags under, sets the extent of completeness that the parameters are displayed for adjustment.

Options: Basic, Advanced
Default: Basic

  • Basic: Only commonly used parameters are displayed for adjustment.

  • Advanced: All parameters are displayed for adjustment.

Advanced parameters are marked with ☆ below.

Key parameters

Introduction

Model and Pick Point

Set the model file path and the geometric center file path

Cloud Orientation Calculation

Choose the calculation method of point cloud orientation

Correspondence Settings

Select the method to match the model point cloud with the object point cloud

Sample Settings

Set the details of sampling the point cloud

Symmetry Settings ☆

Tell the matching algorithm the symmetricity of the objects

Model Weight in Validation ☆

The weight added to object parts with obvious features to facilitate matching

Validation Settings

Settings on matching result validation

Output Settings

Settings on matching result output

Acceleration Settings ☆

Whether to use GPU to accelerate the running

Results Visualization

What parts of the results to visualize

Model and Pick Point

Model and Pick Point set paths of the .ply file that contains the point cloud(s) of a single model (multiple point clouds taken from different perspectives of a single model object when necessary) and the .json file that contains the information of pick point(s) of the model object.

When the paths are changed, the next time this step is called, the files will be re-read. Both absolute paths and relative paths can be recognized.

Model File
Default: model.ply
Description: The file path of the .ply file that contains the model point cloud. Please see Generate Point Cloud Model for details about generating model point clouds.
Geometric Center File
Default: model/pickPoint.json
Description: This file contains the pick point information of the model point cloud. This file can contain geometric center information for testing purposes, but in practice, when the robot performs actual picking, the file must contain pick point information. Please see Add Pick Point for details about generating pick point files.

Cloud Orientation Calculation

Point Cloud Calc Mode
Default: Origin
Options: Origin, StandardMode, EdgeTangent, EdgeNormal
Description: The above options represent four methods for calculating the direction (normal) of the model point cloud required for the matching algorithm. The normal of the point cloud is perpendicular to the image plane of the depth map on which the point cloud is generated.

  • Origin: Directly obtain the normal from the model point cloud when the file contains the normal of the model point cloud.

  • StandardMode: Select a target point, search for the k points closest to the target point in the vicinity of the target point, and find the minimum feature vector as the normal by principal component analysis.

  • EdgeTangent: Calculate the tangent at the edge of the model point cloud, and take the tangent as the normal. This method can distinguish different objects whose outer contours are mirror images of each other.

  • EdgeNormal: Calculate the normal at the edge of the model point cloud. This method is recommended for relatively flat objects.

Note

When using EdgeTangent or EdgeNormal, please ensure that each edge point cloud does not contain multiple objects (each object’s point cloud should be separate and no object should overlap in the point cloud).

Number of Searching Points
Default: 10
Description: This parameter is for StandardMode mentioned above. It sets the k value, i.e., the number of points in the vicinity of the target point to search for. The minimum value is 1. The larger the value, the more accurate the result, the slower the computation.

Correspondence Settings

Correspondence Settings
Default: GMM
Options: GMM, nearest-neighbor
Description: Select the method to match the model point cloud with the object point cloud. The model point cloud will be moved and rotated within the set range to match the object point cloud.
Applicability: In most scenarios, GMM has better anti-interference performance and higher matching speed, so it is generally recommended; in rare cases, when GMM cannot meet the requirement, nearest-neighbor can be selected.

Please see Settings under GMM and Settings under Nearest-Neighbor below for parameter adjustment instrutions of the two methods.

The figure below is an example of a comparison between the input point cloud and the visualized output matching results via nearest-neighbor.

../../../../_images/in_out_comparison.png

Figure 1. Input/output of matching via nearest-neighbor

Settings under GMM

Matching Mode
Default: Standard
Options: HighSpeed, Standard, HighPrecision
Description: please select based on the accuracy and speed requirements.

  • HighSpeed: The fastest but less accurate.

  • Standard: Relatively stable.

  • HighPrecision: The most accurate but slower.

Number of Iterations
Default: 30
Description: Iteration refers to iteratively running a specific instruction until the conditions are met. The number of iterations refers to the number of repetitions of instruction running in this process. The parameter set here is the upper limit of the number of iterations. The larger the value, the greater the number of times of matching calculations, the longer the running time, the higher the accuracy.
Standard Deviation
Default: 0.005 m
Description: This value is a measure of the expected deviation of the matched pose of the model from the actual pose of the object after 3D Coarse Matching. The closer the expectation to the actual value, the faster the calculation can be, but a higher Standard Deviation will lead to less accurate matching results. Normally, this value will be no greater than 0.01 m. | Example: The figure below is a comparison between matching results when Standard Deviation is 0.04 m and 0.01 m respectively.
../../../../_images/standard_deviation.png

Figure 2. Comparison of results under different Standard Deviations

Standard Deviation Update Step Number
Default: 3
Description: During the matching, the standard deviation gradually decreases, until it reaches the Minimum Standard Deviation. The value of this parameter is the number of times the standard deviation is updated in the matching algorithm. Normally this parameter does not need adjustment.
Minimum Standard Deviation
Default: 0.001
Description: The target value the standard deviation needs to reach.
Apply Weight in Iteration
Default: False
Options: True, False
Description: If set to True, the weight of the points in the model point cloud will be used in the iteration to facilitate the matching.

Please see Weight in Point Cloud Matching for why and how weight works.

Settings under Nearest-Neighbor

Iteration Settings

Number of Iterations
Default: 30
Description: Iteration refers to repeatedly running a specific instruction until the conditions are met. The number of iterations refers to the number of repetitions of instruction running in this process. The parameter set here is the upper limit of the number of iterations. The larger the value, the greater the number of times of matching calculations, the longer the running time, the higher the accuracy.
Nearest Point Search Radius
Default: 0.01
Minimum: 0.001
Description: It adjusts the search radius of the closest point, its unit is m, and its value should correspond to the input initial pose deviation, i.e., the pose deviation after 3D coarse matching. If the deviation is large, the search radius should be large. The value of this parameter should be no lower than 0.001; otherwise, some sparsely distributed points of the object point cloud may not be matched to their nearest points in the model point cloud, thus affecting the output result.
Example: When the initial deviation is large while this parameter is set to the minimum value 0.001, the matching will be incomplete. When this parameter is increased to a larger value like 0.01, the search radius increases, and the matching will be comparatively complete.

Below are two parameters that set the standard by which the matching iteration ends. The iteration ends when either of the standards set by MSE Threshold and Window Size is met.

MSE Threshold
Default: 0.001
Description: It adjusts the threshold of the mean square error (MSE), which is a measure of the closeness of the matching. The MSE is recorded and compared with the MSE Threshold after each iteration. When the MSE after an iteration is less than the threshold, the iteration ends.
Window Size
Default: 10
Minimum: 3
Description: When the number of iterations during which the deviation’s fluctuation remains trivial reaches the Window Size, the iteration ends. The larger the Window Size, the longer the computation may take, but a Window Size may lead to suboptimal results. For instance, if the Window Size is set to the minimum value 3, the iteration ends when the deviations after three consecutive iterations show trivial fluctuation, and the matching result may not be optimal because had the iteration continued, yet smaller deviation would have been produced.
Show Corresponse
Default: False
Options: True, False
Description: If it is set to True, the paired point cloud points will be displayed for each iteration.
Complex Object
Default: False
Options: True, False
Description: It decides whether the algorithm will allocate resources to treat complex object point cloud matching. If the target object is a complex shaped object (not a simple geometric shape) and Automatic Weight is set to True, please set it to True.
Automatic Weight
Default: False
Options: True, False
Description: If set to True, the function of automatically applying weight to the point cloud points will be enabled.

Please see Weight in Point Cloud Matching for why and how weight works.

Point Pair Rejection

Reject Pair
Default: False
Options: True, False
Description: Nearest-neighbor search will find multiple matches for one point, so the function of rejecting pairs should be enabled to filter those matches. If set to True, the function will be enabled. N Sigma Threshold and Point Pair Angle Diff Thre are parameters to the function of rejecting invalid pairs. If it is set to False, the two parameters below will not take effect.
N Sigma Threshold
Default: 1.000,0
Description: It sets the threshold for the distance between the points in a pair. Suppose the mean standard deviation of the distances in all point pairs in the point cloud is σ, the mean of the distances in all point pairs in the point cloud is D. If a point pair’s distance is higher than D + threshold*σ, the pair is deemed invalid and is rejected.
Point Pair Angle Diff Thre
Default: 45°
Description: If the angle difference between the normals of the points in a pair is greater than this value, the point pair will be eliminated.
Apply Weights on Pairs
Default: False
Options: True, False
Description: If set to True, weight will be applied in the iteration. Please see Weight in Point Cloud Matching for why and how weight works.
Remove Repeat Correspond
Default: False
Options: True, False
Description: If set to True, the function of removing redundant point pairs will be enabled, and the settings of Remove Repeat Correspond Type will take effect. It is recommended to set to True when there are more points in the model point cloud than in the point cloud of a single object in the scene.
Remove Repeat Correspond Type
Default: MinDis
Options: MinDis, GlobalMin
Description: It is to select the method of finding and deleting the redundant correspondences between a point in the object point cloud and points in the model point cloud.

  • MinDis: Only keep the pair with the shortest distance.

  • GlobalMin: Obtain the mean distance in all pairs, and only keep the pair with the distance closest to the mean distance.

Sample Settings

Sampling Interval
Default: 0.005 m
Description: It is the sampling interval used for downsampling the model point cloud and the object point cloud. The larger the value, the fewer the points will remain in the point clouds, and the lower the accuracy of matching.
Example: Figure 3 is a comparison of the output results when Sampling Interval is set to 0.025m and 0.005m, respectively. The white parts are the input object point clouds, and the colored lines are the matching results. The comparison of the two figures shows that the smaller the sampling interval (right), the more accurate the output matching result can be.
../../../../_images/sampling_interval_compare.png

Figure 3. Comparison of outputs with small and large sampling intervals

Symmetry Settings ☆

Symmetry Axis ☆
Default: ROTATE_BY_Z
Options: ROTATE_BY_X, ROTATE_BY_Y, ROTATE_BY_Z
Description: For objects with symmetry, this parameter is to select the symmetry axis of the model point cloud.
Angle Step (0–360) ☆
Default: 360°
Description: This parameter is to adjust the symmetry angle step. For instance, if an object looks the same before and after rotating 60° around the symmetry axis, its symmetry angle step is 60°.
Min Rotation Angle (-180–180) ☆
Default: -180°
Max Rotation Angle (-180–180) ☆
Default: 180°
Description: Min Rotation Angle and Max Rotation Angle together tell the matching algorithm the range of the possible angle the object is positioned within the angle step around the symmetry axis. For instance, if the angle step is 60°, then the range can be -180° to -120°. Usually, the range does not need to be adjusted.

For more information about symmetry, please see Symmetry.

Model Weight in Validation ☆

File of Model Fragments with Heigh Weight ☆
Description: This .ply file contains the part of the model point cloud that needs to have a higher weight. The weight value is set by Weight of Each Point ☆ below.

For the reason of setting the weight, please see Weight in Point Cloud Matching below.

Weight of Each Point ☆
Default: 2.0
Description: This parameter is to set the weight value for the parts with salient features in the model point cloud. The initial weight of all points is 1. After applying the weight, the points in the target part of the model point cloud will have a weight of this parameter.
Search Radius When Set Weight ☆
Default: 0.003,0
Description: This parameter is used to set the search radius when applying the weight. The weight setting is an operation performed on the original model point cloud. The model point cloud will be down-sampled before being used in the matching process, resulting in the shifting or loss of some points that need to be weighted. When setting the weight of the points near a missing point, the search radius needs to be set by this parameter.

Validation Settings

Confidence Threshold
Default: 0.500
Minimum: 0
Description: The confidence threshold decides the standard by which each match between an object point cloud and its matched model point cloud is considered valid. Matches with confidences higher than the threshold are valid and those lower are not. The confidence of a match is a measure of the spatial closeness between the object point cloud points and its matched model’s point cloud points. Therefore, a lower threshold means more matches are valid for output and the output is less accurate, and a higher one means less valid matches but higher accuracy of the output.
Limitation: Please note mismatches may still occur given a higher threshold, especially when the object’s different parts are of the same shape.

Instructions:

The Confidence Threshold needs to be tuned during testing for the optimal effect, its minimum is 0, but its maximum max should be calculated as follows:

Suppose the value of Weight of Each Point ☆ is w. For the reason of setting the weight, please see Weight in Point Cloud Matching below.

The ratio of weighted points to all the points in a model point cloud is r, that is:

r = point count of File of Model Fragments with Heigh Weight ☆ / point count of Model File.

max = w * r + (1 - r)

Please set Confidence Threshold to a value between 0 and max.

The point count of a point cloud is displayed on the lower left corner of the window that pops up by double clicking on the input/output ports in the graphical programming workspace. as shown below:

../../../../_images/point_count.png

Figure 4. Point count of a point cloud

Example: Figure 4 below is an example showing two valid matches obtained under small and large confidence thresholds respectively. The confidence threshold of (a) is 0.50 and that of (b) is 0.92. The match of (a) is more accurate than that of (b). Even though the comparison is not prominent, as 3D fine matching itself is a process of fine-tuning, the difference can still be observed, especially in the yellow boxes.
../../../../_images/confidence_threshold_comparison_mark.png

Figure 5. Comparison of matching results under different confidence thresholds with marks

Search Radius
Default: 0.01 m
Minimum: 0
Description: It is the search radius when calculating the confidence. It is set by the sparseness of the object point cloud. If the point cloud is sparse, it needs to be set slightly larger.
Apply Angle Deviation in Validation ☆
Default: False
Options: True, False
Description: If set to True, during the validation, the angle difference of the normals of the corresponding points in a pair will be considered, and the number of output results will be reduced, but the output will be more accurate.
Multiply Corresponding Points Ratio in Scene ☆
Default: False
Options: True, False
Description: If set to True, the validation score of matching will be multiplied by the ratio of the number of the points with correspondence to the number of points in the entire point cloud. The validation score is printed in the log for debugging, and is printed only when the log level is set to Debug. It should be set to False for obtaining the poses of multiple objects in a point cloud. Figure 6 is an example of validation scores printed in the log when debugging.
../../../../_images/validation_score.png

Figure 6. Validation scores in the log

Output Settings

Maximum Number of Detected Poses in Each Point Cloud
Default: 1
Description: The larger the value, the more matching pairs will be output.
Example: Figure 7 is a comparison of the output results when it is set to 1 and 4 respectively.
../../../../_images/single_multiple_compare.png

Figure 7. Comparison of single and multiple output results

Acceleration Settings ☆

Acceleration with GPU ☆
Default: False
Options: True, False
Description: If set to True, GPU acceleration will be enabled. Please set by actual needs.

Results Visualization

Show Sampled Model Cloud ☆
Default: False
Options: True, False
Description: If set to True, the sampled model point cloud will be displayed in the output result. Please set by actual needs.
Show Sampled Scene Cloud ☆
Default: False
Options: True, False
Description: If set to True, the sampled object point cloud will be displayed in the output result. Please set by actual needs.
Show Validation Point Correspondences ☆
Default: False
Options: True, False
Description: If set to True, the correspondence between the model and the object point clouds will be displayed in the output result. Please set by actual needs.
Show Matching Results
Default: False
Options: True, False
Description: Display the output model and object point clouds when set to True.

Weight in Point Cloud Matching

Of an object, some parts have salient features, such as the corners, the holes, etc., and those parts should be paid special attention to by the matching algorithm to make the matching easier. Therefore, weights should be applied to the points of those parts in the point cloud so that the algorithm gives importance to those parts. In particular, weight should be applied when matching without weight does not work well.

Normally, among the points in a point cloud in Mech-Vision, a point is either not weighted or weighted by a fixed multiplier.

Yet please note that, after applying weight, mismatches may still occur especially when the object’s different parts are of the same shape or have similar features.

Parameters involved in weighting: