3D Fine Matching

Function

This Step can accurately match the point cloud model with the original point cloud and output more accurate poses of target objects.

Hint

Please refer to Generate Point Cloud Model to create a point cloud model.

../../../../_images/function_description3.png

Usage Scenario

This Step performs fine matching based on the matching results of 3D Coarse Matching, and outputs more accurate object poses, which can be used as pick points.

Input and Output

../../../../_images/input_and_output9.png

Parameter Description

Parameter Tuning Level

Default value: Basic

Value list: Basic, Advanced

Instruction: You can choose from the following two parameter tuning levels.

  • Basic: Basic parameter tuning level, provides a few essential parameters that are frequently adjusted.

  • Advanced: Advanced parameter tuning level, which provides more adjustable parameters.

Basic Parameter Tuning Level

This section introduces the parameters in the Basic parameter tuning level, and these parameters are also available in the Advanced parameter tuning level.

Model Settings

Model Selection
Instructions:

  • Click on the drop-down button to select a point cloud model saved in the model gallery (the \resource\3d_matching directory under the project folder).

  • Once you select a point cloud model, its path and the path of its geometric center file are automatically displayed in Model File and Geo Center Point File below.

Note

The drop-down list of the drop-down button only displays the folders containing point cloud models and the associated geometric center/pick point files in the model gallery.

You can use Matching Model and Pick Point Editor to easily generate, edit and save point cloud models and the corresponding geometric centers/pick points.

Model File
Display the path of the selected point cloud model file.
Instructions: click on icon_file to select a point cloud model file in PLY format.
Geo Center Point File
Display the path of the selected geometric center file.
Instructions: click on icon_file to select a geometric center file in JSON format.

Cloud Orientation Calculation

Point Orientation Calc Mode
Default value: Origin
Instruction: This parameter is used to specify the method to calculate the orientations of points in the point cloud. There are 4 methods as shown below. Please select one according to the actual situation of the project.

Value list

Description

Origin

Directly obtain the normal from the point cloud model when the file contains the normal of the point cloud model directly

StandardMode

Select a target, search for the k points closest to the target in the vicinity of the target point, and find the minimum feature vector as the normal by principal component analysis

EdgeTangent

Calculate the tangent at the edge of the point cloud model, and take the tangent as the normal. This method can distinguish different objects whose outer contours are mirror images of each other. This method is recommended when edge matching is performed on relatively flat objects. There are two example usage scenarios as follows:

Scenario 1: The two sides of the object need to be distinguished. The 3D Fine Matching (Multiple Models) Step can be used and there should be two model point cloud models corresponding to the two sides of the object respectively.

Scenario 2: The two sides of the object need NOT to be distinguished. In this case, the 3D Fine Matching Step with one point cloud model is applicable.

EdgeNormal

Calculate the normal at the edge of the point cloud model. This method is recommended when edge matching is performed on relatively flat objects.

Note

When using EdgeTangent or EdgeNormal, please ensure that each edge point cloud does not contain multiple objects, i.e., objects’ point clouds are separate instead of overlapping with each other.

Number of Searching Points
Default value: 10
Instruction: This parameter is for StandardMode mentioned above. It sets the k value, i.e., the number of points in the vicinity of the target point to search for. The minimum value is 1. The larger the value, the more accurate the result, the slower the computation.

Matching Method

Default value: GMM
Value list: GMM, nearest-neighbor
Description: This parameter is used to select the method to match the point cloud model with the object point cloud. The point cloud model will be moved and rotated within the set range to match the object point cloud in the scene obtained from preceding Steps.
Instruction: In most scenarios, GMM has better anti-interference performance and higher matching speed, so it is generally recommended; in rare cases, when GMM cannot meet the requirement, nearest-neighbor can be selected.
../../../../_images/nearest_neighbor.png

Settings under GMM

Matching Mode
Default setting: Standard
Options: HighSpeed, Standard, HighPrecision
Instruction: There are 3 matching modes in total. Please select based on the accuracy and speed requirements.
HighSpeed: The fastest but less accurate.
Standard: Relatively stable.
HighPrecision: The most accurate but slower.
Number of Iterations
Default setting: 30
Description: Iteration refers to iteratively running a specific instruction until the conditions are met. The number of iterations refers to the number of repetitions of instruction running in this process. The parameter set here is the upper limit of the number of iterations. The larger the value, the greater the number of times of matching calculations, the longer the running time, the higher the accuracy.
Standard Deviation
Default setting: 3.000 mm
Description: This value is a measure of the expected deviation of the matched pose of the model from the actual pose of the object after 3D Coarse Matching. The closer the expectation to the actual value, the faster the calculation can be, but a larger Standard Deviation will lead to less accurate matching results. | Example: The figure below is a comparison between matching results when Standard Deviation is 10.000 mm and 40.000 mm respectively. Normally, this value will be no greater than 10 mm, and you can see that the matching result on the right with an overly large standard deviation is less accurate.
../../../../_images/standard_deviation_gmm.png

Hint

For smaller workpieces, it is recommended to adjust the standard deviation to a smaller value for more accurate matching results.

Standard Deviation Update Step Number
Default setting: 3
Description: During the matching, the standard deviation gradually decreases, until it reaches a minimum standard deviation. The value of this parameter is the number of times the standard deviation is updated in the matching algorithm. Normally this parameter does not need to be adjusted.
Speed Up on Large Object Quantities
Default setting: Disabled
Description: It is recommended to enable this option if there is a large number of objects in the scene. If this option is enabled, the Step will perform faster when there is a large number of objects. However, if the number of objects is small, the processing speed of the Step may be slowed down.

Settings under Nearest-Neighbor

Iteration Settings

Number of Iterations
Default setting: 30
Description: Iteration refers to repeatedly running a specific instruction until the conditions are met. The number of iterations refers to the number of repetitions of instruction running in this process. The parameter set here is the upper limit of the number of iterations. The larger the value, the greater the number of times of matching calculations, the longer the running time, the higher the accuracy.
Nearest Point Search Radius
Default setting: 10.000 mm
Description: It adjusts the search radius of the closest point, its unit is m, and its value should correspond to the input initial pose deviation, i.e., the pose deviation after 3D coarse matching. If the deviation is large, the search radius should be large. The value of this parameter should be no lower than 0.001; otherwise, some sparsely distributed points of the object point cloud may not be matched to their nearest points in the point cloud model, thus affecting the output result.
Example: When the initial deviation is large while this parameter is set to a small value such as 1, the matching will be incomplete. When this parameter is increased to a larger value such as 10, the search radius increases, and the matching will be comparatively complete.

Below are two parameters that set the standard by which the matching iteration ends. The iteration ends when either of the standards set by MSE Threshold and Window Size is met.

MSE Threshold
Default setting: 0.001
Description: It adjusts the threshold of the mean square error (MSE), which is a measure of the closeness of the matching. The MSE is recorded and compared with the MSE Threshold after each iteration. When the MSE after an iteration is less than the threshold, the iteration ends.
Window Size
Default setting: 10
Minimum: 2
Description: When the number of iterations during which the deviation’s fluctuation remains trivial reaches the Window Size, the iteration ends. The larger the Window Size, the longer the computation may take, but a Window Size may lead to suboptimal results. For instance, if the Window Size is set to the minimum value 3, the iteration ends when the deviations after three consecutive iterations show trivial fluctuation, and the matching result may not be optimal because had the iteration continued, yet smaller deviation would have been produced.
Show Corresponse
Default setting: Unselected
Description: If this option is selected, the paired point cloud points will be displayed for each iteration.
Complex Object
Default setting: Unselected
Description: It decides whether the algorithm will allocate resources to treat complex object point cloud matching. If the target object is a complex-shaped object (not a simple geometric shape) and Automatic Weight is selected, please select this option.
Automatic Weight
Default setting: Unselected
Description: If this option is selected, the function of automatically applying weight to the point cloud points will be enabled.

Please see Weight in Point Cloud Matching for why and how weight works.

Point Pair Rejection

Reject Pair
Default setting: Unselected
Description: Nearest-neighbor search will find multiple matches for one point, so the function of rejecting pairs should be enabled to filter those matches. If this option is selected, the function will be enabled. N Sigma Threshold and Point Pair Angle Diff Thre are parameters to the function of rejecting invalid pairs. If it is not selected, the two parameters below will not take effect.
N Sigma Threshold
Default setting: 1.0000
Description: It sets the threshold for the distance between the points in a pair. Suppose the mean standard deviation of the distances in all point pairs in the point cloud is σ, the mean of the distances in all point pairs in the point cloud is D. If a point pair’s distance is higher than D + threshold*σ, the pair is deemed invalid and is rejected.
Point Pair Angle Diff Thre
Default setting: 45°
Description: If the angle difference between the normals of the points in a pair is greater than this value, the point pair will be eliminated.
Apply Weights on Pairs
Default setting: Unselected
Description: If this option is selected, the weight will be applied in the iteration. Please see Weight in Point Cloud Matching for why and how weight works.
Remove Repeat Correspond
Default setting: Unselected
Description: If this option is selected, the function of removing redundant point pairs will be enabled, and the settings of Remove Repeat Correspond Type will take effect. It is recommended to select this option when there are more points in the point cloud model than in the point cloud of a single object in the scene.
Remove Repeat Correspond Type
Default setting: MinDis
Options: MinDis, GlobalMean
Description: It is to select the method of finding and deleting the redundant correspondences between a point in the object point cloud and points in the point cloud model.

  • MinDis: Only keep the pair with the shortest distance.

  • GlobalMean: Obtain the mean distance in all pairs, and only keep the pair with the distance closest to the mean distance.

Sample Settings

Sampling Interval
Default setting: 5.000 mm
Description: It is the sampling interval used for downsampling the point cloud model and the object point cloud. The larger the value, the fewer the points will remain in the point clouds, and the lower the accuracy of matching.
Example: The figure below shows the output results when Sampling Interval is set to 5.000 mm and 25.000 mm respectively. The white parts are the input object point clouds, and the colored lines are the matching results. The comparison of the two figures shows that the smaller the sampling interval, the more accurate the output matching result can be.
../../../../_images/output_onetofour.png

Validation Settings

Confidence Threshold
Default setting: 0.500
Value range: [0, +∞)
Description: The confidence threshold is used to determine whether the matching result is valid. Matching results with confidences higher than the threshold are valid and those lower are not. A lower threshold leads to more valid matching results that can be output but the output results may be less accurate, and a higher threshold leads to less valid matching results but higher accuracy of the output results.
Example: The figure below is an example showing two valid matches obtained under small and large confidence thresholds respectively. The confidence threshold of (a) is 0.50 and that of (b) is 0.92. The match of (a) is more accurate than that of (b). Even though the comparison is not prominent, as 3D fine matching itself is a process of fine-tuning, the difference can still be observed, especially in the yellow boxes.

Attention

Even when a higher threshold is set, there may be no valid matching result to output. It may be because that different parts of the target object have the same shape.

../../../../_images/threshold_finematching.png
Search Radius
Default setting: 10.000 mm
Value range: [0, +∞)
Description: The value of Search Radius affects the pose matching score. The higher the value of the Search Radius, the looser the criterion for evaluating the pose matching result will be, which means that even if the pose matching result is of poor quality, the pose matching score will be high; the lower the value of the Search Radius, the more strict the criterion for evaluating the pose matching result will be. Please set the value according to the actual situation of the point cloud. If the points in the point cloud are sparse, the Search Radius should be set slightly larger.

Tip

During parameter tuning, you can set an original Search Radius first and run the Step. Then you can check the matching result in the Debug Output window and compare the result with the pose matching score. If the matching result does not correspond with the matching score, please adjust the Search Radius according to the actual situation.

Output Settings

Maximum Number of Detected Poses in Each Point Cloud
Default setting: 1
Description: The larger the value, the more matching pairs will be output.
Example: The figure below shows the output results when this parameter is set to 1 and 4 respectively.
../../../../_images/default_setting_finematching.png

Results Visualization

Show Matching Results
Default setting: Selected
Description: Select to display the point cloud model and object point clouds.

Advanced Parameter Tuning Level

This section introduces parameters that are only available in the Advanced parameter tuning level. For parameters in the Basic parameter tuning level, please refer to Basic.

Symmetry Settings

When a part of the workpiece to be recognized is symmetrical, you will need to adjust the symmetry settings to escape a local optimum of the matching result. An axis of the reference frame with the geometric center as its origin will be specified as the rotation axis, and the point cloud model will rotate around the specified axis according to the symmetry angle step and therefore the optimal matching result can be obtained.

Symmetry Axis
Default setting: ROTATE_BY_Z
Options: ROTATE_BY_X, ROTATE_BY_Y, ROTATE_BY_Z
Instruction: This parameter specifies the axis of the reference frame with the geometric center as its origin, and the point cloud model will rotate around the specified axis.
Angle Step
Default setting: 360°
Instruction: This parameter specifies the angle of rotational symmetry.
Example: If an object looks the same before and after rotating 60° around an axis, its angle of rotational symmetry is 60°.
Min Rotation Angle
Default setting: -180°
Instruction: This parameter adjusts the minimum rotation angle.
Max Rotation Angle
Default setting: 180°
Instruction: This parameter adjusts the maximum rotation angle.
Example

As shown in the figure below, the main part of the crankshaft, i.e., the part in the frame, has 180° rotational symmetry along the Y-axis.

../../../../_images/3d_fine_matching_add_example1.png
../../../../_images/3d_fine_matching_add_example2.png

If you do not adjust the symmetry settings, there can be mismatches since the shapes of parts of the point cloud model and the crankshaft are similar, as shown below.

../../../../_images/3d_fine_matching_add_example3.png

Only the middle part of the point cloud model and the crankshaft match with each other. However, if the point cloud model rotates 180°, it will match with the entire crankshaft, and the corresponding matching confidence will be higher.

After ROTATE_BY_Y (the Y-axis of the reference frame with the geometric center as its origin) is set as the Symmetry Axis, and the Angle Step is set to 180°, the point cloud model and the crankshaft match with each other correctly, as shown below.

../../../../_images/3d_fine_matching_add_example4.png

Pose Filtering Settings

Filter Poses by Model Rotation Angle
Default setting: Unselected
Description: In the edge matching mode, the point cloud model may need to be rotated by a certain angle to match with the scene point cloud, and the poses will be filtered by the point cloud model’s rotation angle. When the model’s rotation angle exceeds the Angle Difference Upper Threshold, the pose will be filtered out.

Model Weight in Validation

File of Model Fragments with High Weight
Description: This PLY file contains the part of the point cloud model that has a higher weight. The weight value is set by Weight of Each Point below. You need to select the part of point cloud that has a higher weight according to the actual situation of the point cloud.

For the reason of setting the weight, please see Weight in Point Cloud Matching below.

Weight of Each Point
Default setting: 2.0000
Description: This parameter is to set the weight value for the parts with salient features in the point cloud model. The initial weight of all points is 1. After applying the weight, the points in the target part of the point cloud model will have a weight of this parameter.
Search Radius When Set Weight
Default setting: 3.000 mm
Description: This parameter is used to set the search radius when applying the weight. The weight setting is an operation performed on the original point cloud model. The point cloud model will be down-sampled before being used in the matching process, resulting in the shifting or loss of some points that need to be weighted. When setting the weight of the points near a missing point, the search radius needs to be set by this parameter.

Validation Settings

Apply Angle Deviation in Validation
Default setting: Unselected
Description: If this option is selected, during the validation, the angle difference of the normals of the corresponding points in a pair will be considered, and the number of output results will be reduced, but the output will be more accurate.
Multiply Corresponding Points Ratio in Scene
Default setting: Unselected
Description: If this option is selected, the validation score of matching will be multiplied by the ratio of the number of the points with correspondence to the number of points in the entire point cloud. The validation score is printed in the log for debugging, and is printed only when the log level is set to Debug. It should NOT be selected for obtaining the poses of multiple objects in a point cloud.

Results Visualization

Show Sampled Model Cloud
Default setting: Unselected
Description: If this option is selected, the sampled point cloud model will be displayed in the output result. Please set by actual requirements.
Show Sampled Scene Cloud
Default setting: Unselected
Description: If this option is selected, the sampled object point cloud will be displayed in the output result. Please set by actual requirements.
Show Validation Point Correspondences
Default setting: Unselected
Description: If this option is selected, the correspondence between the model and the object point clouds will be displayed in the output result. Please set by actual requirements.

Weight in Point Cloud Matching

Of an object, some parts have salient features, such as the corners, the holes, etc., and those parts should be paid special attention to by the matching algorithm to make the matching easier. Therefore, weights should be applied to the points of those parts in the point cloud so that the algorithm gives importance to those parts. In particular, weight should be applied when matching without weight does not work well.

Normally, among the points in a point cloud in Mech-Vision, a point is either not weighted or weighted by a fixed multiplier.

Yet please note that, after applying weight, mismatches may still occur especially when the object’s different parts are of the same shape or have similar features.