3D Fine Matching (Multiple Models)

Function

Use multiple models to accurately match objects in the scene, and output accurate pick points of the target objects.

Usage Scenario

This Step performs more accurate matching based on the initial candidate poses output by the Step 3D Coarse Matching (Multiple Models) and outputs accurate poses of the target object.

Input and Output

Parameter Description

Parameter Tuning Level

Default value: Basic

List of values: Basic, Advanced

Instruction: Two different parameter tuning levels can be set:

• Basic: Basic adjustment, with a small amount of adjustable parameters;

• Advanced: Advanced adjustment, with more adjustable parameters.

Basic Mode

This chapter introduces the adjustable parameters in Basic Mode. These parameters can also be configured in Advanced Mode.

Model Settings

The path of model file and pick point file.

Model File (Compulsory)

Default value: model.ply

Instruction: This parameter is used to set the path of the 3D model file. If the file path or name is changed, the file will be re-read when the step is called next time. Both full path and relative path of the model file can be recognized.

Geo Center Point File (Compulsory)

Instruction: The geometric center file in JSON format. You can enter multiple file paths. Please use semicolons to divide different file paths.

Model Label File (Optional)

Instruction: The model label file in JSON format. You can enter multiple file paths. Please use semicolons to divide different file paths.

Example: Ensure that the order of the file paths is consistent, i.e., if the file path of an object with front side facing up is entered first in the Model Label File, the corresponding front side object file path should also be entered before the back side one in both Model File and Geometric Center File, as shown below.

After inputting file paths correctly, the output point cloud image is shown as below. Both the front and back point cloud models of the object can match the scene point cloud well.

Cloud Orientation Calculation

Point Orientation Calc Mode

Default value: Origin

Instruction: This parameter is used to specify the method to calculate the orientations of points in the point cloud. There are 4 methods as shown below. Please select one according to the actual situation of the project.

Value list	Description
Origin	Directly use the original normal direction of the input point cloud
StandardMode	Use the CPU to recalculate the normal direction of the input point cloud. It is recommended to use this mode when the model file doesn’t have normal direction: Search for the k points closest to the target, and use PCA (Principal Component Analysis) to obtain the minimum eigenvector as the point’s normal direction.
EdgeTangent	Calculate the tangent of the input edge point cloud, and use the tangent direction of the point as the normal direction. This mode can be used when matching the edge point cloud of a flat object. There are two usage scenarios as follows:
	Scenario 1: The two sides of the object need to be distinguished. The 3D Fine Matching (Multiple Models) Step can be used and there should be two model point cloud models corresponding to the two sides of the object respectively.
	Scenario 2: The two sides of the object need NOT to be distinguished. In this case, the 3D Fine Matching Step with one point cloud model is applicable.
EdgeNormal	Calculate the normal at the edge of the point cloud model. This method is recommended when edge matching is performed on relatively flat objects.

Note

When using EdgeTangent or EdgeNormal, please ensure that each edge point cloud does not contain multiple objects, i.e., objects’ point clouds are separate instead of overlapping with each other.

Number of Searching Points

Default value: 10

Instruction: This parameter is used to set the number of points in the vicinity of the target point to search for. The minimum value is 1.. The minimum value is 1.

Correspondence Settings

Default value: GMM

List of values: GMM, nearest-neighbor

Instruction: This parameter is used to select the local matching method to match the point cloud model with the scene point cloud. The point cloud model will move and rotate according to the set range to match the point cloud in the scene obtained in the previous step match. In most scenarios, GMM has better anti-interference ability and faster matching speed, so it is recommended to use this mode; in rare cases when GMM cannot meet the demand, the nearest point (nearest-neighbor) mode is worth to try.

GMM Mode

Matching Mode

Default value: Standard

List of values: HighSpeed, Standard, HighPrecision

Instruction: There are three matching modes, and select the appropriate one according to the project requirements:

• HighSpeed: The fastest mode, but with lower precision;

• Standard: A relatively stable mode;

• HighPrecision: The highest precision mode, but with slower speed.

Number of Iterations

Default value: 30

Instruction: Iteration is the process of cyclically running for a specific instruction until the conditions are met. Number of Iterations is the number of repeated instructions in this process. The parameter set here is the maximum number of iterations. The larger this value, the more matching calculation times and the longer running time. However, larger Number of Iterations also gives better matching accuracy.

Standard Deviation

Default value: 0.005

Instruction: The value here needs to macth with the initial pose deviation. When the input initial pose deviation after the previous step (3D Coarse Matching) is greater, a larger standard deviation is required. In this case, the matching speed is faster. If the set value is greater than the deviation of the initial pose, the accuracy of the output result will be greatly reduced. Normally, the value of this parameter setting will not exceed 0.01m.

Hint

When matching small objects, it is recommended to set the standard deviation a smaller value for better performance.

Standard Deviation Update Step Number

Default value: 3

Instruction: During the matching process, the Standard Deviation gradually decreases until it reaches the value of Minimum Standard Deviation. The number of times that the standard deviation decreases in this process is the value of this parameter. For some cases with large standard deviation, the more times of the decrease, the more stable the entire decrease process. Under normal circumstances, this parameter does not need to be adjusted.

Speed Up on Large Object Quantities

Default setting: Disabled

Description: It is recommended to enable this option if there is a large number of objects in the scene. If this option is enabled, the Step will perform faster when there is a large number of objects. However, if the number of objects is small, the processing speed of the Step may be slowed down.

Nearest-Neighbor Mode

Iteration Settings

Number of Iterations

Default value: 30

Instruction: Iteration is the process of cyclically running for a specific instruction until the conditions are met. Number of Iterations is the number of repeated instructions in this process. The parameter set here is the maximum number of iterations. In the nearest point mode, the iteration stops when the calculation meets the accuracy requirements of the algorithm. The larger this value, the more matching calculation times and the longer running time. However, larger Number of Iterations also gives better matching accuracy.

Nearest Points Search Radius

Default value: 0.01 m | Instruction: This parameter is used to adjust the search radius of the nearest point (in m), and the value setting should correspond to the input initial pose deviation. When the input initial pose deviation is large (the deviation between the scene point cloud and the point cloud model is large), this parameter needs to be increased; when the initial pose deviation is very small, this parameter can be adjusted to a small value. The minimum value that can be set for this parameter is 0.001 . If the radius is less than this lower limit, some sparsely distributed points may not find the closest point, which could affect the output result.

Example: When the initial pose deviation is very large and this parameter is set as the minimum value 0.001, the matching result will be incomplete, as shown by the (b) in the figure below. When this parameter is increased to 0.01, the search radius will become larger, and a complete matching result can be obtained, as shown by (a) in the figure below.

MSE Threshold | Default value: 0.001

Instruction: This parameter (Mean Square Error Threshold) is used to adjust the upper limit of the mean square error. This value is recorded after each iteration and used for comparison in subsequent iteration steps. When the mean square error of an iteration is less than the set value, it is considered that the iteration effect has met the requirements and the iteration will be terminated.

Window Size

Default value: 10

Instruction: This parameter refers to the number of consecutive iterations with small error fluctuations in the iterative optimization process. When this parameter is too small, a local optimal solution may appear, which affects the final matching accuracy.

Example: If the window only contains the error of the results of three consecutive iterations, so the error trend can only be obtained by comparing the errors of the three iterations, which may lead to a local optimal solution (the error of the second iteration is less than the error of the first iteration and the first iteration) instead of a global optimal solution. According to the recommended value of the number of iterations which is 30, subsequent iterations with more significant error reduction may occur.

Show Corresponse

Default value: Unselected

List of values: Selected, Unselected

Instruction: If it is selected, the corresponding points will be displayed in each iteration.

Complex Object

Default value: Unselected

List of values: Selected, Unselected

Instruction: Selected if the target object is a complex-shaped object (not a simple geometric shape) or if the Automatic Weight function is required.

Automatic Weight

Default value: Unselected

List of values: Selected, Unselected

Instruction: If it is selected, the weight of iteration points will be calculated automatically. It is recommended to use this function for objects with poor coarse matching results.

Point Pair Rejection

Reject Pair

Default value: Unselected

List of values: Selected, Unselected

Instruction: This parameter decides whether to remove repeated point pairs. There are multiple point pairs in the process of searching for the nearest point, and sometimes the unwanted point pairs need to be filtered out. If this option is selected, this function will be used to remove them through configurating the following N Sigma Threshold and Point Pair Angle Diff Thre parameters; if not checked, these two parameters will not take effect during the matching process.

N Sigma Threshold

Default value: 1.0000

Instruction: This parameter is the multiplier of the standard deviation threshold of the distance between each points. The point pair whose distance is outside N sigma will be eliminated.

Point Pair Angle Diff Thre

Default value: 45°

Instruction: This parameter is the upper limit of the angle difference (in °) between the normals of two points in every matching point pair. If the angle difference is greater than this value, this point pair will be eliminated.

Apply Weights on Pairs

Default value: Unselected

Instruction: If it is selected, the weight of the model will be used in the iteration of the algorithm, so it is more likely to match better.

Remove Repeat Correspond

Default value: Unselected

Instruction: If it is selected, the function of removing repeating corresponding point pairs will be used, and the Remove Repeat Correspond Type parameter will be available and could take effect during the matching process. It is recommended to use this function when the point cloud model has more points than that of a single object in the scene.

Remove Repeat Correspond Type

Default value: MinDis

List of values: MinDis, GlobalMin

Instruction: This parameter is used to select the method of removing repeating corresponding point pairs. There are two methods in total:

MinDis : Calculate the distance between the repetitive point pairs. Then, keep the point pair with the smallest distance, and delete the remaining repetitive point pairs. More repeating corresponding point pairs will be eliminated in this method than the other;

GlobalMin : Take the average value of the distance between the repetitive point pairs as a reference to find out the point pair whose distance is closest to the average value. Then, retain this point pair and eliminate others. Less repeating corresponding point pairs will be eliminated in this method than the other.

Sampling Settings

Sampling Interval

Default value: 0.005

Instruction: This parameter is used for the downsampling of the model and scene point cloud. The bigger the interval, the less the point number of the sampled point cloud, and the lower the accuracy of model estimation.

Example: The left and right figures below are the output results when this parameter is set as 0.005 and 0.025 respectively. The white one is the input scene point cloud. It can be seen from the comparison of the two pictures that the smaller the sampling interval, the more accurate the output result will be.

Search Points Number

Default value: 20

Instruction: When Parameter Tuning Level is Advanced , this parameter can be adjusted. This parameter is reserved for compatibility with old versions and is used to adjust the number of search points in the sampling process.

Validation Settings

Confidence Threshold

Default value: 0.5

Value range: [0, +∞)

Instruction: If the validation score of the result is larger than this threshold (which cannot be negative), the result is regarded as valid. The higher the value is, the more accurate results are retained.

Result evaluation search radius

Default value: 0.01 m

Value range: [0, +∞)

Instruction: This parameter is the search radius when calculating the confidence level, which cannot be negative. It is set according to the point cloud situation of the object and its value cannot be negative. If the point cloud is sparse, this value needs to be set slightly larger.

Output Settings

Maximum Number of Detected Poses in Each Point

Default value: 1

Instruction: The larger the value, the more matching output results.

Example: By adjusting this parameter, different matching output results can be obtained. The figure below shows the output result of the parameter when the value is set to 1.

The figure below shows the output when the parameter is adjusted to 4.

Results Visualization

Show Matching Results

Default value: Selected

Instruction: If this option is selected, the output model and scene point cloud will be displayed.

Advanced Mode

This chapter only introduces the parameters that Basic Mode does not contain. If there are some parameters that are the same as that in the Basic Mode, please refer to Basic Mode.

Symmetry Settings

When a part of the workpiece to be recognized is symmetrical, you will need to adjust the symmetry settings to escape a local optimum of the matching result. An axis of the reference frame with the geometric center as its origin will be specified as the rotation axis, and the point cloud model will rotate around the specified axis according to the symmetry angle step and therefore the optimal matching result can be obtained.

Symmetry Axis

Default value: ROTATE_BY_Z

List of values: ROTATE_BY_X, ROTATE_BY_Y, ROTATE_BY_Z

Instruction: This parameter is used to select the symmetry rotation axis around which the initial pose is changed.

Angle Step

Default value: 360°

Instruction: This parameter is used for changing the initial orientation from Min Rotation Angle to Max Rotation Angle .

Min Rotation Angle

Default value: -180°

Instruction: This parameter is used to adjust the minimum rotation angle.

Max Rotation Angle

Default value: 180°

Instruction: This parameter is used to adjust the maximum rotation angle.

Example

As shown in the figure below, the main part of the crankshaft, i.e., the part in the frame, has 180° rotational symmetry along the Y-axis.

If you do not adjust the symmetry settings, there can be mismatches since the shapes of parts of the point cloud model and the crankshaft are similar, as shown below.

Only the middle part of the point cloud model and the crankshaft match with each other. However, if the point cloud model rotates 180°, it will match with the entire crankshaft, and the corresponding matching confidence will be higher.

After ROTATE_BY_Y (the Y-axis of the reference frame with the geometric center as its origin) is set as the Symmetry Axis, and the Angle Step is set to 180°, the point cloud model and the crankshaft match with each other correctly, as shown below.

Model Weight in Validation

When the workpieces to be recognized have the similar main body, while their specific parts differ greatly, the weight of the “parts” should be higher to output a correctly matched result.

File of Model Fragments with High Weight

Instruction: This file (PLY format) is a fragment of Model File. Based on the actual workpiece, determine which part is selected as the high weight file. The points in this file have a high weight set by the Weight of Each Point .

Weight of Each Point

Default value: 2.0

Instruction: This parameter is used to set the weight value for each point in the weight file, and there is no unit. The higer the value is, the better the matching result of the model fragments tend to be.

Example: Assuming that the weight coefficient of all initial point clouds is 1, after setting this parameter for the target cloud, the weight coefficient of the target cloud will become \(1 X this parameter\), so as to achieve the purpose of emphasis in the subsequent matching steps.

Search Radius When Set Weight

Default value: 0.0030

Instruction: This parameter is used to set the radius for setting weights. The bigger the value is, the more points in the model will have the set weight. The weight setting is an operation performed on the original point cloud model. The point cloud model will be down-sampled before it is used in the matching process, which causes some points to be shifted or lost. In the process of setting the weight of the points near the missing point, the search radius needs to be set, and the search radius is set by this parameter.

Validation Settings

Apply Angle Deviation in Validation

Default value: Unselected

Instruction: If it is selected, the angle difference of the normal vector of the corresponding point pair will be considered in the pose validation process. In this case, although the output is often reduced, it could be more accurate.

Multiply Corresponding Points Ratio in Scene

Default value: Unselected

Instruction: If it is selected, the matching score will be multiplied by the proportional coefficient of the corresponding point to the scene point. It is not suitable for the case of estimating poses of multiple objects in one point cloud.

Results Visualization

Show Sampled Model Cloud

Default value: Unselected

Instruction: If it is selected, the sampled point cloud model will be visualized in the output.

Show Sampled Scene Cloud

Default value: Unselected

Instruction: If it is selected, the sampled scene point cloud will be visualized in the output.

Show Validation Point Correspondences

Default value: Unselected

Instruction: If it is selected, the corresponding relationship between the model and the scene point cloud will be visualized in the output.