3D Fine Matching (Multiple Models)

Function

Use multiple models to accurately match objects in the scene, and output accurate pick points of the target objects.

Sample Scenario

This Step performs more accurate matching based on the initial candidate poses output by the Step 3D Coarse Matching (Multiple models) and outputs accurate poses of the target object.

Input and Output

Parameters

Parameters Tuning Level

Default Value: Basic

List of Values: Basic, Advanced

Instruction: Two different parameter tuning levels can be set:

• Basic: Basic adjustment, with a small amount of adjustable parameters;

• Advanced: Advanced adjustment, with more adjustable parameters.

Basic Mode

This chapter introduces the adjustable parameters in Basic Mode. These parameters can also be configured in Advanced Mode.

Model and Pick Point

The path of model file and pick point file.

Model File

Default Value：model.ply

Instruction：This parameter is used to set the path of the 3D model file. If the file path or name is changed, the file will be re-read when the step is called next time. Both full path and relative path of the model file can be recognized.

Geometric Center File

Default Value：model/pickPoint.json

Instruction：The pick point file is based on the 3D model file. For the method of generating pick point files, please refer to Add Pick Point. If the file path or name is changed, the file will be re-read when the step is called next time. Both full path and relative path of the model file can be recognized.

Model Label File

Instruction: Fill in the path of the model label file in json format. Multiple file paths are allowed to be set and separated by semicolons. If the file path or name is changed, the file will be re-read when this Step is called next time.

Example: Ensure that the order of the file paths is consistent: if the front file path of an object is entered first in the Model Label File, the corresponding positive file path should also be entered before the negative one in both Model File and Geometric Center File, as shown in Figure 1.

Figure 1 Model and Pick Point File Path Example

After inputting file paths correctly, the output point cloud image is shown in Figure 2. Both the front and back templates of the object can match the scene point cloud well.

Figure 2 Multi-Template 3D Fine Matching Output

Cloud Orientation Calculation

Point Orientation Calc Mode

Default Value: Origin

List of Values: Origin, StandardMode, EdgeTangent, EdgeNormal

Instruction: There are four methods for calculating the direction of the point, which can be selected according to the actual situation of the project:

• Origin: Directly use the original normal direction of the input point cloud;

• StandardMode : Use the CPU to recalculate the normal direction of the input point cloud. It is recommended to use this mode when the model file doesn’t have normal direction: Search for the k points closest to the target point, and use PCA (Principal Component Analysis) to obtain the minimum eigenvector as the point’s normal direction.

• EdgeTangent : Calculate the tangent of the input edge point cloud, and use the tangent direction of the point as the normal direction. This mode can be used when matching the edge point cloud of a flat object. There are two usage scenarios as follows:

  • Scenario 1: Distinguish the front (positive) and back (negative) of the object. Use the 3D Fine Matching (Multiple Models) step, and the model point cloud is the positive and negative template of the object;

  • Scenario 2: No need to distinguish the front and back of the object. Use the 3D Fine Matching (single template) step.

• EdgeNormal : Calculate the normal direction of the input edge point cloud. This mode can also be used when matching the edge point cloud of a flat object.

  Note

  When using EdgeTangent or EdgeNormal methods, please ensure that each edge point cloud does not contain multiple objects (each object’s point cloud has been divided already).

Number of Searching Points

Default Value：10

Instruction: This parameter is used to set the number of used points when calculating the orientation. The minimum value is 1.

Correspondence Settings

Default Value: GMM

List of Values: GMM, nearest-neighbor

Instruction: This parameter is used to select the local matching method to match the model point cloud with the scene point cloud. The model point cloud will move and rotate according to the set range to match the point cloud in the scene obtained in the previous step match. In most scenarios, GMM has better anti-interference ability and faster matching speed, so it is recommended to use this mode; in rare cases when GMM cannot meet the demand, the nearest point (nearest-neighbor) mode is worth to try.

GMM Mode

Matching Mode

Default Value: Standard

List of Values: HighSpeed, Standard, HighPrecision

Instruction: There are three matching modes, and select the appropriate one according to the project requirements:

• HighSpeed: The fastest mode, but with lower precision;

• Standard: A relatively stable mode;

• HighPrecision: The highest precision mode, but with slower speed.

Number of Iterations

Default Value: 30

Instruction: Iteration is the process of cyclically running for a specific instruction until the conditions are met. Number of Iterations is the number of repeated instructions in this process. The parameter set here is the maximum number of iterations. The larger this value, the more matching calculation times and the longer running time. However, larger Number of Iterations also gives better matching accuracy.

Standard Deviation

Default Value: 0.005

Instruction: The value here needs to macth with the initial pose deviation. When the input initial pose deviation after the previous step (3D Coarse Matching) is greater, a larger standard deviation is required. In this case, the matching speed is faster. If the set value is greater than the deviation of the initial pose, the accuracy of the output result will be greatly reduced. Normally, the value of this parameter setting will not exceed 0.01m.

Standard Deviation Update Step Number

Default Value: 3

Instruction: During the matching process, the Standard Deviation gets smaller and smaller until it reaches the value of Minimum Standard Deviation. The number of times that the standard deviation decays in this process is the value of this parameter. For some cases with large standard deviation, the more attenuation steps, the more stable the entire attenuation process. Under normal circumstances, this parameter does not need to be adjusted.

Minimum Standard Deviation

Default Value: 0.001

Instruction: The lower limit of the Standard Deviation .

Apply Weight in Iteration

Default Value: False

List of Values: True, False

Instruction: If checked, the weight of the model will be used during the iteration of the algorithm.

Nearest-Neighbor Mode

Iteration Settings

Number of Iterations

Default Value: 30

Instruction: Iteration is the process of cyclically running for a specific instruction until the conditions are met. Number of Iterations is the number of repeated instructions in this process. The parameter set here is the maximum number of iterations. In the nearest point mode, the iteration stops when the calculation meets the accuracy requirements of the algorithm. The larger this value, the more matching calculation times and the longer running time. However, larger Number of Iterations also gives better matching accuracy.

Nearest Points Search Radius

Default Value: 0.01 m

Instruction: This parameter is used to adjust the search radius of the nearest point (in m), and the value setting should correspond to the input initial pose deviation. When the input initial pose deviation is large (the deviation between the scene point cloud and the model point cloud is large), this parameter needs to be increased; when the initial pose deviation is very small, this parameter can be adjusted to a small value. The minimum value that can be set for this parameter is 0.001 . If the radius is less than this lower limit, some sparsely distributed points may not find the closest point, which could affect the output result.

Example: When the initial pose deviation is very large and this parameter is set as the minimum value 0.001, the matching result will be incomplete, as shown by the (b) in Figure 3; When this parameter is increased to 0.01, the search radius will become larger, and a complete matching result can be obtained, as shown by (a) in Figure 3.

Figure 3 Comparison of Matching Results with Different Search Radius

MSE Threshold

Default Value: 0.001

Instruction: This parameter (Mean Square Error Threshold) is used to adjust the upper limit of the mean square error. This value is recorded after each iteration and used for comparison in subsequent iteration steps. When the mean square error of an iteration is less than the set value, it is considered that the iteration effect has met the requirements and the iteration will be terminated.

Window Size

Default Value: 10

Instruction: This parameter refers to the number of consecutive iterations with small error fluctuations in the iterative optimization process. When this parameter is too small, a local optimal solution may appear, which affects the final matching accuracy.

Example: If the window only contains the error of the results of three consecutive iterations, so the error trend can only be obtained by comparing the errors of the three iterations, which may lead to a local optimal solution (the error of the second iteration is less than the error of the first iteration and the first iteration) instead of a global optimal solution. According to the recommended value of the number of iterations which is 30, subsequent iterations with more significant error reduction may occur.

Show Correspondence

Default Value: False

List of Values: True, False

Instruction: If True is checked, the corresponding points will be displayed in each iteration.

Complex Object

Default Value: False

List of Values: True, False

Instruction: Check True if the target object is a complex shape object (not a simple geometric shape) or if the Automatic Weight function is required.

Automatic Weight

Default Value: False

List of Values: True, False

Instruction: If True is checked, the weight of iteration points will be calculated automatically. It is recommended to use this function for objects with poor coarse matching results.

Point Pair Rejection

Reject Pair

Default Value: False

List of Values: True, False

Instruction: This parameter decides whether to remove repeated point pairs. There are multiple point pairs in the process of searching for the nearest point, and sometimes the unwanted point pairs need to be filtered out. If True is checked, this function will be used to remove them through configurating the following N Sigma Threshold and Point Pair Angle Diff Thre parameters; if not checked, these two parameters will not take effect during the matching process.

N Sigma Threshold

Default Value: 1.0000

Instruction: This parameter is the multiplier of the standard deviation threshold of the distance between each points. The point pair whose distance is outside N sigma will be eliminated.

Point Pair Angle Diff Thre

Default Value: 45°

Instruction: This parameter is the upper limit of the angle difference (in °) between the normals of two points in every matching point pair. If the angle difference is greater than this value, this point pair will be eliminated.

Apply Weights on Pairs

Default Value: False

List of Values: True, False

Instruction: If True is checked, the weight of the model will be used in the iteration of the algorithm, so it is more likely to match better.

Remove Repeat Correspond

Default Value: False

List of Values: True, False

Instruction: If True is checked, the function of removing repeating corresponding point pairs will be used, and the Remove Repeat Correspond Type parameter will be available and could take effect during the matching process. It is recommended to use this function when the model point cloud has more points than that of a single object in the scene.

Remove Repeat Correspond Type

Default Value: MinDis

List of Values: MinDis, GlobalMin

Instruction: This parameter is used to select the method of removing repeating corresponding point pairs. There are two methods in total:

• MinDis : Calculate the distance between the repetitive point pairs. Then, keep the point pair with the smallest distance, and delete the remaining repetitive point pairs. More repeating corresponding point pairs will be eliminated in this method than the other;

• GlobalMin : Take the average value of the distance between the repetitive point pairs as a reference to find out the point pair whose distance is closest to the average value. Then, retain this point pair and eliminate others. Less repeating corresponding point pairs will be eliminated in this method than the other.

Sampling Settings

Sampling Interval

Default Value: 0.005

Instruction: This parameter is used for the downsampling of the model and scene point cloud. The bigger the interval, the less the point number of the sampled point cloud, and the lower the accuracy of model estimation.

Example: Figure 4 and Figure 5 are the output results when this parameter is set as 0.005 and 0.025, respectively. The white one is the input scene point cloud. It can be seen from the comparison of the two pictures that the smaller the sampling interval (Figure 4), the more accurate the output result will be.

Figure 4 Output with a Small Sampling Interval

Figure 5 Output with a Large Sampling Interval

Search Points Number

Default Value: 20

Instruction: When Parameters Tuning Level is Advanced , this parameter can be adjusted. This parameter is reserved for compatibility with old versions and is used to adjust the number of search points in the sampling process.

Validation Settings

Confidence Threshold

Default Value: 0.5

Instruction: If the validation score of the result is larger than this threshold (which cannot be negative), the result is regarded as valid. The higher the value is, the more accurate results are retained.

Result evaluation search radius

Default Value: 0.01 m

Instruction: This parameter is the search radius when calculating the confidence level, which cannot be negative. It is set according to the point cloud situation of the object and its value cannot be negative. If the point cloud is sparse, this value needs to be set slightly larger.

Output Settings

Maximum Number of Detected Poses in Each Point

Default Value: 1

Instruction: The larger the value, the more matching output results.

Example: By adjusting this parameter, different matching output results can be obtained. Figure 6 is the output result of the parameter when the default value is 1, Figure 7 is the output when the parameter is adjusted to 4.

Figure 6 Single Result

Figure 7 Multiple Results

Results Visualization

Show Matching Results

Default Value: True

Instruction: If True is checked, the estimated pose of objects (by rendering the transformed model and the scene point cloud) will be visualized.

Advanced Mode

This chapter only introduces the parameters that Basic Mode does not contain. If there are some parameters that are the same as that in the Basic Mode, please refer to Basic Mode.

Symmetry Settings

Symmetry Axis

Default Value: ROTATE_BY_Z

List of Values: ROTATE_BY_X, ROTATE_BY_Y, ROTATE_BY_Z

Instruction: This parameter is used to select the symmetry rotation axis around which the initial pose is changed.

Angle Step

Default Value: 360°

Instruction: This parameter is used for changing the initial orientation from Min Rotation Angle to Max Rotation Angle .

Min Rotation Angle

Default Value: -180°

Instruction: This parameter is used to adjust the minimum rotation angle.

Max Rotation Angle

Default Value: 180°

Instruction: This parameter is used to adjust the maximum rotation angle.

Model Weight in Validation

File of Model Fragments with High Weight

Instruction: This file (ply format) is a fragment of Model File. Based on the actual workpiece, determine which part is selected as the high weight file. The points in this file have a high weight set by the Weight of Each Point .

Weight of Each Point

Default Value: 2.0

Instruction: This parameter is used to set the weight value for each point in the weight file, and there is no unit. The higer the value is, the better the matching result of the model fragments tend to be.

Example: Assuming that the weight coefficient of all initial point clouds is 1, after setting this parameter for the target point cloud, the weight coefficient of the target point cloud will become [1*this parameter], so as to achieve the purpose of emphasis in the subsequent matching steps.

Search Radius When Set Weight

Default Value: 0.0030

Instruction: This parameter is used to set the radius for setting weights. The bigger the value is, the more points in the model will have the set weight. The weight setting is an operation performed on the original model point cloud. The model point cloud will be down-sampled before it is used in the matching process, which causes some points to be shifted or lost. In the process of setting the weight of the points near the missing point, the search radius needs to be set, and the search radius is set by this parameter.

Validation Settings

Apply Angle Deviation in Validation

Default Value: False

List of Values: True, False

Instruction: If True is checked, the angle difference of the normal vector of the corresponding point pair will be considered in the pose validation process. In this case, although the output is often reduced, it could be more accurate.

Multiply Corresponding Points Ratio in Scene

Default Value: False

List of Values: True, False

Instruction: If True is checked, the matching score will be multiplied by the proportional coefficient of the corresponding point to the scene point. It is not suitable for the case of estimating poses of multiple objects in one point cloud.

Accleration Settings

Accleration with GPU

Default Value: False

List of Values: True, False

Instruction: If True is checked, GPU acceleration will be enabled to speed up the process.

Result Visualization

Show Sampled Model Cloud

Default Value: False

List of Values: True, False

Instruction: If you check True, the sampled model point cloud will be visualized in the output.

Show Sampled Scene Cloud

Default Value: False

List of Values: True, False

Instruction: If True is checked, the sampled scene point cloud will be visualized in the output.

Show Validation Point Correspondences

Default Value: False

List of Values: True, False

Instruction: If True is checked, the corresponding relationship between the model and the scene point cloud will be visualized in the output.