Project Construction Framework

In this section, a simple machine tending project will be constructed as an example to show you the framework of project construction.

The 3D matching algorithm will be used in the project to compute the pick points on the workpieces.

Project Framework

The Steps used in this project include:

../../../_images/project_framework_01.png

Steps related to 3D matching is the core of the project as they are used to obtain the pick point of the object.

Create and Save the Project

Please refer to Create and Save Project for detailed instructions.

Add Steps

After having an overview of the project framework and creating the project, you can start to add Steps and construct the project.

Hint

After creating the project, select the Step you would like to add and drag it to the graphical programming workspace. The Step will be displayed in the graphical programming workspace if it is added successfully.

  1. Add the Capture Images from Camera Step, which is used to trigger the camera to capture images and obtain color images and depth maps.

    ../../../_images/project_framework_11.png

    Attention

    Before adding this Step, please make sure that the camera has been mounted properly and has been connected to the IPC successfully. In addition, an image capturing test has been done and the hand-eye calibration has been carried out.

    After adding the Step, select it and click on the Step Parameters tab. Then you can configure the parameters related to the camera, including Camera Type, Camera ID, Camera Calibration Parameters, and the IP Address.

    ../../../_images/project_framework_1_11.png

  2. Add the From Depth Map to Point Cloud Step, which is used to convert the depth maps to point clouds.

    This Step should follow the Step Capture Images from Camera. Connect the output port for depth maps of Capture Images from Camera to the input port for depth maps of From Depth Map to Point Cloud. Connect the input and output ports for color images in the same way. Keep the default settings on the parameters when running the project.

    ../../../_images/project_framework_21.png

    The two Steps should be connected with each other as shown in the figure above. Then you have added the From Depth Map to Point Cloud Step and input the data you need successfully.

    Hint

    Please click on the output port of the prior Step, press and hold the left mouse button, drag the cursor to the input port of the subsequent Step and then release the mouse button to connect two Steps.

    Attention

    Input ports can only be connected with output ports that share the same data type. For example, the Image/Depth output port can only be connected with the Image/Depth input port.

    After adding the Step, a 2D ROI should be selected to avoid interference from background and outliers and therefore reduce the processing time of subsequent Steps.

    Select the Step, click on the Step Parameters tab, and then click on Set 2D ROI. Please refer to Instructions for Setting 2D ROI for detailed instructions on setting 2D ROI.

    ../../../_images/project_framework_2_11.png

  3. Add the Calc Normals of Point Cloud and Filter It Step, which is used to compute normals of each point in the input point cloud. Therefore, the point cloud that only contains position information can be converted to point cloud with normals.

    This Step will folow the From Depth Map to Point Cloud Step. Connect the output port for point cloud of From Depth Map to Point Cloud to the input port for point cloud of Calc Normals of Point Cloud and Filter It.

    ../../../_images/project_framework_31.png

    After obtaining point cloud with normals, the 3D Coarse Matching and 3D Fine Matching Steps can be used to match the input point cloud with the point cloud model.

  4. Add the 3D Coarse Matching Step, which is used to coarsely match the point cloud model with the input point cloud with normals and then generate the corresponding geometric center and pick point according to the matching result.

    This Step should follow the Calc Normals of Point Cloud and Filter It Step. Connect the output port for point cloud with normals of Calc Normals of Point Cloud and Filter It to the input port for point cloud with normals of 3D Coarse Matching.

    ../../../_images/project_framework_41.png

    After connecting the Steps, please input the point cloud model file and geometric center file that have been made before.

    Select the 3D Coarse Matching Step first, and then click on the Step Parameters tab. Click on icon_1 in the Step Parameters panel and then choose the Model File and Geo Center Point File to input.

    ../../../_images/project_framework_4_11.png

    Hint

    Please refer to Matching Model and Pick Point Editor for instructions on making point cloud model file and geometric center file.

  5. Add the 3D Fine Matching Step, which is used to fine-tune the matching result from 3D Coarse Matching and generate a more precise geometric center and pick point.

    This Step should follow the 3D Coarse Matching Step. Connect the output port for point cloud with normals of Calc Normals of Point Cloud and Filter It to the input port for point cloud with normals of 3D Fine Matching, and connect the output port for initial poses of 3D Coarse Matching to the input port for initial poses of 3D Fine Matching.

    ../../../_images/project_framework_51.png

    After obtaining precise poses via 3D Fine Matching, the obtained poses need to be transformed.

    Hint

    The method to input Model File and Geo Center Point File in 3D Fine Matching is the same as that of 3D Coarse Matching.

  6. Add the Transform Poses Step, which is used to convert the input poses in the camera frame of reference into poses in the robot frame of reference.

    This Step should follow the 3D Fine Matching Step. Connect the output port for pick point poses of 3D Fine Matching to the input port for original poses of Transform Poses. Keep the default settings on the parameters when running the project.

    ../../../_images/project_framework_61.png

  7. Add the Procedure Out Step, which is used to send the input data to Mech-Center, which will send the data to relevant software and devices.

    This Step should follow the Transform Poses Step. Connect the output port for transformed poses of Transform Poses to the input port for poses (the first port on the left) of Procedure Out. The <Abstract> poses will turn to <PoseList> poses, as shown in the figure below.

    ../../../_images/project_framework_7.png

    Now you have completed constructing a simple matching project. You can use the shortcut Ctrl + S or go to File ‣ Save Project to save the project.

    Tip

    After adding and connecting all the Steps, you can arrange the layout of the Steps to make the logic of the data flow clearer. There are 3 ways to arrange the layout of the Steps.

    • Select the Step(s) and drag it/them to arrange the layout yourself.

    • Right click on an empty space, and select Format.

    • Use the shortcut Alt + Shift + F.

After the construction of the project is completed, you can start to run and debug the project.