Solution Deployment
This section introduces the deployment of the Small Sheet Metal Parts solution. The overall process is shown in the figure below.
Vision System Hardware Setup
Vision system hardware setup refers to integrating the hardware (camera and industrial PC) into the actual environment to support the normal operation of the vision system.
In this phase, you need to install and set up the hardware of the vision system. For details, refer to Vision System Hardware Setup.
Robot Communication Configuration
Before robot communication configuration, it is necessary to obtain the solution first. Click here to see how to obtain the solution.
-
Open Mech-Vision.
-
In the Welcome interface of Mech-Vision, click Create from solution library to open the Solution Library.
-
Enter the Typical cases category in the Solution Library, click the
icon in the upper right corner for more resources, and then click the Confirm button in the pop-up window. -
After acquiring the solution resources, select the Small Sheet Metal Parts solution under the Randomly-stacked part picking category, fill in the Solution name and Path at the bottom, and finally click the Create button. Then, click the Confirm button in the pop-up window to download the Small Sheet Metal Parts solution.
Once the solution is downloaded, it will be automatically opened in Mech-Vision.
Before deploying a vision solution, you need to set up the communication between the Mech-Mind Vision System and the robot side (robot, PLC, or host computer).
The Small Sheet Metal Parts solution uses Standard Interface communication. For detailed instructions, please refer to Standard Interface Communication Configuration.
Hand-Eye Calibration
Hand-eye calibration establishes the transformation relationship between the camera and robot reference frames. With this relationship, the object pose determined by the vision system can be transformed into that in the robot reference frame, which guides the robot to perform its tasks.
Please refer to Robot Hand-Eye Calibration Guide and complete the hand-eye calibration.
|
Every time the camera is mounted, or the relative position of the camera and the robot changes after calibration, it is necessary to perform hand-eye calibration again. |
Vision Project Configuration
After completing the communication configuration and hand-eye calibration, you can use Mech-Vision to configure the vision project.
The process of how to configure a vision project is shown in the figure below.
Connect to the Camera and Capture Images
-
Connect to the camera.
Open Mech-Eye Viewer, find the camera to be connected, and click the Connect button.
-
Adjust camera parameters.
To ensure that the captured 2D image is clear and the point cloud is intact, you need to adjust the camera parameters. For detailed instructions, please refer to PRO S Camera Parameter Reference.
-
Capture images.
After the camera is successfully connected and the parameter group is set, you can start capturing the target object images. Click the
button on the top to capture a single image. At this time, you can view the captured 2D image and point cloud of the target object. Ensure that the 2D image is clear, the point cloud is intact, and the edges are clear. The qualified 2D image and point cloud of the target object are shown on the left and right in the figure below respectively.
-
Connect to the camera in Mech-Vision.
Select the Capture Images from Camera Step, disable the Virtual Mode option, and click the Select camera button.
In the pop-up window, click the
icon on the right of the camera serial number. When the icon turns into 
, the camera is connected successfully. After the camera is connected successfully, you can select the camera calibration parameter group in the drop-down list on the right, as shown below.
Now that you have connected to the real camera, you do not need to adjust other parameters. Click the
icon on the Capture Images from Camera Step to run the Step. If there is no error, the camera is connected successfully and the images can be captured properly.
3D Target Object Recognition
The Small Sheet Metal Parts solution uses the 3D Target Object Recognition Step to recognize target objects. Click the Config wizard button in the Step Parameters panel of the 3D Target Object Recognition Step to open the 3D Target Object Recognition tool to configure relevant settings. The overall configuration process is shown in the figure below.
Point Cloud Preprocessing
Before point cloud preprocessing, you need to preprocess the data by adjusting the parameters to make the original point cloud clearer, thus improving the recognition accuracy and efficiency.
-
Set an effective recognition area to block out interference factors and improve recognition efficiency.
-
Set the Edge extraction effect, Noise removal level, and Point filter parameters to remove noise.
After point cloud preprocessing, click the Run Step button.
Recognize Target Object
After point cloud preprocessing, you need to create a point cloud model for the target object in the Target Object Editor, and then set matching parameters in the 3D Target Object Recognition tool for point cloud model matching.
-
Create a target object model.
-
Create point cloud model and add the pick point. Click the Open target object editor button to open the editor, import the STL file to generate a point cloud model for the target object.
-
-
Set parameters related to object recognition.
-
Enable Advanced mode on the right side of Recognize target object.
-
Matching mode: Enable Auto-set matching mode. Once enabled, this Step will automatically adjust the parameters under Coarse matching Settings and Fine matching settings.
-
Coarse matching settings: To reduce the time consumed by matching, set the Performance mode to Custom and set the Expected point count of model to 300.
-
Confidence settings: Set the Confidence strategy to Auto and the Confidence threshold to 0.5 for removing incorrect matches.
-
Output—Max outputs: Minimize the number of outputs to reduce matching time, while ensuring that path planning requirements are met. In this solution, the Max outputs parameter is set to 20.
-
Remove coinciding poses and remove overlapped poses: To remove coinciding and overlapping recognition results, enable the Remove poses of coinciding objects and Remove poses of overlapped objects options, and set their respective thresholds to 30% and 20%.
-
After setting the above parameters, click the Run Step button. The matching result is shown in the figure below.
Configure Step Ports
After target object recognition, Step ports should be configured to provide vision results and point clouds for Mech-Viz for path planning and collision detection.
Since the pick points should be processed in the subsequent workflow, select the Port(s) related to pick point option under Select port. To recognize and locate the bin, select the Preprocessed point cloud option. Finally, click the Save button. A new output port is added to the 3D Target Object Recognition Step, as shown below.
Bin Locating
Since the bin shape is approximately rectangular, it can be located directly using the Standard Bin Locating Procedure in the 3D locating category of Hands-on examples in the Solution Library without the need to make a point cloud model.
-
Strings: Name of the bin in Mech-Viz to update the bin pose.
-
Translation Distance: Move the bin pose from the upper surface of the bin down to its center. The translation distance should be set to -57.5mm in this solution.
Translation distance = -1 × 1/2 Bin height -
Run the Procedure to locate the bin and send the bin pose to Mech-Viz for updating.
Adjust Poses
After obtaining the target object poses, you need to use the Adjust Poses V2 Step to adjust the poses. Click the Config wizard button in the Step Parameters panel of the Adjust Poses V2 Step to open the pose adjustment tool for pose adjustment configuration. The overall configuration process is shown in the figure below.
-
To output the target object poses in the robot reference frame, please select the checkbox before Transform pose to robot reference frame to transform the poses from the camera frame to the robot frame.
-
Set the Sorting type to Sort by X/Y/Z value of pose, set Specified value of the pose to Z-coordinate, and sort the poses in Descending order.
-
To reduce the time required for subsequent path planning, target objects that cannot be easily picked need to be filtered based on the angle between the Z-axis of the pose and the reference direction. In this tutorial, you need to set the Max angle difference to 70°.
-
Set the 3D ROI according to the actual situation to filter out target object poses that fall outside the 3D ROI.
-
General settings.
Set number of new ports to 1, and a new input and output port will be added to the Step. Connect the input port to the Pick Point Info output port of the 3D Target Object Recognition Step and connect the output port to the Output Step.
Output Target Object Information
Use the Output Step to output the pick point, pick point information, preprocessed point cloud, bin name, bin pose, etc., to Mech-Viz for path planning.
Path Planning
Once the target object recognition is complete, you can use Mech-Viz to plan a path and then write a robot program for picking the target objects.
The process of path planning configuration is shown in the figure below.
Configure Scene Objects
Scene objects are introduced to make the scene in the software closer to the real scenario, which facilitates the robot path planning. For detailed instructions, please refer to Configure Scene Objects.
To ensure effective picking, scene objects should be configured to accurately represent the real operating environment. The scene objects in this solution are configured as shown below.
Configure Robot Tool
The end tool should be configured so that its model can be displayed in the 3D simulation area and used for collision detection. For detailed instructions, please refer to Configure Tool.
|
Adjust the Workflow
The workflow refers to the robot motion control program created in Mech-Viz in the form of a flowchart. After the scene objects and end tools are configured, you can adjust the project workflow according to the actual requirements. The flowchart of the logical processing when picking the target object is shown below.
When Standard Interface communication is used, the workflow of the project is shown below.
The two Fixed-Point Move Steps at the beginning and end of the workflow are waypoints determined by robot jogging and do not need to be sent to external devices. The other move-type Steps will send waypoints to external devices, and each Smart Path in Bin Step corresponds to two waypoints. Thus, 5 waypoints in total will be sent.
Simulate and Test
Click the Simulate button on the toolbar to test whether the vision system is set up successfully by simulating the Mech-Viz project.
Place the target object randomly in the bin and click Simulate in the Mech-Viz toolbar to simulate picking the target object. After each successful picking, the target object should be rearranged, and 10 simulation tests should be conducted. If the 10 simulations all lead to successful pickings, the vision system is successfully set up.
If an exception occurs during simulation, refer to the Solution Deployment FAQs to resolve the problem.
Robot Picking and Placing
Write a Robot Program
If the simulation result meets expectations, you can write a pick-and-place program for the FANUC robot.
The example program MM_S10_Viz_Subtask for the FANUC robot can basically satisfy the requirements of this typical case. You can modify the example program. For a detailed explanation of the MM_S10_Viz_Subtask program, please refer to the Example Program Explanation.
Modification Instruction
This example program consists of two programs. The secondary program triggers the Mech-Viz project to run to obtain the planned path. The primary program moves the robot based on the planned path. Then, the primary program triggers the secondary program to run when the robot leaves the picking area to obtain the next planned path, shortening the cycle time. Based on the example program, please modify the program files by following these steps:
-
Secondary program
-
Add path planning commands to store all returned waypoints in local variables.
The planned path Mech-Viz returned contains five waypoints: enter-bin point, approach point for picking, pick point, retreat point for picking, and exit-bin point. The example program can only store three waypoints, so you need to add path planning commands to store the rest. Before modification After modification (example) 22: CALL MM_GET_JPS(1,60,70,80) ; 23: CALL MM_GET_JPS(2,61,71,81) ; 24: CALL MM_GET_JPS(3,62,72,82) ;
22: CALL MM_GET_JPS(1,60,70,80) ; 23: CALL MM_GET_JPS(2,61,71,81) ; 24: CALL MM_GET_JPS(3,62,72,82) ; 25: CALL MM_GET_JPS(4,63,73,83) ; 26: CALL MM_GET_JPS(5,64,74,84) ;
-
-
Primary program
-
Sets the tool reference frame. Verify that the TCP on the robot teach pendant matches the TCP in Mech-Viz. Set the currently selected tool frame number to the one corresponding to the reference frame of the actual tool in use.
Before modification After modification (example) 11: !set current tool NO. to 1 ; 12: UTOOL_NUM=1 ;
11: !set current tool NO. to 3 ; 12: UTOOL_NUM=3 ;
Please replace the number with the number of the actual tool being used, where “3” is an example only. -
Specify the IP address and port number of the IPC. Change the IP address and port number in the CALL MM_INIT_SKT command to those in the vision system.
Before modification After modification (example) 16: CALL MM_INIT_SKT('8','127.0.0.1',50000,5) ;16: CALL MM_INIT_SKT('8','127.1.1.2',50000,6) ; -
Adjust the position register IDs for the enter-bin point, approach point for picking, and the pick point.
Before modification After modification 32: !move to approach waypoint ; 33: !of picking ; 34:J PR[60] 50% FINE ; 35: !move to picking waypoint ; 36:J PR[61] 10% FINE ; ... 40: !move to departure waypoint ; 41: !of picking ; 42:J PR[62] 50% FINE ;
32: !move to approach waypoint ; 33: !of picking ; 34:J PR[60] 50% FINE ; 35:J PR[61] 50% FINE ; 36: !move to picking waypoint ; 37:J PR[62] 10% FINE ; ... 42: !move to departure waypoint ; 43: !of picking ; 44:J PR[63] 50% FINE ; 45:J PR[64] 50% FINE ;
In the modified example program, PR[60] stores the enter-bin point, PR[61] stores the approach point for picking, PR[62] stores the pick point, PR[63] stores the retreat point for picking, and PR[64] stores the exit-bin point. -
Set the signal for the DO port to perform picking, i.e., to close the gripper and pick the target object. Note that the DO command should be set according to the actual DO port number used on site.
Before modification After modification (example) 37: !add object grasping logic here, ; 38: !such as "DO[1]=ON" ; 39: PAUSE ;
38: !add object grasping logic here, ; 39: !such as "DO[1]=ON" ; 40: PAUSE ; 41: DO[1]=ON ;
-
Set the DO port to perform placing. Note that the DO command should be set according to the actual DO port number used on site.
Before modification After modification (example) 53: !add object releasing logic here, ; 54: !such as "DO[1]=OFF" ; 55: PAUSE ;
56: !add object releasing logic here, ; 57: !such as "DO[0]=OFF" ; 58: PAUSE ; 59: DO[0]=OFF ;
-
Reference: Modified Example Program
-
Secondary program
1: !-------------------------------- ;
2: !FUNCTION: run Mech-Viz project ;
3: !and get planned path in subtask ;
4: !(run together with ;
5: !MM_S10_Viz_Subtask) ;
6: !Mech-Mind, 2023-12-25 ;
7: !-------------------------------- ;
8: ;
9: F[11]=(ON) ;
10: !trigger Mech-Viz project ;
11: CALL MM_START_VIZ(2,10) ;
12: !get planned path, 1st argument ;
13: !(1) means getting pose in JPs ;
14: CALL MM_GET_VIZ(1,51,52,53) ;
15: !check whether planned path has ;
16: !been got from Mech-Viz ;
17: !successfully ;
18: IF (R[53]=2100) THEN ;
19: !save waypoints of the planned ;
20: !path to local variables one ;
21: !by one ;
22: CALL MM_GET_JPS(1,60,70,80) ;
23: CALL MM_GET_JPS(2,61,71,81) ;
24: CALL MM_GET_JPS(3,62,72,82) ;
25: CALL MM_GET_JPS(4,63,73,83) ;
26: CALL MM_GET_JPS(5,64,74,84) ;
27: ENDIF ;
28: F[11]=(OFF) ;-
Primary program
1: !-------------------------------- ;
2: !FUNCTION: run Mech-Viz project ;
3: !and get planned path in subtask ;
4: !(run together with ;
5: !MM_S10_Sub) ;
6: !Mech-Mind, 2023-12-25 ;
7: !-------------------------------- ;
8: ;
9: !set current uframe NO. to 0 ;
10: UFRAME_NUM=0 ;
11: !set current tool NO. to 3 ;
12: UTOOL_NUM=3 ;
13: !initialize communication ;
14: !parameters(initialization is ;
15: !required only once) ;
16: CALL MM_INIT_SKT('8','127.1.1.2',50000,6) ;
17: !move to robot home position ;
18:J P[1] 100% FINE ;
19: RUN MM_S10_SUB ;
20: LBL[1:LOOP] ;
21: !move to wait position for ;
22: !picking ;
23:L P[2] 1000mm/sec FINE ;
24: !wait until subtask program ;
25: !finished ;
26: WAIT (F[11]=OFF) ;
27: !check whether planned path has ;
28: !been got from Mech-Viz ;
29: !successfully ;
30: IF R[53]<>2100,JMP LBL[99] ;
31: !follow the planned path to pick ;
32: !move to approach waypoint ;
33: !of picking ;
34:J PR[60] 50% FINE ;
35:J PR[61] 50% FINE ;
36: !move to picking waypoint ;
37:J PR[62] 10% FINE ;
38: !add object grasping logic here, ;
39: !such as "DO[1]=ON" ;
40: PAUSE ;
41: DO[1]=ON ;
42: !move to departure waypoint ;
43: !of picking ;
44:J PR[63] 50% FINE ;
45:J PR[64] 50% FINE ;
46: !move to intermediate waypoint ;
47: !of placing, and trigger Mech-Viz ;
48: !project and get planned path in ;
49: !advance ;
50:J P[3] 50% CNT100 DB 10.0mm,CALL MM_S10_SUB ;
51: !move to approach waypoint ;
52: !of placing ;
53:L P[4] 1000mm/sec FINE Tool_Offset,PR[2] ;
54: !move to placing waypoint ;
55:L P[4] 300mm/sec FINE ;
56: !add object releasing logic here, ;
57: !such as "DO[0]=OFF" ;
58: PAUSE ;
59: DO[0]=OFF ;
60: !move to departure waypoint ;
61: !of placing ;
62:L P[4] 1000mm/sec FINE Tool_Offset,PR[2] ;
63: !move back to robot home position ;
64:J P[1] 100% FINE ;
65: JMP LBL[1] ;
66: END ;
67: ;
68: LBL[99:vision error] ;
69: !add error handling logic here ;
70: !according to different ;
71: !error codes ;
72: !e.g.: status=2038 means no ;
73: !point cloud in ROI ;
74: PAUSE ;Picking Test
To ensure stable production in the actual scenario, the modified example program should be run to perform a picking test with the robot. For detailed instructions, please refer to Test Standard Interface Communication.
Before performing the picking test, please teach the following waypoints.
| Name | Variable | Description |
|---|---|---|
Home position |
P[1] |
The taught initial position. The initial position should be away from the objects to be picked and surrounding devices, and should not block the camera’s field of view. |
Image-capturing position |
P[2] |
The taught image-capturing position. The image-capturing position refers to the position of the robot where the camera captures images. At this position, the robot arm should not block the camera’s FOV. |
Intermediate waypoint |
P[3] |
Adding intermediate waypoints can ensure smooth robot motion and avoid unnecessary collisions. |
Placing waypoint |
P[4] |
The position for placing the target object. |
After teaching, arrange the target objects as shown in the table below, and use the robot to conduct picking tests for all arrangements at a low speed.
The picking tests can be divided into three phases:
Phase 1: Test with Single Target Object
Object placement status |
Illustration |
The target object is placed in the middle of the bin, with its front facing up |
|
The target object is placed in the middle of the bin, with its back facing up |
|
The target object is placed in the middle of the bin, with its side facing up |
|
The target object is placed in the corner of the bin |
|
