HALCON: Hand-Eye Calibration

This topic introduces how to perform hand-eye calibration in Windows using the HALCON sample programs provided by Mech-Mind. Mech-Mind provide the following two sample programs:

  • determine_calibration_poses: used to determine the calibration poses used during the hand-eye calibration.

  • perform_hand_eye_calibration: used to perform the hand-eye calibration.

Note

  • Currently, the sample programs only support hand-eye calibration of 6-axis robots.

  • For UHP series, to perform the hand-eye calibration, the Capture Mode parameter must be set to Camera1.

Prerequisites

Before performing the hand-eye calibration, please make sure that the following requirements are satisfied.

  1. The accuracy of the robot is good enough, and the robot can operate properly.

  2. Prepare the calibration board that came with the camera, and mount the camera and calibration board.

  3. Download and install Mech-Eye SDK.

  4. Make sure that the quality of the 2D image and depth map meet the requirements.

  5. Make sure that the camera intrinsic parameters are accurate.

  6. Download the HALCON sample programs: select Code ‣ Download as ZIP.

Edit the Sample Programs

After determining the calibration poses with the determine_calibration_poses sample program, you will use the perform_hand_eye_calibration sample program to perform the hand-eye calibration.

The following information must be identical in the two sample programs:

  • The camera to be connected

  • Calibration board model

Besides, the Euler angle convention in the robot_pose JSON file should be set before determining the calibration poses.

Select the Camera

Before running the sample programs, you need to set the camera to be connected in the sample program to make sure the two programs connect to the same camera. Please follow these steps.

  1. Open the sample program in HDevelop: Open HDevelop, and drag the sample program into HDevelop.

  2. Select the line containing the info_framegrabber operator in Program Windows, and click step_over in the toolbar or press the F6 key to run this line.

  3. In the Control Variables area, double-click DeviceInfos to display a list of all the available cameras.

    ../../_images/camera_information_1.png

  4. In the list, double-click the camera that you want to connect, and copy the camera name after unique_name: or user_name:.

    ../../_images/camera_information_3.png

  5. Locate the following line, and replace MechEye with the copied unique_name or user_name.

    DeviceInfo := 'MechEye'

  6. Run the sample program by clicking run or pressing the F5 key.

Set Calibration Board Model

Please follow these steps to set the calibration board model.

  1. Open the sample program in HDevelop: Open HDevelop, and drag the sample program into HDevelop.

  2. Set the calibration board model: The default model is BDB-5. If you are using another model, locate the following operator, and change BDB-5 to the calibration board model in use.

    set_framegrabber_param (AcqHandle, 'BoardType', 'BDB-5')

Set Euler Angle Convention and Unit

The calibration poses determined using the determine_calibration_poses sample program need to be entered into the robot_pose JSON file. The default Euler angle convention in this file is sxyz, and the default Euler angle unit is degree. Please follow these steps to set the Euler angle convention and unit.

  1. Open the robot_pose JSON file.

  2. Set the Euler angle convention: locate the following line, and replace sxyz with the Euler angle convention of the robot in use. For the Euler angle conventions already supported in the sample program, please refer to Supported Euler Angle Conventions.

    "EulerType":"sxyz"

  3. Set the Euler angle unit: if you need to enter Euler angles in radiants, locate the following line, and replace true with false.

    "FromDegree":true

  4. Save the robot_pose JSON file.

Determine Calibration Poses

Workflow

../../_images/extri_teach_flow.png

Instructions

Before performing the hand-eye calibration, you need to determine at least 15 calibration poses. Please follow these steps to determine the calibration poses.

  1. Run the sample program by clicking run or pressing the F5 key. When the program runs to the line read_char (WindowHandle, Char, ReCode), the program will pause and wait for command input in the following steps.

    Hint

    If the camera cannot be connected, please check if it is already connected by Mech-Eye Viewer or another GenICam client.

  2. Move the robot to a candidate calibration pose with the teach pendant.

    Hint

    For requirements on the calibration poses, you can refer to the Calibration chapter in Help > HALCON Reference.

  3. Press P to obtain the 2D image from the camera.

    • If the calibration board is not entirely captured, please move the robot and obtain the 2D image again.

    • If the entire calibration board is captured, proceed to the next step.

  4. Press T to obtain the 2D image containing the feature detection results.

    • If the camera did not detect the circles on the calibration board, no image will be displayed in HDevelop. In this case, please move the robot and obtain the original 2D image and the 2D image containing the feature detection results again.

    • If the camera can detect the circles on the calibration board, a 2D image containing the feature detection results (as shown below) will be displayed in HDevelop. Please proceed to the next step.

      ../../_images/calib_sample.png

  5. Check the robot teach pendant, and enter the current robot pose to the robot_pose JSON file.

    Hint

    It is recommended to save the pose on the teach pendant. When performing the hand-eye calibration, you can use the saved poses to move the robot.

  6. Repeat steps 2 to 5 to determine more calibration poses.

  7. After determining at least 15 calibration poses, press Q to exit the program.

  8. Set the number of calibration poses in the robot_pose JSON file: Open the robot_pose JSON file, and locate the following line. Replace 15 with the actual number of calibration poses determined.

    "pose_count":15

Perform Hand-Eye Calibration

After determining the robot calibration poses, you can perform the hand-eye calibration by running the perform_hand_eye_calibration sample program.

Set Camera Mounting Method

Before performing the hand-eye calibration, please set the camera mounting method in the sample program.

The default mounting method is Eye in Hand. If your camera is mounted in the Eye to Hand mode, please locate the following operator, and replace EyeInHand with EyeToHand.

set_framegrabber_param (AcqHandle, 'CalibrationType', 'EyeInHand')

Switch Reference Frame

The sample program contains an operator that can switch the reference frame in which the point cloud is output from the camera.

The default setting is not to switch the reference frame. If you would like to switch to the robot reference frame, please locate the following operator in the captureTranformedPointCloud procedure, and replace false with true.

set_framegrabber_param (AcqHandle, 'Scan3dCoordinateTransformEnable',false)

Instructions

Please follow these steps to perform the hand-eye calibration.

  1. Run the sample program by clicking run or pressing the F5 key. The sample program will stop when it runs to the stop operator.

  2. Move the robot to the calibration pose in the robot_pose JSON file.

    Hint

    Please move the robot to the calibration poses in the exact order as they are written in the JSON file. Otherwise, the subsequent calculation of the extrinsic parameters will fail.

  3. Run the sample program by clicking run or pressing the F5 key. The camera will perform image capturing.

  4. In the Control Variables area, check the value of CollectResult.

    • If SUCCESS is displayed, please proceed to the next step.

    • If an error is displayed, please troubleshoot according to the displayed error code, and then determine the calibration poses again.

  5. A message saying “Move the robot to the next calibration pose” will be displayed on the image. Please repeat steps 2 to 3.

    Hint

    After the robot has moved to all the calibration poses in the robot_pose JSON file, when you run the sample program again, the extrinsic parameters will be calculated.

  6. In the Control Variables area, check the value of CalibResult.

    • If SUCCESS is displayed, then the hand-eye calibration has succeeded. In the folder where the sample program is stored, you can find the ExtrinsicParameters TXT file that contains the extrinsic parameters and the a point cloud file whose coordinates have been transformed.

    • If an error is displayed, please troubleshoot according to the displayed error code, and then determine the calibration poses again.

Supported Euler Angle Conventions

The conversion from the following Euler angle conventions to quaternions is already supported in the sample program.

Angle input order

Euler angle convention

Robot brand

Z-Y’-Z’’

rzyz

Kawasaki

Z-Y’-X’’

rzyx

ABB

KUKA

X-Y-Z

sxyz

FANUC

YASKAWA

ROKAE

UR

X-Y’-Z’’

rxyz

/

Z-X’-Z’’

rzxz

/

Note

  • Even using the same Euler angle convention, different robot brands might display the Euler angles in different orders. Please input the Euler angles in the order specified in the above table.

  • If the Euler angle convention used by your robot is not listed in the above table, you need to add the conversion from this Euler angle convention to quaternions. Please refer to the code of the euler_to_quad procedure in the perform_hand_eye_calibration sample program and add your own conversion.

Calibration Parameters

This section introduces the parameters used during hand-eye calibration.

BoardType

This parameter is used to set the model of the calibration board in use.

Value list and descriptions:

Value

Description

BDB-5

The recommended distance from the calibration board to the camera is < 0.6 m

BDB-6

The recommended distance from the calibration board to the camera is 0.6–1.5 m

BDB-7

The recommended distance from the calibration board to the camera is > 1.5 m

OCB-005

Only used for Eye-to-Hand projects with high accuracy requirements

OCB-010

OCB-015

OCB-020

CGB-020

The recommended distance from the calibration board to the camera is < 0.6 m

CGB-035

The recommended distance from the calibration board to the camera is 0.6–1.5 m

CGB-050

The recommended distance from the calibration board to the camera is > 1.5 m

ExtrinErrCode

This read-only parameter is used to check the status codes returned during the hand-eye calibration process.

Status code

Description

SUCCESS

Successfully executed.

POSE_INVALID

The pose format is incorrect. Please input quaternions.

IMAGE2D_EMPTY

The 2D image is invalid.

FIND_CORNERS_FAIL

Feature detection of the 2D image failed. Please adjust the parameters for the 2D image.

DEPTH_EMPTY

The depth map is invalid.

CORNERS_3D_INVALID

Feature detection of the depth map failed. Please adjust the parameters for the depth map.

POSES_INSUFFICIENT

Not enough poses are input. Please input at least 15 calibration poses.