find_box_3dT_find_box_3dFindBox3dFindBox3d (Operator)

Name

find_box_3dT_find_box_3dFindBox3dFindBox3d — Find boxes in 3D data.

Signature

Description

find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d finds boxes in the 3D object model ObjectModel3DSceneObjectModel3DSceneObjectModel3DSceneObjectModel3DSceneobjectModel3DScene and returns the pose of a gripping point GrippingPoseGrippingPoseGrippingPoseGrippingPosegrippingPose, a 3D object model ObjectModel3DBoxObjectModel3DBoxObjectModel3DBoxObjectModel3DBoxobjectModel3DBox a score value ScoreScoreScoreScorescore and a dictionary BoxInformationBoxInformationBoxInformationBoxInformationboxInformation, containing further information about the found boxes, for each found box.

The side lengths of the boxes are passed in SideLen1SideLen1SideLen1SideLen1sideLen1, SideLen2SideLen2SideLen2SideLen2sideLen2, and SideLen3SideLen3SideLen3SideLen3sideLen3. Each length consists of a tuple of two values, indicating the minimum and maximum length of that side. If only a single face of the box is expected to be visible, SideLen3SideLen3SideLen3SideLen3sideLen3 can be set to -1.

The parameter MinScoreMinScoreMinScoreMinScoreminScore sets the minimum score for boxes to be returned. Boxes with a score smaller than this value will not be returned.

ObjectModel3DSceneObjectModel3DSceneObjectModel3DSceneObjectModel3DSceneobjectModel3DScene must contain a XYZ-mapping, like it is, e.g., the case, when creating it with xyz_to_object_model_3dxyz_to_object_model_3dXyzToObjectModel3dXyzToObjectModel3dXyzToObjectModel3d.

Typical Workflow

A typical workflow for detecting 3D boxes in 3D data looks as follows:

1. Obtain the 3D data either as XYZ-images, or directly as a 3D object model with XYZ-mapping.

2. Make sure that the coordinate frames in the XYZ-images are aligned as necessary. See the section “Troubleshooting” below for details on the data axis.

3. Remove as much background and clutter that is not part of any box from the scene as possible, in order to increase robustness and speed. Therefore, e.g., use thresholdthresholdThresholdThresholdThreshold and reduce_domainreduce_domainReduceDomainReduceDomainReduceDomain on the XYZ-images before calling xyz_to_object_model_3dxyz_to_object_model_3dXyzToObjectModel3dXyzToObjectModel3dXyzToObjectModel3d. Further options are described in the section “Troubleshooting” below.

4. If the 3D data exists in the form of XYZ-images, convert them to a 3D object model using xyz_to_object_model_3dxyz_to_object_model_3dXyzToObjectModel3dXyzToObjectModel3dXyzToObjectModel3d.

5. Obtain the approximate box edge lengths that should be found. Note that changing those lengths later on might make it necessary to also change other parameters, such as MinScoreMinScoreMinScoreMinScoreminScore.

6. Call find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d, passing the 3D object model with the scene and the approximate box edge lengths.

7. Use the procedure visualize_object_model_3d to visualize the results, if necessary.

Understanding the Results

The boxes are returned in several ways.

First, the pose of a gripping point is returned in GrippingPoseGrippingPoseGrippingPoseGrippingPosegrippingPose. If only a single side of the box is visible, the center of the gripping pose is in the center of that side, and its z-axis is oriented perpendicular to that side, in the same direction as the z-axis of the scene coordinate system. If multiple sides of the box are visible, the gripping pose lies in the center of the side that is the most oriented towards the z-axis of the scene coordinate system and the z-axis is again perpendicular to that side. The x-axis of the gripping pose is set to the box axis that is the most aligned with the x-axis of the scene coordinate system. The y-axis is computed based on the x- and z-axis.

The box is also returned in triangulated form in ObjectModel3DBoxObjectModel3DBoxObjectModel3DBoxObjectModel3DBoxobjectModel3DBox. This allows a quick visualization of the results.

For each found box, a score between 0 and 1 is returned in ScoreScoreScoreScorescore. The score indicates how well the box and its edges are visible, and how well the found box matches the specified dimensions.

Finally, additional information about the results is returned in the dictionary BoxInformationBoxInformationBoxInformationBoxInformationboxInformation. get_dict_paramget_dict_paramGetDictParamGetDictParamGetDictParam and get_dict_tupleget_dict_tupleGetDictTupleGetDictTupleGetDictTuple can be used to obtain further information about the results. Also, the HDevelop handle inspect window can be used to inspect the returned dictionary.

The dictionary BoxInformationBoxInformationBoxInformationBoxInformationboxInformation contains the following keys:

'results''results''results''results''results':

This key references a dictionary containing the found boxes. They are sorted according to their score in descending order with ascending integer keys starting at 0.

Each box result is a dictionary with the following keys:

'box_pose''box_pose''box_pose''box_pose''boxPose':

This is the box's pose in the coordinate system of the scene. This pose is used for visualizing the found box.

'box_length_x''box_length_x''box_length_x''box_length_x''boxLengthX', 'box_length_y''box_length_y''box_length_y''box_length_y''boxLengthY', 'box_length_z''box_length_z''box_length_z''box_length_z''boxLengthZ':

The side lengths of the found box corresponding to 'box_pose''box_pose''box_pose''box_pose''boxPose'. box_length_xbox_length_xbox_length_xbox_length_xboxLengthX and box_length_ybox_length_ybox_length_ybox_length_yboxLengthY will always contain a positive number. If only a single side of the box is visible, box_length_zbox_length_zbox_length_zbox_length_zboxLengthZ will be set to 0.

'gripping_pose''gripping_pose''gripping_pose''gripping_pose''grippingPose':

The same pose as returned in GrippingPoseGrippingPoseGrippingPoseGrippingPosegrippingPose.

'gripping_length_x''gripping_length_x''gripping_length_x''gripping_length_x''grippingLengthX', 'gripping_length_y''gripping_length_y''gripping_length_y''gripping_length_y''grippingLengthY', 'gripping_length_z''gripping_length_z''gripping_length_z''gripping_length_z''grippingLengthZ':

The side lengths of the found box corresponding to GrippingPoseGrippingPoseGrippingPoseGrippingPosegrippingPose. gripping_length_xgripping_length_xgripping_length_xgripping_length_xgrippingLengthX and gripping_length_ygripping_length_ygripping_length_ygripping_length_ygrippingLengthY will always contain a positive number. If only a single side of the box is visible, gripping_length_zgripping_length_zgripping_length_zgripping_length_zgrippingLengthZ will be set to 0.

'score''score''score''score''score':

The same score as returned in ScoreScoreScoreScorescore.

'one_side_only''one_side_only''one_side_only''one_side_only''oneSideOnly':

Boolean indicating whether only one side of the box is visible ('true'"true""true""true""true") or not.

'gen_param''gen_param''gen_param''gen_param''genParam':

This is a dictionary with the parameters passed to find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d. SideLen1SideLen1SideLen1SideLen1sideLen1, SideLen2SideLen2SideLen2SideLen2sideLen2, and SideLen3SideLen3SideLen3SideLen3sideLen3 are pooled in a tuple with key 'lengths''lengths''lengths''lengths''lengths'. The key 'min_score''min_score''min_score''min_score''minScore' references MinScoreMinScoreMinScoreMinScoreminScore. The other keys are denoted analogously to the generic parameters of the dictionary GenParamGenParamGenParamGenParamgenParam.

'sampled_edges''sampled_edges''sampled_edges''sampled_edges''sampledEdges':

This is the 3D object model with sampled edges. It contains the viewing direction of the edge points as normal vectors.

'sampled_edges_direction''sampled_edges_direction''sampled_edges_direction''sampled_edges_direction''sampledEdgesDirection':

This is the 3D object model with sampled edges (same as for key 'sampled_edges''sampled_edges''sampled_edges''sampled_edges''sampledEdges'. It contains the edge directions of the edge points as normal vectors.

'sampled_scene''sampled_scene''sampled_scene''sampled_scene''sampledScene':

This is the sampled scene in which the boxes are looked for. It can be used for visualization or debugging the sampling distance.

'sampled_reference_points''sampled_reference_points''sampled_reference_points''sampled_reference_points''sampledReferencePoints':

This is a 3D object model with all points from the 3D scene that were used as reference points in the matching process. For each reference point, the optimum pose of the box is computed under the assumption that the reference point lies on the surface of the box.

Generic Parameters

Additional parameters can be passed as key/tuple pairs in the dictionary GenParamGenParamGenParamGenParamgenParam in order to improve the matching process. The following parameter names serve as keys to their corresponding tuples (see create_dictcreate_dictCreateDictCreateDictCreateDict and set_dict_tupleset_dict_tupleSetDictTupleSetDictTupleSetDictTuple).

'3d_edges''3d_edges''3d_edges''3d_edges''3dEdges':

Allows to manually set the 3D scene edges. The parameter must be a 3D object model handle. The edges are usually a result of the operator edges_object_model_3dedges_object_model_3dEdgesObjectModel3dEdgesObjectModel3dEdgesObjectModel3d but can further be filtered in order to remove outliers. If this parameter is not given, find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d will internally extract the 3D edges similar to the operator edges_object_model_3dedges_object_model_3dEdgesObjectModel3dEdgesObjectModel3dEdgesObjectModel3d.

'3d_edge_min_amplitude''3d_edge_min_amplitude''3d_edge_min_amplitude''3d_edge_min_amplitude''3dEdgeMinAmplitude':

Sets the minimum amplitude of a discontinuity in order for it to be classified as an edge. Note that if edges were passed manually with the generic parameter '3d_edges''3d_edges''3d_edges''3d_edges''3dEdges', this parameter is ignored. Otherwise, it behaves identically to the parameter MinAmplitudeMinAmplitudeMinAmplitudeMinAmplitudeminAmplitude of the operator edges_object_model_3dedges_object_model_3dEdgesObjectModel3dEdgesObjectModel3dEdgesObjectModel3d.

Assertion: '3d_edge_min_amplitude''3d_edge_min_amplitude''3d_edge_min_amplitude''3d_edge_min_amplitude''3dEdgeMinAmplitude' >= 0

Default value: 10% of the smallest box diagonal.

'viewpoint''viewpoint''viewpoint''viewpoint''viewpoint', 'max_gap''max_gap''max_gap''max_gap''maxGap':

If no edges are passed with '3d_edges''3d_edges''3d_edges''3d_edges''3dEdges', the operator will extract 3D edges internally. The parameters mentioned above can be used to control the edge extraction.

'viewpoint''viewpoint''viewpoint''viewpoint''viewpoint' and 'max_gap''max_gap''max_gap''max_gap''maxGap' have the same meaning as in edges_object_model_3dedges_object_model_3dEdgesObjectModel3dEdgesObjectModel3dEdgesObjectModel3d.

'remove_outer_edges''remove_outer_edges''remove_outer_edges''remove_outer_edges''removeOuterEdges':

Removes the outermost edges when set to 'true'"true""true""true""true". This is for example helpful for bin picking applications in order to remove the bin.

Default value: 'false'"false""false""false""false"

Possible values: 'false'"false""false""false""false", 'true'"true""true""true""true"

'max_num_boxes''max_num_boxes''max_num_boxes''max_num_boxes''maxNumBoxes':

Limits the number of returned boxes. By default, find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d will return all detected boxes with a score larger than MinScoreMinScoreMinScoreMinScoreminScore. This parameter can be used to limit the number of boxes respectively.

Default value: 0 (return all boxes)

'box_type''box_type''box_type''box_type''boxType':

Sets the type of boxes to search for. The search can be done for boxes with a only single visible side ('single_side_visible'"single_side_visible""single_side_visible""single_side_visible""single_side_visible"), boxes where all three edge lengths can be determined ('full_box_visible'"full_box_visible""full_box_visible""full_box_visible""full_box_visible"), or for both types ('all'"all""all""all""all").

Default value: 'all'"all""all""all""all"

Possible values: 'all'"all""all""all""all", 'single_side_visible'"single_side_visible""single_side_visible""single_side_visible""single_side_visible", 'full_box_visible'"full_box_visible""full_box_visible""full_box_visible""full_box_visible"

Troubleshooting

Visualizing extracted edges and sampled scene:

To debug the box detector, some of the internally used data can be visualized by obtaining it from the returned dictionary BoxInformationBoxInformationBoxInformationBoxInformationboxInformation, using get_dict_tupleget_dict_tupleGetDictTupleGetDictTupleGetDictTuple.

The sampled 3D scene can be extracted with the key 'sampled_scene''sampled_scene''sampled_scene''sampled_scene''sampledScene'. Finding smaller boxes requires a denser sampling and subsequently slows down the box detection.

The sampled 3D edges can be extracted with the key 'sampled_edges''sampled_edges''sampled_edges''sampled_edges''sampledEdges' and 'sampled_edges_directions''sampled_edges_directions''sampled_edges_directions''sampled_edges_directions''sampledEdgesDirections'. Both 3D object models contain the same points, however, 'sampled_edges''sampled_edges''sampled_edges''sampled_edges''sampledEdges' contains the viewing direction of the edge points as normal vectors, while 'sampled_edges_directions''sampled_edges_directions''sampled_edges_directions''sampled_edges_directions''sampledEdgesDirections' contains the edge directions of the edge points as normal vectors. Note that the edge directions should be perpendicular to the edges, pointing outwards of the boxes.

Data axis:

The 3D edge extraction performed internally by find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d or performed externally with edges_object_model_3dedges_object_model_3dEdgesObjectModel3dEdgesObjectModel3dEdgesObjectModel3d requires the image values to increase from left to right in the X-image, from top to bottom in the Y-image and from front to back in the Z-image (meaning that the z-axis of the sensor points forward). The values in the Z-image need to be positive.

If this precondition is not fulfilled, incorrect 3D edges will be extracted, or the extracted 3D edges will have incorrect edge directions.

If only a 3D object model with mappings is available, object_model_3d_to_xyzobject_model_3d_to_xyzObjectModel3dToXyzObjectModel3dToXyzObjectModel3dToXyz with mode 'from_xyz_map'"from_xyz_map""from_xyz_map""from_xyz_map""from_xyz_map" can be used to reconstruct the original images.

The coordinate axis directions can be fixed by, for example, transforming the 3D object model with rigid_trans_object_model_3drigid_trans_object_model_3dRigidTransObjectModel3dRigidTransObjectModel3dRigidTransObjectModel3d, or by scaling the X-, Y- and Z-image with scale_imagescale_imageScaleImageScaleImageScaleImage. In the latter case, it must be ensured that the handedness of the coordinate frame is preserved.

The coordinate frame can be checked by the procedure debug_find_surface_model in the “Automatic Value Check”.

Improve performance:

If find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d is taking too long, the following steps might help to increase its performance.

• Remove more background and clutter: A significant improvement in runtime and detection accuracy can usually be achieved by removing as much of the background and clutter from the 3D scene as possible.

  The most common approaches for removing unwanted data are:

  • Thresholding the X-, Y- and Z-coordinates, either by using thresholdthresholdThresholdThresholdThreshold and reduce_domainreduce_domainReduceDomainReduceDomainReduceDomain on the XYZ-images before calling xyz_to_object_model_3dxyz_to_object_model_3dXyzToObjectModel3dXyzToObjectModel3dXyzToObjectModel3d, or by using select_points_object_model_3dselect_points_object_model_3dSelectPointsObjectModel3dSelectPointsObjectModel3dSelectPointsObjectModel3d directly on the 3D object model that contains the scene.

  • Some sensors return an intensity image along with the 3D data. Filters on the intensity image can be used to remove parts of the image that contain background.

  • Use background subtraction. If the scene is static, for example, if the sensor is mounted in a fixed position over a conveyor belt, the XYZ-images of the background can be acquired once without any boxes in it. Afterwards, sub_imagesub_imageSubImageSubImageSubImage and thresholdthresholdThresholdThresholdThreshold can be used on the Z-images to select parts of the 3D data that are not part of the background.

• Increase minimum score: An increased minimum score MinScoreMinScoreMinScoreMinScoreminScore might lead to more boxes being removed earlier in the detection pipeline.

• Increase the smallest possible box: The smaller the smallest possible box side is, the slower find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d runs. For example, if all boxes are usually seen from a single side, it might make sense to set SideLen3SideLen3SideLen3SideLen3sideLen3 to -1. Additionally, 'box_type''box_type''box_type''box_type''boxType' can be set to limit the type of boxes that are searched.

• Manually computing and filtering edges: The edges of the scene can be extracted manually, using edges_object_model_3dedges_object_model_3dEdgesObjectModel3dEdgesObjectModel3dEdgesObjectModel3d, and passed to find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d using the generic parameter '3d_edges'"3d_edges""3d_edges""3d_edges""3d_edges" (see above). Thus, the manual extraction can be used as a further way of filtering the edges.

Execution Information

• Multithreading type: reentrant (runs in parallel with non-exclusive operators).
• Multithreading scope: global (may be called from any thread).
• Automatically parallelized on internal data level.

This operator supports cancelling timeouts and interrupts.

Parameters

Result

If all parameters are valid and no error occurs, find_box_3dfind_box_3dFindBox3dFindBox3dFindBox3d returns 2 (H_MSG_TRUE). If necessary, an exception is raised.

Possible Predecessors

read_object_model_3dread_object_model_3dReadObjectModel3dReadObjectModel3dReadObjectModel3d, xyz_to_object_model_3dxyz_to_object_model_3dXyzToObjectModel3dXyzToObjectModel3dXyzToObjectModel3d

Possible Successors

gen_box_object_model_3dgen_box_object_model_3dGenBoxObjectModel3dGenBoxObjectModel3dGenBoxObjectModel3d, get_dict_tupleget_dict_tupleGetDictTupleGetDictTupleGetDictTuple

Alternatives

find_surface_modelfind_surface_modelFindSurfaceModelFindSurfaceModelFindSurfaceModel

Module

3D Metrology