| Operators |
reconstruct_surface_stereo — Reconstruct surface from calibrated multi-view stereo images.
reconstruct_surface_stereo(Images : : StereoModelID : ObjectModel3D)
The operator reconstruct_surface_stereo reconstructs a surface from multiple Images, acquired with a calibrated multi-view setup associated with a stereo model StereoModelID. The reconstructed surface is stored in the handle ObjectModel3D.
A summary of the preparation of a stereo model for surface reconstruction:
Obtain calibrated camera setup model (use calibrate_cameras or create_camera_setup_model) and configure it.
Create a stereo model with create_stereo_model by selecting Method='surface_pairwise' or 'surface_fusion' (see 'Reconstruction algorithm').
Configure the rectification parameters with set_stereo_model_param and afterwards set the image pairs with set_stereo_model_image_pairs.
Configure the bounding box for the system with set_stereo_model_param (GenParamName='bounding_box').
Configure parameters of pairwise reconstruction with set_stereo_model_param.
For models with Method='surface_fusion' configure parameters of the fusion algorithm with set_stereo_model_param.
Acquire images with the calibrated cameras setup and collect them in an image array Images.
Perform surface reconstruction with reconstruct_surface_stereo.
Query and analyze intermediate results with get_stereo_model_object and get_stereo_model_object_model_3d.
Readjust the parameters of the stereo model to improve the results with respect to quality and runtime with set_stereo_model_param.
A camera setup model is associated with the stereo model StereoModelID upon its creation with create_stereo_model. The camera setup must contain calibrated information about the cameras, with which the images in the image array Images were acquired: the I-th image from the array corresponds to the camera with index I-1 from the camera setup; the number of images in the array must be the same as the number of cameras in the camera setup. The Images must represent a static scene or they must be taken simultaneously, otherwise, the reconstruction of the surface might be impossible.
A well-calibrated camera setup is the main requirement for a precise surface reconstruction. Therefore, special attention should be paid to obtaining a precise calibration of the cameras in the multi-view stereo setup used. HALCON provides calibration of a multi-view setup with the operator calibrate_cameras. The resulting calibrated camera setup can be accessed with a successive call to get_calib_data. Alternatively, for camera setups with known parameters a calibrated camera setup can be created with create_camera_setup_model.
The proper selection of image pairs (see set_stereo_model_image_pairs) has an important role for the general quality of the surface reconstruction. On the one hand, camera pairs with a small base line (small distance between the camera centers) are better suited for the binocular stereo disparity algorithms. Hence, pairs with small base lines should be preferred for acquiring accurate depth information of the scene. On the other hand, the pairs should provide different points of view, such that if one pair does not see a certain area of the surface, it is covered by another pair. Please note that the number of pairs linearly affects the runtime of the pairwise reconstruction. Therefore, use "as many as needed and just as few as possible" image pairs in order to handle the trade-off between completeness of the surface reconstruction and reconstruction runtime.
A bounding box is associated with the stereo model StereoModelID. For the surface stereo reconstruction, it is required that the bounding box is valid (see set_stereo_model_param for further details). The reconstruction algorithm needs the bounding box for three reasons:
First, it uses the projection of the bounding box into both images of each image pair in order to estimate the values for MinDisparity and MaxDisparity, which in turn are used in the internal call to binocular_disparity and binocular_disparity_ms. In the case of using binocular_disparity_mg as disparity method, suitable values for the parameters InitialGuess and InitialLevel are derived from the above-mentioned parameters. However, the automatic estimation for this method is only used if called with default values for the two parameters. Otherwise, the values as set by the user with set_stereo_model_param are used.
Secondly, the default parameters for the fusion of pairwise reconstructions are calculated based on the bounding box. They are reset in case the bounding box is changed. The bounding box should be tight around the volume of interest. Else, the runtime will increase unnecessarily and drastically.
Thirdly, the surface fragments lying outside the bounding box are clipped and are not returned in ObjectModel3D. A too large bounding box results in a large difference between MinDisparity and MaxDisparity and this usually slows down the execution of binocular_disparity, binocular_disparity_ms or binocular_disparity_mg and therefore reconstruct_surface_stereo. A too small bounding box might result in clipping valid surface areas.
Note that the method 'surface_fusion' will try to produce a closed surface. If the object is only observed and reconstructed from one side, the far end of the bounding box usually determines where the object is cut off.
Setting parameters of pairwise reconstruction before setting parameters of fusion is essential since the pairwise reconstruction of the object is input for the fusion algorithm. For a description of parameters, see set_stereo_model_param. The choice of 'disparity_method' has a major influence. The objects in the scene should expose certain surface properties in order to make the scene suitable for the dense surface reconstruction. First, the surface reflectance should exhibit Lambertian properties as closely as possible (i.e., light falling on the surface is scattered such that its apparent brightness is the same regardless of the angle of view). Secondly, the surface should exhibit enough texture, but no repeating patterns.
get_stereo_model_object can be used to view intermediate results, in particular rectified, disparity and score images. get_stereo_model_object_model_3d can be used to view the result of pairwise reconstruction for models with Method='surface_fusion'. See the paragraph "Troubleshooting for the configuration of a stereo model" on how to use the obtained results.
The operator reconstruct_surface_stereo performs multiple binocular stereo reconstructions and subsequently combines the results. The image pairs of this pairwise reconstruction are specified in StereoModelID as pairs of cameras of an associated calibrated multi-view setup.
For each image pair, the images are rectified before internally one of the operators binocular_disparity, binocular_disparity_mg or binocular_disparity_ms is called. The disparity information is then converted to points in the coordinate system of the from-camera by an internal call of disparity_image_to_xyz. In the next step, the points are transformed into the common coordinate system that is specified in the camera setup model associated with StereoModelID and stored in a common point cloud together with the points extracted from other image pairs.
If the stereo model is of type 'surface_pairwise' (compare create_stereo_model), the point cloud obtained as described above is directly returned in ObjectModel3D. For each point, the normal vector is calculated by fitting a plane through the neighboring 3D points. In contrast to surface_normals_object_model_3d, the neighboring points are not determined in 3D but simply in 2D by using the neighboring points in the X, Y, and Z images. The normal vector of each 3D point is then set to the normal vector of the respective plane. Additionally, the score of the calculated disparity is attached to every reconstructed 3D point and stored as an extended attribute. Furthermore, transformed coordinate images can be sub-sampled. If only one image pair is processed and no point meshing is enabled, reconstruct_surface_stereo stores a 'xyz_mapping' attribute in ObjectModel3D, which reveals the mapping of the reconstructed 3D points to coordinates of the first image of the pair. This attribute is required by operators like segment_object_model_3d or object_model_3d_to_xyz (with Type='from_xyz_map'). In contrast to the single pair case, if two or more image pairs are processed, reconstruct_surface_stereo does not store the 'xyz_mapping' attribute since single reconstructed points would originate from different image pairs. The presence of the attribute in the output object model can be verified by calling get_object_model_3d_params with GenParamName='has_xyz_mapping'.
The so-obtained point cloud can be additionally meshed in a post-processing step. The object model returned in ObjectModel3D then contains the description of the mesh. The used meshing algorithm depends on the type of the stereo model. For a stereo model of type 'surface_pairwise', only a Poisson solver is supported which can be activated by setting the parameter 'point_meshing' to 'poisson'. It creates a water-tight mesh, therefore surface regions with missing data are covered by an interpolated mesh.
If the stereo model is of type 'surface_fusion', the point cloud obtained as described above is processed further. The goal is to obtain a preferably smooth surface while keeping form fidelity. To this end, the bounding box is sampled and each sample point is assigned a distance to a so-called isosurface (consisting of points with distance 0). The final distance values (and thus the isosurface) are obtained by minimizing an error function based on the points resulting from pairwise reconstruction. This leads to a fusion of the reconstructed point clouds of all camera pairs (see the second paper in References below).
The calculation of the isosurface can be influenced by set_stereo_model_param with the parameters 'resolution', 'surface_tolerance', 'min_thickness' and 'smoothing'. The distance between sample points in the bounding box (in each coordinate direction) can be set by the parameter 'resolution'. The parameter 'smoothing' regulates the 'jumpiness' of the distance function by weighting the two terms in the error function: Fidelity to the initial point clouds obtained by pairwise reconstruction on the one hand, total variation of the distance function on the other hand. Note that the actual value of 'smoothing' for a given data set to be visually pleasing has to be found by trial and error. Too small values lead to integrating many outliers into the surface even if the object surface then exhibits many jumps. Too large values lead to loss of fidelity towards the point cloud of pairwise reconstruction. Fidelity to the initial surfaces obtained by pairwise reconstruction is not maintained in the entire bounding box, but only in cones of sight of cameras to the initial surface. A sample point in such a cone is considered surely outside of the object (in front of the surface) or surely inside the object (behind the surface) with respect to the given camera if its distance to the initial surface exceeds a given value which can be set by the parameter 'surface_tolerance'. The length of considered cones behind the initial surface can roughly be set by the parameter 'min_thickness' (see set_stereo_model_param for more details). 'min_thickness' always has to be larger than or equal to 'surface_tolerance'.
| (1) | (2) |
Each 3D point of the object model returned in ObjectModel3D is extracted from the isosurface where the distance function equals zero. Its normal vector is calculated from the gradient of the distance function. While the method 'surface_fusion' requires the setting of more parameters than simple pairwise reconstruction, postprocessing of the obtained point cloud representing the object surface will probably get a lot simpler. In particular, suppression of outliers, smoothing, equidistant sub-sampling and hole filling can be handled nicely and often in high quality by this method. The same can be said about the possible internal meshing of the output surface, see the next paragraph. Note that the algorithm will try to produce a closed surface. If the object is only observed and reconstructed from one side, the far end of the bounding box usually determines where the object is cut off. The method 'surface_fusion' may take considerably longer than simple pairwise reconstruction, depending mainly on the parameter 'resolution'.
Additionally, the so-obtained point cloud can be meshed in a post-processing step. The object model returned in ObjectModel3D then contains the description of the mesh. For a stereo model of type 'surface_fusion', the algorithm 'marching tetrahedra' is used which can be activated by setting the parameter 'point_meshing' to 'isosurface'. The wanted meshed surface is extracted as the isosurface where the distance function equals zero. Note that there are more points in ObjectModel3D if meshing of the isosurface is enabled even if the used 'resolution' is the same.
It is possible to provide color information for 3D object models that have been reconstructed with reconstruct_surface_stereo from the input images. The computation of the color depends on the chosen method set with set_stereo_model_param (see explanation in the list there). Each 3D point is assigned a color value consisting of a red, green and blue channel which are stored as attributes named 'red', 'green' and 'blue' in the output 3D object model ObjectModel3D. These attributes can for example be used in the procedure visualize_object_model_3d with GenParamName = 'red_channel_attrib', 'green_channel_attrib' and 'blue_channel_attrib'. They can also be queried with get_object_model_3d_params or be processed with select_points_object_model_3d or other operators that use extended attributes. If the reconstruction has been performed using gray value images, the color value for the three channels is identical. If multi-channel images are used, the reconstruction is performed using the first channel only. The remaining channels are solely used for the calculation of the color values.
If stereo models of type 'surface_fusion' are used, the reconstruction will contain points without a direct correspondence to points in the images. These points are not seen by any of the cameras of the stereo system and are therefore "invisible". A color value for these points is derived by assigning the value of the nearest visible neighbor. Normally, the nearest neighbor search is not very time-consuming and can remain active. However, it may happen that the value for the parameter 'resolution' is considerably finer than the available image resolution. In this case, many invisible 3D points are reconstructed making the nearest neighbor search very time consuming. In order to avoid an increased runtime, it is recommended to either adapt the value of 'resolution' or to switch off the calculation for invisible points. This can be done by calling set_stereo_model_param with GenParamName='color_invisible' and GenParamValue= 'false'. In this case, invisible points are assigned 255 as gray value.
The proper configuration of a stereo model is not always easy. Please follow the workflow above. If the reconstruction results are not satisfactory, please consult the following hints and ideas:
If you enable the 'persistence' mode of stereo model (call set_stereo_model_param with GenParamName='persistence') a successive call to reconstruct_surface_stereo will store intermediate iconic results, which provide additional information. They can be accessed by get_stereo_model_object_model_3d and get_stereo_model_object.
If the camera setup was obtained by calibrate_cameras, it stores some quality information about the camera calibration in form of standard deviations of the camera internal parameters. This information is then carried in the camera setup model associated with the stereo model. It can be queried by first calling get_stereo_model_param with GenParamName='camera_setup_model' and then inspecting the camera parameter standard deviations by calling get_camera_setup_param with GenParamName='params_deviations'. Unusually big standard deviation values might indicate a bad camera calibration.
After setting the stereo model 'persistence' mode, we recommend inspecting the rectified images for each image pair. The rectified images are returned by get_stereo_model_object with a camera index pair [From, To] specifying the pair of interest in the parameter PairIndex and the values 'from_image_rect' and 'to_image_rect' in ObjectName, respectively. If the images are properly rectified, all corresponding image features must appear in the same row in both rectified images. A discrepancy of several rows is a serious indication for a bad camera calibration.
Make sure that the bounding box is tight around the volume of interest. The algorithm uses the projection of the bounding box into both images of each image pair in order to estimate the values for MinDisparity and MaxDisparity, which in turn are used in the internal call to binocular_disparity and binocular_disparity_ms. These values can be queried using get_stereo_model_param. In the case of using binocular_disparity_mg as disparity method, suitable values for the parameters InitialGuess and InitialLevel are derived from the bounding box. However, these values can also be reset using set_stereo_model_param. Use the procedures gen_bounding_box_object_model_3d to create a 3D object model of your stereo model, and inspect it in conjunction with the reconstructed 3D object model to verify the bounding box visually.
After setting the stereo model 'persistence' mode (see above), inspect the disparity and the score images for each image pair. They are returned by get_stereo_model_object with a camera index pair [From, To] specifying the pair of interest in the parameter PairIndex and the values 'disparity_image' and 'score_image' in ObjectName, respectively. If both images exhibit significant imperfection (e.g., the disparity image does not really resemble the shape of the object seen in the image), try to adjust the parameters used for the internal call to binocular_disparity (the parameters with a 'binocular_' prefix) by modifying set_stereo_model_param until some improvement is achieved.
Alternatively, a different method to calculate the disparities can be used. Besides the above-mentioned internal call of binocular_disparity, HALCON also provides the two other methods binocular_disparity_mg and binocular_disparity_ms. These methods feature e.g. the calculation of disparities in textureless regions at an expanse of the reconstruction time if compared with cross-correlation methods. However, for these methods, it can be necessary to adapt the parameters to the underlying dataset as well. Dependent on the chosen method, the user can either set the parameters with a 'binocular_mg_' or a 'binocular_ms_' prefix until some improvement is achieved.
A detailed description of the provided methods and their parameters can be found in binocular_disparity, binocular_disparity_mg or binocular_disparity_ms, respectively.
If the result of pairwise reconstruction as inspected by get_stereo_model_object_model_3d can not be improved anymore, begin to adapt the fusion parameters. For a description of parameters see also set_stereo_model_param. Note that the pairwise reconstruction is sometimes not discernible when the fusion algorithm can still tweak it into something sensible. In any case, pairwise reconstruction should yield enough points as input for the fusion algorithm.
In order to improve the runtime, consider the following hints:
The bounding box should be tight around the volume of interest. Else, the runtime will increase unnecessarily and - for the method 'surface_fusion' - drastically.
The stereo model parameter 'rectif_sub_sampling' (see set_stereo_model_param) controls the sub-sampling in the rectification step. Setting this factor to a value > 1.0 will reduce the resolution of the rectified images compared to the original images. This factor has a direct impact on the succeeding performance of the chosen disparity method, but it causes loss of image detail. The parameter 'rectif_interpolation' could have also some impact, but typically not a significant one.
There is a trade-off between completeness of the pairwise surface reconstruction on the one hand and reconstruction runtime on the other. The stereo model offers three different methods to calculate the disparity images. Dependent on the chosen method, the stereo model provides a particular set of parameters that enables a precise adaption of the method to the used dataset. If the method binocular_disparity is selected, only parameters with a 'binocular_' prefix can be set. For the method binocular_disparity_mg, all settable parameters have to exhibit the prefix 'binocular_mg_', whereas for the method binocular_disparity_ms only parameters with 'binocular_ms_' are applicable.
Parameters using the method binocular_disparity
Each of these parameters of binocular_disparity has a corresponding stereo model parameter with the prefix 'binocular_', and has, some more or others less, impact on the performance. Adapting them properly could improve the performance.
Parameters using the method binocular_disparity_mg
Each of these parameters of binocular_disparity_mg has a corresponding stereo model parameter with the prefix 'binocular_mg_', and has, some more or others less, impact on the performance and the result. Adapting them properly could improve the performance.
Parameters using the method binocular_disparity_ms
Each of these parameters of binocular_disparity_ms has a corresponding stereo model parameter with the prefix 'binocular_ms_', and has, some more or others less, impact on the performance and the result. Adapting them properly could improve the performance.
Besides adapting the sub-sampling it is also possible to exclude points of the 3D reconstruction because of their computed disparity score. In order to do this, the user should first query the score images for the disparity values by calling get_stereo_model_object using GenParamName = 'score_image'. Dependent on the distribution of these values, the user can decide whether disparities with a score beneath a certain threshold should be excluded from the reconstruction. This can be achieved with set_stereo_model_param using either GenParamName = 'binocular_score_thresh'. The advantage of excluding points of the reconstruction is a slight speed-up since it is not necessary to process the entire dataset. As an alternative to the above-mentioned procedure, it is also possible to exclude points after executing reconstruct_surface_stereo by filtering reconstructed 3D points. The advantage of this is that at the expense of a slightly increased runtime, a second call to reconstruct_surface_stereo is not necessary.
For the method 'surface_pairwise', you can use a larger sub-sampling step for the X,Y,Z data in the last step of the reconstruction algorithm by modifying GenParamName='sub_sampling_step' with set_stereo_model_param. The reconstructed data will be much sparser, thus speeding up the post-processing.
For the method 'surface_fusion', enlarging the parameter 'resolution' will speed up the execution considerably.
Note that if a 3D object model is no longer needed or should be overwritten, the memory has to be freed first by calling the operator clear_object_model_3d.
This operator returns a handle. Note that the state of an instance of this handle type may be changed by specific operators even though the handle is used as an input parameter by those operators.
An image array acquired by the camera setup associated with the stereo model.
Handle of the stereo model.
Handle to the resulting surface.
create_stereo_model, get_calib_data, set_stereo_model_image_pairs
get_stereo_model_object_model_3d
M. Kazhdan, M. Bolitho, and H. Hoppe: “Poisson Surface Reconstruction.”
Symposium on Geometry Processing (June 2006).,
C. Zach, T. Pock, and H. Bischof: “A globally optimal algorithm for robust
TV-L1 range image integration.”
Proceedings of IEEE International Conference on Computer Vision (ICCV 2007).
3D Metrology
| Operators |