gen_dl_model_heatmap — Infer the sample and generate a heatmap.
gen_dl_model_heatmap( : : DLModelHandle, DLSample, HeatmapMethod, TargetClasses, GenParam : DLResult)
The operator gen_dl_model_heatmap infers the sample given
by DLSample and generates a heatmap for the class specified by
TargetClasses.
DLSample can be a single dictionary or a tuple of dictionaries for
different samples.
To do so, the operator uses the deep learning model given by
DLModelHandle.
The parameter HeatmapMethod determines, which method is used
to calculate the heatmap.
The operator returns DLResult with a result dictionary for every
given sample.
Note, DLResult will be newly created
and already existing parameters with this name will be overwritten.
A heatmap can be useful to analyze which parts of an image have a strong
influence for the inference into a certain class.
Shape and area of these parts and therewith the heatmaps may vary widely
for differing networks especially if their architectures differ.
There are different methods how a heatmap is calculated.
The following ones can be selected using HeatmapMethod:
Gradient-weighted Class Activation Mapping (Grad-CAM). See reference cited below for further information.
The input parameter TargetClasses determines the target class, the
class for which the heatmap is generated.
As value the class ID has to be set.
Alternatively, an empty tuple can be handed over. In this case, the heatmap
is calculated for the class with the highest confidence value, thus the
top inferred class.
The following entries can be set in the dictionary GenParam:
target_conv_layer:
Specifies the convolution layer whose activation and gradient are
used for the heatmap.
The convolution layer can be specified using its NAME,
as returned using 'summary' in get_dl_model_param,
e.g., 'conv10'.
As default, the last (deepest) convolution layer of the network with a
non-trivial activation (width and height not equal to 1) is used.
As a general rule, the activation of the deepest convolution layer
is most suitable for calculating the heatmap, therefore it is
recommended to keep the default layer.
use_conv_only:
In case the convolution layer (selected with target_conv_layer)
is fused with a directly following ReLU activation layer, this parameter can
be used to determine, from which of these layers the activation and gradient
will be used for the heatmap. The following values are supported:
'true': convolution layer
'false': ReLU layer
As default, use_conv_only is set to 'true'.
scaling:Sets the scaling method for the heatmap. The following values are supported:
'scale_after_relu': Negative values of the heatmap are set to 0 and then all values are scaled within the range [0,1]. As a consequence, areas within the heatmap can attain very high values close to 1 although their contribution to the classification result might be small.
'scale_before_relu': All values of the heatmap are divided by the maximum absolute value and then negative values are set to 0. This leads to values for the heatmap in [0,1]. However, areas of less activation (and hence with a small contribution to the classification result) may not tend to attain values near 1.
'none': The heatmap values are not scaled.
As default, scaling is set to 'scale_after_relu'.
Every output dictionary in DLResult contains the inference results
as obtained using apply_dl_model.
Additionally it includes a nested dictionary under the key
heatmap_method, where 'method' is the name of the
specified method as given in HeatmapMethod.
In this nested dictionary the heatmap is saved under the key
heatmap_image_class_classID, where classID is the
ID of the target class.
System requirements:
To run this operator on GPU by setting 'runtime' to
'gpu' (see get_dl_model_param), cuDNN and cuBLAS are required.
For further details, please refer to the “Installation Guide”,
paragraph “Requirements for Deep Learning and Deep-Learning-Based Methods”.
Alternatively, this operator
can also be run on CPU by setting 'runtime' to 'cpu'.
The heatmap should be used as a tool for visualizing and better
understanding classification results. It is not intended as a
segmentation tool.
Moreover, gen_dl_model_heatmap currently only supports models with
'type'='classification'.
DLModelHandle (input_control) dl_model → (handle)
Handle of a Deep learning model.
DLSample (input_control) dict-array → (handle)
Dictionaries with the sample input data.
HeatmapMethod (input_control) string → (string)
Method to be used for the heatmap calculation.
Default value: 'grad_cam'
TargetClasses (input_control) tuple(-array) → (integer)
ID of the target class.
Default value: []
GenParam (input_control) dict → (handle)
Dictionary for generic parameters.
Default value: []
DLResult (output_control) dict(-array) → (handle)
Dictionaries with the result data.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra: "Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 618-626. doi: 10.1109/ICCV.2017.74
Deep Learning Inference