train_class_mlpT_train_class_mlpTrainClassMlpTrainClassMlptrain_class_mlp (Operator)

Name

train_class_mlpT_train_class_mlpTrainClassMlpTrainClassMlptrain_class_mlp — Train a multilayer perceptron.

Signature

train_class_mlp( : : MLPHandle, MaxIterations, WeightTolerance, ErrorTolerance : Error, ErrorLog)

Herror T_train_class_mlp(const Htuple MLPHandle, const Htuple MaxIterations, const Htuple WeightTolerance, const Htuple ErrorTolerance, Htuple* Error, Htuple* ErrorLog)

void TrainClassMlp(const HTuple& MLPHandle, const HTuple& MaxIterations, const HTuple& WeightTolerance, const HTuple& ErrorTolerance, HTuple* Error, HTuple* ErrorLog)

double HClassMlp::TrainClassMlp(Hlong MaxIterations, double WeightTolerance, double ErrorTolerance, HTuple* ErrorLog) const

static void HOperatorSet.TrainClassMlp(HTuple MLPHandle, HTuple maxIterations, HTuple weightTolerance, HTuple errorTolerance, out HTuple error, out HTuple errorLog)

double HClassMlp.TrainClassMlp(int maxIterations, double weightTolerance, double errorTolerance, out HTuple errorLog)

def train_class_mlp(mlphandle: HHandle, max_iterations: int, weight_tolerance: float, error_tolerance: float) -> Tuple[float, Sequence[float]]

Description

train_class_mlptrain_class_mlpTrainClassMlpTrainClassMlpTrainClassMlptrain_class_mlp trains the multilayer perceptron (MLP) given in MLPHandleMLPHandleMLPHandleMLPHandleMLPHandlemlphandle. Before the MLP can be trained, all training samples to be used for the training must be stored in the MLP using add_sample_class_mlpadd_sample_class_mlpAddSampleClassMlpAddSampleClassMlpAddSampleClassMlpadd_sample_class_mlp or read_samples_class_mlpread_samples_class_mlpReadSamplesClassMlpReadSamplesClassMlpReadSamplesClassMlpread_samples_class_mlp. If after the training new additional training samples should be used a new MLP must be created with create_class_mlpcreate_class_mlpCreateClassMlpCreateClassMlpCreateClassMlpcreate_class_mlp, in which again all training samples to be used (i.e., the original ones and the additional ones) must be stored. In these cases, it is useful to save and read the training data with write_samples_class_mlpwrite_samples_class_mlpWriteSamplesClassMlpWriteSamplesClassMlpWriteSamplesClassMlpwrite_samples_class_mlp and read_samples_class_mlpread_samples_class_mlpReadSamplesClassMlpReadSamplesClassMlpReadSamplesClassMlpread_samples_class_mlp, respectively. A second training with additional training samples is not explicitly forbidden by train_class_mlptrain_class_mlpTrainClassMlpTrainClassMlpTrainClassMlptrain_class_mlp. However, this typically does not lead to good results because the training of an MLP is a complex nonlinear optimization problem, and consequently the second training with new data will very likely lead to the fact that the optimization gets stuck in a local minimum.

If a rejection class has been specified using set_rejection_params_class_mlpset_rejection_params_class_mlpSetRejectionParamsClassMlpSetRejectionParamsClassMlpSetRejectionParamsClassMlpset_rejection_params_class_mlp, before the actual training the samples for the rejection class are generated.

During the training, the error the MLP achieves on the stored training samples is minimized by using a nonlinear optimization algorithm. If the MLP has been regularized with set_regularization_params_class_mlpset_regularization_params_class_mlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpset_regularization_params_class_mlp, an additional weight penalty term is taken into account. With this, the MLP weights described in create_class_mlpcreate_class_mlpCreateClassMlpCreateClassMlpCreateClassMlpcreate_class_mlp are determined. Furthermore, if an automatic determination of the regularization parameters has been specified with set_regularization_params_class_mlpset_regularization_params_class_mlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpset_regularization_params_class_mlp, these parameters are optimized as well. As described at set_regularization_params_class_mlpset_regularization_params_class_mlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpset_regularization_params_class_mlp, training the MLP with automatic determination of the regularization parameters requires significantly more time than training an unregularized MLP or an MLP with fixed regularization parameters.

create_class_mlpcreate_class_mlpCreateClassMlpCreateClassMlpCreateClassMlpcreate_class_mlp initializes the MLP weights with random values to make it very likely that the optimization converges to the global minimum of the error function. Nevertheless, in rare cases it may happen that the random values determined with RandSeed in create_class_mlpcreate_class_mlpCreateClassMlpCreateClassMlpCreateClassMlpcreate_class_mlp result in a relatively large optimum error, i.e., that the optimization gets stuck in a local minimum. If it can be conjectured that this has happened the MLP should be created anew with a different value for RandSeed in order to check whether a significantly smaller error can be achieved.

The parameters MaxIterationsMaxIterationsMaxIterationsMaxIterationsmaxIterationsmax_iterations, WeightToleranceWeightToleranceWeightToleranceWeightToleranceweightToleranceweight_tolerance, and ErrorToleranceErrorToleranceErrorToleranceErrorToleranceerrorToleranceerror_tolerance control the nonlinear optimization algorithm. Note that if an automatic determination of the regularization parameters has been specified with set_regularization_params_class_mlpset_regularization_params_class_mlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpset_regularization_params_class_mlp, these parameters refer to one training within one step of the evidence procedure. MaxIterationsMaxIterationsMaxIterationsMaxIterationsmaxIterationsmax_iterations specifies the maximum number of iterations of the optimization algorithm. In practice, values between 100 and 200 should be sufficient for most problems. WeightToleranceWeightToleranceWeightToleranceWeightToleranceweightToleranceweight_tolerance specifies a threshold for the change of the weights per iteration. Here, the absolute value of the change of the weights between two iterations is summed. Hence, this value depends on the number of weights as well as the size of the weights, which in turn depend on the scaling of the training data. Typically, values between 0.00001 and 1 should be used. ErrorToleranceErrorToleranceErrorToleranceErrorToleranceerrorToleranceerror_tolerance specifies a threshold for the change of the error value per iteration. This value depends on the number of training samples as well as the number of output variables of the MLP. Also here, values between 0.00001 and 1 should typically be used. The optimization is terminated if the weight change is smaller than WeightToleranceWeightToleranceWeightToleranceWeightToleranceweightToleranceweight_tolerance and the change of the error value is smaller than ErrorToleranceErrorToleranceErrorToleranceErrorToleranceerrorToleranceerror_tolerance. In any case, the optimization is terminated after at most MaxIterationsMaxIterationsMaxIterationsMaxIterationsmaxIterationsmax_iterations iterations. It should be noted that, depending on the size of the MLP and the number of training samples, the training can take from a few seconds to several hours.

On output, train_class_mlptrain_class_mlpTrainClassMlpTrainClassMlpTrainClassMlptrain_class_mlp returns the error of the MLP with the optimal weights on the training samples in ErrorErrorErrorErrorerrorerror. Furthermore, ErrorLogErrorLogErrorLogErrorLogerrorLogerror_log contains the error value as a function of the number of iterations. With this, it is possible to decide whether a second training of the MLP with the same training data without creating the MLP anew makes sense. If ErrorLogErrorLogErrorLogErrorLogerrorLogerror_log is regarded as a function, it should drop off steeply initially, while leveling out very flatly at the end. If ErrorLogErrorLogErrorLogErrorLogerrorLogerror_log is still relatively steep at the end, it usually makes sense to call train_class_mlptrain_class_mlpTrainClassMlpTrainClassMlpTrainClassMlptrain_class_mlp again. It should be noted, however, that this mechanism should not be used to train the MLP successively with MaxIterationsMaxIterationsMaxIterationsMaxIterationsmaxIterationsmax_iterations = 1 (or other small values for MaxIterationsMaxIterationsMaxIterationsMaxIterationsmaxIterationsmax_iterations) because this will substantially increase the number of iterations required to train the MLP. Note that if an automatic determination of the regularization parameters has been specified with set_regularization_params_class_mlpset_regularization_params_class_mlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpset_regularization_params_class_mlp, ErrorErrorErrorErrorerrorerror and ErrorLogErrorLogErrorLogErrorLogerrorLogerror_log refer to the last training that was executed in the evidence procedure. If the error log should be monitored within the individual iterations of the evidence procedure, the outer iteration of the evidence procedure must be implemented explicitly, as described at set_regularization_params_class_mlpset_regularization_params_class_mlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpset_regularization_params_class_mlp.

Execution Information

This operator modifies the state of the following input parameter:

During execution of this operator, access to the value of this parameter must be synchronized if it is used across multiple threads.

Parameters

MLPHandleMLPHandleMLPHandleMLPHandleMLPHandlemlphandle (input_control, state is modified)  class_mlp HClassMlp, HTupleHHandleHTupleHtuple (handle) (IntPtr) (HHandle) (handle)

MLP handle.

MaxIterationsMaxIterationsMaxIterationsMaxIterationsmaxIterationsmax_iterations (input_control)  integer HTupleintHTupleHtuple (integer) (int / long) (Hlong) (Hlong)

Maximum number of iterations of the optimization algorithm.

Default value: 200

Suggested values: 20, 40, 60, 80, 100, 120, 140, 160, 180, 200, 220, 240, 260, 280, 300

WeightToleranceWeightToleranceWeightToleranceWeightToleranceweightToleranceweight_tolerance (input_control)  real HTuplefloatHTupleHtuple (real) (double) (double) (double)

Threshold for the difference of the weights of the MLP between two iterations of the optimization algorithm.

Default value: 1.0

Suggested values: 1.0, 0.1, 0.01, 0.001, 0.0001, 0.00001

Restriction: WeightTolerance >= 1.0e-8

ErrorToleranceErrorToleranceErrorToleranceErrorToleranceerrorToleranceerror_tolerance (input_control)  real HTuplefloatHTupleHtuple (real) (double) (double) (double)

Threshold for the difference of the mean error of the MLP on the training data between two iterations of the optimization algorithm.

Default value: 0.01

Suggested values: 1.0, 0.1, 0.01, 0.001, 0.0001, 0.00001

Restriction: ErrorTolerance >= 1.0e-8

ErrorErrorErrorErrorerrorerror (output_control)  real HTuplefloatHTupleHtuple (real) (double) (double) (double)

Mean error of the MLP on the training data.

ErrorLogErrorLogErrorLogErrorLogerrorLogerror_log (output_control)  real-array HTupleSequence[float]HTupleHtuple (real) (double) (double) (double)

Mean error of the MLP on the training data as a function of the number of iterations of the optimization algorithm.

Example (HDevelop)

* Train an MLP
create_class_mlp (NumIn, NumHidden, NumOut, 'softmax', \
                  'normalization', 1, 42, MLPHandle)
read_samples_class_mlp (MLPHandle, 'samples.mtf')
train_class_mlp (MLPHandle, 100, 1, 0.01, Error, ErrorLog)
write_class_mlp (MLPHandle, 'classifier.mlp')

Result

If the parameters are valid, the operator train_class_mlptrain_class_mlpTrainClassMlpTrainClassMlpTrainClassMlptrain_class_mlp returns the value 2 (H_MSG_TRUE). If necessary, an exception is raised.

train_class_mlptrain_class_mlpTrainClassMlpTrainClassMlpTrainClassMlptrain_class_mlp may return the error 9211 (Matrix is not positive definite) if Preprocessing = 'canonical_variates'"canonical_variates""canonical_variates""canonical_variates""canonical_variates""canonical_variates" is used. This typically indicates that not enough training samples have been stored for each class.

Possible Predecessors

add_sample_class_mlpadd_sample_class_mlpAddSampleClassMlpAddSampleClassMlpAddSampleClassMlpadd_sample_class_mlp, read_samples_class_mlpread_samples_class_mlpReadSamplesClassMlpReadSamplesClassMlpReadSamplesClassMlpread_samples_class_mlp, set_regularization_params_class_mlpset_regularization_params_class_mlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpSetRegularizationParamsClassMlpset_regularization_params_class_mlp

Possible Successors

evaluate_class_mlpevaluate_class_mlpEvaluateClassMlpEvaluateClassMlpEvaluateClassMlpevaluate_class_mlp, classify_class_mlpclassify_class_mlpClassifyClassMlpClassifyClassMlpClassifyClassMlpclassify_class_mlp, write_class_mlpwrite_class_mlpWriteClassMlpWriteClassMlpWriteClassMlpwrite_class_mlp, create_class_lut_mlpcreate_class_lut_mlpCreateClassLutMlpCreateClassLutMlpCreateClassLutMlpcreate_class_lut_mlp

Alternatives

train_dl_classifier_batchtrain_dl_classifier_batchTrainDlClassifierBatchTrainDlClassifierBatchTrainDlClassifierBatchtrain_dl_classifier_batch, read_class_mlpread_class_mlpReadClassMlpReadClassMlpReadClassMlpread_class_mlp

See also

create_class_mlpcreate_class_mlpCreateClassMlpCreateClassMlpCreateClassMlpcreate_class_mlp

References

Christopher M. Bishop: “Neural Networks for Pattern Recognition”; Oxford University Press, Oxford; 1995.
Andrew Webb: “Statistical Pattern Recognition”; Arnold, London; 1999.

Module

Foundation