This is the developer tutorial for implementing a new neural network cross validation.
In this tutorial, you will learn how to create the generator file "*.gen.m" for a new neural network cross validation, which can then be compiled by braph2genesis. All kinds of neural network cross validation are (direct or indirect) extensions of the base element NNCrossValidation. Here, you will use as examples NNRegressorMLP_CrossValidation (a cross validation for multi-layer perceptron regressors) and NNClassifierMLP_CrossValidation (a cross validation for multi-layer perceptron classifiers).
Cross validation for regressors (
NNRegressorMLP_CrossValidation)Cross validation for classifiers (
NNClassifierMLP_CrossValidation)
Cross validation for regressors (NNRegressorMLP_CrossValidation) ⬆
You will start by implementing in detail NNRegressorMLP_CrossValidation, which is a direct extension of NNCrossValidation.
NNRegressorMLP_CrossValidation performs a procedure designed for evaluating multi-layer perceptron regressors using cross-validation.
Code 1. NNRegressorMLP_CrossValidation element header. The
headersection of the generator code for "_NNRegressorMLP_CrossValidation.gen.m" provides the general information about theNNRegressorMLP_CrossValidationelement.%% ¡header! NNRegressorMLP_CrossValidation < NNCrossValidation (nncv, neural network cross-validation) is a process for evaluating multi-layer perceptron regressors using cross-validation. ① %%% ¡description! A cross validation for multi-layer perceptron regressors (NNRegressorMLP_CrossValidation) is a process that facilitates the evaluation of multi-layer perceptron regressors using cross-validation. It involves splitting a dataset into multiple subsets (folds), training the model on some folds while validating on others, and then repeating the process for all combinations of folds. This helps in assessing the generalization performance of the model and detecting overfitting. To train all the neural networks for all folds, use: nncv.get('TRAIN') %%% ¡build! 1① defines
NNRegressorMLP_CrossValidation.genas a subclass ofNNCrossValidation. The moniker will benncv.
Code 2. NNRegressorMLP_CrossValidation element prop update. The
props_updatesection of the generator code for "_NNRegressorMLP_CrossValidation.gen.m" updates the properties of theNNRegressorMLP_CrossValidationelement. This defines the core properties of the cross validation.%% ¡props_update! %%% ¡prop! NAME (constant, string) is the name of the cross-validation. %%%% ¡default! 'NNRegressorMLP_CrossValidation' %%% ¡prop! DESCRIPTION (constant, string) is the description of the cross-validation. %%%% ¡default! 'A cross validation for multi-layer perceptron regressors (NNRegressorMLP_CrossValidation) is a process that facilitates the evaluation of multi-layer perceptron regressors using cross-validation. It involves splitting a dataset into multiple subsets (folds), training the model on some folds while validating on others, and then repeating the process for all combinations of folds. This helps in assessing the generalization performance of the model and detecting overfitting.' %%% ¡prop! TEMPLATE (parameter, item) is the template of the nerual cross-validation. %%%% ¡settings! 'NNRegressorMLP_CrossValidation' %%% ¡prop! ID (data, string) is a few-letter code for the cross-validation. %%%% ¡default! 'NNRegressorMLP_CrossValidation ID' %%% ¡prop! LABEL (metadata, string) is an extended label of the cross-validation. %%%% ¡default! 'NNRegressorMLP_CrossValidation label' %%% ¡prop! NOTES (metadata, string) are some specific notes about the cross-validation. %%%% ¡default! 'NNRegressorMLP_CrossValidation notes' %%% ¡prop! NN_TEMPLATE (parameter, item) is the neural network template to set all neural network parameters. %%%% ¡settings! 'NNRegressorMLP' ① %%% ¡prop! NNEVALUATOR_TEMPLATE (parameter, item) is the neural network evaluator template to set all evalutor parameters. %%%% ¡settings! 'NNRegressorMLP_Evaluator' ② %%% ¡prop! NN_LIST (result, itemlist) contains the neural network models corresponding to k folds. %%%% ¡calculate! d_list = nncv.get('D_LIST'); if isempty(d_list) value = {}; else for i = 1:length(d_list) ③ d_training_set{i} = d_list; d_training_set{i}(i) = []; % Exclude the i-th element d_training_set{i} = NNDatasetCombine('D_LIST', d_training_set{i}).get('D'); end d_training_set = d_training_set'; if ~isa(nncv.getr('NN_TEMPLATE'), 'NoValue') nn_template = nncv.get('NN_TEMPLATE'); ④ else nn_template = NNRegressorMLP( ... ⑤ 'EPOCHS', nncv.get('EPOCHS'), ... 'BATCH', nncv.get('BATCH'), ... 'SHUFFLE', nncv.get('SHUFFLE'), ... 'SOLVER', nncv.get('SOLVER'), ... 'VERBOSE', nncv.get('VERBOSE'), ... 'PLOT_TRAINING', nncv.get('PLOT_TRAINING')); end value = cellfun(@(d) NNRegressorMLP( ... ⑥ 'TEMPLATE', nn_template, 'D', d), ... d_training_set, 'UniformOutput', false); end %%% ¡prop! EVALUATOR_LIST (result, itemlist) contains the evaluators corresponding to k folds. %%%% ¡calculate! %%% ¡prop! EVALUATOR_LIST (result, itemlist) contains the evaluators corresponding to k folds. %%%% ¡calculate! d_list = nncv.get('D_LIST'); nn_list = nncv.get('NN_LIST'); if ~isa(nncv.getr('NNEVALUATOR_TEMPLATE'), 'NoValue') nne_template = nncv.get('NNEVALUATOR_TEMPLATE'); ⑦ else nne_template = NNRegressorMLP_Evaluator( ... ⑧ 'P', nncv.get('P')); end value = cellfun(@(d, nn) NNRegressorMLP_Evaluator('TEMPLATE', nne_template, 'D', d, 'NN', nn), ... ⑨ d_list, nn_list, 'UniformOutput', false);① defines the neural network template to be
NNRegressorMLP, , which will be used in ⑥ to set the relevant parameters.② defines the neural network evaluator template to be
NNRegressorMLP_Evaluator, which will be used in ⑦ to set the relevant parameters.③ constructs the training sets iteratively by concatenating all of the remaining folds after exlcuding the one as the validation set.
④ and ⑤ set the parameters to all the
NNRegressorMLPs based on either theNN_TEMPLATEor the parameters from this cross validation.⑥ initializes all of the
NNRegressorMLPs.⑦ and ⑧ sets the parameters to all the
NNRegressorMLP_Evaluators based on either theNNEVALUATOR_TEMPLATEor the parameters from this cross validation.⑨ initializes all of the
NNRegressorMLP_Evaluators.
Code 3. NNRegressorMLP_CrossValidation element props. The
propssection of generator code for "_NNRegressorMLP_CrossValidation.gen.m" defines the properties to be used in "NNRegressorMLP_CrossValidation".%% ¡props! %%% ¡prop! AV_CORR (result, rvector) provides the metric of the correlation of coefficients. ① %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); value = cellfun(@(e) e.get('CORR'), ... e_list, 'UniformOutput', false); if isempty(value) value = []; else value = mean(cell2mat(value), 1); end %%% ¡prop! AV_DET (result, rvector) provides the coefficient of determination, a measure showing how well the predictions are replicated by the model. ② %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); value = cellfun(@(e) e.get('DET'), ... e_list, 'UniformOutput', false); if isempty(value) value = []; else value = mean(cell2mat(value), 1); end %%% ¡prop! AV_MAE (result, rvector) provides the metric of the mean absolute error. ③ %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); value = cellfun(@(e) e.get('MAE'), ... e_list, 'UniformOutput', false); if isempty(value) value = []; else value = mean(cell2mat(value), 1); end %%% ¡prop! AV_MSE (result, rvector) provides the metric of the mean squared error. ④ %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); value = cellfun(@(e) e.get('MSE'), ... e_list, 'UniformOutput', false); if isempty(value) value = []; else value = mean(cell2mat(value), 1); end %%% ¡prop! AV_RMSE (result, rvector) provides the metric of the root mean squared error. %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); value = cellfun(@(e) e.get('RMSE'), ... e_list, 'UniformOutput', false); if isempty(value) value = []; else value = mean(cell2mat(value), 1); end①-④ calculate the average metrics, including the correlation of coefficients, the coefficient of determination, the mean absolute error, the mean squared error, and the root mean squared error.
Code 4. NNRegressorMLP_CrossValidation element tests. The
testssection from the element generator "_NNRegressorMLP_CrossValidation.gen.m". A test for creating example files should be prepared to test the properties of the data point. Furthermore, additional test should be prepared for validating the value of input and target for the data point.%% ¡tests! %%% ¡test! %%%% ¡name! evaluate a regressor cross-validation with the example data %%%% ¡code! % ensure the example data is generated if ~isfile([fileparts(which('NNDataPoint_CON_REG')) filesep 'Example data NN REG CON XLS' filesep 'atlas.xlsx']) create_data_NN_REG_CON_XLS() % create example files end % Load BrainAtlas im_ba = ImporterBrainAtlasXLS( ... 'FILE', [fileparts(which('NNDataPoint_CON_REG')) filesep 'Example data NN REG CON XLS' filesep 'atlas.xlsx'], ... 'WAITBAR', true ... ); ba = im_ba.get('BA'); % Load Groups of SubjectCON im_gr = ImporterGroupSubjectCON_XLS( ... 'DIRECTORY', [fileparts(which('NNDataPoint_CON_REG')) filesep 'Example data NN REG CON XLS' filesep 'CON_Group_XLS'], ... 'BA', ba, ... 'WAITBAR', true ... ); gr = im_gr.get('GR'); % create a item list of NNDataPoint_CON_REG it_list = cellfun(@(x) NNDataPoint_CON_REG( ... 'ID', x.get('ID'), ... 'SUB', x, ... 'TARGET_IDS', x.get('VOI_DICT').get('KEYS')), ... gr.get('SUB_DICT').get('IT_LIST'), ... 'UniformOutput', false); % create a NNDataPoint_CON_REG DICT dp_list = IndexedDictionary(... 'IT_CLASS', 'NNDataPoint_CON_REG', ... 'IT_LIST', it_list ... ); % create a NNData containing the NNDataPoint_CON_REG DICT d = NNDataset( ... 'DP_CLASS', 'NNDataPoint_CON_REG', ... 'DP_DICT', dp_list ... ); kfolds = 3; nncv = NNRegressorMLP_CrossValidation('KFOLDS', kfolds, 'D', d); nn_list = nncv.get('NN_LIST'); assert(length(nn_list) == kfolds, ... ① [BRAPH2.STR ':NNRegressorMLP_CrossValidation:' BRAPH2.FAIL_TEST], ... 'NNRegressorMLP_CrossValidation does not calculate the neural network list correctly.' ... ) e_list = nncv.get('EVALUATOR_LIST'); assert(length(e_list) == kfolds, ... ② [BRAPH2.STR ':NNRegressorMLP_CrossValidation:' BRAPH2.FAIL_TEST], ... 'NNRegressorMLP_CrossValidation does not calculate the evaluator list correctly.' ... )① and ② check whether the data, regressors, and evaluators are initialized according to the user-specified number of folds.
Cross validation for classifiers (NNClassifierMLP_CrossValidation) ⬆
We can now use NNRegressorMLP_CrossValidation as the basis to implement theNNClassifierMLP_CrossValidation. The parts of the code that are modified are highlighted.
A cross validation for multi-layer perceptron classifier (NNClassifierMLP_CrossValidation) is a procedure designed for evaluating multi-layer perceptron classifiers using cross-validation.
Code 5. NNClassifierMLP_CrossValidation element header. The
headersection of the generator code for "_NNClassifierMLP_CrossValidation.gen.m" provides the general information about theNNClassifierMLP_CrossValidationelement.%% ¡header! NNClassifierMLP_CrossValidation < NNCrossValidation (nncv, neural network cross-validation) is a process for evaluating multi-layer perceptron classifiers using cross-validation. %%% ¡description! A cross validation for multi-layer perceptron classifiers (NNClassifierMLP_CrossValidation) is a process that facilitates the evaluation of multi-layer perceptron classifiers using cross-validation. It involves splitting a dataset into multiple subsets (folds), training the model on some folds while validating on others, and then repeating the process for all combinations of folds. This helps in assessing the generalization performance of the model and detecting overfitting. To train all the neural networks for all folds, use: nncv.get('TRAIN') %%% ¡build! 1
Code 6. NNClassifierMLP_CrossValidation element prop update. The
props_updatesection of the generator code for "_NNClassifierMLP_CrossValidation.gen.m" updates the properties of theNNClassifierMLP_CrossValidationelement. This defines the core properties of the cross validation.%% ¡props_update! %%% ¡prop! NAME (constant, string) is the name of the cross-validation. %%%% ¡default! 'NNClassifierMLP_CrossValidation' %%% ¡prop! DESCRIPTION (constant, string) is the description of the cross-validation. %%%% ¡default! 'A cross validation for multi-layer perceptron classifiers (NNClassifierMLP_CrossValidation) is a process that facilitates the evaluation of multi-layer perceptron classifiers using cross-validation. It involves splitting a dataset into multiple subsets (folds), training the model on some folds while validating on others, and then repeating the process for all combinations of folds. This helps in assessing the generalization performance of the model and detecting overfitting.' %%% ¡prop! TEMPLATE (parameter, item) is the template of the cross-validation. %%%% ¡settings! 'NNClassifierMLP_CrossValidation' %%% ¡prop! ID (data, string) is a few-letter code for the cross-validation. %%%% ¡default! 'NNClassifierMLP_CrossValidation ID' %%% ¡prop! LABEL (metadata, string) is an extended label of the cross-validation. %%%% ¡default! 'NNClassifierMLP_CrossValidation label' %%% ¡prop! NOTES (metadata, string) are some specific notes about the cross-validation. %%%% ¡default! 'NNClassifierMLP_CrossValidation notes' %%% ¡prop! NN_TEMPLATE (parameter, item) is the neural network template to set all neural network parameters. %%%% ¡settings! 'NNClassifierMLP' ① %%% ¡prop! NNEVALUATOR_TEMPLATE (parameter, item) is the neural network evaluator template to set all evalutor parameters. %%%% ¡settings! 'NNClassifierMLP_Evaluator' ② %%% ¡prop! NN_LIST (result, itemlist) contains the neural network models corresponding to k folds. %%%% ¡calculate! d_list = nncv.get('D_LIST'); if isempty(d_list) value = {}; else for i = 1:length(d_list) d_training_set{i} = d_list; d_training_set{i}(i) = []; % Exclude the i-th element d_training_set{i} = NNDatasetCombine('D_LIST', d_training_set{i}).get('D'); end d_training_set = d_training_set'; if ~isa(nncv.getr('NN_TEMPLATE'), 'NoValue') nn_template = nncv.get('NN_TEMPLATE'); else nn_template = NNClassifierMLP( ... 'EPOCHS', nncv.get('EPOCHS'), ... 'BATCH', nncv.get('BATCH'), ... 'SHUFFLE', nncv.get('SHUFFLE'), ... 'SOLVER', nncv.get('SOLVER'), ... 'VERBOSE', nncv.get('VERBOSE'), ... 'PLOT_TRAINING', nncv.get('PLOT_TRAINING')); end value = cellfun(@(d) NNClassifierMLP( ... 'TEMPLATE', nn_template, 'D', d), ... d_training_set, 'UniformOutput', false); . . . . . %%% ¡prop! EVALUATOR_LIST (result, itemlist) contains the evaluators corresponding to k folds. %%%% ¡calculate! d_list = nncv.get('D_LIST'); nn_list = nncv.get('NN_LIST'); if ~isa(nncv.getr('NNEVALUATOR_TEMPLATE'), 'NoValue') nne_template = nncv.get('NNEVALUATOR_TEMPLATE'); else nne_template = NNClassifierMLP_Evaluator( ... 'P', nncv.get('P')); . . . . . value = cellfun(@(d, nn) NNClassifierMLP_Evaluator('TEMPLATE', nne_template, 'D', d, 'NN', nn), ... d_list, nn_list, 'UniformOutput', false);① and ② define the
NN_TEMPLATEasNNClassifierMLPand theNNEVALUATOR_TEMPLATEasNNClassifierMLP_Evaluator.
Code 7. NNClassifierMLP_CrossValidation element props. The
propssection of generator code for "_NNClassifierMLP_CrossValidation.gen.m" defines the properties to be used in "NNClassifierMLP_CrossValidation".%% ¡props! %%% ¡prop! ① AV_AUC (result, rvector) provides the average value of the area under the receiver operating characteristic curve across k folds. %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); aucs = cellfun(@(e) e.get('AUC'), ... e_list, 'UniformOutput', false); if isempty(aucs) value = []; else value = mean(cell2mat(aucs), 1); end %%% ¡prop! ② AV_MACRO_AUC (result, scalar) provides the metric of the average macro AUC value across k folds. %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); macro_aucs = cellfun(@(e) e.get('MACRO_AUC'), ... e_list, 'UniformOutput', false); if isempty(macro_aucs) value = 0; else value = mean(cell2mat(macro_aucs), 1); end %%% ¡prop! ③ C_MATRIX (result, matrix) provides the confusion matrix across k folds. %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); c_matrices = cellfun(@(e) e.get('C_MATRIX'), ... e_list, 'UniformOutput', false); combined_c_matrix = cellfun(@(x) double(x), c_matrices, 'UniformOutput', false); value = sum(cat(3, combined_c_matrix{:}), 3); %%% ¡prop! AV_FEATURE_IMPORTANCE (result, cell) averages the feature importances across k folds. %%%% ¡calculate! e_list = nncv.get('EVALUATOR_LIST'); all_fi = cellfun(@(e) cell2mat(e.get('FEATURE_IMPORTANCE')), ... e_list, 'UniformOutput', false); if isempty(cell2mat(all_fi)) value = {}; else average_fi = zeros(size(all_fi{1})); for i = 1:numel(all_fi) % Add the current cell contents to the averageCell average_fi = average_fi + all_fi{i}; end average_fi = average_fi / numel(all_fi); value = {average_fi}; end① and ② calculate the average value of the area under the receiver operating characteristic curve across k folds.
③ aggregates the confusion matrix across k folds.
Code 8. NNClassifierMLP_CrossValidation element tests. The
testssection from the element generator "_NNClassifierMLP_CrossValidation.gen.m". A test for creating example files should be prepared to test the properties of the data point. Furthermore, additional test should be prepared for validating the value of input and target for the data point.%% ¡tests! %%% ¡test! %%%% ¡name! evaluate a classifier cross-validation with the example data %%%% ¡code! % ensure the example data is generated if ~isfile([fileparts(which('NNDataPoint_CON_CLA')) filesep 'Example data NN CLA CON XLS' filesep 'atlas.xlsx']) create_data_NN_CLA_CON_XLS() % create example files end % Load BrainAtlas im_ba = ImporterBrainAtlasXLS( ... 'FILE', [fileparts(which('NNDataPoint_CON_CLA')) filesep 'Example data NN CLA CON XLS' filesep 'atlas.xlsx'], ... 'WAITBAR', true ... ); ba = im_ba.get('BA'); % Load Groups of SubjectCON im_gr1 = ImporterGroupSubjectCON_XLS( ... 'DIRECTORY', [fileparts(which('NNDataPoint_CON_CLA')) filesep 'Example data NN CLA CON XLS' filesep 'CON_Group_1_XLS'], ... 'BA', ba, ... 'WAITBAR', true ... ); gr1 = im_gr1.get('GR'); im_gr2 = ImporterGroupSubjectCON_XLS( ... 'DIRECTORY', [fileparts(which('NNDataPoint_CON_CLA')) filesep 'Example data NN CLA CON XLS' filesep 'CON_Group_2_XLS'], ... 'BA', ba, ... 'WAITBAR', true ... ); gr2 = im_gr2.get('GR'); % create item lists of NNDataPoint_CON_CLA [~, group_folder_name] = fileparts(im_gr1.get('DIRECTORY')); it_list1 = cellfun(@(x) NNDataPoint_CON_CLA( ... 'ID', x.get('ID'), ... 'SUB', x, ... 'TARGET_IDS', {group_folder_name}), ... gr1.get('SUB_DICT').get('IT_LIST'), ... 'UniformOutput', false); [~, group_folder_name] = fileparts(im_gr2.get('DIRECTORY')); it_list2 = cellfun(@(x) NNDataPoint_CON_CLA( ... 'ID', x.get('ID'), ... 'SUB', x, ... 'TARGET_IDS', {group_folder_name}), ... gr2.get('SUB_DICT').get('IT_LIST'), ... 'UniformOutput', false); % create NNDataPoint_CON_CLA DICT items dp_list1 = IndexedDictionary(... 'IT_CLASS', 'NNDataPoint_CON_CLA', ... 'IT_LIST', it_list1 ... ); dp_list2 = IndexedDictionary(... 'IT_CLASS', 'NNDataPoint_CON_CLA', ... 'IT_LIST', it_list2 ... ); % create a NNDataset containing the NNDataPoint_CON_CLA DICT d1 = NNDataset( ... 'DP_CLASS', 'NNDataPoint_CON_CLA', ... 'DP_DICT', dp_list1 ... ); d2 = NNDataset( ... 'DP_CLASS', 'NNDataPoint_CON_CLA', ... 'DP_DICT', dp_list2 ... ); % combine the two datasets d = NNDatasetCombine('D_LIST', {d1, d2}).get('D'); kfolds = 7; nncv = NNClassifierMLP_CrossValidation('KFOLDS', kfolds, 'D', d); nn_list = nncv.get('NN_LIST'); assert(length(nn_list) == kfolds, ... ① [BRAPH2.STR ':NNClassifierMLP_CrossValidation:' BRAPH2.FAIL_TEST], ... 'NNClassifierMLP_CrossValidation does not calculate the neural network list correctly.' ... ) e_list = nncv.get('EVALUATOR_LIST'); assert(length(e_list) == kfolds, ... ② [BRAPH2.STR ':NNClassifierMLP_CrossValidation:' BRAPH2.FAIL_TEST], ... 'NNClassifierMLP_CrossValidation does not calculate the evaluator list correctly.' ... )① and ② check whether the data, classifiers, and evaluators are initialized according to the user-specified number of folds.