xfer.BnnRepurposer

class xfer.BnnRepurposer(source_model: mxnet.module.module.Module, feature_layer_names, context_function=<function cpu>, num_devices=1, bnn_context_function=<function cpu>, sigma=100.0, num_layers=1, n_hidden=10, num_samples_mc=3, learning_rate=0.001, batch_size=20, num_epochs=200, start_annealing=None, end_annealing=None, num_samples_mc_prediction=100, verbose=0)

Bases: xfer.meta_model_repurposer.MetaModelRepurposer

Perform Transfer Learning through a Bayesian Neural Network (BNN) meta-model which repurposes the source neural network.

Parameters:
  • source_model (mxnet.mod.Module) – Source neural network to do transfer learning from.
  • feature_layer_names (list[str]) – Name of layer(s) in source_model from which features should be transferred.
  • context_function (function(int)->:class:mx.context.Context) – MXNet context function that provides device type context. It is used to extract features with the source model.
  • num_devices (int) – Number of devices to use to extract features from source_model.
  • bnn_context_function (function(int)->:class:mx.context.Context) – MXNet context function used to train the BNN.
  • sigma (float) – Standard deviation of the Gaussian prior used for the weights of the BNN meta-model (w_i sim N(0, sigma^2)).
  • num_layers (int) – Number of layers of the BNN meta-model.
  • n_hidden (int) – Dimensionality of the hidden layers of the BNN meta-model (all hidden layers have the same dimensionality).
  • num_samples_mc (int) – Number of samples used for the Monte Carlo approximation of the variational bound.
  • learning_rate (float) – Learning rate for the BNN meta-model training.
  • batch_size (int) – Mini-batch size for the BNN meta-model training.
  • num_epochs (int) – Number of epochs for the BNN meta-model training.
  • start_annealing (int) – To help the training of the BNN meta-model, we anneal the KL term using a weight that varies fromm zero to one. start_annealing determines the epoch at which the annealing weight start to increase linearly until it reaches one in the epoch given by end_annealing.
  • end_annealing (int) – Determines the epoch at which the annealing process of the KL term ends.
  • step_annealing_sample_weight (float) – Amount that the annealing weight is incremented in each epoch (from start_annealing to end_annealing).
  • num_samples_mc_prediction (int) – Number of Monte Carlo samples to use on prediction.
  • verbose (bool) – Flag to control whether accuracy monitoring is logged during repurposing.
  • annealing_weight (float) – Annealing weight in the current epoch.
  • train_acc (list[float]) – Accuracy in training set in each epoch.
  • test_acc (list[float]) – Accuracy in validation set in each epoch.
  • moving_loss_total (list[float]) – Total loss (negative ELBO) smoothed across epochs.
  • average_loss (list[float]) – Average loss (negative ELBO) per data point.
  • anneal_weights (list[float]) – Annealing weight used in each epoch.

Methods

__init__ Initialize self.
deserialize Uses dictionary to set attributes of repurposer.
get_features_from_source_model Extract feature outputs from feature_layer_names in source_model, merge and return all features and labels.
get_params Get parameters of repurposer that are in the constructor.
predict_label Predict class labels on test data using the target_model (repurposed meta-model).
predict_probability Predict class probabilities on test data using the target_model (repurposed meta-model).
repurpose Train a meta-model using features extracted from training data through the source neural network.
save_repurposer Serialize the repurposed model (source_model, target_model and supporting info) and save it to given file_path.
serialize Saves repurposer (excluding source model) to file_prefix.json, file_prefix_posterior.json, file_prefix_posterior_params.npz.

Attributes

end_annealing Determines the epoch at which the annealing process of the KL term ends.
feature_layer_names Names of the layers to extract features from.
source_model Model to extract features from.
start_annealing Determines the epoch at which the annealing process of the KL term starts.
get_params()

Get parameters of repurposer that are in the constructor.

Return type:dict
start_annealing

Determines the epoch at which the annealing process of the KL term starts.

end_annealing

Determines the epoch at which the annealing process of the KL term ends.

feature_layer_names

Names of the layers to extract features from.

get_features_from_source_model(data_iterator: mxnet.io.io.DataIter)

Extract feature outputs from feature_layer_names in source_model, merge and return all features and labels.

In addition, return mapping of feature_layer_name to indices in feature array.

Parameters:data_iterator (mxnet.io.DataIter) – Iterator for data to be passed through the source network and extract features.
Returns:features, feature_indices_per_layer and labels.
Return type:MetaModelData
predict_label(test_iterator: mxnet.io.io.DataIter)

Predict class labels on test data using the target_model (repurposed meta-model).

Parameters:test_iterator (mxnet.io.DataIter) – Test data iterator to return predictions for.
Returns:Predicted labels.
Return type:numpy.ndarray
predict_probability(test_iterator: mxnet.io.io.DataIter)

Predict class probabilities on test data using the target_model (repurposed meta-model).

Parameters:test_iterator (mxnet.io.DataIter) – Test data iterator to return predictions for.
Returns:Predicted probabilities.
Return type:numpy.ndarray
repurpose(train_iterator: mxnet.io.io.DataIter)

Train a meta-model using features extracted from training data through the source neural network.

Set self.target_model to the trained meta-model.

Parameters:train_iterator – Training data iterator to use to extract features from source_model.
save_repurposer(model_name, model_directory='', save_source_model=None)

Serialize the repurposed model (source_model, target_model and supporting info) and save it to given file_path.

Parameters:
  • model_name (str) – Name to save repurposer to.
  • model_directory (str) – File directory to save repurposer in.
  • save_source_model (boolean) – Flag to choose whether to save repurposer source model. Will use default if set to None. (MetaModelRepurposer default: True, NeuralNetworkRepurposer default: False)
source_model

Model to extract features from.

serialize(file_prefix)

Saves repurposer (excluding source model) to file_prefix.json, file_prefix_posterior.json, file_prefix_posterior_params.npz.

Parameters:file_prefix (str) – Prefix to save file with.
deserialize(input_dict)

Uses dictionary to set attributes of repurposer.

Parameters:input_dict (dict) – Dictionary containing values for attributes to be set to.