dingo.core.utils package

Submodules

dingo.core.utils.backward_compatibility module

dingo.core.utils.backward_compatibility.check_minimum_version(version_str: str, raise_exception: bool = False) → None

Check that the version string is greater than a certain minimum value.

By default, logs a warning. Optionally, raises an exception.

This is used to handle major code changes that may break backwards compatibility with previously trained models or generated results.

Parameters:

• version_str (str) – Version string to check, e.g., “version=0.8.5” or “0.8.5”.

• raise_exception (bool) – If True, raise an exception if the version is below the minimum required version.

dingo.core.utils.backward_compatibility.torch_available_devices() → List[Literal['meta', 'cuda', 'mps', 'hip', 'cpu']]

Returns a list of all available PyTorch devices, ordered: cuda, mps, hip, cpu Note: ‘meta’ is not included from the returned list, even if supported.

Return type:

List of available device identifiers

dingo.core.utils.backward_compatibility.torch_load_with_fallback(filename: str, preferred_map_location: Literal['meta', 'cuda', 'mps', 'hip', 'cpu'] = 'cuda') → Tuple[Dict, device]

Loads a PyTorch file with fallback behavior: 1. Tries preferred_map_location (default: cuda) 2. Falls back to CUDA/MPS/HIP if available 3. Finally falls back to CPU

Return type:

Loaded model and torch device on which it has been loaded

dingo.core.utils.backward_compatibility.update_model_config(model_settings: dict)

Update the model settings to ensure backwards compatibility with networks trained using previous versions of Dingo.

Parameters:

model_settings (dict) – Model settings to be updated.

dingo.core.utils.condor_utils module

dingo.core.utils.condor_utils.copy_logfiles(log_dir, epoch, name='info', suffixes=('.err', '.log', '.out'))

dingo.core.utils.condor_utils.copyfile(src, dst)

dingo.core.utils.condor_utils.create_submission_file(train_dir, filename='submission_file.sub')

TODO: documentation :param train_dir: :param filename: :return:

dingo.core.utils.condor_utils.create_submission_file_and_submit_job(train_dir, filename='submission_file.sub')

TODO: documentation :param train_dir: :param filename: :return:

dingo.core.utils.condor_utils.resubmit_condor_job(train_dir, train_settings, epoch)

TODO: documentation :param train_dir: :param train_settings: :param epoch: :return:

dingo.core.utils.gnpeutils module

class dingo.core.utils.gnpeutils.IterationTracker(data=None, store_data=False)

Bases: object

property pvalue_min

update(new_data)

Append new_data to self.data.

Parameters:

new_data (dict) – dict with numpy arrays to append to data

dingo.core.utils.logging_utils module

dingo.core.utils.logging_utils.check_directory_exists_and_if_not_mkdir(directory, logger)

Checks if the given directory exists and creates it if it does not exist

Parameters:

• directory (str) – Name of the directory

• bilby-pipe (Borrowed from)

dingo.core.utils.logging_utils.setup_logger(outdir=None, label=None, log_level='INFO')

Setup logging output: call at the start of the script to use

Parameters:

• outdir (str) – If supplied, write the logging output to outdir/label.log

• label (str) – If supplied, write the logging output to outdir/label.log

• log_level (str, optional) – [‘debug’, ‘info’, ‘warning’] Either a string from the list above, or an integer as specified in https://docs.python.org/2/library/logging.html#logging-levels

• bilby-pipe (Borrowed from)

dingo.core.utils.misc module

dingo.core.utils.misc.call_func_strict_output_dim(func: Callable[[int], Iterable[Iterable]], num_request: int, buffer_fraction: float = 0.02) → Iterable[Iterable]

Repeatedly calls a function until the output shape is the size of num_samples. This can be useful when a user requests N samples from a function, but because of failures in that function, only M < N sample are returned.

Parameters:

• func (Callable[[int], Iterable[Iterable]]) –

  Function to repeatedly call. This should take an int and iterable of iterables. The int tells the function how many samples to generate.

  So for example if one passes 10 to the function, a possible output could be, [np.ndarray, pd.DataFrame, dict{“key”:np.array}], where each entry has len of 10.

• num_request (int) – The output size of the function

• buffer_fraction (float, optional) – The fraction of extra samples to generate given we know the fraction of failed waveforms.

Returns:

final_output – Iterable of iterables where each (sub) iterable has the len equal to num_request.

Return type:

Iterable[Iterable]

dingo.core.utils.misc.get_version()

dingo.core.utils.misc.recursive_check_dicts_are_equal(dict_a, dict_b)

dingo.core.utils.plotting module

dingo.core.utils.plotting.get_latex_labels(prior: PriorDict) → dict

Get the latex labels for prior parameters. If no latex label exists within the prior object, try to choose based on parameter key. Finally, return the parameter key.

Labels are sanitized to fix double-backslash escaping and missing braces that can occur with calibration parameters after prior serialization roundtrips.

Parameters:

prior (PriorDict)

Return type:

dict of latex labels

dingo.core.utils.plotting.plot_corner_multi(samples, weights=None, labels=None, filename: str = 'corner.pdf', latex_labels_dict: dict | None = None, **kwargs)

Generate a corner plot for multiple posteriors.

Parameters:

• samples (list[pd.DataFrame]) – List of sample sets. The DataFrame column names are used as parameter labels.

• weights (list[np.ndarray or None] or None) – List of weights sets. The length of each array should be the same as the length of the corresponding samples.

• labels (list[str or None] or None) – Labels for the posteriors.

• filename (str) – Where to save samples.

• latex_labels_dict (dict) – Dictionary of latex labels.

• legend_font_size (int) – Font size used in legend. Defaults to 50.

• corner.corner. (Also contains additional parameters forwarded to)

dingo.core.utils.pt_to_hdf5 module

dingo.core.utils.pt_to_hdf5.main()

dingo.core.utils.pt_to_hdf5.parse_args()

dingo.core.utils.torchutils module

dingo.core.utils.torchutils.build_train_and_test_loaders(dataset: Dataset, train_fraction: float, batch_size: int, num_workers: int)

Split the dataset into train and test sets, and build corresponding DataLoaders. The random split uses a fixed seed for reproducibility.

Parameters:

• dataset (torch.utils.data.Dataset)

• train_fraction (float) – Fraction of dataset to use for training. The remainder is used for testing. Should lie between 0 and 1.

• batch_size (int)

• num_workers (int)

Return type:

(train_loader, test_loader)

dingo.core.utils.torchutils.fix_random_seeds(_)

Utility function to set random seeds when using multiple workers for DataLoader.

dingo.core.utils.torchutils.get_activation_function_from_string(activation_name: str)

Returns an activation function, based on the name provided.

Parameters:

activation_name – str name of the activation function, one of {‘elu’, ‘relu’, ‘leaky_rely’}

Returns:

function corresponding activation function

dingo.core.utils.torchutils.get_lr(optimizer)

Returns a list with the learning rates of the optimizer.

dingo.core.utils.torchutils.get_number_of_model_parameters(model: Module, requires_grad_flags: tuple = (True, False))

Counts parameters of the module. The list requires_grad_flag can be used to specify whether all parameters should be counted, or only those with requires_grad = True or False. :param model: nn.Module

model

Parameters:

requires_grad_flags – tuple tuple of bools, for requested requires_grad flags

Returns:

number of parameters of the model with requested required_grad flags

dingo.core.utils.torchutils.get_optimizer_from_kwargs(model_parameters: Iterable, **optimizer_kwargs)

Builds and returns an optimizer for model_parameters. The type of the optimizer is determined by kwarg type, the remaining kwargs are passed to the optimizer.

Parameters:

• model_parameters (Iterable) – iterable of parameters to optimize or dicts defining parameter groups

• optimizer_kwargs – kwargs for optimizer; type needs to be one of [adagrad, adam, adamw, lbfgs, RMSprop, sgd], the remaining kwargs are used for specific optimizer kwargs, such as learning rate and momentum

Return type:

optimizer

dingo.core.utils.torchutils.get_scheduler_from_kwargs(optimizer: Optimizer, **scheduler_kwargs)

Builds and returns an scheduler for optimizer. The type of the scheduler is determined by kwarg type, the remaining kwargs are passed to the scheduler.

Parameters:

• optimizer (torch.optim.optimizer.Optimizer) – optimizer for which the scheduler is used

• scheduler_kwargs – kwargs for scheduler; type needs to be one of [step, cosine, reduce_on_plateau], the remaining kwargs are used for specific scheduler kwargs, such as learning rate and momentum

Return type:

scheduler

dingo.core.utils.torchutils.perform_scheduler_step(scheduler, loss=None)

Wrapper for scheduler.step(). If scheduler is ReduceLROnPlateau, then scheduler.step(loss) is called, if not, scheduler.step().

Parameters:

• scheduler – scheduler for learning rate

• loss – validation loss

dingo.core.utils.torchutils.set_requires_grad_flag(model, name_startswith=None, name_contains=None, requires_grad=True)

Set param.requires_grad of all model parameters with a name starting with name_startswith, or name containing name_contains, to requires_grad.

dingo.core.utils.torchutils.split_dataset_into_train_and_test(dataset, train_fraction)

Splits dataset into a trainset of size int(train_fraction * len(dataset)), and a testset with the remainder. Uses fixed random seed for reproducibility.

Parameters:

• dataset (torch.utils.data.Datset) – dataset to be split

• train_fraction (float) – fraction of the dataset to be used for trainset

Return type:

trainset, testset

dingo.core.utils.torchutils.torch_detach_to_cpu(x)

dingo.core.utils.trainutils module

class dingo.core.utils.trainutils.AvgTracker

Bases: object

get_avg()

update(x, n=1)

class dingo.core.utils.trainutils.EarlyStopping(patience: int = 5, verbose: bool = False, delta: float = 0.0, metric: Literal['training', 'validation'] = 'validation')

Bases: object

Implement early stopping during training, once the validation loss stops decreasing for a certain number of epochs (the patience).

If val_loss > min_val_loss - delta for more than patience epochs, then returns early stopping occurs.

Parameters:

• patience (int = 5) – Number of epochs to wait before stopping.

• verbose (bool = False) – Whether to print counter increments.

• delta (float = 0.0) – Amount by which loss must decrease in patience epochs.

• metric (Literal["training", "validation"]) – Whether to use the training loss to determine early stopping (“training”) or the test loss (“validation”)

class dingo.core.utils.trainutils.LossInfo(epoch, len_dataset, batch_size, mode='Train', print_freq=1)

Bases: object

get_avg()

print_info(batch_idx)

update(loss, n)

update_timer(timer_mode='Dataloader')

class dingo.core.utils.trainutils.RuntimeLimits(max_time_per_run: float | None = None, max_epochs_per_run: int | None = None, max_epochs_total: int | None = None, epoch_start: int | None = None)

Bases: object

Keeps track of the runtime limits (time limit, epoch limit, max. number of epochs for model).

Parameters:

• max_time_per_run (float = None) – maximum time for run, in seconds [soft limit, break only after full epoch]

• max_epochs_per_run (int = None) – maximum number of epochs for run

• max_epochs_total (int = None) – maximum total number of epochs for model

• epoch_start (int = None) – start epoch of run

limits_exceeded(epoch: int | None = None)

Check whether any of the runtime limits are exceeded.

Parameters:

epoch (int = None)

Returns:

limits_exceeded – flag whether runtime limits are exceeded and run should be stopped; if limits_exceeded = True, this prints a message for the reason

Return type:

bool

local_limits_exceeded(epoch: int | None = None)

Check whether any of the local runtime limits are exceeded. Local runtime limits include max_epochs_per_run and max_time_per_run, but not max_epochs_total.

Parameters:

epoch (int = None)

Returns:

limits_exceeded – flag whether local runtime limits are exceeded

Return type:

bool

dingo.core.utils.trainutils.copyfile(src, dst)

copy src to dst. :param src: :param dst: :return:

dingo.core.utils.trainutils.save_model(pm, log_dir, model_prefix='model', checkpoint_epochs=None)

Save model to <model_prefix>_latest.pt in log_dir. Additionally, all checkpoint_epochs a permanent checkpoint is saved.

Parameters:

• pm – model to be saved

• log_dir (str) – log directory, where model is saved

• model_prefix (str = 'model') – prefix for name of save model

• checkpoint_epochs (int = None) – number of steps between two consecutive model checkpoints

dingo.core.utils.trainutils.write_history(log_dir, epoch, train_loss, test_loss, learning_rates, aux=None, filename='history.txt')

Writes losses and learning rate history to csv file.

Parameters:

• log_dir (str) – directory containing the history file

• epoch (int) – epoch

• train_loss (float) – train_loss of epoch

• test_loss (float) – test_loss of epoch

• learning_rates (list) – list of learning rates in epoch

• aux (list = []) – list of auxiliary information to be logged

• filename (str = 'history.txt') – name of history file

Module contents