Model Comparison

Utilities for comparing the performance of several ML models on the same dataset.

The module provides:

  • WrapperFactory - builds lightweight model wrapper classes that share a common fit / predict / validate interface backed by a precomputed SMILES-to-descriptor dictionary.

  • get_performance_stats() - aggregates per-fold \(R^2\) and MAE values for a set of methods evaluated on the same cross-validation splits.

  • make_tukey_plot() - draws a Tukey HSD “simultaneous” plot to visualize pairwise differences between methods.

class WrapperFactory[source]

Factory for building model wrapper classes with a uniform interface.

The generated wrappers all expose fit(train), predict(test), and validate(train, test) methods that read SMILES from a SMILES column and look up precomputed descriptors in a shared dictionary. This lets a benchmark loop treat heterogeneous scikit-learn-style models uniformly.

static create_wrapper_class(model_class, descriptor_dict, class_name='DynamicMLWrapper', **model_kwargs)[source]

Dynamically create a wrapper class around model_class.

Any extra keyword arguments are forwarded to the underlying model constructor each time the wrapper is instantiated.

Parameters:
  • model_class – A scikit-learn-style estimator class implementing fit and predict.

  • descriptor_dict – Mapping from SMILES string to a fixed-length descriptor / fingerprint vector (anything np.stack can stack).

  • class_name – Name assigned to the dynamically created class.

  • model_kwargs – Keyword arguments forwarded to model_class when the wrapper is instantiated.

Returns:

A new class whose instances accept y_col (the target column name) and expose fit, predict, and validate.

get_performance_stats(df, y_col, method_list)[source]

Calculate R^2 and MAE statistics for each method in the method list.

Parameters:
  • df (DataFrame) – The input DataFrame containing the data.

  • y_col (str) – The name of the column containing the true values.

  • method_list (list[str]) – A list of method names to evaluate.

Returns:

Two DataFrames containing the R^2 and MAE values for each method.

Return type:

tuple[DataFrame, DataFrame]

make_tukey_plot(df, y_col, ax=None, method_col='method', xlim=None, title='')[source]

Draw a Tukey HSD “simultaneous” plot comparing methods on a metric.

The best-performing method is chosen as the comparison reference: highest mean for r2 (higher is better) or lowest mean for mae (lower is better). The x-axis label is set accordingly.

Parameters:
  • df – Long-format DataFrame with one row per (fold, method) result.

  • y_col – Metric column to compare. "r2" and "mae" are recognized and receive appropriate labels and sort directions.

  • ax – Matplotlib axes to draw on. A new figure/axes is created if None.

  • method_col – Column identifying the method/group for each row.

  • xlim – Optional (xmin, xmax) to apply to the x-axis.

  • title – Title for the axes.

plot_r2_mae(r2_combo_df, mae_combo_df, figwidth=15, figheight=5)[source]

Plot side-by-side boxplots of R² and MAE across assays and methods.

Parameters:
  • r2_combo_df – Combined DataFrame with ‘assay’, ‘r2’, and ‘method’ columns.

  • mae_combo_df – Combined DataFrame with ‘assay’, ‘mae’, and ‘method’ columns.

  • figwidth – Figure width in inches (default 15).

  • figheight – Figure height in inches (default 5).

Returns:

The matplotlib Figure object.

plot_tukey_stats(r2_df_list, mae_df_list, assay_list, r2_xlim=(-1, 1), mae_xlim=(0, 1), fig_width=8, fig_height=1, title_height=0.9)[source]

Create Tukey HSD comparison plots for multiple assays.

For each assay a two-panel figure is produced: left panel compares methods by R² and right panel compares methods by MAE. The function calls the shared Tukey plotting helper for each metric and returns the created figures.

Parameters:
  • r2_df_list – Sequence of dataframes with ‘r2’ and ‘method’ columns.

  • mae_df_list – Sequence of dataframes with ‘mae’ and ‘method’ columns.

  • assay_list – Sequence of assay names corresponding to the dataframes.

  • r2_xlim – Optional (xmin, xmax) for R² plots.

  • mae_xlim – Optional (xmin, xmax) for MAE plots.

  • fig_width – Width in inches for each figure.

  • fig_height – Height in inches for each figure.

  • title_height – Vertical position for the assay title (default 0.9).

Returns:

List of matplotlib Figure objects, one per assay.