Model Comparison
Utilities for comparing the performance of several ML models on the same dataset.
The module provides:
WrapperFactory- builds lightweight model wrapper classes that share a commonfit/predict/validateinterface backed by a precomputed SMILES-to-descriptor dictionary.get_performance_stats()- aggregates per-fold \(R^2\) and MAE values for a set of methods evaluated on the same cross-validation splits.make_tukey_plot()- draws a Tukey HSD “simultaneous” plot to visualize pairwise differences between methods.
- class WrapperFactory[source]
Factory for building model wrapper classes with a uniform interface.
The generated wrappers all expose
fit(train),predict(test), andvalidate(train, test)methods that read SMILES from aSMILEScolumn and look up precomputed descriptors in a shared dictionary. This lets a benchmark loop treat heterogeneous scikit-learn-style models uniformly.- static create_wrapper_class(model_class, descriptor_dict, class_name='DynamicMLWrapper', **model_kwargs)[source]
Dynamically create a wrapper class around
model_class.Any extra keyword arguments are forwarded to the underlying model constructor each time the wrapper is instantiated.
- Parameters:
model_class – A scikit-learn-style estimator class implementing
fitandpredict.descriptor_dict – Mapping from SMILES string to a fixed-length descriptor / fingerprint vector (anything
np.stackcan stack).class_name – Name assigned to the dynamically created class.
model_kwargs – Keyword arguments forwarded to
model_classwhen the wrapper is instantiated.
- Returns:
A new class whose instances accept
y_col(the target column name) and exposefit,predict, andvalidate.
- get_performance_stats(df, y_col, method_list)[source]
Calculate R^2 and MAE statistics for each method in the method list.
- Parameters:
- Returns:
Two DataFrames containing the R^2 and MAE values for each method.
- Return type:
tuple[DataFrame,DataFrame]
- make_tukey_plot(df, y_col, ax=None, method_col='method', xlim=None, title='')[source]
Draw a Tukey HSD “simultaneous” plot comparing methods on a metric.
The best-performing method is chosen as the comparison reference: highest mean for
r2(higher is better) or lowest mean formae(lower is better). The x-axis label is set accordingly.- Parameters:
df – Long-format DataFrame with one row per (fold, method) result.
y_col – Metric column to compare.
"r2"and"mae"are recognized and receive appropriate labels and sort directions.ax – Matplotlib axes to draw on. A new figure/axes is created if
None.method_col – Column identifying the method/group for each row.
xlim – Optional
(xmin, xmax)to apply to the x-axis.title – Title for the axes.
- plot_r2_mae(r2_combo_df, mae_combo_df, figwidth=15, figheight=5)[source]
Plot side-by-side boxplots of R² and MAE across assays and methods.
- Parameters:
r2_combo_df – Combined DataFrame with ‘assay’, ‘r2’, and ‘method’ columns.
mae_combo_df – Combined DataFrame with ‘assay’, ‘mae’, and ‘method’ columns.
figwidth – Figure width in inches (default 15).
figheight – Figure height in inches (default 5).
- Returns:
The matplotlib Figure object.
- plot_tukey_stats(r2_df_list, mae_df_list, assay_list, r2_xlim=(-1, 1), mae_xlim=(0, 1), fig_width=8, fig_height=1, title_height=0.9)[source]
Create Tukey HSD comparison plots for multiple assays.
For each assay a two-panel figure is produced: left panel compares methods by R² and right panel compares methods by MAE. The function calls the shared Tukey plotting helper for each metric and returns the created figures.
- Parameters:
r2_df_list – Sequence of dataframes with ‘r2’ and ‘method’ columns.
mae_df_list – Sequence of dataframes with ‘mae’ and ‘method’ columns.
assay_list – Sequence of assay names corresponding to the dataframes.
r2_xlim – Optional (xmin, xmax) for R² plots.
mae_xlim – Optional (xmin, xmax) for MAE plots.
fig_width – Width in inches for each figure.
fig_height – Height in inches for each figure.
title_height – Vertical position for the assay title (default 0.9).
- Returns:
List of matplotlib Figure objects, one per assay.