HUGIML Benchmark Analysis

A 100-dataset tabular classification benchmark (50 real-world + 50 synthetic) comparing HUGIML with tuned XGBoost, LightGBM, RandomForest, EBM, and RuleFit baselines. Budgeted tree variants are constrained by learned model complexity, measured as terminal leaves across the fitted ensemble. HUGIML complexity is measured as selected patterns.

Primary metric: AUC Datasets: 100 Models: 9 Friedman p-value: 5.558e-44 Tree budget: ≤100 leaves HUGIML topK: 50 / 100
Best mean AUC
0.8805
HUGIML
HUGIML mean AUC
0.8805
mean model inspection units: 81.3; mean instance inspection units: 18.5
Best mean rank
3.58
HUGIML
Global rank test
5.558e-44
Significant global rank differences among models
Pairwise vs HUGIML
3 significant
HUGIML vs 8 baselines, Holm-adjusted
Budget violations
0
budgeted fits exceeded 200 model inspection units

Mean AUC by model

Standard-deviation intervals shown

AUC versus model complexity

Leaves for tree ensembles and RuleFit rules; EBM active cells; selected patterns for HUGIML

Benchmark interpretation

Statistical findings
Global comparison: The Friedman test shows significant global rank differences among models. It is a global rank test and does not identify which model pairs differ.
HUGIML-specific comparison: 3 of 8 HUGIML-vs-baseline comparisons are significant after Holm correction.

Complexity-budget check

Budgeted tree variants checked after fitting

Interactive dataset explorer

Filter datasets, toggle models, and inspect metrics, complexity, and tuned settings

Filtered dataset table

Metric values and winners

Selected dataset: tuned settings

Includes fitted model complexity

HUGIML depth and topK

Selected configuration frequencies

HUGIML AUC gap

HUGIML minus best non-HUGIML baseline

Pairwise tests versus HUGIML

Paired Wilcoxon tests; Holm correction within this family

Overall summary

AUC, ranks, wins, and model complexity

Significant all-pair comparisons

21 significant pair(s) after Holm correction across all 36 pairwise tests

Rows below are the statistically significant differences among all model pairs after Holm correction. This table can include HUGIML pairs if they are significant; use the HUGIML-focused comparison above for only HUGIML-vs-baseline tests.

Summary results by dataset scope

Overall, public real-world/package datasets, and reproducible synthetic datasets.
Methodology and parameter search space

Cross-validation protocol

  • Dataset panel: 100 binary-classification tasks (50 real-world and 50 synthetic).
  • Rows per dataset: all available rows.
  • Outer evaluation: 5-fold StratifiedKFold with shuffle=True and random_state=42.
  • Inner selection: 3-fold StratifiedKFold within every outer-training partition, with shuffle=True and random_state=42.
  • Selection metric: mean inner-fold ROC-AUC. The selected candidate is refitted on the complete outer-training partition and evaluated once on the untouched outer-test partition.
  • Reported dataset AUC: arithmetic mean of the 5 outer-fold ROC-AUC values.
  • Statistical comparisons use dataset-level outer-CV aggregates, Friedman ranking, paired Wilcoxon tests, and Holm adjustment where shown.

Preprocessing and timing

  • HUGIML receives the native pandas table, including categorical columns, and performs its own binning and pattern construction inside each training fold.
  • Non-HUGIML models use a fold-local pipeline: numeric median imputation; categorical most-frequent imputation followed by dense one-hot encoding with unknown categories ignored.
  • Source dataframe indexes are discarded before modelling; statsmodels-backed tasks use only columns declared in load_pandas().data.
  • Preprocessing is fitted only on the relevant training partition. No test-fold information is used for preprocessing or parameter selection.
  • fit_seconds measures the selected estimator's final outer-fold refit; tune_seconds measures inner-CV search; predict_seconds measures outer-test inference.

Model search spaces

HUGIML — Augmented pair path

Search space: performance_ho · 16 candidate configurations
ParameterValues considered
B-1
adaptive_binningTrue
L1, 2
topK50, 100
feature_modeoriginal_plus_patterns
G0.01, 0.001
convert_binary_to_categoricalFalse
augmented_pair_transformsTrue
topk_budget_strictFalse
base_estimatorLogistic regression (L1 penalty, liblinear solver, C=1.0), OneVsRestClassifier(RPTE; leaf_config=3xD, depth=4, adaptive lookahead)

Constant settings: execution_mode=production; n_jobs=1; Linear base estimator: logistic regression with L1 penalty, liblinear solver, C=1.0, random_state=0, and max_iter=500.; Inner cross-validation selects between the stated logistic-regression base estimator and one-vs-rest RPTE.; Numeric 0/1 columns remain numeric and eligible for augmented-pair transforms.; RPTE uses leaf indicators together with downstream inputs not selected in accepted tree splits.; RPTE lookahead is adaptive for this path.

Complexity definition: Model inspection units represent the complete fitted HUGIML model that a reviewer must inspect. Linear branches count active source contributions; RPTE branches count conditions across active terminal paths plus direct terms. Intercepts are excluded, and fitted numeric components are active when their absolute value exceeds 1e-12.

HUGIML — Interaction-relaxed path

Search space: interpretability_ho · 16 candidate configurations
ParameterValues considered
B-1
adaptive_binningTrue
L1, 2
topK50, 100
feature_modepatterns_only
G0.01, 0.001
interaction_relaxed_miningTrue
augmented_pair_transformsFalse
convert_binary_to_categoricalTrue
topk_budget_strictFalse
base_estimatorLogistic regression (L1 penalty, liblinear solver, C=1.0), OneVsRestClassifier(RPTE; leaf_config=3xD, depth=4, adaptive lookahead)

Constant settings: execution_mode=production; n_jobs=1; Linear base estimator: logistic regression with L1 penalty, liblinear solver, C=1.0, random_state=0, and max_iter=500.; Inner cross-validation selects between the stated logistic-regression base estimator and one-vs-rest RPTE.; Numeric 0/1 indicators are treated categorically for the pattern-mining surface.; Interaction relaxation is performed by the pattern miner; augmented pairs are disabled.; RPTE uses the sequential backend with lookahead inactive for this path.

Complexity definition: Model inspection units represent the complete fitted HUGIML model that a reviewer must inspect. Linear branches count active source contributions; RPTE branches count conditions across active terminal paths plus direct terms. Intercepts are excluded, and fitted numeric components are active when their absolute value exceeds 1e-12.

XGB standard

Search space: XGBoost · 16 candidate configurations
ParameterValues considered
n_estimators100, 200
max_depth3, 4
learning_rate0.03, 0.1
min_child_weight1, 5

Constant settings: eval_metric=logloss; verbosity=0; n_jobs=1; random_state=42; Unspecified estimator settings use the library defaults.

Complexity definition: Model inspection units sum the conditions across complete root-to-leaf paths with active terminal outputs.

LightGBM standard

Search space: LightGBM · 16 candidate configurations
ParameterValues considered
n_estimators100, 200
learning_rate0.03, 0.1
num_leaves15, 31
min_child_samples10, 20

Constant settings: verbose=-1; n_jobs=1; random_state=42; Unspecified estimator settings use the library defaults.

Complexity definition: Model inspection units sum the conditions across complete root-to-leaf paths with active terminal outputs.

RandomForest standard

Search space: RandomForest · 16 candidate configurations
ParameterValues considered
n_estimators200, 400
max_depth4, 8
min_samples_leaf1, 5
max_featuressqrt, 0.5

Constant settings: n_jobs=1; random_state=42; Unspecified estimator settings use the library defaults.

Complexity definition: Model inspection units sum the conditions across every complete root-to-leaf path in the fitted forest.

XGB complexity-budgeted

Search space: XGBoost · 16 candidate configurations
ParameterValues considered
n_estimators25, 50
max_depth1, 2
learning_rate0.03, 0.1
min_child_weight1, 5

Constant settings: eval_metric=logloss; verbosity=0; n_jobs=1; random_state=42; Complexity budget=200; candidates within the budget are preferred during inner-CV selection.; Unspecified estimator settings use the library defaults.

Complexity definition: Model inspection units sum the conditions across complete root-to-leaf paths with active terminal outputs.

LightGBM complexity-budgeted

Search space: LightGBM · 16 candidate configurations
ParameterValues considered
n_estimators25, 50
num_leaves2, 4
learning_rate0.03, 0.1
min_child_samples10, 20

Constant settings: verbose=-1; n_jobs=1; random_state=42; Complexity budget=200; candidates within the budget are preferred during inner-CV selection.; Unspecified estimator settings use the library defaults.

Complexity definition: Model inspection units sum the conditions across complete root-to-leaf paths with active terminal outputs.

RandomForest complexity-budgeted

Search space: RandomForest · 16 candidate configurations
ParameterValues considered
n_estimators25, 50
max_leaf_nodes2, 4
min_samples_leaf1, 5
max_featuressqrt, 0.5
max_depthNone

Constant settings: n_jobs=1; random_state=42; Complexity budget=200; candidates within the budget are preferred during inner-CV selection.; Unspecified estimator settings use the library defaults.

Complexity definition: Model inspection units sum the conditions across every complete root-to-leaf path in the fitted forest.

EBM

Search space: EBM · 8 candidate configurations
ParameterValues considered
learning_rate0.01, 0.05
max_bins32, 64
interactions0, 5
max_rounds500

Constant settings: outer_bags=4; max_rounds=500; n_jobs=1; random_state=42; Unspecified estimator settings use the library defaults.

Complexity definition: Model inspection units count finite nonzero term-score cells.

RuleFit

Search space: RuleFit · 8 candidate configurations
ParameterValues considered
n_estimators50, 100
max_rules50, 100
tree_size5, 10

Constant settings: alpha=None; random_state=42; Unspecified estimator settings use the library defaults.

Complexity definition: Model inspection units count active linear terms and every condition in each active rule.