HUGIML Benchmark Analysis

Cross-validation protocol

• Dataset panel: 100 binary-classification tasks (50 real-world and 50 synthetic).
• Rows per dataset: all available rows.
• Outer evaluation: 5-fold StratifiedKFold with shuffle=True and random_state=42.
• Inner selection: 3-fold StratifiedKFold within every outer-training partition, with shuffle=True and random_state=42.
• Selection metric: mean inner-fold ROC-AUC. The selected candidate is refitted on the complete outer-training partition and evaluated once on the untouched outer-test partition.
• Reported dataset AUC: arithmetic mean of the 5 outer-fold ROC-AUC values.
• Statistical comparisons use dataset-level outer-CV aggregates, Friedman ranking, paired Wilcoxon tests, and Holm adjustment where shown.
• Each dataset uses the same recorded worker allocation for every model. Full-run wall-time comparisons use datasets with uninterrupted timings for every selected model.

Preprocessing and timing

• HUGIML receives the native pandas table, including categorical columns, and performs its own binning and pattern construction inside each training fold.
• HUGIML applies exact downstream canonicalization followed by one training-only VIF analysis. Original terms are preserved; patterns and augmented pairs with VIF greater than 5 are reduced in that order when an earlier preferred term explains at least 80 percent of their variance.
• Non-HUGIML models use a fold-local pipeline: numeric median imputation; categorical most-frequent imputation followed by dense one-hot encoding with unknown categories ignored.
• Source dataframe indexes are discarded before modelling; statsmodels-backed tasks use only columns declared in load_pandas().data.
• Preprocessing is fitted only on the relevant training partition. No test-fold information is used for preprocessing or parameter selection.
• fit_seconds measures the selected estimator's final outer-fold refit; tune_seconds measures inner-CV search; predict_seconds measures outer-test inference.

Model search spaces