Breast Cancer Response Prediction — AMD ROCm

"looks like the work on the project started a month before the actual start of the hackathon, with a marginal improvement of the previous result during the time of the competition Material issues 1. README↔code discrepancy on pretraining data. [confidence: high] The README states: "Duke Breast MRI (n=922) ... Single timepoint, used for Phase LSTM pre-training." The actual pretrain_backbone.py loads only ispy2_features.pkl / ispy2_features_radimagenet.pkl — no Duke pickle, no Duke path, no Duke loader anywhere in the repo. Following the Quick Start reproduces a pipeline pretrained on I-SPY 2 itself, not on Duke. Either the README is wrong about what was actually done, or the reproducing code is missing. Either way, a reader cannot verify the claim. 2. The phase encoder was pretrained on the same 199 patients used in 5-fold CV. [confidence: high] pretrain_backbone.py does train_test_split(all_pids, test_size=0.2) over the I-SPY 2 feature dict. Then train_cv_v2.py does StratifiedKFold(n_splits=5) over the same patients and loads the encoder frozen. Every patient in the CV has had their imaging features fed to the autoencoder during pretraining. This isn't label leakage (the autoencoder reconstructs features), but it is representation leakage: the frozen encoder was tuned partly on patients it will later be evaluated against. At n=199 this matters. The clean fix is patient-disjoint pretraining (which is presumably what the Duke claim was meant to provide — see issue 1). 3. The advertised sensitivity/specificity are val-tuned and have optimistic bias. [confidence: high — the code says so explicitly] In train_cv_v2.py: # NOTE: Threshold optimized on val set → metrics (sens/spec/ppv/npv) have optimistic bias. AUC is the primary threshold-free metric for comparison. Yet the abstract leads with "81% sensitivity" and the README headlines "Sensitivity 81.3% / Specificity 70.7%." Per-fold thresholds were chosen via Youden's J on the held-out fold itself, so these are upper-bounded best-case operating points, not generalizable operating points. The honest headline number is AUC 0.771 ± 0.011 with the asterisk that operating-point claims require a held-out threshold-fitting set, which doesn't exist here. 4. The "0.926 best single fold" is not a metric. [confidence: high] With ~40 patients per fold and ~12 positives, fold-level AUC has very high sampling variance. Reporting the maximum across 5 folds is order statistics of a noisy estimator. It belongs in a results dump for diagnostic purposes, not in the headline table next to the mean. Its presence reads as cherry-picking. 5. "±0.011" is being interpreted as a confidence interval but isn't one. [confidence: high] That ± is the standard deviation of mean-AUC across 3 random seeds (each seed reruns 5-fold CV). The real statistical uncertainty about AUC given n_pos ≈ 60 and AUC ≈ 0.77 has SE ≈ √(0.77·0.23/60) ≈ 0.054, giving a 95% CI roughly [0.66, 0.88]. The relevant uncertainty is the patient-sampling uncertainty, not the seed-sampling uncertainty. ±0.011 makes the result look ~5× more precise than it is. 6. HR+/HER2− subgroup AUC is not distinguishable from random. [confidence: high] Subgroup table: AUC 0.571, 95% CI [0.386, 0.744]. 0.5 is inside the CI. The abstract describes this as "challenges from severe class imbalance" — the substantive statement is that the model has no demonstrated signal in this subtype, which is ~40% of the cohort. Framing this as a difficulty rather than a failure understates the practical limitation: for the largest molecular group, the model is, on the available evidence, indistinguishable from a coin flip. 7. The clinical claim is not supported by the numbers, and is the dangerous kind of overclaim. [confidence: high] Abstract: "...identifying pCR patients who may avoid unnecessary surgery." Apply Bayes at base rate ~30% pCR with the reported (optimistically biased) sens=0.81, spec=0.71: PPV ≈ (0.30 · 0.81) / (0.30 · 0.81 + 0.70 · 0.29) ≈ 54% So among model-predicted pCR patients, roughly 46% in fact have residual disease. Routing those patients away from surgery would mean leaving active cancer in situ in nearly half of the "avoid surgery" group. The clinical implication the abstract floats — even hedged with "may" — is not a defensible reading of these numbers, and the asymmetry of harms (missed cancer vs. unnecessary surgery) makes the framing worse, not better. The number you'd want here is at the high-specificity operating point — what's the PPV when spec ≥ 95%? — and that's not what's reported. 8. No external validation, but "validated" language. [confidence: high] The abstract says "validated on 199 patients from the I-SPY 2 clinical trial." In clinical ML the convention is that "validated" means a held-out external cohort. This is internal 5-fold CV. "