case study · maximum likelihood · event-history

Time to Event

Survival analysis on the NCCTG lung-cancer cohort — estimating how long patients live and what shifts that risk, across three lenses: Kaplan-Meier curves, a Weibull accelerated-failure model, and a Cox proportional-hazards model, each built from the likelihood up.

R / survivalPython · numpy · scipyMLE from scratchPartial likelihoodEfron ties

228

patients (165 deaths · 63 censored)

270 / 426

median survival, men / women (days)

0.60

hazard ratio, women vs men

// the problem

Survival (or "event-history" / "duration") analysis answers a different question than ordinary regression: not whether something happens, but how long until it does. The outcome pairs a time with an event status — here, days of follow-up and whether the patient died or was still alive when observation ended.

The wrinkle that breaks standard tools is right-censoring: 63 of the 228 patients had not died when the study closed. We don't know their true survival time, only that it exceeds what we observed. Survival methods are built to use censored observations correctly, assuming censoring is independent of risk given covariates.

two functions describe the whole process

Survival S(y) = Pr(Y > y): the probability of lasting beyond time y. Hazard h(y) = f(y) / S(y): the instantaneous rate of failure given survival so far. Every model below is really a statement about the hazard.

// non-parametric · Kaplan-Meier

The Kaplan-Meier estimator is the product-limit count of who's still at risk: at each death time, multiply survival by the fraction of the risk set that made it through. No distribution assumed — censored observations included. I implemented the product-limit estimator and the log-rank test directly.

Fig 1. KM survival by sex. Women's curve sits above men's throughout: median survival of 426 vs 270 days. The log-rank test (p = 0.001) rejects equality of the two curves.

// modeling approaches

To bring covariates in, we assume a form for the hazard. The three standard choices trade assumptions for flexibility:

Exponential — h(y) = τ, a constant hazard. No duration dependence.
Weibull — adds a shape parameter so the hazard can rise or fall with time. Fit here as an accelerated failure time model: positive coefficients mean longer survival.
Cox proportional hazards — leaves the baseline hazard unspecified and models only how covariates multiply it. Positive coefficients mean higher hazard. Buys flexibility at the cost of the proportional-hazards assumption.

// parametric · Weibull AFT

The Weibull AFT writes log(T) = Xβ + σW with W an extreme-value error. I coded its censored log-likelihood — events contribute a density, censored cases a survival probability — and maximized it with scipy.optimize, recovering standard errors from the observed information. It converges to the published log-likelihood of −1147.1 and matches survreg's coefficients to the decimal.

Weibull accelerated-failure-time model · Surv(time, status) ~ age + sex
term	value	std. err	z	p
(Intercept)	6.275	0.481	13.04	<0.001^*
age	−0.012	0.007	−1.76	0.078⁺
sex (female)	0.382	0.127	3.00	0.003^*
Log(scale)	−0.282	0.062	−4.56	<0.001^*

scale = 0.754 · log-likelihood = −1147.1 · N = 228 · positive coefficient → longer survival

Both signs match the curves: being a woman lengthens survival (β = 0.38), and each additional year of age shortens it slightly. The model also yields quantities of interest — 90% of 65-year-old men survive past 64 days, versus 94 days for women.

Fig 2. The parametric Weibull curves (thick) smooth the non-parametric KM steps (thin) for a 62-year-old, preserving the sex gap. Where KM reads the data, the Weibull imposes a single survival law.

// semiparametric · Cox PH

The Cox model is the workhorse: it never commits to a shape for the baseline hazard, estimating covariate effects from the partial likelihood instead. I implemented it with Newton-Raphson, including Efron's correction for tied event times (coxph's default) and the analytic information matrix for standard errors — reproducing coxph exactly.

Cox proportional-hazards model · Efron ties · 165 events
term	coef	HR = e^β	std. err	z	p	95% CI (HR)
age	0.017	1.017	0.009	1.85	0.065⁺	0.999 – 1.036
sex (female)	−0.513	0.599	0.167	−3.07	0.002^*	0.431 – 0.831

concordance = 0.603 · LR test = 14.12 on 2 df, p < 0.001 · ^* p < .05, ⁺ p < .10

Fig 3. Hazard ratios on a log scale. Women face 0.60× the hazard of men (CI 0.43–0.83, excludes 1) — a ~40% lower rate of death at any given time. Age nudges hazard up ~2% per year, but its interval grazes 1.

takeaway

All three lenses agree, which is the point: the non-parametric KM, the fully-parametric Weibull, and the semiparametric Cox each recover the same finding — women with advanced lung cancer survive markedly longer than men — while making progressively different assumptions. Knowing which assumption each method buys, and what it costs, is the actual skill.

// method notes

Censored-data likelihood — handling right-censoring correctly in the Kaplan-Meier estimator and the Weibull MLE.
Partial likelihood & tie handling — a from-scratch Cox model with Efron's approximation and an analytic information matrix, matching coxph's coefficients and standard errors exactly.
Three modeling philosophies — non-parametric, parametric, and semiparametric estimators side by side, with assumptions made explicit.