Eric Chen

LLM judges are promising for evaluating reasoned mathematical solutions, yet their scores remain prompt-sensitive and unstable. We evaluate 13 grading strategies on Putnam-AXIOM-Grading, a new 500-problem benchmark with 1,000 mathematically annotated candidate solutions, and IMO-GradingBench. The strategies span direct scoring, structured chain-of-thought, and multi-call pipelines such as G-Eval, Chain-of-Verification, and Debate. No fixed strategy dominates, and the strongest single default, comparative prompting, wins only on average. In hindsight, the best strategy per problem differs from the best fixed default in 76% of cases, revealing substantial oracle headroom for adaptive selection. Motivated by this, we train Mixture of Graders, a ModernBERT-large strategy router, using the fixed-strategy score matrix and human labels, then deploy it using only problem and reference features. On Putnam-AXIOM-Grading, MoG improves over the best fixed strategy across all open-weight judge models, with relative MSE reductions up to 27.8% across both Qwen2.5-72B-Math-Instruct and Mistral-Large-Instruct-2411 student responses. It also generalizes across students, with a router trained on Qwen responses outperforming the best fixed default on Mistral responses. The prescription for difficult mathematical grading is not a single best prompt, but a learned mixture over grader strategies.

LLM judges are a promising way to evaluate reasoned mathematical solutions, but their scores are prompt-sensitive, unstable and hard to interpret. We challenge two prevailing assumptions: that chain-of-thought adds little to agreement with human grades, and that multi-step pipelines such as debate or ensembles beat single-step evaluation. On two human-graded competition-mathematics benchmarks, unconstrained chain-of-thought tends to hurt evaluation consistency without moving performance much, while deliberately structured chain-of-thought recovers and often improves agreement — and comparative prompting, a single pass that weighs a solution against a reference, consistently beats the multi-step pipelines.

Existing mathematical reasoning benchmarks are nearing saturation, with some models clearing 90%, and are increasingly compromised by training-set contamination. Putnam-AXIOM pairs 522 university-level Putnam competition problems with a companion set of 100 functional variants, produced by programmatically perturbing variables and constants — a protocol that yields an endless supply of equally hard, unseen instances. OpenAI's o1-preview, the strongest model tested, scores 41.9% on the original set but loses 19.6 points on the paired variations; the other eighteen models all decline the same way, ten of them with non-overlapping 95% confidence intervals.

Telling human-written text from machine-generated text gets harder as language models improve. Subtask B of SemEval-2024 Task 8 asks not just whether a passage was machine-generated but which model produced it. We compared RoBERTa and DeBERTa encoders on the task; our DeBERTa model beat the RoBERTa baseline by more than ten points of leaderboard accuracy. PCA and t-SNE projections of the learned embeddings show the model separating the outputs of different LLMs into distinct clusters.

Telling apart the agents that modified bone at paleoanthropological sites underpins much of what we can say about how early hominins exploited large animals. Fracture pattern analysis has struggled with equifinality, and researchers dispute the replicability of existing methods. We present a new approach that extracts a far richer and more transparent feature set from 3D models of fragmentary bone, then applies supervised machine learning to separate hominin breakage from carnivore breakage, averaging around 77% accuracy across tests.

We investigate how the accessory parameter of the Heun equation relates to the properties of its monodromy groups, asking specifically which values of that parameter produce unitary groups — ones preserving a Hermitian inner product. Extending earlier work on the Lamé equation, we prove for a broad class of Heun equations that unitarity is tied to the traces of certain monodromy matrices, and turn that theorem into an algorithm for locating the accessory parameters that yield unitary monodromy. We report the values found, draw convergence maps showing the algorithm is robust, and derive an asymptotic formula matching the numerics.

Simulating fibrinogen in solvent atom by atom is prohibitively slow. We built a coarse-grained representation that groups atoms into larger interaction sites, cutting simulation cost while keeping the structural behavior that matters for how the protein folds and binds.