From Parallel Token Prediction to Scalable Language Models

15. December 2026 11:30 - 12:30 | Zürich Oerlikon, OAT X 11, ETH AI Center Foyer

Autoregressive language models generate text one token at a time, creating an inherent bottleneck that limits inference speed. Can we overcome this constraint without sacrificing generation quality? In this talk, I will present a line of work toward parallel language generation, progressing from Parallel Token Prediction (PTP) to Parallel Language Models (PLMs). Both approaches build on the insight that dependencies between tokens can be mediated through auxiliary random variables rather than sequential computation, enabling multiple mutually consistent tokens to be generated in a single forward pass. PTP establishes the theoretical foundations and explores distillation and inverse autoregressive training, while PLMs extend these ideas to end-to-end training through a variational objective. We demonstrate that PLMs can be pretrained at scales of up to 3 billion parameters while approaching autoregressive modeling quality. Interestingly, inference speedups follow a power law with model size, suggesting increasing benefits at larger scales. Our post-trained models match or exceed autoregressive performance on downstream tasks while achieving inference speedups of up to 5.7×, pointing toward a scalable alternative to sequential language generation.