Forays into Reliability, Efficiency, and Energy of Language Models

abstract:

Language model based AI systems have given us much to admire: complex reasoning, ability to write code, use tools, agentic solutions and much more. Reliability and efficiency have become key hurdles to making the leap from admiration to impactful adoption. In this talk, I will present three threads of research we pursue in addressing these hurdles:

1. Reliability -- We have created a range of benchmarks over the years to assess reliable reasoning. I will briefly present AppWorld, our benchmark for interactive coding agents, and a recent extension AppWorld-UL that requires user interaction to reduce uncertainty. I will also discuss our recent work on self-expressed uncertainty in LLM reasoning and its calibration issues.

2. Efficiency -- Small Language Models present a viable alternative in multi-agent settings. However, their learning capacities limit effective use for complex problems. I will present our recent work on progressive subtask training to improve SLM abilities.

3. Energy -- The massive compute requirements of LLMs translate directly into energy requirements. I will present our recent work on predicting energy consumption of models and in inferring macro-impacts of LLM training on the power grids in the US.

Bio:

Niranjan is an Associate Professor in the Computer Science department at Stony Brook University, where he heads the Language Understanding and Reasoning lab (LUNR). Prior to joining Stony Brook, he was a post-doctoral researcher in the University of Washington, and was one of the early members of the Allen Institute for Artificial Intelligence. He completed his PhD in Computer Science from the University of Massachusetts Amherst. His team’s work on AppWorld received the Best Resource Paper ACL 2024, his work on energy efficiency of mobile networks and search applications received a Test of Time award at IMC 2023.

When he is not writing abstracts, he paints them: www.instagram.com/b__niranjan/