Human-Like Visual Attention Improves Gaze-Augmented Reward Modeling for Large Language Models

Professor Sang Ho Lee’s group has developed a new approach to reward modeling for large language models (LLMs) that incorporates a psychologically grounded model of visual attention during reading. In reinforcement learning from human feedback (RLHF), which is a widely used approach for aligning LLMs with human preferences, a reward model learns from people’s reported preferences between LLM outputs. Recent approaches have incorporated eye-tracking information into reward models, using gaze behavior as an additional signal that can help predict human preferences. Existing methods, however, typically assign gaze information only to the individual token being fixated, despite extensive evidence from cognitive psychology that visual information is acquired from a broader region surrounding the fixation point.

Professor Lee’s group has modeled visual attention as a spatially distributed signal that extends to surrounding text. Incorporating this representation into consistently improved human preference prediction across different LLMs.

“Existing approaches use eye-tracking information as a token-level signal, but human attention does not operate in such an isolated manner,” said Prof. Sang Ho Lee. “Our results show that representing gaze in a way that more closely reflects how people actually distribute visual attention can improve reward modeling. More broadly, this study shows how findings from cognitive psychology can provide principled guidance for designing AI systems.”

Specifically, the team introduced a learnable asymmetric Gaussian distribution that redistributes gaze information from each fixated token to neighboring tokens. The widths of the leftward and rightward attentional distributions were learned directly during reward-model training.

Experiments showed that the proposed approach improved preference-prediction accuracy over the token-wise gaze representation in all tested LLMs and datasets. The largest improvement was observed with Llama-3-8B-Instruct, where accuracy increased from 68.67% to 74.69%. Importantly, the learned attentional distributions tended to extend farther to the right than to the left, qualitatively matching the asymmetry reported in human attention during left-to-right reading.

These findings suggest that established theories of human cognition can provide a useful foundation for designing more effective AI systems. At the same time, they raise the possibility that AI systems built on cognitively grounded frameworks may independently develop information-processing patterns that resemble those observed in humans. This work has been accepted to NeurIPS 2026.