Value-Aware Recommendation Modeling Through Reinforcement Learning and Offline Interaction Data

Abstract

This paper studies the modeling and optimization of recommendation algorithms driven by reinforcement learning, abstracting the recommendation process into a sequential decision-making task under continuous interaction to maximize long-term cumulative rewards. In this framework, user history and item-side information are encoded as state representations. Recommendation actions are generated by a policy network in the candidate space, while a value network and a critic network are used to estimate future rewards and construct low-variance learning signals, thereby improving the stability and effectiveness of policy updates. To adapt to the training characteristics of offline interaction logs, the method performs bounded and normalized processing on the feedback signal at the reward level, and combines baseline and advantage modeling to mitigate the bias caused by gradient variance and short-sighted decision-making, achieving synergistic optimization of policy learning and value estimation. Experimentally, a unified evaluation setting is constructed on a public dataset. Multi-method comparisons show that the proposed framework consistently outperforms the compared baselines on ranking quality and hit-rate metrics, indicating that it leverages historical interactions more effectively to make robust recommendation decisions. Overall, this paper provides a reinforcement learning recommendation modeling paradigm oriented towards long-term value optimization, offering structured methodological support for the design of trainable recommendation strategies in dynamic preference scenarios.