Abstract: Interactive recommendation with natural-language feedback can provide richer user feedback and has demonstrated advantages over traditional recommender systems. However, the classical online paradigm involves iteratively collecting experience via interaction with users, which is expensive and risky. We propose an offline interactive recommendation to exploit arbitrary experience collected by multiple unknown devices (policies). Direct policy learning with such fixed experience suffers from the distribution shift. To tackle this issue, we leverage a neural correction estimator via adversarial training, which can be seamlessly incorporated into the standard RL objective. Empirical results on real-world datasets demonstrate the effectiveness of our proposed framework.