Abstract: In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. However, phasing out of the traditional methods of human-vehicle interaction raises the question of the interaction between the self-driving car and the passenger, and with the increased degree of control and autonomy being given to the AI, makes it that much more critical. Recent works have tried to tackle this issue by allowing the passenger to give commands that refer to specific objects in the visual scene. Nevertheless, this is only half the task as the car should also understand the physical destination of the command, which is what we focus on in this paper. We propose an extension where we annotate the 3D end location that the car needs to reach after executing the given command, and evaluate multiple different baselines on predicting this destination location. From our results, we conclude that our dataset and proposed task are challenging. Additionally, we also propose a model that outperforms the prior works adapted for this particular setting.