Abstract: Current state-of-the-art salieny detection models rely heavily on large datasets of accurate pixel-wise annotations, which cost a lot of time to prepare. There are some weakly supervised methods developed for alleviating the problem, such as image label, bounding box label, and scribble label, while point label still has not to be explored in this field. In this paper, we propose a new point-supervised dataset (P-DUTS) by relabeling the DUTS dataset. In P-DUTS, there is only one labeled point for each salient object. To infer the saliency map, we first design a adaptive masked flood filling algorithm to generate pseudo labels. Then we design a point-supervised saliency detection model based on the transformer to produce the first round of saliency maps. However, we find that due to the sparseness of the label, the weakly supervised model tends to degenerate into a general foreground detection model. To address this issue, we propose a Non-salient suppression (NSS) step to optimize the erroneous sliency maps generated in the first round, and leverage them for the second round of training. Comprehensive experiments on the five largest benchmark datasets demonstrate our method outperforms the previous state-of-the-art methods trained with the stronger supervision and even surpassed several fully supervised state-of-the-art models. The code is available.