Abstract: Understanding how a city’s physical appearance and environmental surroundings impact society traits, such as safety, is an essential issue in social artificial intelligence. To demonstrate the relationship, most existing studies utilize subjective human perceptual attributes, categorization only for a few violent crimes, and images taken from still shot images. These lead to difficulty in identifying location-specific characteristics for urban safety. In this work, to address this problem, we propose a large-scale dataset and a novel method by adopting a concept of “Deviance" which explains behaviors violating social norms, both formally (e.g. crime) and informally (e.g. civil complaints). We first collect a geo-tagged dataset consisting of incident report data for seven metropolitan cities, with corresponding sequential images around incident sites obtained from Google street view. We also design a convolutional neural network that learns spatio-temporal visual attributes of deviant streets. Experimental results show that our framework can reliably recognize real-world deviance in various cities. Furthermore, we analyze which visual attribute is important for deviance identification and severity estimation. We have released our dataset and source codes at our project page: https://deviance-project.github.io/DevianceNet/