Abstract: Neighborhood gentrification plays a significant role in shaping the social and economic status of both individuals and communities. While some efforts have been made to detect gentrification in cities, existing approaches mainly relies on estimated measures from survey data and requires substantial work of human labeling yet fails to characterize the physical appearance of neighborhoods. To this end, we introduce a novel approach to incorporate data like street view images and POI features to represent urban neighborhoods comprehensively at each timestamp. We show the effectiveness of the proposed methods with previous research on gentrification measures: each neighborhood representation we trained not only indicates its gentrification status, but also could become supplementary parts for the current measures and valid resource for researchers and policy makers.