Domain transfer through deep activation matching

We introduce a layer-wise unsupervised domain adaptation approach for semantic segmentation. Instead of merely matching the out- put distributions of the source and target domains, our approach aligns the distributions of activations of intermediate layers. This scheme ex- hibits two key advantages. First, matching across intermediate layers introduces more constraints for training the network in the target do- main, making the optimization problem better conditioned. Second, the matched activations at each layer provide similar inputs to the next layer for both training and adaptation, and thus alleviate covariate shift. We use a Generative Adversarial Network (or GAN) to align activation dis- tributions. Experimental results show that our approach achieves state- of-the-art results on a variety of popular domain adaptation tasks, in- cluding (1) from GTA to Cityscapes for semantic segmentation, (2) from SYNTHIA to Cityscapes for semantic segmentation, and (3) adaptations on USPS and MNIST for image classification.