Abstract: Realistic fine-grained multi-agent simulation of real world complex systems is crucial for many downstream tasks such as reinforcement learning. Recent work has used generative models (GANs in particular) for providing high fidelity simulation of real world systems. However, such generative models are often monolithic and miss out on modeling the interaction in multi-agent systems. In this work, we take a first step towards building multiple interacting generative models (GANs) that reflects the interaction in real world. We build and analyze a hierarchical set-up where a higher level GAN is conditioned on the output of multiple lower level GANs. We present a technique of using feedback from the higher level GAN to improve performance of lower level GANs. We mathematically characterize the conditions under which our technique is impactful, including understanding the transfer learning nature of our set-up. We present three distinct experiments on synthetic data, time-series data, and image domain, revealing the wide applicability of our technique.