The author rebuttal went a long way towards satisfying the reviewers and all of them have recommended acceptance after discussions. The authors should go through the suggestions given by the reviewers (esp. about harder dataset experiments, better derivations, better ablations) and incorporate as much as possible in the camera-ready version of this paper.