In this paper, we present a simple yet effective rotation averaging pipeline, termed Incremental Rotation Averaging (IRA), which is inspired by the well-developed incremental Structure from Motion (SfM) techniques. Unlike the traditional rotation averaging methods which estimate all the absolute rotations simultaneously and focus on designing either robust loss function or outlier ﬁltering strategy, here the absolute rotations are estimated in an incremental way. Similar to the incremental SfM, our IRA is robust to relative rotation outliers and could achieve accurate rotation averaging results. In addition, we propose several key techniques, such as initial triplet and Next-Best-View selection, Weighted Local/Global Optimization, and ReRotation Averaging, to push the rotation averaging results one step further. Ablation studies and comparison experiments on the 1DSfM, Campus, and San Francisco datasets demonstrate the effectiveness of our IRA and its advantages over the state-of-the-art rotation averaging methods in accuracy and robustness.