In this paper, we investigate the formation control of multiple unmanned aerial vehicles (UAVs), specifically unmanned aircraft, in an obstacle-laden environment. The main contribution of this paper is to integrate the formation control, trajectory tracking, and obstacle/collision avoidance into one unified optimal control framework. A nonquadratic avoidance cost is innovatively constructed via an inverse optimal control approach, which leads to an analytical, distributed, and optimal formation control law. The stability and optimality of the closed-loop system are proven. In addition, the proposed optimal control law is dependent only on the information from the local neighbors, rather than all UAVs' information. Simulation of multiple UAVs' formation flying demonstrates the effectiveness of the integrated optimal control design with desired behaviors including formation flying, trajectory tracking, and obstacle/collision avoidance.