In the free space laser communication(FSO) system, pointing error and atmospheric turbulence seriously affect the communication link of the system, which are the two main factors for the deterioration of the link performance. Based on the theoretical analysis of the joint channel, the channel model was established, and the expression of the average channel capacity of the direct detection system and coherent detection system was derived by using the Meijer G function. By comparing the simulation results in Matlab, given the pointing error and atmospheric turbulence conditions, the signal light beam divergence angle is not as small as possible, but there is an optimal beam divergence angle. The beam divergence angle can ensure the system performance, making the transmission power minimum. The coherent detection system suppress the influence of atmospheric turbulence on the system, thus the performance of the system is effectively improved.