Using Jones polarization theory, the polarization models of optical terminal and atmospheric transmission channel were carried out for a space coherent optical communication system, and their effect on transformation of the signal light was analyzed quantitatively. The relationship between the polarization mismatch of signal light and local oscillator light and the mixing efficiency was derived in this paper. In addition, using the polarization control algorithm and the PCDM02 type piezoelectric ceramic polarization control device (PCD), an automatic feedback polarization control compensation device was proposed, which can effectively avoid the polarization mismatch, thus ensuring a high frequency mixing efficiency and improving the overall performance of system.