The complex dynamic characteristics and changeable marine environment pose great challenges to the design of the unmanned underwater vehicle (UUV) controller. In practical application, the parameter of the controller will be fixed after frequent manual debugging, which is unable to adapt to the changes of the environment during the control. In view of the above problems, a parameter adaptive S-plane control method based on the reinforcement learning is proposed referring to the adaptive control idea. The adaptive control method is used to optimize and automatically tune the controller parameter under different environments. This method is trained by Q-learning algorithm. The optimal mapping between the input status and output action is sought through the self-learning mechanism of Q-learning. Simulation results show that the proposed method can adjust the controller parameter in real time with excellent control effect and environment adaptability.