The position of ships in Euclidean space can be described by Euler angles, and the equilibrium equations of floatation are therefore established by the three independent variables: waterline height, pitch angle and roll angle, according to Newton's Law. The solution of the nonlinear system of equations can be converted to a numerical optimization problem, which can be solved by the unconstrained optimization method. Through the theoretical research and calculation example based on the MATLAB programming, it shows that the traditional Newton's method can run into the problem of irreversibility of Jacobian matrix, which may cause the interruption of iteration, whereas the trust-region method can guarantee the convergence of the objective function with less dependence on the initial guess, and thus it is more reliable and suitable for the industrial problems. In addition, this method can also be applied to other stability problems, such as the calculation of the righting arm curves of the intact stability and damage stability, and the variational axial heel of the platform.