In this paper we propose a new hierarchical evolutionary algorithm that combines binary encoding and floating-point encoding to automatically select the beam directions and determine the weights of the selected beams. With traditional optimization methods the beam directions are fixed a priori by the operator in recognition of the fact that computer selection of beam directions is a difficult problem. In this investigation, we used a hybrid-encoding scheme. The binary encoding part of each chromosome was used to select the beam directions, and its corresponding floating-point encoding part of the same chromosome was used to determine the weights of those selected beams. Before beginning the optimization process, we set a constraint on the number of the beam directions we wanted in the final solution. We present three examples to verify this method. These examples differ with each other in tumor sites, problem sizes, and optimization parameters. Three-dimensional optimization results and statistical data showed that this method is feasible. We think this method can be easily extended to solve more complex target problems (such as nonconvex target problems). © 2000 American Association of Physicists in Medicine.