Radiation therapy aims at delivering a prescribed dose to cancerous targets using high-energy radiation beams, while sparing dose to surrounding normal tissues and organs at risks. For this purpose, a treatment plan is customized for each individual patient, where parameters in a treatment plan, e.g. beam direction and fluence, are adjusted. Such a problem is mathematically formulated as an optimization problem and is solved with numerical algorithms. This talk will first give an introduction to the treatment plan optimization problem in radiotherapy, including intensity-modulated radiation therapy (IMRT) and volumetric modulated arc therapy (VMAT). It will then focus on a particular problem in IMRT, beam orientation optimization (BOO), which tries to find a solution that contains nonzero fluence map at only a small number of beam angles to achieve a dosimetric objective. We noticed that the objective of the BOO problem is equivalent to finding a fluence map that is sparse at the beam angle level. As such, we introduce a sparsity energy into the total energy function, which takes an L2 norm of beamlet intensities within each angle and then takes a weighted L1 norm over angles. Such an energy term favors solutions with nonvanishing fluence map at only a few beam angles. During optimization, the weighting factors in the L1 norm are adaptively adjusted. Starting with all candidate angles, the optimization process identifies unimportant orientations gradually and removes them without largely sacrificing the dosimetric objective. The whole process terminates when a target number of beams is achieved. The developed BOO algorithm is found to be effective for identifying important beam angles, which leads to better plan qualities than unoptimized beam configurations.