When the amount of population diminishes, the expressions negative population growth and negative population growth rate are used.
See also Overpopulation