The same principle as when splitting blocks applies: The largest doesn't
have to be in the queue, because we already refine with respect to it
when refining the second-to-last block.