5 lbs. of fat is much bigger than the 5 lbs. of muscle.
Five pounds are still five pounds. Muscle does not weigh more than fat but fat takes up more space (volume) than five pounds of muscle.
A person with more body fat will always look bigger even if they weigh the same as someone who is leaner.
It is important to stop being obsessed with body weight and start paying attention to body composition. The quickest way to change body composition is through the diet and regular exercise.