The 6dB gain from balanced drive is huge, and obvious to headphone listeners with any experience. Of course it has to be adjusted for. The sonic differences I've noticed, regardless of volume, are significant; some amps more than others. In particular, the 2006 Headroom Maxed Home and Ray Samuels Apache showed a striking improvement in balanced mode; their SE modes were very mediocre, if not disappointing. There were rumors the Ray Samuels amp circuits were based off an older Headroom Home design, so it's not surprising they behaved similarly in this regard.
Some headphones also benefit more or less from balanced drive. For example, the ATH-L3000 "Leatherhead" benefits some from balanced drive, but not as much as the Sennheiser HD600/650/800, which have always been particularly responsive to balanced drive.
And then cables. I've compared many kinds, both SE and balanced, on the Sennhieser HD600/650. The differences here are generally MUCH more subtle than those between amps, or between SE/balanced. But the Zu Mobius is a single exception -- and I've owned a few copies over the years, both SE and XLR. This cable significantly alters the tonality of the headphones.
Balanced, done right, will double the amp's slew rate, reject common mode noise (lower noise floor), and reduce distortion. So for "all out assault" headphone systems, it's almost always the right choice. And anyways, electrostatic headphones absolutely require balanced drive, and they're the best headphones
