Actually if you add a delay after Sidekick on the voice channel, it does delay the voice but doesn't delay the send to the music bed. So the music bed level can be set to drop just before the voice is heard.
It's quite interesting to do it like that - you have to extend the attack time of the ducking quite a bit to get an acceptable fade on the music, and the delay time you choose is quite critical too (I used about 100ms) - but it works fine.