It is normal for wireless headsets to take around 1-2 seconds for the link to establish (for example when first answering a call).
A solution to get around this would be to programmatically raise the audio link during an incoming ringing signal so the link is live by the time the user answers. To be able to do this using the native interface you would have to disable audio sensing via IAdvanceSettings interface (AudioSensing = false). This only needs doing once, when you first attach to the device.
To activate the audio link, use IDevice.HostCommand.AudioState = MonoOn and to deactivate the audio link set to MonoOff.
However, if using this approach you might need to play your own PCM ring signal through the headset, because depending on the product the Plantronics "internal" headset ring tone may not be heard once the wireless audio link is established.