Wi-Fi calling works ONLY if both parties are on Wi-Fi ?

I have xfinity mobile for a long time and never had an issue with Wi-Fi calling from home but since last few months I started having call quality issues (choppy audio, dropped calls) so I started looking into it and found that its not using wifi calling even if its enabled on my phone. Also figured out that it works fine when both parties are on wifi but does not work if the other party is not on wifi. This is not how its supposed to work. Wifi calling should work for whoever is on wifi and has it enabled. 

As expected, customer service gave me the run around for several calls and no resolution yet. Have tried everything - reset network settings twenty thousand times. If I chat from the app they completely ignore the problem I am talking about and start selling me stuff!! yeah right first get the stuff you already sold me working properly.

Has anyone experienced this wifi calling issue ? Really frustrated by this and hoping to find some resolution here.

Thanks!