Voice chat active when I join parties

I have my voice chat on push to talk. Whenever I join a party, I have to press my push to talk button once to turn off the microphone. Otherwise it always starts mic permanently on until I press the push to talk button. (Im not talking about auto connect)