Bard portals being blocked

It's happened a couple of times where when going through bard's E, Magic Journey, it at first makes the sound shows you going into the portal, but stops and spits you out before you travel, where you have to try and go in again, maybe causing you to lose a sticky situation.