Is there a way that we can be able to be controlled while the toy user is “Invisible”? ?

I know it allows us to play on our own while in that status, but I don’t see why it can’t work while being controlled...