Yeah, but if no one ever comes back, then what? Do they just disappear mid-sentence or whatever? If we assume people are just being put back to the time they came from, for it to be possible for someone to disappear, then there would have to be timelines where we actually do disappear in our worlds. Time and space aren't linear, so when other universes start getting involved, everything that has happened and can happen has to be accounted for.

[The Infinite Universe Theory. Or part of it, at least. But in that vein, there should be infinite copies of this world out there too, which seemed incomprehensible given how many other worldly influences there are here...]

That's the only way different worlds and universes can impact each other. And the reason they're normally kept apart.