This s a mansion/hotel/some big building that's in between dimensions that likes to pull people from different dimensions. However, it has specific taste and only pulls from a select group of people, so it ends up pulling someone's alternates. That group of people seems to be people we know in one way or another.

No one knows how to get back and it just sends off and pulls in people when whatever controls this place wants too.