Everyone has a room here, but you don't have to stay in the Temple. I think a few gems actually do live in town.

[He leads her down the hallways, eyes searching the doors.]

Every door has a gem that matches the room's owner, and it only open to that person. Well, unless you connect rooms with someone else. It's pretty awesome in terms of security, but one downside is that the rooms are not soundproof.