What is Restricted Mode of Instance Startup ?

Restricted mode startup limits access to only users with DBA permission, it's designed for maintenance, not security. To start a restricted instance, use the command:

STARTUP RESTRICT

To limit access to certain users, just use the normal Oracle access control. First, ensure that your users all have their own username/password, if they're sharing access credentials, you can't tell them apart so can't control access. Then you can just unlock or lock the accounts of people you want to allow or deny access:

ALTER USER scott ACCOUNT LOCK;
ALTER USER scott ACCOUNT UNLOCK;

Beyond that, you can get more sophisticated and control access to different objects within the database.