To remove and delete contacts on Nexus 4 (or other Android devices) you need to do as follows. Since this action is not reversible note that you always sync your contact list with your Gmail account or create a backup from your contacts.
1. Open "People" App
2. Tap the name of the contact you want to remove (You need to tap the name and NOT the picture)
3. Tap the three dot icon on the top right
4. Select "Delete"
5. Confirm the Process