On Hangouts for Android, new messages are notified by a sound or vibration or both. To completely disable hangouts notifications or for a specific conversation, do as follows:
To manage Hangouts Notifications (Which affects all your conversations):
1. Open Hangouts and tap the 3-line icon on the top left
2. Go to the Settings and select the account you want to manage notifications
3. Under the Hangout Notifications, you can change the notifications for message & invites and video calls separately
4. for example, if you want to disable sound notifications for video calls and enable the vibration, you need to make sure the Vibrate option is marked, and the Sound option is set to None (tap Sound , select Media Storage if asked, and then select the None option)
To manage Hangout Notifications for a Specific Conversation:
1. Open Hangouts on your Android device
2. Tap to open the specific conversation you want to manage the notifications
3. Tap the 3dot icon on the top right, and go to the People & Optioins settings
4. If you want to disable Hangouts notifications for this specific chat and mute it, make sure to unmark the Notifications option
5. Otherwise tap the Chat message sound or Call ringtone option and manage the sound or ringtone