Agree with Loop for most hotels, but have also been in Nam Plas situation wherein the key keeping practises look wanting in a busy hotel.
If all yer valuables are in the room safe, then leave the key, no ones gunna steal your nice shirts.. but if its really lax, then take the key with you, front pocket , next to your cash.
I tend to keep my right hand in my pocket a lot, so I prefer there.