yeah, i'm with you regarding our first world problem.
i had the same concerns, so i attached only my house key to the leather fob case to minimize its size. all additional keys stay in my bag. with the leather case, the key is bulky in my pocket but it's worth it to me in order to minimize scratches on the fob. yes, i'm a bit ocd.