I only wear gloves for construction, handling grungy stuff, and for inspections. Back in the day, on the bike, I was an idiot and rarely wore gloves. Never have in the car, though I have thought about it. Three main pair right now - in the cars: old black Mechanix gloves, probably a decade old. Two pairs of Stihl leather gloves, largest size they make - I think my hands are small but can only rarely find gloves that aren't too tight. These were actually factory defects (over sized for marked size) left over at one of my in-laws implement dealerships. Mother in law salvaged them.
At home: fifty year old pair of white mules, made in the USA and stilll going strong.
At the store: various leather work gloves for emergency use.
If you like the gloves in the car, they make you feel more comfortable, not stress about the leather, etc..., wear them. who gives a shit what anyone else thinks.