All holsters will give some wear - that's why I don't holster most of my guns - I typically only have 1 gun that is my carry gun anyway - so, I have no reason to holster the rest.
But, I would imagine that out of everything, nylon would wear the least. Also, if U can find one, I occassionally see nylon holsters with a terry-cloth type of lining. I have 1 like that that I bought in the late 1990s - it was made for a gun w/ a lasersight attached.
Anyway, maybe U can find a nylon holsyter w/ that lining at a gun show or something....
1 insertion of my USP compact into a leather holster made some small marks. So, leather will tear it up quick.