Sure I'm a Stage Manager on Broadway now, and all my xp is into this but i have high survival because at one point in my life i used to be a park ranger' That shit should be kept, not washed away to minmax better.
This. Skill atrophy is totally a thing, but it mostly strikes really low skills (yeah technically I took a sign language class ages ago, but I'd be hard-pressed to remember much now) or really high ones (it's easy to slip down from Olympic levels if you slack off). In the middle, humans are actually pretty amazing at retaining old skills and knowledge. Not always, mind you, but in general. I don't really see that as a compelling reason for allowing respecs.
But at the same time... if somebody came in as an elite sniper but wanted to do a story where over the next few months they stopped their hard-core marksmanship training and instead focused on working out, I'd have no problem letting them shift a point from firearms to athletics. Those sorts of minor, story-driven changes aren't jarring and I definitely think they should be considered on a case-by-case basis. They're just not "respecs" in my mind.