some survival advise please (or words of encouragement)

i am lvl 20 and struggling with a stealth melee build on survival.
my special stats are
3 strength with big leagues rank 3
4 perception with lock picking 2
1 endurance with no perks
3 charisma with lone wanderer 2
1 intelligence no perks
9 agility with sneak 3, ninja 2 and blitz 1
9 luck with bloody mess 1, idiot savant 2, better crits 1 and crit banker 1

i dont want to lower the difficulty because i think ill enjoy the game more later but my slow progress is frustrating. i rushed to the railroad to try and get Ballistics weave asap but i cant seem to survive any missions in the city yet. is my build bad? have a specked wrong or should i just be more patient?