You have to figure out how to let go of the hold others' opinions have on your actions. For years that was a huge obstacle for me. There are so many things I wish I had done or wish I had done sooner. I would be worried about having to fight with all the people around me. Eventually I just decided, ---fudgesicle--- it, if I consider it a positive thing, and it's not hurting anyone, and it's not negatively affecting my family in some way, I'm going to do it. And if the people around me want to give me a hard time then I'll just stop answering the phone. I don't care if relatives say I'm crazy, or if the neighbors see me working out in the yard, or if people give me crap because I'm not fitting into the box they want me in. You just can't live your life that way and be happy.
Eventually I noticed that it doesn't even matter what you're doing. You could be going back to college, studying a martial art, building an addition on the house, anything. The people around you that are doing nothing will try to find ways to pull you back to their level. Ignore them and do your thing.