It’s never too late to reevaluate and remove abusive relationships from your life. Whether old friends, mates from school, parents or siblings, absolutely no one is entitled to your time and emotional energy. By giving yourself some distance, you may realize contacts maintained out of social obligation or fear of loneliness were actually anchors preventing you from seeking your own happiness.
Nothing wrong with continuing them as is or returning to them after some time to find clarity, either. But it is worth remembering, all of this is entirely under your control.