When the ego attacks, it does not see you for who you really are.
Its belief in sin and guilt heavily skews its perception. It perceives in you a sinful guilty person, which isn't really in you.
It then addresses that sinful guilty person which it is hallucinating, and thinks that it is talking to a real person, or to "you".
It's not about you. It's not really personal.
The ego cannot see you in your real self. Your real self is invisible to it, just as the ego is invisible to the real self because it does not exist.
When the ego is seeming to attack, it is actually inside its own private world where nothing is real. It literally cannot even fight its way out of a paper bag.
So really when someone is attacking, they are attacking their own projected hallucination. They're not attacking the you that you really are.
Your only real issue is if YOU believe that you are sinful and guilty, because if you do so, you will perceive that there is a very close correlation between what the other person is claiming about "you" - which is really about their hallucinated sinful self - and what YOU think about you. Not being able to tell the two apart - your hallucination vs their hallucination, you'll bridge the gap and perceive that they are attacking you.
YOU are only ever attacking you, and THEY are only ever attacking themselves.
To be fully free of such attacks, claim your innocence! Then you will know you are not being targeted by attackers and will recognize that they are actually just hallucinating, attacking themselves, and are against themselves. It's not even about you at all!
How free and unafraid you will be when you learn to see that you are not the target of the ego at all, because you are not the ego. And all the people who ever seemed to attack you, and all of your belief that you even COULD be attacked, will disappear in the light of the truth that people ONLY do stuff to themselves.