The main reason why you automatically assume that other people do stuff to you, and that it affects you, is because you assume that you are currently aware of the entire content of your own mind.
Your little 'conscious self' believes that everything about you, everything that is in you, you must be conscious of right now. If you're not conscious of it, you believe it does not exist and is not a part of you, except that maybe you accept you have 'memories' stored somewhere. But you don't generally accept that you have 'reactions' stored somewhere, or 'wounds', or 'emotional content'.
This is how most of us perceive and engage and live our lives, as little conscious selves that believe we are fully aware of every part of ourselves, as though any and all previous trauma or upset simply does not exist in you anymore.
So then what happens is, someone comes along and offers to be a victimizer and does something 'to you'. You refer to your vast catalog of immediate personal awarewness and you do not have awareness of anything `in you` that even remotely resembles upsets, guilts, fears, sins, hurts, pains, unhappiness, grief, etc. All you're currently aware of is your present mood and a handful of thoughts and that's about it.
So this person does something and all of a sudden, inside you there appears a big giant elephant made of guilt and fear. It feels terrible. It came out of NOWHERE! And because it was not there before, you're pretty sure that must mean the victimizer PUT IT THERE or caused it to be there. You don't even think it is YOUR upset. Everything about it is their fault because, if this whole scenario is the truth, and if this content was not in you previously, then obviously it MUST be their fault.
So this is our default ego reaction mode. Stuff just pops into our experience internally that we're convinced was not there before, and it doesn't feel good, so we blame its 'creation' on someone else - they are the cause.
So when you look at it, this all stems from the fact that you have a stinking heap of shit LURKING in your mind, OUTSIDE your awareness, unconsciously lingering in the background. Out of sight, out of mind. Its hidden behind denial and repression and suppression. It's covered over with layer upon layer of rejection and avoidance. And each layer of it conceals and hides even more deeper layers that you don't even realize are `in there`.
If it weren't for this unconsciousness, and if you were fully aware right now of the ENTIRE content of your mind, and if someone came along and 'did something to you', you would immediately know that NOTHING CHANGED. Your content was already there. All your guilts and hurts and fears were already there in your awareness and you were already experiencing them fully. The person did not add anything, so that means.... you know they are innocent.
Long story short.... you blame everyone and everything for your own experience of life because they keep causing your own stuff to come into your awareness and you FORGOT it was there, because you buried it and hoped it would go away. Other people come along to RE-MIND you - to get you to look at your shit and become aware of the stuff you DO NOT WANT, so that you will actually take productive steps to really heal it and undo your belief in it being true.
So everyone is just trying to help you. Everything is for your benefit. And for that you can be grateful.
Someday when you are done purifying your mind, releasing old pain, healing wounds and undoing all trace that anything even seemed to happen at all, you WILL be walking around in FULL awareness of all of yourself, because you will be awake. And you WILL know that everyone is innocent, and you will know that nobody can add to or take away from what is already in you.