You are not being ripped off, you are being exploited and coerced by a criminal.
+1. Stay well away from this people, the farther shot you can get of this person/people the better. When I first worked, I started off working independently and for about 6 months I had absolutely no problems worth noting (aside from TW's and a couple of d*cks kicking off in texts)
I then went to work for an agency because my friend at the time insisted they send you lots of rich and nice clients, lots of work and really good bookings, and I had more problems in 2 months than I did for the entire time I worked for myself. They sent me alsorts of horrible people and I felt so unsafe. I had to kick people out, threaten clients with the police and alsorts.
I'm working again now and I'm working independently. What I learnt from that was that there is no better judge than your own instinct, and nobody cares about your own safety the same way you do.
It may seem scarier working independently, but you'll be in charge of your own bookings this time. Nobody else. Just make sure you have a security buddy and procedures to make sure you're safe.