This organization needs a change of attitude IMMEDIATELY!!! If you’re on the Board of Directors, maybe you can "give back" by helping to fix it.

The COACHES should be policing their players and their parents. The UMPIRES should have ABSOLUTE AUTHORITY over the game, the coaches, parents and other spectators, including the authority remove ANYONE from the premises. If they need backup, the other responsible and reasonable parents and coaches need to STEP UP and support their authority PERIOD. YOU and the board should be available to resolve disputes and address appeals at your regularly scheduled meetings.

Maybe I’m not telling you anything you don’t already, know. I know it’s not "the good days" of Little League either. Having been involved in youth athletic associations all my life, as a player, umpire, coach, and fan, I realize things will not always go as outlined but what you’ve got is a serious problem waiting to happen.
If you can’t affect some change, maybe you should consider resigning. As the "sole" authority, you are going to be the lightning rod and you may find yourself involved in some serious personal and maybe even legal predicaments.