Proposals to relocate a rugby club as part of a scheme to build more than 350 homes has been met with huge opposition.