A popular guest house’s plan for a large extension, which it argues is “vital” to help it fight competition from budget hotels, has been rejected.