I don't think there are any circumstances under which you can withhold payment of your fees. If new installations are introduced which you did not vote for, you can opt out, i.e. not contribute to the cost, on the understanding that you forfeit your right to use the new facility, but otherwise, I think by not paying your fees in full you may find yourself on the wall of shame (debtors list). If you want to make a legal case of it, you may be able to pay your fees to a court, pending the outcome of the case, but by simply not paying, you would be in breach of your obligations according to the Gospel of Horizontal Law.
As you rightly say, there must have been a saving in costs by not opening the pool, but I can only surmise that the savings were swallowed up elsewhere on unexpected things that were not budgeted for. The fact that the annual fees were increased despite the saving on pool maintenance would seem to confirm that the overall budget fell short. Perhaps there were emergency repairs in some other area? If not, then at the very least there should be a positive bank balance accumulating, which is no bad thing in any community. When (not "if") some unexpected repair does crop up, perhaps you won't have to cough up an "extra" payment to cover it.
Of course you are entitled to a justification of the fees you pay. The normal time to discuss this would be at the AGM, but you should be able to approach your president or administrator at any time if you have concerns about how the budget is being used.