In the worst-case scenario, you might be faced with a lawsuit from a disgruntled user who wasn’t able to access your website. If you are a government organization, agency or contractor of theirs, you are obliged to comply and will be fined if you don’t do so.
If you are a private entity, you might get away with it but not for long. Not only because the laws are changing from day to day and what was a “recommendation” yesterday might become a “requirement” tomorrow but also because you are not providing equal opportunities for all of your site visitors (and yes, potential customers if you run a business). Plus, it might potentially affect your website ranking (see below).