This is actually easy to do, if you know this one weird trick!
Turn on Enable Minimum and/or Maximum Dates and then set Minimum Date: to -1 days ago.
You have to choose a value of 1 or larger for the maximum date. If you leave that field blank it will be set to 0 and you won't be able to choose any dates at all.
This is what the date picker looks like with the settings above when clicked on today (April 30, 2020):
(Aren't I nice? Screen shots and everything! )