Your memory is partially right.
You can retract your votes within five minutes (not 100% on the exact time here, but it's a relatively short window) of casting them. After that, you're committed to it unless the post is edited.
@bjb568 The short version is that it's in part to prevent abuse (downvoting a bunch of answers to make yours stand out better, then retracting those votes, for example) and in part to encourage thinking before voting.
@Anna But we're talking about tactical downvoting. The post may never be improved (it may be fine), and if it is improved I probably won't know (5 minutes delay is long after me leaving the page). So I would use tactical downvoting, never retract it, and probably nobody will ever find out (anon voting).
@bjb568 The point here is "don't vote for the wrong reasons and think before you click". Unlimited vote retraction doesn't make the scenario you just described any better.
Actually, it does. I would undownvote out of sympathy after my answer is (maliciously) promoted to the top spot. And people don't think, so "making them think before voting" doesn't really work. Especially if they don't know about it.
"I would undownvote out of sympathy after my answer is (maliciously) promoted to the top spot." - wait, what? I think you severely overestimate how many people feel bad about their malicious behaviour.
But I'm question banned and it's impossible to get out of it because all my questions are crap and deleted.
Feature request: Make self-deletion be positive for question-banning. Removing crap is good, not bad.
Feature request: Make votes not count toward question-banning on meta, instead use closed questions. Voting is different on meta, and great posts can be downvoted.
@bjb568 Not counting self-deletion opens the door wide for abuse - self-deleting crap to post more crap. (Not saying that you'd do that, but there are a lot of people who would.)
Anyway, the system can probably do a better job at determining if I'm evil… Other activity (answers, comments) can help. The score of the answers of the question can help. Accept rate can help. Maybe it needs to be manual (or at least mostly manual, the system can easily find evil people by spam/offensive/off-topic flags), since question-banning meta users seems to not work...