I think I’ve settled on the latter. Disagreement is maybe best communicated by the absence of an upvote? And downvotes work best when they signal something that is just off base, and while not reportable, is not appreciated at a broad cultural level.
The thing is, downvotes mean whatever the person downvoting feels like they mean. Personally I don’t downvote anything, only upvote.
It would be interesting to have a bot that looks for bad actors—by that I mean users who abuse the downvote and do not use it the way the community agrees that it means. And have a mod review and take action if necessary.