Unfortunately its even earlier meaning still has confections to describing special needs people in formal medical terms. In that retard = slow, retarded = slowed. And of course the other meaning like flame retardant, to slow fire from spreading.

But you’re right that it shouldn’t be a slur.