If you keep using that janky starter long enough, the starter gear might pop out of there completely, and will bang around the bellhousing.
(VERY noisy, and the next time you hit the starter, you hear it freespin)
No chance the starter IS engaged, just not powered? The extra drag maybe causing...