Removing the favorites from a child removes any votes given by each favoriter, if those vote records still exist. But like BrokenEagle said, after 90 days the vote records are gone, so when there are no upvote records to remove, the score won't decrease.
It could be changed so that removing a gold favorite adds a downvote to decrease the score. But you can't generally assume that each Gold favorite gave one upvote. Favorites added when a person was a Member didn't give upvotes. And up to a few years ago (can't remember when it changed), it used to be possible for a person to give either +1 by favoriting, or +2 by favoriting and upvoting. And to be really pedantic, it's technically possible to favorite a post then downvote it.
So in general, it's difficult to fix the scores accurately. I think ultimately the idea of replacing the image itself (issue #2949) will be a better route to go.