OP here. Found a workaround for anyone with similar issue. You just have to put referrerPolicy="no-referrer" in your image tag.
Relevant stackoverflow post: https://stackoverflow.com/questions/53536329/images-are-loading-okay-on-a-chrome-tab-but-returning-403-in-react-code
tldr of the post:
"The server might have a referer check, IOW: it doesn't want you using direct links to this resource, but maybe download and host yourself, so that there server is not unduly bashed."
"The server checks the referer sent by the browser and will not return the image when the referer is from a different domain. To prevent this, in many current browsers you can use the referrerpolicy attribute of the img tag to ask the browser to not send the referer:
<img referrerPolicy="no-referrer" src="https://www.host.net/image.png" />"
"NineBerry is correct, but it is best practice to add an explicit cross origin resource sharing permission to the headers in the server that is hosting the images (assuming you own that server), rather than ignoring the referer."
Not sure if the same origin referer check is intended by Danbooru but adding the referrerPolicy to the img tag works.