Simple stuff like color profiles is important for displaying the image, but there's a lot of other stuff that isn't. I worked on a game that used PNGs for map files, and we stored collision and entity data in a kind of comment chunk. Image hosting sites would retain all the metadata but downscale the maps. Since they still had the metadata they looked like ordinary maps to the game, but they were broken because the graphical image (itself an ordinary piece of mediocre pixel art) was downscaled.
https://en.wikipedia.org/wiki/Portable_Network_Graphics#Ancillary_chunks
For PNG, I think all the text chunks that can hold long arbitrary data should be filtered out of sample images. Not sure what's available in JPEG metadata.