Switch to Sharp instead of imagemagick?
https://sharp.pixelplumbing.com/
Sharp is much faster than imagemagick and doesn't need all of install files of imagemagick.
I did some quick testing:
Sharp: 400ms for 574kb file
Imagemagick + mozjpeg: 1.6s for 323kb file
We'd have to figure out if we can make the file size any smaller to match what we are getting now. Sharp also currently doesn't support custom profiles (https://github.com/lovell/sharp/issues/1323) but their default jpeg profile is the latest web-friendly sRGB ICC profile.