>>>find . -name '*.jpg' \! -exec bash -c 'test -f `expr substr "$0" 1 \( length "$0" - 4 \)`.mod' {} \; -print
It might be overkill to call bash for each jpg found using exec, especially when there are many jpgs...
@OP, in bash, for one directory. I leave you to find a way for recursive.
for i in *.[jm][op][gd]
do
prefix=${i%%.*}
if [ -f ${prefix}.jpg -a -f ${prefix}.mod ];then
continue
else
echo "$i" #do your image processing here.
fi