The criminal pictures are probably mugshots. The non-criminal pictures are not.
This can introduce unwanted bias. Mugshots are taken while the subject is under stress, being just arrested and meeting the prospect of jail time, and probably feeling anger, guilt or regret (either for their deed, or about being caught). Mugshots are taken against their will, and this means you’ll look different on a mugshot then when you voluntarily pose for a nice portrait picture.
This reflects in the facial expression - eyes and mouth, and a generally rather tense, frightened or defiant expression. A person usually will not smile on a mugshot for example. When viewing these pictures, our brains pick up on those stress signals - we rate the facial expression as “non-normal” and in the context of a test like this then proceed to judge “criminal”.
So all this test probably does, is prove we can recognize someone under stress (and photographed against his will) in pictures. Not that we can recognize criminals.
To even out the test, all non-criminal pictures in the test should be of people under stress, for example pictures of people who just heard their doctor say “you have cancer”. They should be taken against their will as well.
Added to this: in addition, things like whether the picture is taken with warm light (sunlight or studio light) or a cold crimelab flash, and demanded straight and frontal posure instead of more casual head orientation, makes quite a difference in how we perceive them.
So I’d say a lot of critique is possible about the scientific design of this “test”.
