Help solve a crime

Three years ago, a young girl was kidnapped and killed in Denmark. The police believe they got a grainy surveillance video of the car used that night. Unfortunately, not much detail is available - but what if the video pixels could be projected onto a moving 3D surface thus revealing new detail? Can threejs be setup to animate a moving object along a scene with video pixels being thrown at it from a specific view point?

The car is arriving from the left, making a u-turn, and leaving the scene again. In between camera and car are various obstacles like door panels, signs and vegetation.

Here is a link to the video in question: https://newsbreak.dk/wp-content/uploads/2017/06/200617-video-hvid-bil.mp4

This is the location in Google Maps: https://bit.ly/2sfHwn4

Any ideas on how to approach this would be highly appreciated.

Um, I don’t understand how this should reveal new details. Are you referring to an existing approach in context of computer vision (maybe documented in a scientific paper)?

In any event, the Danish police should have specialists capable enough to sufficiently analyze such videos. May I ask why are you asking for help in such a forum? Keep in mind that three.js is an engine for real-time rendering. It’s focus is not on (artifical) computer vision. Maybe asking in the OpenCV space makes more sense.

3 Likes

Yes. This is what is done in my viper demo (on static geometry): Projecting a photo onto geometry to facilitate photo manipulation In that case, it depends on knowing (being able to guess) the geometry, motion and camera projection beforehand, though. Some of the necessary information can be obtained and applied during a crime investigation, such as camera specifications, position, rotation, building geometry, hopefully knowledge of the car model (ideally a 3D model of the car). In addition to this, street lighting, weather/sunlight (sun direction) at the time and place of the recording etc. will be useful for the analysis.

You are touching onto computer vision and specifically photogrammetry, which is outside the expected expertise of three.js forum members, as @Mugen87 says. And I honestly don’t expect Danish police experts to be able to apply state of the art methods (or develop the methods needed). I suspect they will say “that’s CSI magic, and not possible in real life”. To some extent they are right. You cannot just click “enhance image” to get more detail, but a video of a car contains multiple noisy and aliased (and compressed!) measurements of a rather tidy process. I guess it may be possible to, e.g., extract car plate numbers from videos, even when each frame does not reveal the number sufficiently well. (Two examples I found now are linked below.)

Now I have watched the video. The car is at large distance, and viewed through multiple windows, if I see right. To compensate properly for the refraction in the windows, it may be necessary to make recordings of controlled geometry and texture (e.g. a plate with a chequerboard-pattern) through the same windows. You could thus be able to generate refraction maps for the windows (or for the combination of windows?), and then “invert” the refraction of the original video.

Links:

  1. https://www.researchgate.net/publication/3428025_Superresolution_of_License_Plates_in_Real_Traffic_Videos

  2. https://articles.forensicfocus.com/2019/06/10/how-to-read-a-moving-low-quality-license-plate-using-amped-fives-perspective-stabilization-and-perspective-super-resolution/

PS: I named camera specification among necessary input, but if you can get to the location of the video recording and make detailed measurements of the hallway, you will probably be able to derive camera position, orientation, aspect ratio and FOV even without access to the camera. Likewise, knowing the suspect car model may help deriving refraction maps for the windows, without having to measure them using a reference. But it probably is a lot harder, though, not least because the motion estimation then is coupled to the estimation of refraction.

PPS: Even though I can confirm that it is possible to project an image from a virtual camera onto geometry in a scene, that is not necessarily what you are interested in doing. Collecting good information about the scene and camera will be useful for trying to reproduce the video by simulating (realistic) motion of the same type of car (probably a Hyundai i30 hatchback, according to the Danish police) with different license plate numbers, window refraction maps, outdoors light settings etc., and optimizing all these variables for video reconstruction. (I don’t know if this is really viable.)

1 Like

Thank you for pointing to OpenCV. I was asking the threejs forum, because the software is so versatile with a huge user group. My apologies, if the question is out of place.

I’m essentially trying to verify the car model. About five different models are viable candidates. They are almost identical in shape and looks except for small variations in windows and tail lights. The Danish police ended up concluding that the model probably was a Hyundai i30 “or similar in shape”. It would be exciting, if threejs (or OpenCV) could verify this.

You’re right: Maybe the Danish police have done this already, but at least nothing has been reported publicly.

Thank you @EliasHasle for some very interesting thoughts. I doubt, trying to see details in the number plate is realistic in this case. The ambition is merely to reveal details from the exterior body of the vehicle.

This is the planned approach:

  1. Crop the video
  2. Cache the frames in arrays
  3. Remove static contents
  4. Make some kind of invisible moving 3D-object in threejs
  5. Do preliminary distance, angle and location evaluation
  6. Project each video pixel from the video onto the object (slowly fading)
  7. Run the animation and adjust the 3D-object until all pixels settle on the surface

Hopefully, this will reveal some exterior dimensions comparable to car models available in 2016.

Is this what you have in mind? While my viper demo does project video onto a scene, it does not track the resulting scene/object positions of the video pixels.

@EliasHasle

I think so, but open to suggestions. It could perhaps prove easier to skip the 3D object and just devide the video frames into columns of pixels, place them at a certain distance and altitiude in the scene and adjust from there by “playing” the video frames back and forth…

If you find the project interesting, feel free to join. :slight_smile: