For a video camera, the frame rate and the exposure time are two separate parameters. The frame rate is the frequency at which images are recorded. The exposure time is the duration for which the sensor collects light to build up a single image. For a given frame rate, different exposure times can be configured, but the exposure time has a physical upper limit, because the exposures of consecutive frames can not overlap. For example, for a frame rate of 100 fps (frames per second), the maximum exposure time is 10ms.

The brightness of an image recorded by a camera depends directly on the exposure time. The longer the exposure time, the more light can be collected, the brighter becomes the image. In the regime of high frame rates, the maximum possible exposure times becomes so low, that it can be challenging or undesirable to increase the illumination of the scene to a level that yields a usable brightness of the images.

highrate_longexp makes use of two cameras to overcome the limitation of the exposure time that a single camera has for a given frame rate. Figure 1 shows timing diagrams to illustrate the idea. (a) shows the sequence of frames f(i) that are acquired with a single camera for a desired frame rate. The exposure time is T, making use of the full duration between two frame exposure starting points. (b) shows the sequence of frames g(i) that are acquired with half the desired frame rate. The exposure time is 2T, so in this scenario, the images become brighter by a factor of two. However, any downstream image analysis will now receive images only at half the update rate, so it has a less precise view of what is happening in the observed scene. (c) shows the concept of highrate_longexp. Two cameras are used with an exposure time of 2T. So the image brightness is the same as in (b), but since the cameras start and stop their acquisition in a time interleaved fashion, the total output of frames h(i) has the same update rate as in (a).

Figure 1: Timing diagrams

If an application makes use of a camera to react fast on changes in an observed scene, not only the update rate is relevant, but also the latency. Here, latency means the time difference between the scene changing from one state to another and the camera system providing a frame that reflects this change. (For an example latency measurement see lalelu_drums // Log #17: Latency) To provide a theoretical estimate for the latency in the described scenarios, the time at which the frames f(i), g(i) and h(i) are provided to downstream image analysis is marked with an asterisk (*) in figure 1.

The latest possible time the observed scene can change so that the change is reflected in the frames f(i), g(i) and h(i) is marked with an ampersand symbol (&). In all scenarios, this is at half the respective exposure time before the frame is completely recorded. If the scene changes later, the change can have an influence on the frame, but the frame is dominated by the older state of the scene, since this state was recorded during more than half of the exposure time. The latest possible time the observed scene can change so that the change is reflected in the preceding frames f(i-1), g(i-1) or h(i-1) is marked with a section sign (§). It is half an exposure time before the preceding frame is completely recorded.

The mean latency for the three scenarios can be estimated by taking the average between the duration from (&) to (*) and from (§) to (*), assuming there is no correlation between the frame triggers and the time of the change of the scene. The results are

This means that highrate_longexp (c) is superior to the case of reduced frame rate (b) not only in terms of the update rate, but also in terms of latency. Still, the latency for highrate_longexp is larger than for the case with the original frame rate and a single camera (a).

Another aspect of highrate_longexp is that due to the difference in the position of the cameras, there is a parallax between the images. While this parallax is desired in the case of a stereo camera, it is disadvantageous here. For the planned application of highrate_longexp in my other project lalelu_drums, I will treat this problem in an upcoming log entry there.