When I tried out the Cinemetrics software on the first 12 minutes of The World of Suzie Wong, I straightaway put in an extra couple of key presses through trigger-happiness. But more importantly, I was aware that on a couple of occasions my response was delayed, because my attention had been momentarily seized by what was going on across the cut, from the point of view of film editing technique.
This raises an important point about the use of the Cinemetrics recording software. It is founded on the assumption that each user’s reaction time is constant, to within 0.1 second.
Reaction time has been one of the most extensively studied variables in experimental psychology since it was introduced in 1862 by Wundt. The simplest form of test is for the subject to be presented with a series of flashes of light at random intervals, to which he is asked to respond by pressing a button as quickly as possible. Vast numbers of experiments show that the mean value of reaction time in this situation is about 0.2 seconds. Specifically, with the favoured subjects for psychological experiments, university students, the mean value for all experiments conducted so far is 0.22 seconds. However, reaction time varies from stimulus to stimulus for any individual, and beyond that, different individuals have different mean reaction times. In general, reaction time increases with age. And as distractions are introduced into the test situation, reaction time also increases. The situation when using the Cinemetrics software to record cuts is not the same as the simplest kind of psychological reaction test, because the stimulus has to be picked out from all the other things that you can see going on in the film. You can test your reaction time online with a set-up that has a certain amount of distraction in it by going to
(Alternatively, searching on the web for “sheep dash” will find a number of instances of this game.)
It does not matter for the purposes of Cinemetrics if your mean reaction time is longer or shorter than another person’s reaction time, but it does matter if it is not constant to within 0.1 second.
With this in mind I have checked the recorded Cinemetric results for three films against my frame-accurate measurements. The films analysed were la Signora senza camelie, posted by Yuri Tsivian, Verboten!, posted by John C., and Shadows, posted by Charley Leary. For the first of these I used a rather worn 16 mm. print, which may have some frames missing at the reel changes, and made the measurements on a Steenbeck, and for the other two I digitized repectively a VHS tape and a DVD into a non-linear editing program (an NLE) (Adobe Premiere), and then put markers on every shot transition. For fades and dissolves, I put the marker in the middle of the transition, not at its beginning or end. Then I opened the Adobe Premiere “project file” in a text editor and extracted the numbers corresponding to the cumulative frame counts buried in the middle of all the other stuff in the project file. As another Cinemetrician has already noted, you can probably get the same results by marking a cut on every transition using any NLE program, and then outputting the resulting edit decision list (EDL).
The comparison of the sets of results is not as simple as it might seem at first glance, for a number of reasons, First of all, not everyone starts and ends their counts at the same place. My standard procedure for the last thirty years has always been to start the count immediately after the last credit in the main title sequence (usually the director credit), and stop immediately at the appearance of the end credit. There is an exception to this, which I have not mentioned before this writing, when I am getting Average Shot Lengths. This is that I also count in the shots and their length in any pre-credit sequence running longer than one minute. However this does not apply in the particular cases under consideration, though they do include action under the main credits, which I do not count. The shots in these title sequences have been included in their recordings by the other people concerned, so I had to identify them and cut them out. Then I had to identify what speed the film, DVD or tape that they used was running at, and correct for that. Finally, all these results include missed or non-existent cuts, and this has to be dealt with before they can be aligned with my results. In the case of missing cuts, I have replaced the missing shots lengths with my values before making the comparison. To be specific, la Signora senza camelie misses one cut, and includes one extra, while Verboten! misses twelve cuts, and Shadows records one extra cut that isn’t there. My frame accurate values also have to be rounded down to the nearest tenth of a second for the comparison. Aligning the two sets of results for the same film entails identifying patterns of sequences of shots with approximately the same lengths, and in particular the longer shots, which usually stand out in a sea of much shorter shots in most films. After doing all this, the aligned results showed that I and the other Cinemetricians were indeed working with copies originating from the same source prints, even though Charley Leary was apparently using an NTSC transfer of Shadows, and I was using a PAL DVD. And we were working with the authorised revised and re-shot version of the film, not the recently rediscovered first attempt at making it. In the case of Verboten!, it appears that both John C. and I were working with PAL video copies. The procedures I have described were adopted to show the Cinemetric results for the various films in the best possible light.
It is not of course possible to determine the mean reaction time for each recorder from their results, but one can get the errors they made in the measurement of the shot lengths.
To give a flavour of this, I quote the differences between my results and those of Yuri Tsivian for the whole 102 shots under consideration in la Signora senza camelie:

0 0 0.1 0.1 0 0.8 1.2 -0.2 -1.7 1.7 0.4 -0 0.2 -0.2 0.2 0.2 -0.1 0 -0.2 0.4 -0.1 0.1 -0.4 0.4 0 -0.7 0.6 0 0 0.2 0.3 0.2 1.7 0.1 0 0.2 -0.3 0.3 -.10 0.2 0 0.4 -0.2 0.2 0 0 0 0 -0.7 -0.4 -0.2 0.1 -0.1 -0.2 0.4 0 0 -0.9 0.6 0 0 0 -0.2 0.6 0 -0.2 1.3 0.4 -0.2 0 -0.1 0 -0.1 -0.2 0.1 0.1 -0.2 1.4 3.5 0.4 -0.6 0.2 0 -0.1 0.1 0.1 -0.3 0 0.1 0.6 1 0 -0.1 0 -0.1 0 0 -0.1 0.2 -0.1 4.3

To get a better sense of this it is best to show the distribution of these errors in a graph:



One would expect the measurement errors to be normally distributed, and indeed the above shows a very rough approximation to a Normal (or Gaussian) distribution. If there were no errors greater than 0.1 second, then there would be just be a single very tall bar straddling the zero point in the above graph, and nothing else. An important variable describing error distributions are their Standard Deviations, which is 0.70 in this case.

The other two films studied have a larger, and more usual, number of shots in them, so I will just illustrate the error graphs for them, starting with Verboten!


The Standard Deviation for the errors about the correct values of the shots lengths in this case is 1.06, which is larger than the previous example.

For Shadows, the results are a lot better.


In this case the Standard Deviation is 0.54, and the error distribution is much more like a Normal distribution.

The general indications are that as long as the Cinemetrics shot length recording system is used, as it is at the moment, for getting general indications of cutting rates throughout a movie, then it is satisfactory. But even with the most accurate Cinemetrics recording so far found by me, which is for Shadows, the errors are still of a size which could make the results noticeably unreliable if they occurred when analysing a film with large numbers of short shots (down below 2 seconds). That is; when the possible errors are of the same order as the shot lengths being recorded. In particular, this would apply to the whole length of the very fastest cut films of recent times, which have ASLs below 2 seconds. The trick of running the DVD at half speed will of course cut the size of the errors in half, though not eliminate them. Of course, all the above assertions could do with experimental verification, as usual.
The alternative method that I have used to get the frame-accurate results for this study does take longer, but in the case of Verboten! and Shadows, it only took me about three hours to mark the cuts on each film.
I think it would be very helpful if in future, as already has been mooted, the people using Cinemetrics would note whether the source they have used is PAL or NTSC, or if the Cinemetrics recording was done during the projection of an actual film. And an indication of the start and end points of the Cinemetrics recording would be helpful as well.

Barry Salt,
January 2008