11. Data compression and data reduction

For long-term preservation purposes, target formats employing data reduction (often called, incorrectly, data “compression”) should not be used when encoding from original analogue or linear digital recordings. Such so-called “lossy codecs” based on perceptual coding result in the irretrievable loss of parts of the primary information. The results of such data reduction may sound and look identical or very similar to the unreduced linear signal, but further use of the data-reduced signal will be much more likely to result in degradation of the primary content.

While there is no objection in principle to the use of lossless (fully reversible) compression, any resultant saving in storage costs would need to be offset against the increased risk that the tools required to decode the files might be unavailable or insufficiently supported in the future. Both lossy and lossless data compression schemes produce data streams that are more susceptible to minor read errors than are linear encoded streams, and therefore the content of those compressed streams is more likely to be corrupted to a much larger extent by those errors than a linear encoded stream would be.

This archival principle should also be applied, whenever possible, to the creation of original recordings made with the intention of being archived. However, if content comes to an archive having been recorded in a data reduced, non-linear format however, it must be preserved faithfully as-is.

Comment:

Data reduction is a powerful tool in the dissemination of audiovisual content. Its use for preservation, however, is counter to the ethical principle of preserving as much of the primary information as possible. Data reduction does not permit the restoration of the signal to its original condition and will, in addition, limit the further use of the recording because of the artefacts generated when cascading perceptually coded material, for example, in the making of a new programme incorporating the original sounds and images.

Because of the sheer amount of data required to store digital video signals, the use of data reduction for production formats has been and remains widespread. Ideally, non-linear encoding formats should be preserved in their original form. A major problem may arise however, when the format of origination is of a proprietary character such as the MiniDisc and DVCAM (see IASA-TC 04, 5.5.12.1; IASA-TC 06). The primary information of such recordings may be logically migrated to a preservation supported format, or the coding may be kept as-is. This will often be the decision of the archives digital preservation policies.