Toward a theory of information processing

Abstract

Information processing is performed when a system preserves some aspects of the information encoded by its input while it suppresses others. To describe a system's information processing capability, input and output need to be compared in a way invariant to the signal's form and how it represents information. We describe an approach to quantify information processing based on applying controlled changes to the input and observing the corresponding output. Information-theoretic distance measures---those that reflect the data processing theorem---are calculated on the input and output separately and compared. The best candidate for the distance measure to describe signal processing is the information theoretic distance measure called the Kullback-Leibler distance (also known as the relative entropy). Properties of the resulting information transfer ratio are used to derive fundamental information processing properties of systems and interconnected systems.