In the present communication, we have obtained the optimum probability distribution with which the messages should be delivered so that the average redundancy of the source is minimized. Here, we have taken the case of various generalized mean codeword lengths. Moreover, the upper bound to these codeword lengths has been found for the case of Huffman encoding.
Any message that brings a specification in a problem which involves a certain degree of uncertainty is called information and it was Shannon [
then there exists a uniquely decodable code with these lengths, which means that any sequence
where
Later, Campbell [
and
respectively, where
is Campbell’s [
is Kapur’s [
is Renyi’s [
Recently, Parkash and Kakkar [
and
Further, the authors provided two source coding theorems which show that for all uniquely decipherable codes, the mean codeword lengths
and
respectively where
is a Kapur’s [
is measure of entropy developed by Parkash and Kakkar [
This is to emphasize that in the entire literature of source coding theorems, one can observe that the mean codeword length is lower bounded by the entropy of the source and it can never be less than the entropy of the source but can be made closer to it. This phenomenon provides the idea of absolute redundancy which is the number of bits used to transmit a message minus the number of bits of actual information in the message, that is, the mean codeword length minus the entropy of the source. The objective of the present communication is to minimize this redundancy in order to increase the efficiency of the source encoding. For this purpose we have made use of the concept of escort distribution as follows:
If
where
The aim of the present paper is to obtain the optimum probability distribution with which the source should deliver messages in order to minimize the absolute redundancy. To obtain our goal, we have taken into consideration the above mentioned generalized mean codeword lengths. Moreover, the upper bound to these codeword lengths has been found for Huffman [
Let us assume that for discrete source
where
In order to minimize the average redundancy, we resort to the following theorem:
Theorem 1: The optimum probability distribution that minimizes the absolute redundancy
Proof: To minimize the redundancy, we need to minimize
subject to the constraint
To prove this, we first of all, find the extremum of
So, in order to extremize
where
Now
Letting
Substituting (2.6) in (2.4), we get
Substituting (2.7) in (2.6), we get the result (2.2).
Now,
We see that
Also,
So,
Thus,
Thus, the minimum value is given by
Again, the necessary condition for the construction of uniquely decipherable codes is given by
Therefore, from (2.9), we have
NOTE: It is to be noted that
Therefore, for this case, (2.2) becomes
Similarly, if we consider the codeword length
where
Theorem 2. The optimum probability distribution that minimizes the absolute redundancy
Proof: We will find the extremum of
Let us consider the Lagrangian given by
where
For an extremum, let
Using (2.14), we get
Substituting (2.17) in (2.16), we get (2.13).
Also,
and
So,
that is,
Note: Again in this case also, if the source is Huffman [
Next, we will find the upper bound on the codeword lengths
Theorem 3. The exponentiated codeword length
if the source is encoded using Huffman procedure.
Proof: The exponentiated codeword length
where
Considering (2.12), (2.19) becomes
where
We need to find the extremum of
For this purpose, we first of all, find the extremum of
So, we consider the Lagrangian given by
where
Letting
Now,
Using (2.23) in (2.22), we get
that is,
Now,
We see that
Also,
So,
Therefore,
Thus, the maximum value is given by
Theorem 4. The mean codeword length
if the source is encoded using Huffman procedure.
Proof: The exponentiated codeword length
We need to find the extremum of
So, we consider the Lagrangian given by
where
Letting
Since
Substitute (2.28) in (2.27), we get
Now,
Also,
So, the mean codeword length
Note-I: For the case of Campbell’s codeword length
where
The absolute redundancy in the case of Campbell’s [
Note-II: Absolute redundancy when we use Kapur’s[
where
Theorem 5: The optimum probability distribution that minimizes the absolute redundancy of the source with entropy
Proof: To minimize the redundancy, we need to minimize
subject to the constraint
To prove this, we first of all find the extremum of
So, in order to extremize
where
Letting
Substituting (2.33) in (2.31), we get
Substituting (2.34) in (2.33), we get the result (2.29).
Now,
We see that
Also,
So,
Therefore,
The minimum value is given by
Theorem 6. The Kapur’s [
if the source is encoded using Huffman procedure.
Proof: Proceeding as in Theorem 2.3, we can prove the Theorem 6.
The authors are thankful to Council of Scientific and Industrial Research, New Delhi, for providing the financial assistance for the preparation of the manuscript.