Journal of Modern Physics
Vol.07 No.02(2016), Article ID:63237,16 pages
10.4236/jmp.2016.72025
Category Theoretic properties of the A. Rényi and C. Tsallis Entropies
György Steinbrecher1, Alberto Sonnino2,3, Giorgio Sonnino4,5
1Physics Department, Faculty of Science, University of Craiova, Craiova, Romania
2Karlsruhe Institute of Technologies (KIT), Department of Electrical Engineering and Information Technologies, Karlsruhe, Germany
3Ecole Polytechnique de Louvain (EPL), Université Catholique de Louvain (UCL), Louvain-la-Neuve, Belgium
4Department of Theoretical Physics and Mathematics, Université Libre de Bruxelles (ULB), Brussels, Belgium
5Military School (RMS), Brussels, Belgium

Copyright © 2016 by authors and Scientific Research Publishing Inc.
This work is licensed under the Creative Commons Attribution International License (CC BY).
http://creativecommons.org/licenses/by/4.0/



Received 9 December 2015; accepted 26 January 2016; published 29 January 2016
ABSTRACT
The problem of embedding the Tsallis, Rényi and generalized Rényi entropies in the framework of category theory and their axiomatic foundation is studied. To this end, we construct a special category MES related to measured spaces. We prove that both of the Rényi and Tsallis entropies can be imbedded in the formalism of category theory by proving that the same basic partition functional that appears in their definitions, as well as in the associated Lebesgue space norms, has good algebraic compatibility properties. We prove that this functional is both additive and multiplicative with respect to the direct product and the disjoint sum (the coproduct) in the category MES, so it is a natural candidate for the measure of information or uncertainty. We prove that the category MES can be extended to monoidal category, both with respect to the direct product as well as to the coproduct. The basic axioms of the original Rényi entropy theory are generalized and reformulated in the framework of category MES and we prove that these axioms foresee the existence of an universal exponent having the same values for all the objects of the category MES. In addition, this universal exponent is the parameter, which appears in the definition of the Tsallis and Rényi entropies. It is proved that in a similar manner, the partition functional that appears in the definition of the Generalized Rényi entropy is a multiplicative functional with respect to direct product and additive with respect to the disjoint sum, but its symmetry group is reduced compared to the case of classical Rényi entropy.
Keywords:
Rényi Entropy, Generalized Rényi Entropy, measured spaces,monoidal Category

1. Introduction
The discovery of two related generalizations of the classical Shannon entropy [1] is a remarkable coincidence in the history of abstract probability theory and statistical physics. A. Rényi introduced a possible generalization [2] of the classical Shannon entropy by pure axiomatic extension of the Fadeev axioms [3] [4] that uniquely defined the Shannon entropy. On the other hand, the generalized entropy [5] [6] introduced by C. Tsallis was useful to extend the classical maximum entropy principle such that the heavy tailed distributions observed in a large scale of physical processes [7] -[10] , could be derived from (generalized) maximum entropy principles. The interest in the study of the generalizations of the Shannon entropy in the recent years is due to the multiple applications of the Tsallis and Rényi entropy or the associated Rényi divergence [7] [8] [11] [12] . We also mention that similar to the classical H theorem of L. Boltzmann, the generalizations of the Rényi entropy, as well as the original Rényi entropy, are Liapunov functions for a large class of stochastic processes described by generalized Fokker-Planck equations, more exactly by Fokker-Planck equation where the drift term and the diffusion tensor are itself dependent on some external random variable [13] . We mention that in the case of suitable singular limiting procedure, both the Tsallis and Rényi entropies give the same limit: the Shannon entropy. The classical and generalized Rényi entropies are additive while the Tsallis entropy is not. Despite the Rényi and Tsallis entropies give the same results in the case of problems associated to the determination of the probability density function from the Maximum Entropy principles, because they are algebraically related by simple formulae, the non- additivity of the Tsall is entropy generated many discussions in the physical literature. On the other hand, by formulating the basic axioms [2] , A. Rényi introduced new concepts (incomplete random variables and incomplete distributions) that were not included in the standard terminology of the probability theory. Also the formulation of the Postulate 5’ [2] , is not the simplest, mathematically natural.
Because the measure of information is a basic scientific concept, in this work we develop a formalism in the framework of the category theory [14] [15] for the study of generalized entropies. The category theory is the branch of mathematics that plays a central role in the logical foundation and synthesis of the whole contemporary mathematics. In particular, the category theory allows avoiding the paradoxes of the classical set theory. Category theory has application in informatics [16] . In order to highlight the natural structures related to generalized entropies, we use the central concepts of the modern mathematics.
The paper is organized as follows. In the Section 2, Subsection 2.1, we define a special category related to measurable spaces (referred to as MES), enabling the introduction an associated basic functional zp (see the forthcoming Section for his exact definition). Both the Tsallis and Rényi entropies, as well as the distance in lp spaces, may be expressed in terms of this functional. In the Subsection 2.2, we define the direct product of the objects in MES and we prove that the functional zp satisfies a compatibility relation with respect to this product i.e., it is multiplicative. This multiplicative property is equivalent to the additivity of the Rényi entropy. In the Subsection 2.3, we define the disjoint sum (or the coproduct) of the objects in MES, and we prove that the functional zp satisfies a compatibility relation with respect to coproduct i.e., it is additive. Note that this property is equivalent to one of the postulates characterizing the Rényi entropy. The proofs that both product and coproduct possess a universal property and that the direct product and coproduct can also be defined for morphisms of the category MES, can be found in the Subsection 2.4. In the Subsection 2.5 we show that, by extending the category MES with the introduction of the unit object and the null object, the category MES becomes to a monoidal category.
Section 3 deals with the axiomatic characterization of the functional zp. We demonstrate that there exists a universal exponent p (the same for all the objects of the category) that characterizes completely the functional zp (hence, also the Tsallis or Rényi entropies) up to an arbitrary multiplicative factor. In Section 4, it is proven that the main properties of the Rényi entropy, which are used in the axiomatic and category theoretic formulation, can be reformulated in order to be generalized to the case of the generalized Rényi entropy (GRE). The symmetry properties of GRE are studied in Subsection 4.1. Appendix 1 shows that the Rényi divergence can be expressed in terms of the Rényi entropy. The proof of the universality (with respect to all the objects of the category MES) of the exponent defining the Rényi or Tsallis entropies can be found in Appendix 2. In Appendix 3 some algebraic results related to the symmetry of GRE are proved.
2. The Category-theoretic properties Related to Rényi and Tsallis Entropies
2.1. Definitions
Our definitions include as a particular case the original definition of the generalized entropies [5] [6] and [2] . Our basic construction that will play the role of the object of the category MES is derived from the well known concept of measurable space [17] [18] . Guided by statistical ideas, in order to take into account the negligible sets we specify also an sub-ideal of the σ-algebra of measurable sets. The objects of the category MES consist of triplets
with X denoting the phase space (for instance, it is a symplectic manifold in the case of statistical physics or, in the case of elementary probability models, finite or denumerable set) and
is the σ-algebra generated by a family of subsets of X, respectively. We also denote with
an ideal of the σ-algebra
having the meaning of negligible sets. Let us now postulate the completeness property. From
and
results
. The morphisms of the category MES with the source
and range
are the measurable maps Φ from x to y, which are nonsingular, i.e. such that
. From the completeness property results the ideal property, i.e. if
and
then
. Note that it is possible that
contains only the empty set (as, for example, in the case of atomic spaces).
Remark 1 At first sight it would be more natural to consider the objects as measure space triplet
containing the measure
We denote with













finite norm (pseudo norm, respectively): more precisely,

for some non-negative density


where 




In this framework, for a given measurable space 


which in the case of discrete distribution, X a denumerable set, 

For a given measurable space


the density ρ in the Banach space for 




These relations give the geometrical interpretation of the generalized entropies (for further information Refs to [13] ).
Remark 2 The study of the generalized entropies helps us to better understand the classical entropy. For








The corresponding generalized entropy



Consider now a measure space 


Note that the Rényi divergence [2] [12]

is related to the Rényi entropies (see Appendix 1). Note that when x is a finite or denumerable set, if we denote with 






Remark that, in this particular case, 


2.2. Direct Product of Measurable spaces and the multiplicative Property of Zp[MX, μX, ρX]
In the framework of the our formalism, the multiplicative property is the counterpart of the Postulate 4 in the Rényi theory [2] . In the following we overload the tensor product notation “




Here 











Consider now the measures




with


We have the following basic proposition
Proposition 3 Let


Then we have


The validity of this statement follows directly from the definitions of the direct product, the Rényi entropy and the functional zp.
2.3. Coproduct of measurable spaces and theadditivity of the Functional Zp[MX, μX, ρX]
Let us study now the property encoded in the Postulate 5’ related to the Rényi entropy theory (Ref. [2] ), trans- cribed in the measure theoretic and category language and re -expressed in the term of the functional 


Definition 4 The coproduct of measurable spaces 



Here, 

















Let 


We restrict our definition of coproduct to finite terms. An example of (denumerable infinite) coproduct is the grand canonical ensemble.
Remark 5 If 



From the previous definition of the direct sum and the functional 
Proposition 6 The reformulation of the Postulate 5’ (Ref. [2] ) reads: the functional 

2.4. Universal Properties of the direct product and Direct Sum in the category of Measurable spaces
In the following we prove that the basic binary operations on measurable spaces, the direct product and the direct sum, defined in the previous section, have universality properties in the category of measurable spaces MES.
Consider the direct product 






Proposition 7 In the category MES the applications



Proof. The measurability of 


From the previous Proposition 7 results immediately the following Theorem
Theorem 8 In the category MES, the direct product has the universal property. Let







where

Proof. The morphism θ is induced by the application 












In conclusion the direct product operation has the natural functorial property, so the multiplicative property Equation (24) of the functional 
Proposition 9 In the category MES, consider the objects





Proof. The injections










By reversing the arrows, in analogy to the Theorem 8, we obtain the following result.
Theorem 10 In the category mes the direct sum of the objects has the following universality property. Let denote with





where

Proof. The morphism 










In conclusion, the direct sum operation has natural category theoretic properties. Hence, the additivity property Equation (29) of the functional 
2.5. The Monoidal Categories associated to product and Coproduct
We recall the following
Proposition 11 [15] Let 








For the pair of morphisms 





We denoted with








Let 

If in the category 

Similarly, by duality arguments, we have the following result for the direct sum (coproduct)
Proposition 12 [15] Let 





For the pair of morphisms





We denoted with










If in the category 

We emphasize that, despite the fact that the construction of the direct sum is dual to the direct product, from the previous proposition (12) the functor G is a covariant functor. In the category mes we have an unit object as well as the null object. The unit object is denoted with





Conclusion 13 The category MES is a monoidal category both with respect to the product 

3. Axioms
We expose another approach, based on category theory, to the problem of the naturalness of the choice of the family of functions 






Note that 

Then, the Postulate 1 (the symmetry property) and Postulate 5’ (the additivity property expressed in Propo- sition 6) can be generalized as follows. Postulate 1 & Postulate 5’


for some Borel measurable function 

The last requirement result by considering the case when the support of 



In our settings, the analog of the Postulate 4 (the additivity property) [2] is the multiplicative property given by Equation (24) and Proposition 3. By using Equations (24), (34), (36) and (37), and by continuity of the functions




By arguments similar to the proof of the uniqueness, from Theorem 2 [2] ), we get Equation (33) (for details see Appendix 2): there exists an universal family of functions, independent of X, parametrized by the positive parameter p such that



4. The Generalized Rényi Entropy (GRE)
Remark that all of the definitions of the classical, Rényi, Tsallis entropies contains only set theoretic and mea- sure theoretic concepts, no supposition on the auxiliary algebraic or differentiable structure associated to the measure space are assumed, so their definitions can be used t, continuos or discrete distributions. In the case of discrete measured space the classical definitions of the entropies Equations (7), (13)-(15) are invariant under the permutation group of the elements of the discrete set. This invariance encodes the assumption of complete apriory lack of information about the physical system, this absolute ignorance is lifted by the specification of the probability density function. On the other hand, consider the case when the measure space has the product structure

such that



Suppose that the probability measure on 

The GRE’s associated are [13]



We remark that in the definitions Equation (48), the role of the variables 

In the limit case

4.1. Symmetry Properties of GRE
In order to prove that in the case of the GRE the symmetry group is reduced to some subgroup, we consider only a special case: the spaces 






We use the array notation 

It is invariant under the transformation (see Lemma 16)


where the transformation 





Suppose we are in general case, when the indices i, a has completely different physical interpretation. Its is clear that the measure of information of such a system cannot be invariant under the permutation group 













Similarly we are interested to find the subgroup 



By using the Corollary 17, we obtain the following conclusion concerning the symmetry group of GRE, com- pared to the symmetry group of the classical Rényi or Tsallis entropies.
Proposition 14 The symmetry group 



where 







where the map 










is the direct product 


where 


In conclusion, in this particular case, the symmetry group associated to GRE’s 



4.2. The Additivity of GRE, Multiplicative property of
According to Equations (42)-(49), the additivity of the GRE is equivalent to the multiplicative property of the functionals
Under these assumptions and with the notations Equations (47) and (49), we have the following functorial property with respect to the direct product:
4.3. Additivity of the functionals 
It is possible to extend, partially, the additivity property from Proposition 6. Consider the measured space defined in Equations (42)-(46) and suppose that the space X and the related objects has the following decom- position in direct sum, similar to the Definition 4

We define the measure








similar to Definition 4


Under previous conditions Equations (67)-(71), we have the following additivity result:

We obtain a similar result for the functional 

5. Summary and Conclusions
We proved that the most natural setting for treating the axiomatic approach to the study of definitions of measures of information or uncertainty, is the formalism of measure spaces and of the category theory. The Rényi divergence can be reduced to the Rényi entropy in our measure theoretic formalism. Category theory was invented for the most difficult, apparently contradictory aspects of the foundation of mathematics. In this respect, we introduced a category of measurable spaces MES. We proved that in the category MES existed the direct product and the direct sum, having universal properties. We proved that the functional 
The main conclusions may be summarized as follows:
1) The natural measure of the quantity of information is the family of functionals 


2)The category MES is the natural framework for treating the problems related to the measure of the infor- mation, in particular in reformulating the Rényi axioms;
3) The category MES is a monoidal category with respect to direct product and coproduct and the functional 
4) Up to a multiplicative constant, it is possible to recover the exact form of the functional 
5) The GRE 
6) The symmetry group of 



7) The Postulate 5'’of the classical Rényi entropy appears in the case of GRE as the additivity property of the functional 
Acknowledgements
The authors are grateful to Prof. M. Van Schoor and Dr D. Van Eester from Royal Military School, Brussels. György Steinbrecher is grateful to Prof. C. P. Niculescu from Mathematics Department, University of Craiova, Romania, and S. Barasch for discussions on category theory. Giorgio Sonnino is also grateful to Prof. P. Nar- done and Dr. P. Peeters of the Université Libre de Bruxelles (ULB) for useful discussions and suggestions.
Cite this paper
GyörgySteinbrecher,AlbertoSonnino,GiorgioSonnino,11,11, (2016) Category Theoretic Properties of the A. Rényi and C. Tsallis Entropies. Journal of Modern Physics,07,251-266. doi: 10.4236/jmp.2016.72025
References
- 1. Shannon, C.E. (1948) Bell System Technical Journal, 27, 379-423, 623-656.
http://dx.doi.org/10.1002/j.1538-7305.1948.tb01338.x - 2. Rényi, A. (1960) On Measures of Information and Entropy. Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability, 20 June-30 July 1960, Volume I, University of California Press, Berkeley and Los Angeles, 547-561.
- 3. Fadeev, D.K. (1957) Zum Begriff der Entropie einer endlichen Wahrscheinlichkeitsschernas. (Arbeiten zur Informationstheorie I, Berlin, Deutscher Verlag der Wissenschaften).
- 4. Feinstein, A. (1958) Foundations of Information Theory. McGraw-Hill, New York.
- 5. Tsallis, C. (1988) Journal of Statistical Physics, 52, 479-487.
http://dx.doi.org/10.1007/BF01016429 - 6. Tsallis, C., Mendes, R.S. and Plastino, A.R. (1998) Physica, A261, 534-554.
http://dx.doi.org/10.1016/S0378-4371(98)00437-3 - 7. Tsallis, C. (2009) Introduction to Nonextensive Statistical Mechanics: Approaching a Complex World. Springer, Berlin.
- 8. Gell-Mann, M. and Tsallis, C. (2004) Nonextensive Entropy—Interdisciplinary Applications. Oxford University Press, Oxford.
- 9. Steinbrecher, G. and Weyssow, B. (1999) Physical Review Letters, 92, 12503.
Rhodes, T.L., et al. (1999) Physics Letters A, 253, 181-186.
http://dx.doi.org/10.1016/S0375-9601(99)00034-1 - 10. Steinbrecher, G., Garbet, X. and Weyssow, B. (2010) Large Time Behavior in Random Multiplicative Processes. e-Print arXiv:1007.0952v1
- 11. Maszczyk, T. and Duch, X. (2008) Lecture Notes in Computer Science, 5097, 643-651.
http://dx.doi.org/10.1007/978-3-540-69731-2_62 - 12. van Erven, T. and Harremoës, P. (2014) Rényi Divergence and Kullback-Leibler Divergence. arXiv:1206.2459v2 [hep-ph]
- 13. Sonnino, G. and Steinbrecher, G. (2014) Physical Review E, 89, Article ID: 062106.
http://dx.doi.org/10.1103/PhysRevE.89.062106 - 14. Bucur, I.A. (1968) Introduction to the Theory of Categories and Functors. Wiley, London.
- 15. Mac Lane, S. (1997) Categories for the Working Mathematician. Second Edition, Springer, New York.
- 16. Pierce, B.C. (1991) Basic Category Theory for Computer Scientists. MIT Press, Cambridge and London.
- 17. Rudin, W. (1987) Real and Complex Analysis. Third Edition, McGraw Hill Inc., New York.
- 18. Reed, M. and Simon, B. (1980) Methods of Modern Mathematical Physics. Academic Press, New York.
- 19. Luschgy, H. and Pagès, G. (2008) Electronic Communication in Probability, 13, 422-434.
http://dx.doi.org/10.1214/ECP.v13-1397 - 20. Lesche, B. (1982) Journal of Statistical Physics, 27, 419-422.
http://dx.doi.org/10.1007/BF01008947 - 21. Dunford, N. and Schwartz, J.T. (1957) Linear Operators Part I: General Theory. J. Wiley & Sons, New York.
Appendix
A1. Rényi Divergence and entropy
Suppose to have a measurable space 




Consider now a measurable space 



The Rényi divergence reads

According to the Equations (73, 74, 76) and normalization Equation (75), we get

A2. Solution of the functional Equation Equation (38)
Using Equation (35) with



Hence, Equation (38) reads

In the particular case 

From Equations (81), (82) results

We select in Equation (83)

and the following equation results

Remark t hat putting in Equation (84) 


The general solution of corresponding homogenous equation

may be found by using again the continuity of the function 


Here 
The general solution of the Equation (85) reads

and similarly we have for all of the object of the category mes


By using Equations (81), (89), (90), (91), we get the universal linear slope p
and, by Equations (78)-(80), up to undetermined multiplicative constants

A3. Some Algebraic Result
Lemma 16 Let 


where 



Proof. We proceed by induction. For 




By using the previous Lemma 16 in two successive steps, with 

Corollary 17 Suppose that for all 

where




where the map 




































