mutual information optimization problem
in [DSP]
|
Prev: Call For Participation: WORLDCOMP'08 (CS and CE conferences), July 14-17, 2008, Las Vegas
Next: www.nikeadishoes.com
From: Sylvia on 24 Jun 2008 10:56 Suppose I have two vectors x and y, x is known and y is unknown.I want to find a y such that it gives maximum mutual information with x.To find a unique solution y, I have some constraints(suppose R y=d) where R is a known matrix and d is the known vector.Is there any way to solve this constrained optimization problem?Any anaylitical expression to find partial derivatives of mutual information function wrt y? Sylvia
From: julius on 24 Jun 2008 21:03 On Jun 24, 9:56 am, "Sylvia" <sylvia.za...(a)gmail.com> wrote: > Suppose I have two vectors x and y, x is known and y is unknown.I want to > find a y such that it gives maximum mutual information with x.To find a > unique solution y, I have some constraints(suppose R y=d) where R is a > known matrix and d is the known vector.Is there any way to solve this > constrained optimization problem?Any anaylitical expression to find partial > derivatives of mutual information function wrt y? > Sylvia If x and y are both deterministic, then isn't the mutual information zero? Or were you referring to optimizing over the distribution of the vector y, i.e., p(y) subject to some linear constraint? I'm confused .... Julius
From: Andor on 25 Jun 2008 05:38 On 25 Jun., 03:03, julius <juli...(a)gmail.com> wrote: > On Jun 24, 9:56 am, "Sylvia" <sylvia.za...(a)gmail.com> wrote: > > > Suppose I have two vectors x and y, x is known and y is unknown.I want to > > find a y such that it gives maximum mutual information with x.To find a > > unique solution y, I have some constraints(suppose R y=d) where R is a > > known matrix and d is the known vector.Is there any way to solve this > > constrained optimization problem?Any anaylitical expression to find partial > > derivatives of mutual information function wrt y? > > Sylvia > > If x and y are both deterministic, then isn't the mutual information > zero? > > Or were you referring to optimizing over the distribution of the > vector > y, i.e., p(y) subject to some linear constraint? > > I'm confused .... I am too, I fear we don't understand Sylvia correctly. Even under the assumption that the "vectors" x and y are discrete probability distributions for two random variables X and Y, ie. P[X = x_k] = x[k] (and the same for Y and y) the task is underspecified. The concept of mutual information is based on the joint probability distribution so having only marginal distributions is insufficient to determine the mutual information. For the general case, we don't need two vectors x and y but a matrix to describe the distribution of X and Y. For the simple case where the joint probability is the product of the marginal probabilities and therefore two vectors suffice to describe the statistical relation between X and Y, the mutual information cannot be maximized because it is equal to zero (X and Y are independent). Regards, Andor
From: Sylvia on 25 Jun 2008 11:42 Ry=d is under-determined system,there will be many solutions, the information I have is: the distribution of unknown y is close to distribution of x.Hence if i minimize the mutual information between distribution of x and y subject to linear constraints Ry=d, i will get to the true solution x.Is this problem solvable? I will appreciate your comments. I know the distribution of y,but i dont have any information about joint probability >On 25 Jun., 03:03, julius <juli...(a)gmail.com> wrote: >> On Jun 24, 9:56 am, "Sylvia" <sylvia.za...(a)gmail.com> wrote: >> >> > Suppose I have two vectors x and y, x is known and y is unknown.I want to >> > find a y such that it gives maximum mutual information with x.To find a >> > unique solution y, I have some constraints(suppose R y=d) where R is a >> > known matrix and d is the known vector.Is there any way to solve this >> > constrained optimization problem?Any anaylitical expression to find partial >> > derivatives of mutual information function wrt y? >> > Sylvia >> >> If x and y are both deterministic, then isn't the mutual information >> zero? >> >> Or were you referring to optimizing over the distribution of the >> vector >> y, i.e., p(y) subject to some linear constraint? >> >> I'm confused .... > >I am too, I fear we don't understand Sylvia correctly. Even under the >assumption that the "vectors" x and y are discrete probability >distributions for two random variables X and Y, ie. > >P[X = x_k] = x[k] > >(and the same for Y and y) the task is underspecified. The concept of >mutual information is based on the joint probability distribution so >having only marginal distributions is insufficient to determine the >mutual information. > >For the general case, we don't need two vectors x and y but a matrix >to describe the distribution of X and Y. For the simple case where the >joint probability is the product of the marginal probabilities and >therefore two vectors suffice to describe the statistical relation >between X and Y, the mutual information cannot be maximized because it >is equal to zero (X and Y are independent). > >Regards, >Andor
From: Andor on 26 Jun 2008 02:57
Sylvia wrote: > Ry=d is under-determined system,there will be many solutions, the > information I have is: the distribution of unknown y is close to > distribution of x. Are x and y are random vectors or are they distributions of random variables? |