Feature2 set4

[xiyuyi]

Now let's think about the alpha value.
The alpha might need to be dependent on the max(bpjres), other than mu.

[xiyuyi]

how to set alpha?

when mu is higher than max(bpjres)

when mu is higher than the max(bpjres), the first back projection shouldn't yield any coefficents.
then the cum-bpjres increases, until it passes mu.

Then the excess amount is re-scaled and added to the coefficients of x.

when mu and max(bpjres) is similar, the first excess amount will be approximately the amplitude of the whole profile of bpjres.

So each time... the excess amount [that changes from 0 to something].. is the bpjres. Others just cumulated under the sea level.

so maybe, alpha, being the attenuation factor of the excess amount of cumulative bpjres, should be dependent on the max(bpjres) from the beginning ... instead of depending on mu?
but mu is also depending on the max(bpjres), that making them mutually dependent.

So my current parameter selection strategies are OK for now.

when mu is not higher than max(bpjres)

the first bpjres will pass the shrinkage.......
then if it is not greatly attenuated, the convergence path would be expected to oscillate.
so in this case, alpha is not related to bpjres. Because we don't know the true signal level
[maybe should be related to the bpjres and further projection of the bpjres.. A'A, check the paper...]

[xiyuyi]

how to set alpha?

from https://www.caam.rice.edu/~optimization/linearized_bregman/fixed_step/lbreg_fixedstep.html
we have:

There exists alpha0 so that any alpha >= alpha0 gives the solution to minimize |x|_1 subject to Ax = b.
% The alpha depends on the data, but a typical value is 1 to 10 times the estimate of norm(solution_x,inf)

So we start with a big alpha and choose the smallest alpha that works?
Does A needs to be normalized?

nah, it is different.

The demo code quoted above uses unit shrinkage while having that stepsize to control the amplitude of the cumulated back projected residuals. But our implementation is based on this paper: https://arxiv.org/pdf/1104.0262.pdf where we specify the shrinkage threshold. But have non-attenuated back projected residuals/
See algorithm 2 from https://arxiv.org/pdf/1104.0262.pdf

[xiyuyi]

Keep thinking about alpha... in our case.... it depends on the amplitude of the x, with respect to the back-projected residuals for sure.

[xiyuyi]

See this:

Seems like in this regard, the way to set this "stepsize" is just like my way of setting mu, the shrinkage threshold. Set the shrinkage threshold to be .. slightly above the maxi(bpjres) or below?

Now check:

% How to set opts.stepsize:
% Too large will cause divergence; too small will cause slow convergence.

this opt.stepsize is actually squeezing the firs bpjres to be below 1, given that the implementation set shrinkage threshold to be 1. So, it is actually trying to set the onset of y to be just right at the shrinkage threshold. When stepsize is small, that means it will be too far away from the shrinkage threshold, so it converges slowly. When stepsize is too big, it passes the shrinkage threshold... it may cause divergence.
So it is true that we want to send the max(bpjres) to be close to the shrinkage threshold.

our strategy for mu has been correct.

set mu to be slightly above max(bpjres), or around.

after all, A'A is not the same with max(bpjres0)

[xiyuyi]

commit 73c903d
Set an extra range of parameters with smaller mu and bigger alpha.
based on sneak peek of Feature1 set6.

smaller mu to have better connectivity [less sparse].
bigger alpha to increase convergence speed.

use absolute values for alpha, and mu.

If i set mu to be right above the shrinkage threshold, then it will be too tight..!
so for get about the mu rule.

set it to be several of orders of magnitute below or above the max(bpjres)