[Scilab-users] Surface smoothing in Scilab, immune to outliers

[Stéphane Mottelet]

Hello,

I have written a little script making the comparison between L1 and L2 
norm. For the L1 case, I have used leastsq and cheated by returning the 
square root of the abs of the residue and the 'nd' option.

function  r=resid(coef,x,y,z)
     r=sqrt(abs(z-(coef(1)+coef(2)*x+coef(3)*y)));
     r=r(:);
endfunction

n=10;
[x,y]=meshgrid(linspace(-1,1,n));
z=1+x+y+rand(x,'normal')/10;
z(n,n)=20;  // outlier

// L2 case

A=[x(:)^0  x(:)  y(:)];
coefl2=A\z(:);
disp(coefl2);

// L1 case

[l1norm,coefl1]=leastsq(list(resid,x,y,z),coefl2,'nd');
disp(coefl1);


Things are not supposed to work so well because 'nd' is only meaning 
that the function to minimize in non-differentiable at the optimum (and 
only there....). However, it seems to work well (true value is [1,1,1]') :

L2 case :

     1.165072
     1.4380897
     1.398408

L1 case :

     0.9954939
     1.0441875
     0.9939874

S.

Le 04/03/13 18:51, Rafael Guerra a écrit :
> Thanks Stéphane for the useful L1-references and for the insight on
> iterative L2 methods and to the others for their repplies.
>
> PS:
> Strong outliers or spikes have infinite bandwidth and therefore bandpass
> filtering/convolution does not seem, a priori, to be the most effective
> method to remove them.
>
> Regards,
> Rafael
>
> -----Original Message-----
> From: users-bounces at lists.scilab.org [mailto:users-bounces at lists.scilab.org]
> On Behalf Of Stéphane Mottelet
> Sent: Monday, March 04, 2013 10:14 AM
> To: users at lists.scilab.org
> Subject: Re: [Scilab-users] Surface smoothing in Scilab, immune to outliers
>
> Hello,
>
> Replacing the squared L2 norm by the L1 norm in the linear regression gives
> a good robustness to outliers (cf. Donoho and al. papers). The problem is
> then non differentiable but you can implement it by iteratively reweighting
> the classical L2 method (IRLS method), or by writing an equivalent linear
> program.
>
> S.
>
>
> Le 04/03/13 13:23, Dang, Christophe a écrit :
>> Hello,
>>
>> De la part de Rafael Guerra
>> Envoyé : lundi 4 mars 2013 04:37
>>
>>> Does somebody know if there are Scilab functions [...] that smooths
>>> experimental data z=f(x,y) and is immune to strong outliers.
>> imho, the problem with smoothing and outliers is that the definition
>> of a outlier depends on the field.
>>
>> How can Scilab know what a "strong outlier" is?
>>
>> I personally would try Fourier filtering:
>> a strong outlier means a steep slope
>> and therefore correspond to a high frequency.
>>
>> Thus fft2, set high frequencies to 0
>> (with possibly a smooth transition),
>> then inverse fft2 -- ifft2 does not exist, I never used 2-dimension
>> Fourier transform so I don't know if the inverse is easy to perform...
>>
> _______________________________________________
> users mailing list
> users at lists.scilab.org
> http://lists.scilab.org/mailman/listinfo/users
>
> _______________________________________________
> users mailing list
> users at lists.scilab.org
> http://lists.scilab.org/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/users/attachments/20130304/835df7ab/attachment.htm>