[Scilab-users] "Smoothing" very localised discontinuities in (scilab: to exclusive) (scilab: to exclusive) curves.

Mon Apr 4 16:45:22 CEST 2016

Rafael/Stepahane/Tom,

The problem with using a median filter -- and actually any continuous filter -- is that it implies that the median value of any n-group of adjacent values is "more reliable" than the actual value *for every value in the dataset*. And I'm really not convinced that is true for this data.

In other words. Continuous filtering can adjust all the values in the dataset; rather than just adjusting or rejecting the anomalous ones. One (large) erroneous data point early in the dataset would impose an influence upon the rest of the entire dataset causing a subtle shift in one direction or the other. If there are multiple erroneous values that all tend to be in the same direction -- as appears to be the case with these data -- then that shift accumulates through the dataset.

And as an engineer, that feels wrong. If you're taking a set of measurements and some external influence messes with one of them -- a fly blocks your sensor -- you reject that single data point; not spread some percentage of it through the rest of your readings.

I'm going to put in a request to the manufacturer of the equipment that produces this data, to request an explanation of the cause of the discontinuities; in the hope that might shed some light on the best way to deal with them. (With luck they'll have some standard mechanism for doing so.) 

(I've been trying to word the request all weekend, but its difficult to phrase it correctly.  These are the pre-eminent people in their field; they don't know me, and I don't have an introduction; and their equipment defines the standard for these types of measurements. It is extremely difficult to formulate the request such that it does not imply some shortcoming in their equipment or techniques.)

The data is magnetic field intensity vs field strength for samples of amorphous metal. The measurement involves ramping the surrounding field with one set of coils, and measuring the field strength induced in the material with another set of coils. The samples have hysteresis; the coils have hysteresis; the ambient surrounding can influence. The equipment goes to great pains to adjust the speed of ramping and sampling to try and eliminate discontinuities due to hysteresis and eddy current effects. 

I believe (at this point) that the discontinuities are due to these effects "settling out"; and the right thing to do is to essentially ignore them. My problem is how to go about that.

I've come up with something. (It almost certainly can be written in a less prosaic way; but I'm still finding my feet in SciLab):

    plot2d(  ptype, h*1000, b, style = [ rgb( i ) ] );
    e = gce(); e.children.mark_style = 2;

    h1 = [h(1)]; b1 = [b(1)];
    for n=2:size(h,'r') 
        if( (b(n) - b(n-1)) / (h(n) - h(n-1) + %eps) > 0 ) then
             h1 = [ h1, h(n) ]; b1 = [ b1, b(n) ];
        end
    end
    plot2d( ptype, h1*1000, b1, style = [ rgb( i + 1 ) ] );

    h = h1'; b = b1'; 
    h1 = [h(1)]; b1 = [b(1)];
    for n=2:size(h,'r') 
        if( (b(n) - b(n-1)) / (h(n) - h(n-1) + %eps) > 0 ) then
             h1 = [ h1, h(n) ]; b1 = [ b1, b(n) ];
        end
    end
    plot2d( ptype, h1*1000, b1, style = [ rgb( i + 2 ) ] );

See the attached png. The black Xs are the raw data. 
The red is the results of the first pass.
The green is the results of the second pass.
The purple are hand-drawn "what I think I'd like" lines.

What I like about this is that it only adjust (currently omits; but it could interpolate replacements) points that fall outside the criteria. As you said of the median filter; it doesn't guarantee monotonicity after one pass (or even 2), but it only makes changes where they are strictly required, leaving most of the raw data intact. 

(Note: At this stage I'm not saying that is the right thing to do; just that it seems to be :)

I'm not entirely happy with the results:

a) I think the had-drawn purple lines are a better representation of the replaced data; but I can't divine the criteria to produce those?
b) I've hard coded two passes for this particular dataset; but I need to repeat until no negative slopes remain; and I haven't worked out how to do that yet.

Comments; rebuttals; referrals to the abuse of SciLab/math police; along with better implementations of what I have; or better criteria for solving my problem all actively sought.

Thanks, Buk.

> -----Original Message-----
> From: scilab.browseruk.b28bd2e902.jrafaelbguerra#hotmail.com at ob.0sg.net
> Sent: Mon, 4 Apr 2016 14:58:47 +0200
> To: users at lists.scilab.org
> Subject: Re: [Scilab-users] "Smoothing" very localised discontinuities in
> (scilab: to exclusive) (scilab: to exclusive) curves.
> 
> If your data is not recorded in real-time, you can sort it (along the
> x-axis)
> and this does not imply that the "y(x) function" will become monotonous.
> See
> below.
> 
> As suggested, by Stephane Mottelet, see one 3-point median filter
> solution below
> applied to data similar to yours:
> 
> 
> M = [1.0  -0.2;
>         1.4   0.0;
>         2.1   0.2;
>         1.7   0.45;
>         2.45  0.5;
>         2.95  0.6;
>         2.5   0.75;
>         3.0   0.8;
>         3.3   1.2];
> x0 = M(:,1);
> y0 = M(:,2);
> clf();
> plot2d(x0,[y0 y0],style=[5 -9]);
> [x,ix] = gsort(x0,'g','i'); // sorting input x-axis
> y = y0(ix);
> k =1; // median filter half-lenght
> n = length(x);
> x(2:n+1)=x; y(2:n+1)=y;
> x(1)=x(2); y(1)=y(2);
> x(n+2)=x(n+1); y(n+2)=y(n+1);
> n = length(x);
> for j = 1:n
>     j1 = max(1,j-k);
>     j2 = min(n,j+k);
>     ym(j) = median(y(j1:j2));
> end
> plot2d(x,ym+5e-3,style=[3],leg="3-point median filtering@"); // shift for
> display purposes
> 
> 
> 
> This gets rid of obvious outliers but does not guarantee a monotonous
> output
> (idem for the more robust LOWESS technique, that can be googled).
> 
> Rafael
>

____________________________________________________________
Can't remember your password? Do you need a strong and secure password?
Use Password manager! It stores your passwords & protects your account.
Check it out at http://mysecurelogon.com/password-manager
-------------- next part --------------
A non-text attachment was scrubbed...
Name: disconsRemoved.png
Type: image/png
Size: 60006 bytes
Desc: not available
URL: <https://lists.scilab.org/pipermail/users/attachments/20160404/ebaf3d9b/attachment.png>