Stephan-<br>Sounds like you are working with data sets that resemble some that I work with frequently. About the same size, non-stationary seasonality (which I prefer to call low-frequency periodic trends), high, non-stationary temporally variable "noisy" signals. I usually have a priori knowledge that there is likely aliasing due to the fact that I am limited in sampling rate vs. total time period (varying the sampling rate for different collection instances can sometimes resolve at least part of this issue). I haven't figured out how to get the aliasing out of the system, although there should be some way to do this based on "unfolding" the spectrum.<br>
<br>Anyway, the approach I take is this. Starting with a COPY of the raw data, I run it through <a href="http://www.itl.nist.gov/div898/software/dataplot/">NIST Dataplot</a> to get a feel for the data, using what they call the 4-plot. This gives me a feel for the periodicity, auto-correlation, statistical distribution, and helps me identify the "ouliers" (i.e., spikes, 0's, any data point that is "unusual"), which are then filtered out by replacing specific data points with a "more reasonable" value (which is why I use a copy of the raw data, rather than the original). Usually use a spreadsheet for this. I might run the data through Dataplot a couple of times to evaluate the effects my "filtering" have had. Usually, still in the spreadsheet, I next remove the "DC offset" by subtracting one of the <a href="http://en.wikipedia.org/wiki/Pythagorean_means">Pythagorean means</a>. Which is appropriate depends on the nature of the data. This reduces the low end of the spectrum, which is where the "trends" are located. <br>
<br>For the Fourier analysis, I could stay with Dataplot, but I find it much easier to extract information from the Scilab approach. Furthermore, Scilab offers an alternative DFT processcalled MESE.<font face="Times New Roman, serif" size="2"> The
Maximum Entropy Spectral Estimate (MESE),
designed to produce high-resolution, low-bias spectral estimate
(refer to page 128 of the <a href="http://wiki.scilab.org/Tutorials%20archives?action=AttachFile&do=view&target=signal.pdf"><i><b>Signal processing With Scilab</b></i></a> manual, or available <a href="http://wiki.scilab.org/Tutorials%20archives">here</a>).
MESE incorporates no information in the estimated spectrum about the
autocorrelation lags. That is to say that the bias resulting from the
leakage from the window sidelobes should be eliminated (or at least
minimized in some sense).</font>
In other words, one tends to get cleaner "spikes" in the spectrum. It is much easier to pick out the lower-frequency components of the signals with this procedure. One then subtracts these components from the working data, and repeats the process. Ideally, you have extracted all of the available information from the data when the residual is Gaussian white noise (a point I have never actually reached in practice). <br>
<br>I don't believe "windowing" techniques will work with low-frequency components, although I could be mistaken in this. I have toyed with windowing when I have reduced my residual to what has the appearance of a frequency-modulated signal- but, then, I am looking to characterize such events. The information you are trying to extract will ultimately dictate what approach you take.<br>
<br>I am attaching a "working document" that I have put together giving more detail on this approach.<br><br>Charlie<br><br><div class="gmail_quote">2011/11/21 Ginters Bušs <span dir="ltr"><<a href="mailto:ginters.buss@gmail.com">ginters.buss@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Better stick with DFT, smoothed DFT or try seasonal adjustment freeware Demetra+ - that's what official statisticians might do.<br>
<br>gin<br><br><br><div class="gmail_quote"><div class="im">On Mon, Nov 21, 2011 at 10:00 AM, Schreckenbach Stephan <span dir="ltr"><<a href="mailto:s.schreckenbach@truma.com" target="_blank">s.schreckenbach@truma.com</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<u></u>
<div link="blue" vlink="blue" lang="DE">
<div><div class="im">
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">Filtering temporal spikes is a good idea, since there are some of them. I will try that.<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">The data sample as around 7000 data points, the frequency I look for is around 1/10 * sample rate.<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">May be there are methods that are better suited for identifying frequency components in that kind of data?<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">FFT always describes the time series by harmonic oszillations, which might not work well<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">if oscillations are not (strictly) harmonic.<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">What about wavelets (don’t know much about it yet, though)?<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"><u></u> <u></u></span></font></p>
</div><div>
<div>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">Stephan</span></font><font color="navy"><span style="color: navy;" lang="EN-GB"><u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font color="navy" face="Times New Roman" size="3"><span style="font-size: 12pt; color: navy;" lang="EN-GB"> <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font color="navy" face="Times New Roman" size="3"><span style="font-size: 12pt; color: navy;" lang="EN-GB"> </span></font><span lang="EN-GB"><u></u><u></u></span></p>
</div>
</div>
<div>
<div class="MsoNormal" style="text-align: center;" align="center"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">
<hr size="2" width="100%" align="center">
</span></font></div>
<p class="MsoNormal"><b><font face="Tahoma" size="2"><span style="font-size: 10pt; font-family: Tahoma; font-weight: bold;">Von:</span></font></b><font face="Tahoma" size="2"><span style="font-size: 10pt; font-family: Tahoma;"> Charles Warner [mailto:<a href="mailto:cwarner.cw711@gmail.com" target="_blank">cwarner.cw711@gmail.com</a>]
<br><div class="im">
<b><span style="font-weight: bold;">Gesendet:</span></b> Samstag, 19. November 2011 05:12</div></span></font></p><div><div><font face="Tahoma" size="2"></font></div><div><font face="Tahoma" size="2"><br><div><div class="h5">
<b><span style="font-weight: bold;">An:</span></b> <u></u><a href="mailto:users@lists.scilab.org" target="_blank">users@lists.scilab.org</a><u></u><br>
<b><span style="font-weight: bold;">Betreff:</span></b> Re: [scilab-Users] saisonality in time series</div></div></font></div></div><u></u><u></u><p></p>
</div><div><div class="h5"><div><div></div><div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"><u></u> <u></u></span></font></p>
<p class="MsoNormal" style="margin-bottom: 12pt;"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">Another trick I have found that greatly reduces FFT noise it to temporarily mask any localized "spikes" in the data (such spikes, with a narrow
temporal profile have a very broad spectral distribution). One can also try to eliminate any offset by subtracting the mean (or the geometric mean or harmonic mean- the appropriate mean would be dictated by the nature of the data). This should hopefully
reduce the scale of the FFT amplitude, making it easier to spot any (especially low-frequency, or seasonal) potential frequency components.<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">On Fri, Nov 18, 2011 at 3:09 AM,
<u></u>Schreckenbach Stephan<u></u> <<a href="mailto:s.schreckenbach@truma.com" target="_blank">s.schreckenbach@truma.com</a>> wrote:<u></u><u></u></span></font></p>
<div link="blue" vlink="blue">
<div>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;">Hi,</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">sorry, of course I meant seasonality.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">The time series consists of longer term trends, short term noise
and short time seasonality. </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">oscillations / seasonality, if any, it is most likely to be nonharmonic.
I look for distinct frequencies.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">When I did a FFT plot of the original time series there was noise
only in the spectrum.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">I will give it a run with the differenciated series / the log
of the data. </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">There is still the question how to test for significance of the
found seasonality. </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"> </span></font><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;">Stephan</span></font><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><font color="navy" face="Times New Roman" size="3"><span style="font-size: 12pt; color: navy;"> </span></font><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><font color="navy" face="Times New Roman" size="3"><span style="font-size: 12pt; color: navy;"> </span></font><u></u><u></u></p>
</div>
</div>
<div>
<div class="MsoNormal" style="text-align: center;" align="center"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">
<hr size="2" width="100%" align="center">
</span></font></div>
<p class="MsoNormal"><b><font face="Tahoma" size="2"><span style="font-size: 10pt; font-family: Tahoma; font-weight: bold;">Von:</span></font></b><font face="Tahoma" size="2"><span style="font-size: 10pt; font-family: Tahoma;">
Charles Warner [mailto:<a href="mailto:cwarner.cw711@gmail.com" target="_blank">cwarner.cw711@gmail.com</a>]
<br>
<b><span style="font-weight: bold;">Gesendet:</span></b> Freitag, 18. November 2011 00:34<br>
<b><span style="font-weight: bold;">An:</span></b> <a href="mailto:users@lists.scilab.org" target="_blank">
users@lists.scilab.org</a><br>
<b><span style="font-weight: bold;">Betreff:</span></b> Re: [scilab-Users] saisonality in time series</span></font><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> <u></u><u></u></span></font></p>
<p class="MsoNormal" style="margin-bottom: 12pt;"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">Although "seasonality" is not the term I use for long term trends hidden in noisy data, I have had some success by
taking the log of the data, and running an FFT on the log data. Usually, I have some prior knowledge of the long-term periodic trends I expect, so it is relatively easy to determine quickly if this method works. Plotting the log of the data also gives one
a good feel for whether the data is stationary, or whether there are windows of data that can be treated as stationary. Any changing magnitude effect is, of course, reduced when on works with logs, but such effects can help one understand what the raw data
is really telling you.<br>
<br>
Charlie<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">On Thu, Nov 17, 2011 at 12:40 PM, Mike Page <<a href="mailto:Mike@page-one.waitrose.com" target="_blank">Mike@page-one.waitrose.com</a>>
wrote:<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">Hi,<br>
<br>
I don't know much about this application, but the Cepstrum can be used to<br>
find hidden periodicity in time series. Might be worth trying? I have used<br>
it for finding rotational components in the vibration signatures from<br>
rotating machinery. There's a simple example here<br>
(<a href="http://www.dliengineering.com/downloads/cepstrum%20analysis.pdf" target="_blank">http://www.dliengineering.com/downloads/cepstrum%20analysis.pdf</a>).<br>
<font color="#888888"><span style="color: rgb(136, 136, 136);"><br>
Mike.</span></font><u></u><u></u></span></font></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom: 12pt;"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"><br>
<br>
-----Original Message-----<br>
From: Petter Wingren [mailto:<a href="mailto:petterwr@gmail.com" target="_blank">petterwr@gmail.com</a>]<br>
Sent: 17 November 2011 17:18<br>
To: <a href="mailto:users@lists.scilab.org" target="_blank">users@lists.scilab.org</a><br>
Subject: Re: [scilab-Users] saisonality in time series<br>
<br>
<br>
Did a quick search but couldnt find anything obvious. I suppose the<br>
word you are looking for is seasonality - maybe that helps in finding<br>
something useful.<br>
<br>
On Thu, Nov 17, 2011 at 3:36 PM, <u></u>Schreckenbach Stephan<u></u><br>
<<a href="mailto:s.schreckenbach@truma.com" target="_blank">s.schreckenbach@truma.com</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> I look for a test of saisonality in time series.<br>
> The time series might be instationary and nonlinear and the saisonality<br>
> / oscillation might have a changing amplitude. Furthermore the<br>
> distribution<br>
> might be unknown as well.<br>
> I need something to test for significant saisonality without knowing /<br>
> estimating a (linear) model of the time series.<br>
><br>
> ideas I got so far: Chi Square Test for independency:<br>
> I could test for independence of saison and mean value of the data<br>
><br>
> Chi Square Test to test for different means of two data groups.<br>
> I could test for a difference of the mean between several seasons.<br>
><br>
> Any more or better ideas?<br>
><br>
> Thanks in advance, Stephan<br>
><br>
><u></u><u></u></span></font></p>
</div>
</div>
</div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> <u></u><u></u></span></font></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"><u></u> <u></u></span></font></p>
</div></div></div></div></div>
</div>
</blockquote></div><br>
</blockquote></div><br>