\section{Introduction}


\section{Basic Concept}

The program is loosely related to the ObsPy package. Some functions are (almost) identically taken from this package. Among them are the read functions and the spectrogram. ObsPy is an open-source project dedicated to provide a Python framework for processing seismological data. It provides parsers for common file formats, clients to access data centers and seismological signal processing routines which allow the manipulation of seismological time series
Beyreuther et al. 2010, Megies et al. 2011

facilitate rapid application development for seismology.

    Moritz Beyreuther, Robert Barsch, Lion Krischer, Tobias Megies, Yannik Behr and Joachim Wassermann (2010), ObsPy: A Python Toolbox for Seismology, SRL, 81(3), 530-533.
    Tobias Megies, Moritz Beyreuther, Robert Barsch, Lion Krischer, Joachim Wassermann (2011), ObsPy – What can it do for data centers and observatories? Annals Of Geophysics, 54(1), 47-58, doi:10.4401/ag-4838.


Development using the Subversion environment (e.g. providing version control, development
access, tickets, wiki documentation, etc.)
Demonstration script is available. Operational tests are conducted within the initial phase of
instrumental setup at the Conrad Observatory

\subsection{DI measurements}

\subsection{Variometer and Scalar measurements}

\subsection{The Programs Layout}


\section{Input and Output Formats}

\subsection{Supported input types}

Any data loaded into the MagPy analysis software is transferred into an internal data structure. This structure represents a time series with a number of keywords which represent individual columns of the original data. The following keywords (table) are generally available for all input data, although most input data only use a few of them.

KEYLIST = ['time','x','y','z','f','t1','t2','var1','var2','var3','var4','var5','dx','dy','dz','df','str1','str2','str3','str4','flag','comment','typ','sectime']


\subsection{Supported output types}

\subsection{Using databases}


\section{Basic geomagnetic data analysis}

\subsection{Reading and Plotting}

Plotting all data from a certain directory:
\begin{verbatim}
    from dev_magpy_stream import *
    st = pmRead(path_or_url=os.path.normpath('datapath\\*'))
    st.pmplot(['x','y','z'])
\end{verbatim}
The first line denotes the import line for the required package. The second line contains the read command which load the selected data into an internal memory. The data will be stored as a structured time series, as stream ``st'', for further treatment. ``datapath'' can either be an absolute path (e.g. c:$\\$data$\\$instrument$\\$*, /data/instrument/*) or a relative path (e.g. ..$\\$data$\\$instrument\\*, ../data/instrument/*). Any function is then performed on the stream ``st'' by using the following structure: st.function(options). For plotting the time series the pmplot function is used. This function requires the

Trimming data with starttime and/or endtime limits. Starttime and endtime formats:
\begin{verbatim}
    st = pmRead(path_or_url=os.path.normpath('datapath\\*'),starttime='2011-7-8',endtime=datetime(2011,7,18,13,0,0,0))
    st.pmplot(['x','y','z'])
\end{verbatim}

\subsection{Writing data}

\begin{verbatim}
    from dev_magpy_stream import *
    st = pmRead(...)
    st.DoUsefulThings(options)
    st.pmwrite('outputpath')
\end{verbatim}

\subsection{Filtering/Smoothing/Interpolation}

Filtering and smoothing are similar subjects: Recovering a signal from a time series is called smoothing if we have data from all time points available to work with. If we only know x(t) up to the current time point t(n) then the problem is called filtering. MagPy includes three different functions, of which one, the nfilter function is the most general and useful. An old "brute force" version is also available (filter), only supporting gaussian and flat filter, which should be used only if the input signal is very unevenly spaced. The smooth function is similarly included into the nfilter function but does not modify the header information.
   
Supported filters:
'flat','barthann','bartlett','blackman','blackmanharris','bohman','boxcar','cosine','flattop','hamming','hann','nuttall','parzen','triang','gaussian','wiener','spline','butterworth'


The first example shows how to filter data using gaussian and linear filters.
\begin{verbatim}
    st = read(...)
    st = st.nfilter(filter_type='gaussian',filter_width=timedelta(minutes=1))
    st.plot(['x','y','z'])
    st.nfilter(filter_type='flat',filter_width=timedelta(minutes=60),filter_offset=timedelta(minutes=30))
    st.plot(['x','y','z'])
\end{verbatim}

Smoothing:
\begin{verbatim}
    st = pmRead(path_or_url=os.path.normpath('datapath\\*'),starttime='2011-7-8',endtime='2011-7-9')
    st = st.smooth(['x'],window_len=21)
    st.pmplot(['x','y','z'])
\end{verbatim}

Approximating functions and fitting:
\begin{verbatim}
    st = pmRead(path_or_url=os.path.normpath('datapath\\*'),starttime='2011-7-8',endtime='2011-7-9')
    func = st.fit(['f'],fitfunc='spline',knotstep=0.05)
    st.pmplot(['f'],function=func)
\end{verbatim}

\begin{verbatim}
    st = pmRead(path_or_url=os.path.normpath('datapath\\*'),starttime='2011-7-8',endtime='2011-7-9')
    func = st.interpol(['x','y','z'])
    st.pmplot(['f'],function=func)
\end{verbatim}

\subsection{Removing spikes}

Outlier detection is based on quartiles. Calculated is the interquartile range ($IQR$, which is the difference between the first and third quartile ($Q3 - Q1$). The interquartile range is termed to be a relatively robust statistic compared to the range and standard deviation. $Q1$ is based on the lower 25\% of data, $Q3$ on the upper part, after using medians to divide the data set. Using an optional threshold value outliers are then characterized by exceeding the limit defined by
\begin{equation}\label{outlier}
    L = M \pm t*IQR
\end{equation}
where $L$ denotes the limit, $M$ is the median of the values and $t$ is the optional threshold value. By default the threshold value is set to 4, which keeps all data except prominent spikes even during stormy conditions in a noisy environment. The outlier function accepts four optional keywords for modifying threshold, time window, and the column key. Shown below are the default values:
\begin{verbatim}
stream = stream.routlier(keys=['f'], timerange=timedelta(hours=1), threshold = 4.0)
\end{verbatim}
Outlying data is \emph{not} removed from the data set but flagged (see next section). The log file (see section \ref{logging}) contains a list of all flagged data points.

\subsection{Flagging data}\label{flagging}

\begin{verbatim}
st = pmRead(path_or_url=os.path.normpath('datapath\\*'),starttime='2011-7-8',endtime='2011-7-9')
st = st.routlier()
st = st.flag_stream('f',3,"Moaing",datetime(2010,7,18,12,0,0,0),datetime(2010,7,18,13,0,0,0))
st = st.remove_flagged()
st.pmplot(['f'],function=func)
st = st.get_gaps(gapvariable=True)
st.pmplot(['f','var2'])
\end{verbatim}


\section{DI and Baselines}

\subsection{Calculating absolutes and and collimation angles}

\begin{verbatim}
abso = analyzeAbsFiles(path_or_url=source, alpha=3.25, beta=0.0, variopath=os.path.normpath('..\\dat\\lemi025\\*'), scalarpath=os.path.normpath('..\\dat\\didd\\*'), archivepath=os.path.normpath('..\\dat\\absolutes\\analyzed'))
abso.pmwrite('..\\dat\\output\\absolutes\\',coverage='all',mode='replace',filenamebegins='absolutes_lemi')
func = abso.fit(['dx','dy','dz'],fitfunc='spline',knotstep=0.05)
abso.pmplot(['dx','dy','dz'],function=func, plottitle = "Ex 12 - Analysis of absolute values and spline fit")
\end{verbatim}
The analyzeAbsFiles function reads DI files located in directory source (e.g. os.path.normpath('..\\dat\\absolutes\\raw')). For calculation, the variometer data in variopath is used and if no scalar values are provided along with the DI file, scalar data from scalarpath is used. If the analysis is successful, DI files are moved to the archive directory. Alpha and beta describe the rotation angles. Alpha is the horizontal angle. In case of an HDZ oriented instrument, alpha corresponds to D at the time of instrument orientation. Beta describes deviations from horizontal configurations and is usually zero in case of suspended systems.

The analyzeAbsFiles function is located in the absolutes package and reads special DI file formats. The output of the function is a stream class of the stream package, and thus can be treated/visualized using any basic stream analysis routine as described in table xxx.

For the write command the following options are now used: Coverage$=$all saves all data to one file and does not include any date in the filename. This requires option filenamebegins to be set, which determines the actual file name. Mode$=$replace indicates that any already existing input in an existing summary file at the same save-location will be replaced by new analyzes. All others remain untouched.


\subsection{Baseline calculation}
\begin{verbatim}
abslemi = pmRead(path_or_url=os.path.normpath(r'..\\dat\\output\\absolutes\\absolutes_lemi.txt'))
func = abslemi.fit(['dx','dy','dz'],fitfunc='spline',knotstep=0.05)
abslemi.pmplot(['dx','dy','dz'],function=func, plottitle = "Ex 13 - Baseline values and spline fit")
lemi = pmRead(path_or_url=os.path.normpath('..\\dat\\lemi025\\*'),starttime='2011-09-1',endtime='2011-9-30')
lemi = lemi.rotation(alpha=3.3,beta=0.0)
lemi = lemi.baseline(abslemi,knotstep=0.05,plotbaseline=True)
lemi.pmplot(['x','y','z'])
\end{verbatim}

Baseline correction is done by the baseline function. Reading the file containing absolute data corrected with the variometer is done in the first line. The fields dx, dy, and dz contain the baseline values for the specified variometer. As shown above a spline fit can be easily calculated for the absolute components. Then the variometer data is loaded and rotated according to its orientation matrix (HDZ here). The baseline correction is then done using the baseline function.


\subsection{Baseline stability}


\subsection{Quasi-definite data}


\section{Auxiliary data}


\section{Advanced data analysis}

\subsection{Merging and Comparing records}

\subsection{Spectral analysis}

\subsection{Variation anaylsis/Strom recognition}

\begin{verbatim}
    st = pmRead(...)
    func = st.fit(['x','y','z'],fitfunc='spline',knotstep=0.1)
    st = st.aic_calc('x',timerange=timedelta(hours=1))
    st.pmplot(['x','y','z','var2'],function=func)
    fmi = st.k_fmi(fitdegree=2)
    fmi = st.k_fmi(fitfunc='spline',knotstep=0.4)
    col = st._get_column('var2')
    st = st._put_column(col,'y')
    st = st.differentiate()
    st.pmplot(['x','var2','dy','t2'],symbollist = ['-','-','-','z'])
\end{verbatim}


\subsubsection{Akaika-Information-Criterion}\label{AIC}

MagPy implements picking of storm onsets using the Akaike Information Criterion (AIC) which is well known from seismic data analysis for determining p-wave onsets (citations).
The AIC picker principally compares the variance content of two parts of one data stream and aims to identify any change in the information content of the data stream. For that purpose, a 1-dimensional array of a data stream, for storm recognition usually the H column, is taken. A window of suitable length is then used to define segments to investigate of this stream. For each segement, a point of interest is moved along the segement and log of variance is calculated before and after this point. The difference of these two variance values defines the AIC value. The minimum AIC points to the occurrence of the most significant change in information content. Beside recognition of storm onsets, this method can be used fro any other monitoring of information content, e.g. chang in noise, etc. Beside the actual AIC value, the function also determines the relative AIC, normalized to the maximum of each segment, which is useful to get an idea of the relative amplitude of the information change. Finally, this relative value can be stacked with the values of repeated runs of the AIC function (e.g. using different time windows).

        Required:
        :type key: string
        :param key: needs to be an element of KEYLIST

        Optional keywords:
        :type timerange: timedelta object
        :param timerange: defines the length of the time window examined by the aic iteration
                        default: timedelta(hours=1)
        :type aic2key: string 
        :param aic2key: defines the key of the column where to save the aic values (default = var2)
        :type aicmin2key: string
        :param aicmin2key: defines the key of the column where to save the aic minimum val
                        default: key = var1
        :type aicminstack: bool
        :param aicminstack: if true, aicmin values are added to previously present column values

\section{Automizing analysis procedures}

\subsection{Scripting}

\subsection{Logging and Notification}\label{logging}

use the python logging package

\section{Discussion}


\section{Summary of supported routines}

Show table containing all functions, their purpose and the available keys.

\section{Appendix}

\subsection{Creating import/export filters}

\subsection{Adding Analysis Functions}


