%\section{The Imager PIPELINE and REPORT tools}
\label{sec:all}

With \ALMA{} or \NOEMA{}, you may end up with an observational data set
that contains several sources, each of them observed with
a number of spectral windows of different spectral resolutions,
covering many spectral lines. 

The bookkeeping of such data sets can be intricate. To help the
users to focus on the science, we have developped a \com{PIPELINE} 
(namely a suite of scripts named \script{all-*.ima}) that automates 
the whole imaging process in this case, 
and the \com{REPORT} command that can create printable (PDF format) files
that summarize the data content and properties. 


As a (strong) bonus, the \qom{PIPELINE} builds up on all the knowledge 
and tools described in previous sections to provide an excellent solution
given the data set, unless the source and/or data have very special properties.

This pipeline can be run through the \com{PIPELINE} command,
or through a control panel launched by the \comm{PIPELINE}{/WIDGET} command.

The pipeline control parameters are available in the \sicvar{ALL\%} global structure,
and can be checked using the \qom{PIPELINE ?} command. 

The pipeline is intended to handle 
\textbf{all spectral sub-bands for a single receiver tuning of the interferometer}, 
with the data (possibly with several observing dates and configurations)
in UVFITS or GILDAS UV format. Data from each sub-band and each source should
be stored in a separate file. 
%

The principle is to gather all UV tables in a sub-directory (named
\texttt{./RAW/} by default), and to store the results of the
various processing steps (Self Calibration,  Spectral line
extraction, Imaging) in different sub-directories (respectively 
named \file{./SELF/, ./TABLES/, ./MAPS/} by default).
%
The \com{REPORT} command uses the same architecture, and generates PDF reports
in the \file{./REPORT/} sub-directory. 


The behaviour of the \qom{PIPELINE} Imaging step depends on the pipeline mode, \sicvar{ALL\%MODE}, that
can be controlled by the \comm{PIPELINE}{/MODE} option. It also depends on
whether a \com{CATALOG} is used or not, and on the \sicvar{ALL\%RANGE[1:3]} variable that
controls the velocity span and resolution. 
\begin{itemize}
\item{CONTINUUM} Mode: \\
In this mode no data cube is produced. Only continuum images are created, with no attempt
to remove any contaminating line emission.  
\item{SURVEY} Mode \\
In this mode, there is also no spectral line emission filtering. Full data cubes are
produced, at the spectral resolution specified by \sicvar{ALL\%RANGE[3]}. No continuum
image is produced: continuum can be later extracted through the \qom{MAP\_CONTINUUM}
command.
\item{SPLIT} Mode\\
\begin{itemize}
\item{Without} a spectral line catalog, the Imaging step will make
no attempt to identify spectral lines, and thus no attempt to image
each line separately.  Instead, each ``high spectral resolution'' UV table will be imaged 
completely.  An automatic estimate of the continuum level is made,
and is imaged separately from the continuum-subtracted data. At
the end of the process, the deconvolved ``continuum'' image
is added back to the spectral image.```Low spectral resolution'' data only produce
a continuum image. 
This mode is appropriate for e.g. low spectral resolution data, 
very wide lines, and/or nearly confusion limited data with many spectral lines.
\item{With} a line catalog, a user-specified velocity range is imaged
separately around each line in the catalog that falls into the
observed frequency coverage.  Continuum and Line are separated, but
no attempt is made to add back the continuum data at end.
\end{itemize}
\item{ALL} Mode \\
This mode is similar to the SPLIT mode, except that no attempt is made to produce
continuum-free line data cubes. The line data cubes also include the continuum data.
This mode is appropriate when the continuum emission has spatial variations of
its spectral index. 
\end{itemize}
Naming conventions apply to identify which data set covers which spectral line. 


\subsection{Preparing the data: \qom{ORGANIZE} step}

For \NOEMA{}, the UV data can be created from the \file{.IPB, .hpb}
raw data files using the \clic{}  script \qom{@}\script{~all-tables}. The 
resulting UV tables (\file{.uvt} files) should be placed in your
working directory. 

For \ALMA{},  all relevant Measurement Sets should first be concatenated
in a single one. UVFITS files should be created from this concatenated Measurement
Set (\texttt{.ms} directory) using the \texttt{casagildas()} Python tool 
in CASA, and moved tin your working directory. Data for each source/spectral window
combination is labelled by \texttt{SourceName-Frequency} where Frequency is the
window central frequency in MHz (rounded to nearest integer).

It is recommended to separate data from different sources into different
directory structures, to simplify further processing. 

In both cases, once your working directory is loaded with the \file{.uvt} or 
\file{.uvfits} files, you can start using the \script{all-widget} script,
usually through the \comm{PIPELINE}{/WIDGET} command. 

Then, the \comm{PIPELINE}{ORGANIZE} command should be run to organize the data
in the appropriate way, in particular convert UVFITS files to native
UV table format.


\subsection{Before using the \qom{PIPELINE}: the \qom{REPORT} command}

By default, the pipeline will run to produces images that encompass
the whole field of view and at the nominal spatial and spectral
resolution offered by the data. This avoids ignoring some
information, but unfortunately maximizes data sizes and compute time.

The pipeline run time is heavily dependent on the size of
images, and (obviously), the size of its output even more. It
is thus \textbf{highly recommended} to define a proper map size,
as well as proper imaging arguments, before running the pipeline.
This involves variables \sicvar{MAP\_FIELD, MAP\_CELL} and \sicvar{MAP\_ROBUST}.


Similarly, the pipeline run time and required space scales quasi-linearly
with the number of output channels. It is thus also \textbf{strongly recommended}
to specify the proper values of the velocity coverage in \sicvar{All\%Range}.

To do so, it is best to select a representative UV data set
and explore it interactively, in order to define the best compromise
(smallest possible field of view given the source size, 
largest acceptable cell size to avoid undersampling, appropriate
velocity resolution and coverage).

The \comm{REPORT}{SUMMARY} command will suggest you some possibilities for the imaging parameters, list a summary of 
the spectral coverage, and explore the feasibility of self-calibration based on the currently specified integration 
times (see \qom{CHECK} and \qomm{SELF} steps in Sec.\ref{all:sub:check}-\ref{all:sub:self} for details) using continuum 
data. It uses the \comm{PIPELINE}{EXPLORE} step to evaluate image sizes and weighting choice, see 
Sec.\ref{all:sub:explore}. Results can be found in the \file{REPORT/summary.pdf} file. You may adjust the weighting 
parameters \sicvar{MAP\_ROBUST} and/or Self-calibration parameters and re-run the \qomm{REPORT}{SUMMARY} if needed.

The \comm{REPORT}{COVERAGE} command will produce a detailed view
of the spectral coverage, using all spectral windows and the spectral lines
from the user-specified \com{CATALOG}. Plots will be found in
the \file{REPORT/coverage.pdf} file.

\subsection{The \com{PIPELINE} Widget}

This section describes the widget that can be used to control
the image processing in a systematic way for an ensemble of 
UV tables.   The (selected) UV tables must come from
observations of the same source \textbf{and} correspond to a single
frequency setup, so that all spectral windows share the same
\textit{uv} coverage and observing dates and times.

\subsubsection{Getting started: \comm{PIPELINE}{/WIDGET}}

\qomm{PIPELINE}{/WIDGET} creates a Widget that is used to customize
the process and launch the various steps. It also creates
(through a call to \qom{@}\script{~all-create}) the \sicvar{ALL\%} structure
and its components.
\begin{figure}[!h]
  \centering
  \includegraphics[height=16cm]{all-widget.png}
  \caption{The Pipeline control widget. The two first lines
  of the widget cannot be modified directly. The \texttt{Telescope}
  information is derived by button \texttt{ORGANIZE} from the data
  set. The \texttt{Spectral Line Catalog} is defined through command \qom{CATALOG}.
\label{fig:allwidget}}
\end{figure}

\subsubsection{\texttt{ORGANIZE} step}
The \button{ORGANIZE} button or sub-panel will move the initial files
(\file{.uvfits} and \file{.uvt} files) into an appropriate sub-directory
structure. By default, \file{.uvfits} files will be in \file{./UVFITS/}
(name controlled by \sicvar{all\%uvfits})
sub-directory, while \file{.uvt} files will be in \file{./RAW/} sub-directory
(name controlled by \sicvar{all\%raw}).

If absent, the \file{.uvt} files will be created from the \file{.uvfits} ones,
using default parameters (i.e. assuming unpolarized emission).

\subsubsection{\texttt{SETUP} step}
The \button{SETUP} performs all actions in the \button{FIND, SELECT, EXPLORE} and
\button{SCAN} sub-panels, preparing the selected data for further processing
with the \button{COMPUTE} and/or \button{IMAGE} button or sub-panels.

Alternatively, the user can perform the individual actions one by one in sequence
him/herself.

\subsubsection{\texttt{FIND} sub-step}

The \button{FIND} sub-panel will explore the directory containing
the initial UV tables (\file{./RAW/} by default) to identify
\textit{Wide Band} UV tables, i.e. those that cover enough 
bandwidth to provide enough sensitivity
for self-calibration on continuum flux.

This information will be used later to derive the self-calibration
solutions from some UV tables, and apply it to others, typically
narrow band spectral line data.

\subsubsection{\texttt{SELECT} sub-step}

The \button{SELECT} sub-panel will restrict the subsequent work on
the files matching the \texttt{"File Filter"} field value,
and build the corresponding list of \textit{Wide Band}
UV tables. 

\subsubsection{\texttt{EXPLORE} sub-step}
\label{all:sub:explore}

The \button{EXPLORE} sub-panel will select the first Wide band UV table
to evaluate data sizes and recommend adjustments in Field and Map size
(variables \sicvar{MAP\_FIELD, MAP\_SIZE, MAP\_CELL}), 
and optionally Robustness parameter \sicvar{MAP\_ROBUST} according to the effective UV
coverage. 

If the data size remains reasonable, it will provide appropriate
default values, but you may chose different values according
to your science objectives.

\subsubsection{\texttt{CHECK} step}
\label{all:sub:check}
The \button{CHECK} sub-panel will use the list of \textit{Wide Band}
UV tables to check whether Self calibration may be needed and possible.
The method is currently approximate and will be improved in future
releases. You may use the \sicvar{SELF\_MINTIME} return variable
to adjust your \sicvar{SELF\_TIMES} values to something more reasonable.
%
This step may turn off Self-Calibration by setting \sicvar{all\%doself} to 
\sicvar{NO}. You can override it under your own responsibility !...

\subsubsection{\texttt{SELF-CALIBRATION} button}

The \button{SELF-CALIBRATION} button will perform all self-calibration
related actions in proper sequence (\button{CHECK, COMPUTE, COLLECT, APPLY, SHOW}).

\subsubsection{\texttt{SELF} or \texttt{COMPUTE} step}
\label{all:sub:self}

The \button{COMPUTE} sub-panel will use the list of \textit{Wide Band}
UV tables to compute a (phase and amplitude) Self calibration
solution, and apply it to all UV tables. It will identify which
UV tables correspond to a given Wide Band one, so that the proper
self calibration solution is applied.

In \texttt{SPLIT, ALL} and \texttt{CONTINUUM} mode, self-calibration
is computed on continuum only data, after line filtering. 
In \texttt{SURVEY} mode, if no line catalog exists, the whole
bandwidth is used. This may be inappropriate if a strong
line dominates the integrated flux and is spatially extended.

The self calibrated UV tables are placed in another sub-directory
(\file{./SELF/} by default, controlled by \sicvar{all\%self}).

A prefix (controlled by \sicvar{all\%prefix\_self}) is automatically
added to the file names to remind the user that they have been self-calibrated.\\


\textbf{Caveats: CURRENT LIMITATIONS} \\
- No provision is made to attempt to use spectral line flux when 
the continuum flux is too low. You may use the \texttt{SURVEY} mode in such a case.\\
- The filtering mechanism (based on \com{UV\_PREVIEW}) may leave too few channels for 
sensitivity in case of strong line contamination and/or steep spectral index. You 
can change that by using manually \com{UV\_PREVIEW} with different parameters
on one wide-band window prior to relaunching the Pipeline. As \qom{UV\_PREVIEW} parameters
are sticky, your selected parameters will be used by the Pipeline.\\
- The pipeline evaluates to first order the quality of the self-calibration
to check whether it should be applied or not. However, there are
cases that escape such automatic estimate. Visual inspection is
recommended, and if the Self-calibration solution is not good, you should not apply
it. Also, the \button{COLLECT} step may help at this stage,
by improving the S/N on some bands.

\subsubsection{\texttt{SHOW} step}

This step will display the phase and amplitude correction that have been
computed in the Self Calibration step. Optionally, these plots can be
saved on Hardcopies (\file{.pdf} files for Pdf, \file{.eps} for EPS).

\subsubsection{\texttt{COLLECT} step}

Most NOEMA data include several wide bands (4 with PolyFix in its standard
mode), and an independent Self-calibration solution is computed for each
of them. The \button{COLLECT} sub-panel will merge all these solutions
into a single one, assuming the phase errors are due to pathlength changes. 

This improves the S/N of the self-calibration solution, and can
be beneficial when the S/N is the limiting factor. On the other hand,
if the data has very high S/N, this step may result in lower dynamic
range than the application of the separate self-calibration solutions.

This also applies for most ALMA data.

\subsubsection{\texttt{APPLY} step}
The \button{APPLY} sub-panel will apply the Self-calibration solution
found in the previous steps (\button{SELF} and optionally \button{COLLECT})
to all UV tables. The selection of the solution to be applied is based
on matching frequencies between the UV table and the solutions.

If the \button{COLLECT} step has been performed, one may still
use the individual solutions by setting variable \sicvar{ALL\%COMBINE} to \texttt{NO}. 

\subsubsection{\texttt{TIME} step}

The \button{TIME} button/sub-panel will time average the self calibrated
UV tables to save space and speed up further processing.

The integration time can be specified, or left to 0. In this case,
the data will be time-averaged to an appropriate sampling time 
given the longest baselines and field of view.

\subsubsection{\texttt{IMAGING} step}

The \button{IMAGING} button/sub-panel will scan all spectral windows to
identify whether they can be used to produce a Continuum or
spectral line image. 

  Low resolution spectral windows, identified as those whose
resolution is coarser than \sicvar{ALL\%MINFRES} will be used to
produce continuum images, by filtering any detected spectral
signature in the data before.

  For High resolution spectral windows, identified as those whose
resolution is better than \sicvar{ALL\%MINFRES}, data cubes will
be produced.

In \texttt{SPLIT} Mode, if a Line Catalog is present, these high spectral resolution
windows will be scanned
for line identifications. For each spectral line in the current 
catalog(s) (defined by command \com{CATALOG}) 
that fall in a spectral window, a continuum-free UV table will be
created, covering the velocity range specified by the user (by
variable \sicvar{ALL\%RANGE}) around the line frequency. The naming
convention is the following:
\begin{verbatim}
  original-molecule-I-X
\end{verbatim}
where \\
- \texttt{original} is the name of the initial high resolution
spectral window\\
- \texttt{molecule} is the name of the spectral line in the catalog\\
- \texttt{I} is a sequence number, incremented each time a new
line is found from the same original UV table.\\
- \texttt{X} is a character code, equal to \texttt{D} if the line
is suspected to be detected, and to \texttt{U} if not. When several
lines are too close together, the \texttt{D} status may be incorrectly
affected, but this is just a naming convention, not a strong result.

  In addition an
\begin{verbatim}
  original-C
\end{verbatim}
file that contains (presumably) line-free emission only is created for each
original UV table. 

  The imaging results are stored in a specific sub-directory 
(\texttt{./MAPS/} by default, controlled by \sicvar{all\%maps}). Only 
the Clean Component Table (\file{.cct}) and Clean image 
(\file{.lmv-clean}) files are written.

  When no Line catalog is present, only 3 images are produced for each UV table:
starting from the \texttt{original}  name
\begin{verbatim}
  original-C       original-U    and   original-A
\end{verbatim}
The \ldots\texttt{-C} contains (an estimate of) line-free emission, averaged
over the bandwidth of the UV data. The \ldots\texttt{-U} is the continuum
subtracted image, while in \ldots\texttt{-A} the pseudo-continuum image
obtained after \ldots\texttt{-C} has been added back. The \ldots\texttt{-A}
thus contains all the signal, and can be used to provide a better estimate
of the continuum level using e.g. the \com{MAP\_CONTINUUM} command.

  Spectral resampling can be performed by appropriate setting of \sicvar{ALL\%RANGE}.
If non zero, \sicvar{ALL\%RANGE[3]} indicates the desired spectral resolution.
Furthermore, in the \qom{CATALOG} case, \sicvar{ALL\%RANGE[1:2]}, if non zero, specify the Min and Max velocities
imaged around each spectral line.
  
  In \texttt{ALL} mode, the behaviour is similar, but only the \texttt{-C} and \texttt{-A}
products are created. 
  
  In \texttt{CONTINUUM} mode, line emission is assumed to be negligible, and only the \texttt{-C}
continuum images are produced. Running the Pipeline in this mode first can be
a quick way to evaluate the data content.
  
\subsubsection{\texttt{TABLES} step}

The \button{TABLES} button performs the same scanning and
identification process as the \button{IMAGING} button, but stores
the resulting UV Tables (instead of the .cct and .lmv-clean files for
the \texttt{IMAGING} button) in a sub-directory, named \file{./TABLES/}
by default, controlled by \sicvar{all\%tables}.

  This step is optional, and only needed if the user intends to 
perform some UV data analysis (like UV plane fitting, line stacking,
continuum spectral index determination, direct modeling, etc...).

  Extraction of the velocity range specified by \sicvar{ALL\%RANGE[1:2]}
is performed if a \qom{CATALOG} is provided by the user. Spectral
resampling according to \sicvar{ALL\%RANGE[3]} is performed if and
only if \sicvar{ALL\%RESAMPLE} is set to \texttt{YES}. With no \qom{CATALOG},
the full spectral coverage is preserved.

  Except in \texttt{CONTINUUM} mode, line-free continuum UV tables are
also produced. 

\subsubsection{\texttt{SKY} step}

The \button{SKY} button will apply primary beam correction to the
deconvolved images. This button has no effect for Mosaics, as sky brightness
distributions are directly produced in this case.


\subsection{The non interactive \com{PIPELINE}}


The whole series of actions available through the Pipeline Widget can be
performed in series through the \qom{PIPELINE} command.

Command \qom{PIPELINE *} will perform all the actions, without
any user interaction. The \button{PIPELINE} button in the Pipeline widget will do the same
(and also the \button{GO} button, as usual in {\mbox{{\color{red} \sc Gildas}}} widgets).

A specific step can be performed by command \qomm{PIPELINE}{StepName}.
The available step sequence is 
\button{ORGANIZE}, \button{SETUP} (the suite of \button{FIND}, \button{SELECT}, \button{EXPLORE}, 
and \button{SCAN}, 
\button{CHECK}, \button{COMPUTE}, \button{COLLECT}, \button{APPLY}, \button{SHOW}, \button{TIME}, 
\button{TABLES}, \button{IMAGING},  and ultimately \button{SAVE} to remember the parameters used
in the processing. \qomm{PIPELINE}{LAST} will repeat the last step (presumably
after you changed a control variable !...) while \qomm{PIPELINE}{NEXT} will
execute the next one.

\subsection{User control: the \qom{STOP} symbol}
User interaction in the Pipeline is controlled by the \qom{STOP} symbol.
In non-interactive mode, this symbol is automatically defined as equivalent
to command \qom{CONTINUE}, so that the pipeline proceeds unattended.
In Widget mode, it is defined as command \qom{PAUSE}, so that the user can
look at the result before proceeding. 

This default behaviour can be modified at will if the user pre-defines
symbol \qom{STOP} before using the \qom{PIPELINE} command. It can also
be changed by using the \button{STOP?} menu in the main Widget bar.


\subsection{Mosaics and other limitations.}

\begin{itemize}
\item{Mosaics:} The pipeline handles mosaics, but with several limitations.
\begin{enumerate}
\item The data has to fit in memory. This is often the case for NOEMA data, but
may not be true for ALMA data. Splitting the data in Frequency chunks may be
needed.
\item Self-Calibration is ignored for Mosaics, although there could
be cases where sufficient S/N is available for this. There is no inherent impossiblity
to do so, since self-calibration behaves in the same way for Single-fields and Mosaics.
But this has not been sufficiently tested, and is planned for a future release. Contact us
should this need clearly arise.
\end{enumerate}

\item{Short Spacings}
The pipeline assumes the available UV tables are complete. They may
or may not contain short spacing information. However the pipeline
has no specific tool to add short spacings to pure interferometric UV tables.
This is up to the user to do so.

\item{Proper Motion}
Similarly to short spacings, it is up to the user to correct the UV tables for
source proper motions if needed. It should be done on the RAW UV table prior to any further processing.
\footnote{Conceptually, it would be easy to implement this in an automatic
way in the \button{ORGANIZE} step, usinq some naming convention for  the
file storing the proper motion information.}

\item{UV table consistency:}
Also, the \qom{PIPELINE} is intended for simultaneous observations with a single
tuning and single source.  Although multiple sources can be handled by using the appropriate
file filter, it is recommended to process them in separate sub-directories.

Non simultaneous observations with different tunings can lead to puzzling inconsistencies.
Imagine what might happen if you store data obtained at the same time by two different arrays !...
\end{itemize}

\subsection{After using the \qom{PIPELINE}: the \qom{REPORT} command}

The \comm{REPORT}{RESULTS} command will produce a view
of the spectral coverage for each \textit{detected} spectral line
and a View-based display on the corresponding data cubes. 
Plots will be found in the \file{REPORT/views.pdf} file.

It is recommended to adjust the \qom{VIEW} control parameters before using
this command, in particular the field of view, displayed velocity ranges
and color look up table for consistent displays. 

