\label{sec:uvtables}

The main goal of \imager{} is to convert interferometric measurements 
stored in \uv{} tables  (see \ref{sub:uvtable}) into images suitable for astrophysical interpretation. 
\uv{} tables contain a set of visibilities. \uv{} tables being the
starting point, \imager{} contains a number of commands to read and handle them.

\subsection{In a nutshell}

For \NOEMA{}, create with \clic{} a UV table from the current
index of cross-correlation observations and selected spectral window, 
and read it with \imager{}
\begin{verbatim}
$ clic
CLIC> (FILE ; FIND ; SET )   ! Build your index and spectral Window selection
CLIC> TABLE MyTable.uvt [NEW] [/MOSAIC]  !  Create a UV table
CLIC> EXIT
$ imager
IMAGER> READ MyTable.uvt [/Options ]
\end{verbatim}
A complete set of UV tables for all spectral windows and all sources
can be automatically created from an ensemble of \texttt{.hpb} data files by
\begin{verbatim}
$ clic
CLIC> @ all-tables
CLIC> EXIT
$ imager
IMAGER> SIC FIND *.uvt
IMAGER> READ UV 'dir%file[1]'  ! For the first one...
\end{verbatim}

For \ALMA{}, create with \casa{} a list of UVFITS files from a 
Measuremet Set
\begin{verbatim}
$ casa
CASA <2>: vis='MyMeasurementSet.ms'
CASA <3>: casagildas()
CASA <4>: casagildas("Do") 
CASA <5>: exit()
\end{verbatim}
Once you have the UVFITS files, convert them with \imager{}
\begin{verbatim}
$ imager
IMAGER> sic find *.uvfits
IMAGER> for string /in dir%file
IMAGER:   @ fits_to_uvt 'string'
IMAGER: next
\end{verbatim}
or, even simpler, by using the \com{PIPELINE} command.
\begin{verbatim}
$ imager
IMAGER> PIPELINE ORGANIZE
\end{verbatim}
Conventions for file naming are described in Section \ref{uv:casa}.
Each \uv{} table can later be read separately.

\subsection{Gildas data format}
Internally, \imager{} works on data represented in the Gildas data format.

The Gildas data format handles images, a generic term to indicate
$N$-Dimensional data cubes (typically up to $N=4$ in most cases). 
Data is stored in direct access files that
contain a compact header, followed by the data array values in binary form. 

The header contains the number of dimensions, their sizes, a conversion
formula between axis pixels and user coordinates, sky projection
information, velocity and frequency information, min and max of the data,
etc... 

The Gildas data format also includes special handling for 2-D arrays,
called Tables, in contains list of values organized like in a tabler
(columns and rows). In general, axis coordinates have no meaning for these
Gildas tables.

An exception is the specific variant of these Gildas Tables called \uv{} Tables, 
which contain visibilities coming from interferometric measurements.

\subsection{FITS data format}
\imager{} is also able to read and write data (images or \uv{} tables) 
in the FITS format (Flexible Image Transport System). 

FITS format being extremely flexible, one should not expect any arbitrary
FITS file can be simply ingested into \imager{}: this is possible only for
files those information can reasonably be matched into Gildas data format.

Images stored in the primary HDU of FITS files will be handled, as well
as UVFITS files. These are files that can be produced by \casa{} and \clic{}.

The FITS access facility is designed as a minimal support. 
FITS files can only be accessed in ReadOnly mode, or written in full.
Thus only \qom{READ} and \qom{WRITE} commands should be expected to work
in full on FITS files. This can allow quick looks at the data, but for 
real processing, data should be explicitely converted to Gildas data format.
In particular, some commands (including those used by the \qom{PIPELINE}) may
require write access to the data or header, and would fail on FITS files.

\subsection{UV table description}
\label{sub:uvtable}

\subsubsection{UV table data format}

A \uv{} table is a specific 2-D Gildas table, with a few additional
informations in the header, and a special interpretation of the data
organisation.

In a standard \uv{} table, each \textit{line} describes a visibility. 
Here a \textit{line} designate either the first or second axis of the 
table, and a \textit{column} the other one. \uv{} tables may appear in 
both orders. The default one is \textit{line} on 1st axis (\texttt{.uvt} 
ordering, used by most application). The \texttt{.tuv} ordering 
obtained by a 21 transposition is used essentially for display, as in 
this case the \textit{column} has the same meaning as for the 
\com{COLUMN} of \greg\ .

The number of lines of a \uv{} table is thus the number of visibilities described
in the table. Each \textit{column} of the table stores a particular property of the
visibilities, namely:
\begin{description}\itemsep 0pt
\item[Column 1] U in meters;
\item[Column 2] V in meters;
\item[Column 3] W in meters or Scan number;
\item[Column 4] Observation date (integer \class{}/\clic{} Day
  Number\footnote{The \class{}/\clic{} is a "radio Julian date" (or "Jansky
    Julian date"), which starts as $-2^{15}$ on the date of the first radio
    observation by Karl Jansky. It is thus the Modified Julian date minus
    60549.  That choice was made to maximize the time interval over which
    radio astronomical data could be usefully stored in an
    \texttt{integer*2}, back when 2 bytes of header space per spectrum were
    a significant consideration.  This date has little meaning outside the
    rather sparse community of souls gathered around the \class{} and
    \clic{} programs, however...});
\item[Column 5] Time in seconds since 0:00 UT of above date;
\item[Column 6] Number of the first antenna used to measure the visibility;
\item[Column 7] Number of the second antenna used to measure the visibility;
\item[Column 8] Real part for the first frequency channel;
\item[Column 9] Imaginary part for the first frequency channel;
\item[Column 10] Weight for the first frequency channel;
\item[Columns 11-13] Same as column 8-10 but for the second frequency
  channel, or for the second Stokes parameter of this channel.
\item[...] etc\ldots for all channels
\item[Columns N-ntrail+1 ... N] Trailing columns after the channel visibilities.
\end{description}
If a \uv{} table describes \texttt{nvis} visibility spectra composed of
\texttt{nchan} frequency channels, each with \texttt{nstokes} Stokes parameters,
the size of the table will thus be:
\texttt{nvis} lines of \texttt{7+3*nchan*nstokes+ntrail} columns, where \texttt{ntrail}
is the number of trailing columns.

\subsubsection{\uv{} header}

A \uv{} table header contains all the informations of a GDF header but some
of these informations have a special meaning in this context. Command
\com{HEADER} is the standard way inside \gildas{} to display in a human
readable way the header of GDF file. For instance, the command \\
\texttt{IMAGER> header gag\_demo:demo-line.uvt}  \\
would display
  %I-GIO_RIH,  File is  [EEEI to IEEE] , Header Version 1 (32 bit)
\begin{verbatim}
 1  W-GDF,  UNKNOWN Velocity type defaulted to LSR
 2  File : /Users/guilloteau/gildas/gildas-exe-dev/demo/demo-line.uvt  REAL*4
 3  Size        Reference Pixel           Value                  Increment       
 4       103   16.0000000000000       220398.688000000     -0.183792725205421
 5      9146   0.00000000000000       0.00000000000000       1.00000000000000
 6  Blanking value and tolerance      1.23455997E+34   0.0000000
 7  Source name         GG_TAU
 8  Map unit            Jy
 9  Axis type           UV-DATA      RANDOM
10  Coordinate system   EQUATORIAL          Velocity    LSR
11  Right Ascension   04:32:30.34200        Declination       17:31:40.5230
12  Lii        0.000000000000000            Bii       0.000000000000000
13  Equinox            2000.0000
14  Projection type     AZIMUTHAL           Angle     0.000000000000000
15  Axis 0     A0     04:32:30.34200        Axis 0     D0     17:31:40.5230
16  Baselines               0.0       0.0
17  Axis 1 Line Name    13CO(21)            Rest Frequency   220398.6880000000
18  Resolution in Velocity   0.25000000     in Frequency        -0.18379273
19  Offset in Velocity        6.3000002     Doppler Velocity     -40.755900
20  Beam                   0.00                0.00                 0.00
21  NO Noise level
22  NO Proper motion
23  NO Telescope section
24  UV Data    Channels:     32, Stokes: 1 None        Visibilities:        9146
25  Column            1 (Size 1) contains U           
26  Column            2 (Size 1) contains V           
27  Column            4 (Size 1) contains DATE        
28  Column            5 (Size 1) contains TIME        
29  Column            6 (Size 1) contains IANT        
30  Column            7 (Size 1) contains JANT        
31  Column            3 (Size 1) contains SCAN        
\end{verbatim}
Comments:
\begin{description}\itemsep 0pt
\item[Line 1] Indicates the velocity frame. If not present in the table
(as here), it is assumed to be LSR.
\item[Line 2] Indicates the filename associated to the currently displayed
  header.
\item[Lines 3-5] Display the dimensions of the associated array. Here it is
  a rank 2 array of dimension 103 \textit{columns} times 9146 \textit{lines}, \ie\ 9146 
  visibility spectra of 32 frequency channels.  Line
  4 describes the frequency axis of the visibility spectra stored in the
  \uv{} table. Be careful that this is a convention, \ie\ it must be
  decoded using the particular form of the table. In our case, each spectra
  has 32 frequency channels of width -183.8~kHz, the frequency of the
  reference pixel 16.0 corresponding to 220398.688~MHz. This last
  frequency is the frequency delivered by the correlator, \ie\ seen by the
  observatory. In particular, this is the frequency that must be used to
  compute the primary beam of the interferometer.
\item[Line 8] Indicates the unit of the real and imaginary parts of the
  visibilities, normally the Jansky (Jy).
\item[Line 9] Indicates that this is \uv{} table (\texttt{UV-DATA} and
  \texttt{RANDOM}).
\item[Lines 10-13] Describe the coordinate system.
\item[Lines 14-15] Describe the projection system. In the \uv{} table
  format, \texttt{A0} and \texttt{D0} indicate the phase center while
  \texttt{Right Ascension} and \texttt{Declination} indicate where the
  antenna pointed when acquiring the signal. These informations are in
  general identical for single field imaging and different for mosaicing.
\item[Lines 16] Indicates the baseline range in meters (m). 
\item[Lines 17-19] Describe additional information about the frequency axis
  of the visibility spectra. In particular, the rest frequency (here
  220398.688~MHz, that of the 13CO J=2-1 line) corresponding to a 
  velocity of 6.3~km/s in the velocity frame indicated at line 1 (in general LSR).
  Frequencies are always in MHz, and velocities always in km/s.
\item[Line 20] Indicates the primary beam size of the interferometer in
  radian. This is an obsolescent way to pass the size of the interferometer
  antennas.
\item[Line 21] The noise section has no meaning for the UV table.
\item[Line 22] If present, proper motions are given in mas/yr. The epoch
is used as the time origin.
\item[Line 23] If the TELESCOPE section is present, this line would 
indicate telescope name, its geographic coordinates and the antenna 
diameter (in m). This section contains also the information
to compute the primary beam.
\item[Line 24] UV data section: number of channels, number of Stokes
parameters and number of visibilities.
\item[Line 25 to end] Special columns description, including
the 7 first ones and the \texttt{ntrail} trailing ones.
\end{description}
In particular, \textbf{Mosaic} \uv{} tables contain two trailing 
columns named \texttt{L\_PHASE\_OFF, M\_PHASE\_OFF}
for the so-called ''Phase Offset Mosaics'', or 
\texttt{X\_POINT\_OFF, Y\_POINT\_OFF} for the ''Pointing Offset Mosaics'',
which contains the angular offsets of the field centers with respect
to the Phase reference. 

\subsection{NOEMA: UV Tables from CLIC}
\label{uv:clic}

For \NOEMA{}, creating UV tables suitable for \imager{} is straightforward
in \clic{}. From a list of selected cross-correlation scans, the command 
\qom{TABLE} creates a  \uv{} table with the appropriate format
(or adds to an existing one). Mosaics are treated in much the same way:
it is sufficient to add the \qom{/MOSAIC} option to it.

\subsection{\ALMA : UV Tables from \casa{}}
\label{uv:casa}

Importing \uv{} data from \casa{} to \imager{} is a little more complex, because
of the totally different design philosophy of the two packages.
\casa{} intends to solve the \textit{Measurement Equation}, whatever the
complexity of this process.  It is an all-in-one package for this purpose,
where calibration and imaging are deeply intermixed and use a
unified data format.
As a result, a \casa{} Measurement Set is a complex architecture
encompassing relations between many components stored as Tables
in a directory-like tree. It can handle calibrated data, calibration
tables, multisource data sets, raw data in the same architecture,
allowing to retain all information to process complex images, such
as multi-frequency synthesis of polarized emission observed in
a mosaic of fields.

On the contrary, \imager{} works on calibrated data only, and
with a single source (though possibly a mosaic) and single spectral setup at once.

The importation process goes through UVFITS data files, produced by
\texttt{exportuvfits} from \casa{}, and imported through the \com{FITS}
command of \sic{}.  However, the UVFITS format is an incomplete standard, and 
to recover properly all associated informations, these two steps have
been encapsulated in sophisticated scripts on each side.

\subsubsection{On the \casa{} side}

The proper exportation is done by invoking the 
\texttt{casagildas()} tool. \index{casagildas} 
\footnote{This tool is made available to \casa{} by 
\imager{}  (\imager{} does not need to be active, but must have been 
executed once before)}. \texttt{casagildas()} scans the Measurement Set 
using the \texttt{listobs()} tool, 
then uses a Python script 
that parses the output of 
\texttt{listobs()} to prepare a Python script named  
\texttt{casa\_uvfits.py}.

It is then up to the user to execute this Python script, using the
\texttt{casagildas("Do")} command that creates one UVFITS 
file per Source and Spectral window in the input Measurement Set. Thus, 
the sequence
\begin{verbatim}
  vis='MyMeasurementSet.ms'   # Setup the input Measurement Set filename
  casagildas()                # List its content and create the export script
  casagildas("Do")            # Execute the export script
\end{verbatim}
will create a set of
\begin{verbatim}
  Source-Frequency.uvfits
\end{verbatim}
files, where \texttt{Source} is the source name and \texttt{Frequency} is 
the central frequency of the spectral window in MHz, for all combinations
of sources and spectral windows.

\textbf{Caution:} The recipe above does not work? 
\begin{itemize}
\item  First, be sure to use a recent enough Casa version (Python 3 based),
\item  Second, Tthe \texttt{vis} argument only accepts one Measurement set, not a list.
Use \texttt{concat} before if needed.
\item  If none of the above works, this is probably because of an obsolete version.
Remove \file{\$HOME/.casa/casa2gildas.py}, then launch \imager{} once, and then \casa{} again. 
This will re-install an up-to-date version of the \texttt{casagildas()} tool.
\item the \texttt{exportuvfits} task of \casa\ has bugs when used on some measurement
sets, and may deliver incomplete UVFITS files. See section \ref{app:imager-casa} 
for details and possible work-arounds.
\item if all of that still fails, contact \imager{} support.
\end{itemize}

\subsubsection{On the \imager{} side}

The UVFITS files have first to be
converted to \uv{} tables. The easiest way is to use the \comm{PIPELINE}{ORGANIZE} 
command. Alternatively, this can be done by a simple script
\begin{verbatim}
  sic find *.uvfits
  for string /in dir%file
    @ fits_to_uvt 'string'
  next
\end{verbatim}
Each \file{Source-Frequency.uvfits} file will be converted to a 
\file{Source-Frequency.uvt} \uv{} table (the original 
\file{.uvfits} file is kept too).

The \script{fits\_to\_uvt} \index{fits\_to\_uvt} script is based on the \com{FITS} command
of \sic{}, but augmented with a number of tests to recognize the
proper layout of the UVFITS file, as this layout depends on which
\casa{} version was used, and whether it is a single source or
multi source file.

The \script{fits\_to\_uvt} script has a number of options. Help can
be obtained by typing
  
  \qom{@ }\script{fits\_to\_uvt }\qom{?}
  
Direct reading of UVFITS data is also supported, but with limited capabilities: 
see Section \ref{sub:uvread}.


\subsubsection{Current assumptions and limitations}
\gildas{} works in the \texttt{LSR} velocity frame and has limited 
polarization capabilities (\imager{} can only process 
one polarization state at a time, see Section \ref{sec:polar} for 
details). Thus the \textit{casagildas() -}\script{fits\_to\_uvt} 
sequence makes several assumptions:
\begin{itemize}
\item FDM (Frequency Division Mode) spectral windows are converted to 
LSR frame using the \textit{mstransform} tool
\item TDM (Time Division Mode) spectral windows, which have low 
spectral resolution (15 MHz), are assumed to be pure continuum
data, and remain in the default frame of the original measurement
set.
\item The conversion from UVFITS to \uv{} tables assumes that the data 
is unpolarized\footnote{unless the \texttt{/STOKES} option is present, 
see Section \ref{sec:polar}}, and merges the initial polarization 
states in an optimal way from the signal to noise point of view (i.e. 
using the respective weights of the two parallel hand states). 
\item \textit{casagildas()} takes the same input parameters
as the \textit{listobs()} facility. Source and/or spectral
window selection can thus be made by the user at this stage
\item \textit{casagildas()} takes the same input parameters
as the \textit{mstransform()} facility. Time integration
may be done at this stage, but this may limit the performance
of self-calibration at a later stage.
\end{itemize}

\subsection{Reading UV tables}
\label{sub:uvread}

\uv{} tables are simply read into \imager{} by \comm{READ}{UV}.
The rest frequency to be used to compute the velocity scale
is normally found in the table, but can be overridden 
using the \comm{READ}{/FREQUENCY} option.
By default, the whole table is read, but a subset of the
channels can be read using the \comm{READ}{/RANGE} option (with
the velocity scale as above).

UVFITS file may even be read directly as \uv{} tables, applying the same
conversion methods than the \script{fits\_to\_uvt} script. However,
this capability is still only partially supported. While it will work for
basic (i.e. in memory) handling, full access to the \uv{} data is only possible
on the native format: this includes subset selection, as well as direct operations
on \uv{} data files through the \qom{/FILE} option of many \qom{UV\_...} commands.

\subsection{UV Table handling}

Besides the \comm{READ}{UV} and \comm{WRITE}{UV} commands to read
or write \uv{} tables, \imager{} has a number of commands to manipulate
the current \uv{} table buffer. These commands have names starting
by \qom{UV\_}. Most of them are in the \lang{CLEAN} language, some
in the \lang{ADVANCED} one. 

\imager{} works using UV buffers. Most commands only work on 
the current UV buffer, but some of them keep track of the previous
buffer to allow the user to revert the operation.

\begin{description}\itemsep 0pt

\item[Data inspection and editing:]
\begin{itemize}\itemsep 0pt
\item \comm{SHOW}{COVERAGE} display the \uv{} coverage
\item \comm{SHOW}{UV} display the \uv{} data
\item \com{UV\_FLAG} allow flagging visibilities
\item \com{UV\_PREVIEW} provides a quick view of the visibilities as a function
of frequencies, and attempts to automatically find the continuum level
and parts of the bandwidth with spectral line emissions.
\end{itemize}

\item[Data size reduction routines:]
\begin{itemize}\itemsep 0pt
\item \com{UV\_COMPRESS} is a simple spectral smoothing, providing only channel 
averaging by integer number of channels.
\item \com{UV\_RESAMPLE} provides a more flexible spectral smoothing and resampling
facility. 
\item \com{UV\_TIME}  can be used to time-average the UV data set, leading
to faster processing. However, using \com{UV\_TIME} too early may limit
your ability to perform accurate phase self-calibration.
\end{itemize}

\item[Continuum processing commands:]
\begin{itemize}\itemsep 0pt
\item \com{UV\_BASELINE} allows to remove the continuum baseline, by
0th or 1st order baseline fitting of each visibility. 
\item Conversely, \com{UV\_FILTER} will filter the spectral line
range to leave only the channels with continuum emission.
Both \com{UV\_BASELINE} and \com{UV\_FILTER} can use the results provided
by \com{UV\_PREVIEW} to specify where spectral lines may be found.
\item \com{UV\_CONTINUUM} converts a spectral line \uv{} table into a bandwidth 
synthesis continuum \uv{} table. \com{UV\_CONTINUUM} is only useful
for UV plane analysis: see Section \ref{sec:uvplane} for details.
\item \comm{UV\_MERGE}{/FILE} can merge several UV continuum 
tables with a specified spectral index to optimize the sensitivity.
\end{itemize}

\item[Image preparation:]
\begin{itemize}\itemsep 0pt
\item \com{UV\_CHECK} inspects the \uv{} data to figure out how many
different synthesized beams are needed.
\item \com{UV\_SHORT} adds the short (or zero) spacing information provided by
an additional single dish data, read by \comm{READ}{SINGLE}.
\item \com{UV\_STAT} evaluates the impact of robust weighting
and tapering on the synthesized beam. It provides recommendations for
the image and pixel sizes.
\item \com{UV\_TRUNCATE} restricts the \uv{} baseline length range.
\end{itemize}

\item[UV Plane analysis:]
\begin{itemize}\itemsep 0pt
\item \com{UV\_FIT} fit simple source models to the visbilities.
\item \comm{SHOW}{UV\_FIT} display the fit results, usually as a function
of frequency, but also in other ways.
\end{itemize}


\item[Miscellaneous:]
\begin{itemize}\itemsep 0pt
\item \com{UV\_DEPROJECT} de-projects the $(u,v)$ coordinates given
a specified phase center, orientation and inclination. This can be
useful for inclined, flattened, nearly axi-symmetric structures such as
proto-planetary disks or galaxies. 
\item \com{UV\_CIRCLE} and \com{UV\_RADIAL} compute the azimutal average of the 
visibilities. They are useful for rotationally symmetric structures such 
as proto-planetary disks or circumstellar envelopes, for example.
\item \com{UV\_REWEIGHT} changes the visibility weights.
\item \com{UV\_SHIFT} changes the phase center. This can be useful
for UV plane analysis.
\item \comm{UV\_MERGE}{/FILE} can merge several UV tables, Line or Continuum.
It also allows stacking different spectral lines, by aligning them in
velocity, as well as stacking emission from different sources.
\end{itemize}

\end{description} 

The remaining \com{UV\_...} commands are related to imaging and 
deconvolution: \com{UV\_MAP} computes the dirty image, 
\com{UV\_RESTORE} computes the Clean image from a Clean component list 
by removal of the Clean components in the \uv{} plane, and imaging of 
the residuals. \com{UV\_RESIDUAL} just computes the residuals by 
subtraction of the Clean components. 

Finally, \com{UV\_SELF}, in the \lang{CALIBRATE} language, is a 
specific variant of \com{UV\_MAP} used to compute the intermediate 
images required for self-calibration. It is not intended for direct use 
by normal users.

