\documentclass[11pt,a4paper]{report} 
\usepackage[latin1]{inputenc} 
\usepackage[american]{babel}
\usepackage{verbatim}
\usepackage{url}
% %pour creer des lien vers des pages web
\usepackage[pdftex,colorlinks=true, urlcolor=cyan,pdfstartview=FitH]{hyperref}
%%pour inserer du code
\usepackage{listings}\usepackage{ulem}

\setcounter{secnumdepth}{4}
\setcounter{tocdepth}{6}

\lstset{ %
         language=XML,
         showspaces=false,
         showstringspaces=false,
         showtabs=false,
         basicstyle=\footnotesize,
         }

%% pour afficher des images avec pdflatex utiliser le package pdftex
\usepackage{graphicx}

%% MARGES
\oddsidemargin -1cm
\marginparwidth 0cm \textwidth 18.5cm
\topmargin -0.5cm
\headheight -0.8cm \headsep 0cm
%%\footskip 0cm
\textheight 26.8cm
\pagestyle{plain}

\date{ Mobyle 1.0 }

\title{Mobyle Configuration Guide}

\begin{document}

\maketitle

\tableofcontents

\pagebreak 

\chapter{General Configuration}
The Mobyle configuration is set in the file \textbf{\$MOBYLEHOME/Local/Config/Config.py}. It is written in Python, so be very careful to validate the syntax of your file.

\section{Link Mobyle with a web server}
Three values have to be carefully updated in the configuration to integrate it correctly with your web server.
\textbf{ROOT\_URL}: the root url and port of the Mobyle server. (mandatory)\\
\textbf{HTDOCS\_PREFIX}: the extra path to the htdocs Mobyle directory. (mandatory)\\
\textbf{CGI\_PREFIX}: the extra path to the cgi-bin Mobyle directory. (mandatory)\\

You can potentially use any web server, just as long as it is multithreaded and CGI-enabled. Following are two configuration examples that use Apache 2, one that hosts Mobyle in a virtual host, the other in a subdirectory.

\subsection{Example of Mobyle configuration using a virtual host}

\subsubsection{Apache configuration}
\begin{verbatim}
<VirtualHost 192.168.0.3:83>
        ServerName server.domain.ext:83
        ScriptAlias "/cgi-bin" "/var/www/localhost/cgi-bin/mobyle/"
        DocumentRoot "/var/www/localhost/htdocs/mobyle/"
        DirectoryIndex index.html index.xml
        ErrorLog "/var/log/apache2/mobyle_error_log"
        TransferLog "/var/log/apache2/mobyle_acces_log"
</VirtualHost>
\end{verbatim}

\subsubsection{Mobyle installation}
\begin{verbatim}
python setup.py install --install-htdocs=/var/www/localhost/htdocs/mobyle/ \
                                     --install-cgis=/var/www/localhost/cgi-bin/mobyle/\
                                     --install-core=/any/where
\end{verbatim}

\subsubsection{Mobyle configuration}

\begin{verbatim}
ROOT_URL = ``http://server.domain.ext:83''
HTDOCS_PREFIX = ``''
CGI_PREFIX = ``cgi-bin/mobyle''
\end{verbatim}

\subsection{Example of Mobyle configuration using a web subdirectory}

\subsubsection{Apache configuration}
\begin{verbatim}
ServerName server.domain.ext
ScriptAlias "/cgi-bin" "/var/www/localhost/cgi-bin/"
DocumentRoot "/var/www/localhost/htdocs/"
DirectoryIndex index.html index.xml
\end{verbatim}

\subsubsection{Mobyle installation}
\begin{verbatim}
python setup.py install --install-htdocs=/var/www/localhost/htdocs/mobyle/ \
                                     --install-cgis=/var/www/localhost/cgi-bin/mobyle/\
                                     --install-core=/any/where
\end{verbatim}

\subsubsection{Mobyle configuration}
\begin{verbatim}
ROOT_URL = ``http:mobyle.mydomain.ext/mobyle''
HTDOCS_PREFIX = ``mobyle''
CGI_PREFIX = ``cgi-bin/mobyle''
\end{verbatim}

\subsection{Additional web server configuration}

\subsubsection{Specific MIME types}

In order to be able to upload/visualize some specific data types, such
as PDB files, you need to overload their mime type. The default is the
\texttt{chemical/x-pdb} in \texttt{/etc/mime.types}, but this mime type
causes most browsers to try opening these files with an external tool.

As a workaround, you can force these files to be sent as ``plain text''
files by declaring them as \texttt{text/plain} files. In Apache, for instance,
this is done by adding this directive in the \texttt{/etc/apache2/mods-available/mime.conf}
Apache configuration file:

\begin{verbatim}
"AddType text/plain .pdb".
\end{verbatim}

\subsubsection{Download button}

The "save" button that is available in job results and data bookmarks can
 automatically open a "save as" prompt (instead of opening the file in the browser),
 given that you use a little trick to the web server configuration, that modifies the 
HTTP response headers.

For instance, in Apache, you can add these few lines in an \texttt{.htaccess} file (if your general Apache
configuration permits it ) in the directory where the jobs are stored (--install-hdocs value + '/data/jobs' by default).

\begin{verbatim}
RewriteEngine on
RewriteCond    %{REQUEST_URI}	 ^HTDOCS_PREFIX/data/jobs(\.*)
RewriteCond %{QUERY_STRING}	^save$
RewriteRule   (.*)/([^/]+)$    $1/$2 [E=SAVEDFILENAME:$2]
Header set Content-Disposition "attachment; filename=\"%{SAVEDFILENAME}e\"" env=SAVEDFILENAME
\end{verbatim}

\subsubsection{Directory indexes}

As the jobs are stored in web-accessible subtrees, data confidentiality relies on the use of
a unique key as the job identifier. Thus, it is strongly recommended to forbid the directory indexes
 of the \texttt{"[--install-htdocs]/data/jobs"} subtree. You can do it in the Apache general configuration
 or in \texttt{.htaccess} files with the directive \texttt{Options -Indexes}.

\subsubsection{Hidden files}

In job directories, Mobyle uses some "hidden files" for administration purposes.
- the unix command generated (the \texttt{.command} file)
- some information relative to the job (the \texttt{.admin} file)
- some internal informations to Mobyle execution (the \texttt{.forChild.dump} file or the \texttt{ADMINDIR} subtree)

These files are not intended to be viewed by the users, and it is thus highly recommended to mask them. 
You can mask them with a rewrite rule placed in an \texttt{.htaccess} file in the \texttt{[--install-htdocs]/data/jobs} subtree.
  
\begin{verbatim}
# Do not show hidden files content
RewriteCond     %{REQUEST_URI}  /\. [OR]
RewriteCond     %{REQUEST_URI}  ADMINDIR
RewriteRule     .*              - [F,L]
\end{verbatim}

\section{Mail}

Mobyle occasionally sends emails to users, to:
\begin{itemize}
 \item validate their email adresses when creating ``authenticated'' accounts,
 \item send job status or help request notifications. 
\end{itemize}

It is mandatory to configure correctly this mail to be able to run Mobyle. The values are the following:

\textbf{MAILHOST}: the mail transfert agent used by Mobyle to send an email. (mandatory)\\
\textbf{MAXMAILSIZE}:  if the results size is over MAXMAILSIZE, job results are not sent. Rather, only a notification of the end 
of his job is send to the user (containing a link to download it), in bytes. (default=2097152 (2Mo))\\
\textbf{MAINTAINER}: the emails list which will receive alert emails when problems occur in Mobyle. (mandatory)\\
\textbf{HELP}: the email adress that receives help requests from users. (mandatory)\\
\textbf{SENDER}: the email address representing Mobyle which sends long job notifications, results etc\ldots (for further
details, see the MailTemplate section)\\ 


\section{Execution system}
Mobyle can execute jobs either locally or on the SGE or PBS Distributed Resource Management systems. 
The execution system has been rewritten to be highly flexible, so its configuration
has completely changed. Please refer to the execution\_system documentation to learn how to configure it.  

\section{Module}(\textit{new in 1.5})
Mobyle is abble to execute programs which are manage via Module (see \url{http://modules.sourceforge.net/}).
To enable Module in Mobyle you must define two variables in Config.py.
\begin{itemize}
	\item MODULE\_INIT: The location of the master Modules package file, the module command initialization shell script.
	\item MODULE\_LOAD: The module load to perform before to execute the command line
\end{itemize}
MODULE\_LOAD is a dictionary wher the keys are the service name and the values the load module command to do before to execute the command.
The MODULE\_LOAD dictionnary accpet a special key wich DEFAULT. when a service name is not find in MODULE\_LOAD the DEFAULT entry is used.
the MODULE\_LOAD value can be either a string or None. If the value is None the module will not be used. If it's a string it could be a python string template.
Mobyle provide 4 variables for interpolation of the template.
\begin{itemize}
  \item PACK\_NAME: The name of the package if it's defined in the xml service description otherwise the program name is used.
  \item PACK\_VERS: The version of the Package if it's defined in the xml service description otherwise the program version is used.
  \item PROG\_NAME: The name of the program as defined in the xml service description.
  \item PROG\_VERS: The version of the program as defined in the xml service description.
\end{itemize}

for instance if we have 3 services as following:
\begin{itemize}
  \item service1: in the xml the name is service1 and version = 2.0
  \item service2: in the xml the name is service2 and version = 2.11-beta
  \item service3: in the xml the name is service3 and version = 1.2.7
\end{itemize}

and the configuration
\begin{verbatim}
MODULE_INIT = "/path/to/Modules/init/sh"
MODULE_LOAD = { 'service1' : "module load dependency/1 && module load %(PROG_NAME)s/%(PROG_VERS)s",
                'service2' : None ,
                'DEFAULT': "module load %(PACK_NAME)s/%(PACK_VERS)s" }
\end{verbatim}

service2 will not use module system for it's execution.
Before to execute service1 Mobyle will execute 2 module load instructions: 
module load dependency/1 then module load service1/2.0 
Before to execute service3 a module load service3/1.2.7 will be performed.

Warning: If MODULE\_LOAD is defined but not MODULE\_INIT a ConfigError is raised (the portal will not displayed).


\section{Logging}
\textbf{LOGDIR} is the directory where are located the different file loggers. Beware, given that the default directory is \texttt{/dev/null}, tracing potential problems can be hard if you do not set this value.
The most important log files in this directory are:
\begin{itemize}
  \item \texttt{access\_log}: to log the launched jobs,
  \item \texttt{error\_log}: to log the errors that occur on the Mobyle server,
  \item \texttt{build\_log}: to log all the steps leading to the construction of the command line (when debug $>=$ 2) 
\end{itemize}
If \textbf{ACCOUNTING} is set to True an \texttt{account\_log} will be created to log some statistics about jobs.
This log file can be used to fine-tune the execution system.\\

\section{Data converter management}

You can define with the \textbf{DATACONVERTER} variable which converter(s) you want to use to
manage the format conversion and verification properties.\\
For each Datatype, you can provide an ordered list of converter(s) as follow:
\begin{verbatim}
DATACONVERTER = {
				'Datatype1': [ converter1_class('/path/to/bin/converter1'),
							  converter2_class('/path/to/bin/converter2')     ] , 
				'Datatype2': [ converter3_class('/path/to/bin/converter3') ] 
                }
\end{verbatim}

Mobyle includes by default two ready-to-use converter classes: \emph{squizz\_alignment} and
\emph{squizz\_sequence} to manage respectively the Alignment and Sequence formats with the \texttt{squizz} program. 
The use of \texttt{squizz} is highly recommended in production environments.

The existing converter classes are located in \texttt{\$MOBYLEHOME/Src/Mobyle/Converter}.\\

Example to use the provided converters:\\
\begin{verbatim}
DATACONVERTER = {
                'Sequence': [ squizz_sequence('/path/to/bin/squizz') ] , 
                'Alignment': [ squizz_alignment('/path/to/bin/squizz') ] 
                }
\end{verbatim}


\section{Debug}
\textbf{DEBUG} allows to set the default debug level in Mobyle.
The default debug level is 0. Beware, only debug levels 0 and 3 allow the execution of a job.

\begin{description}
  \item[\textit{Level 0}] used in production:
  \begin{itemize}
    \item the command line is built
    \item the build log is NOT filled
    \item the job is executed
  \end{itemize}
  \item[\textit{Level 1}] to test a program definition XML (e.g. python syntax in code, precond\ldots):
  \begin{itemize}
    \item the command line is built
    \item the build log is NOT filled
    \item the job is NOT executed
  \end{itemize}  
  \item[\textit{Level 2}] to debug a program definition XML:
  \begin{itemize}
    \item the command line is built
    \item the build log is filled
    \item the job is NOT executed
  \end{itemize} 
  \item[\textit{Level 3}] to test the program definition XML, the job execution and its results:
  \begin{itemize}
    \item the command line is built
    \item the build log is filled
    \item the job is executed
  \end{itemize}     
\end{description}

To test/debug a program or its interface, the debug level can be overloaded for this service using the feature \textbf{PARTICULAR\_DEBUG}
in conjunction with \textbf{AUTHORIZED\_SERVICES}, which allows to define restricted access by IP address 
(see \ref{service_access_restriction}). To test the execution and results, set the \textbf{PARTICULAR\_DEBUG} 
to 2 or 3 for the program in question and restrict the access to this program to your own machine. This way, you will
be the only one who can access the interface in the portal, and the \texttt{build\_log} will register all steps of 
the command line generation, which can be useful to debug an interface. 

In the following example the general debug level is 0 except for \texttt{clustalw} which has a debug level set to 2.
\begin{verbatim}
DEBUG= 0
PARTICULAR_DEBUG={ 'clustalw' : 2 }
\end{verbatim} 

\noindent \large \textbf{Warnings}:
\normalsize
\begin{enumerate}
 \item during the debugging phase, don't forget to reinstall the XML file after each modification with \texttt{mobdeploy}:
\begin{verbatim}
 python mobdeploy.py -s local -p myNewProgram deploy
\end{verbatim}
  \item once the debugging phase has been completed, don't forget to remove the restricted access and set the debug level to 0.
\end{enumerate}

\section{Binary path}
\textbf{BINARY\_PATH} is a list of strings representing the paths where the program binaries can be found.
Each element of the list must be a valid path. The order of the elements is kept to build the final path.
These paths are added before the canonical PATH. By default, it is empty.

Example: here we add \texttt{/usr/local/bin} to \texttt{\$PATH}:
\begin{verbatim}
BINARY_PATH = ['/usr/local/bin'] 
\end{verbatim} 

\section{Data banks}
\textbf{DATABANKS\_CONFIG} describes the locally available databanks to fetch entries from, using the "databox". It is a dictionary of 
dictionaries, such that:
\begin{itemize}
 \item the \textbf{key} is the \textbf{name of the bank}, and the value is a dictionary where
 \item \textbf{'dataType'} is the Mobyle dataType of the data stored in the bank,
 \item \textbf{'bioTypes'} is the list of Mobyle bioTypes for the data stored in the bank,
 \item \textbf{'label'} is the label of the bank, as shown to the user in the portal,
 \item and \textbf{'command'} is a string template that describes how to generate a command line that will retrieve the bank entry(ies).
The \textbf{db} key is the key of the bank (if it is necessary to specify it in the command line), and \textbf{id} key is the requested value.
\end{itemize}

Example:
\begin{verbatim} 
DATABANKS_CONFIG = { 'WGS' : {  'dataType' : 'Sequence', 
                                'bioTypes' : [ 'Nucleic' ],
                                'label' : 'Genbank - Whole Genome Shotgun',
                                'command' : [ 'golden', '%(db)s:%(id)s' ] },
                      'PDB' : { 'dataType' : '3DStructure', 
                                'bioTypes' : [ 'Protein' ],
                                'label' : 'Protein Data Bank',
                                'command' : [ 'PDBGet.py', '%(id)s' ] } 
                    }
\end{verbatim}

\section{Statistics}
\textbf{GACODE} : Google Analytics. Setting this code will automatically configure Mobyle tracking on Google Analytics. 
For more information about how to set up a Google Analytics account, please refer to \url{http://www.google.com/analytics/}

\section{Portal and Welcome page configuration}

\subsection{Simple forms}
The forms can be displayed in two modes, ``simple'' and ``advanced''. Simple forms display only the parameters which are annotated as
with the ``issimple'' attribute. If the \textbf{SIMPLE\_FORMS} configuration rule is set to True (it is by default set to False), then upon display each form is tested for the
presence of ``simple'' parameters. If simple parameters are defined, by default only these ones will be displayed in the portal. The ``advanced''
parameters can then be displayed with a toggle (``simple form''/``advanced form'') located at the top of the form. If some advanced parameters 
have a modified value in the form, they will not be hidden, even when the form is switched to the ``simple'' mode.

\subsection{Portal appearance customization}
You can customize the appearance of the portal multiple ways:
\begin{itemize}
 \item \textbf{CUSTOM\_PORTAL\_HEADER} enables you to replace the top left "banner" zone which by default contains a "Mobyle" 
 title with any HTML content.
 \item \textbf{CUSTOM\_PORTAL\_FOOTER} enables you to add a footer zone below the portal which contains any HTML content.
\end{itemize}

Additionally, the CSS rules of the Mobyle portal can be enriched/overriden by modifying the \texttt{portal/css/local.css} file 
located in the htdocs section of the mobyle portal.

\subsection{Welcome page configuration}
The welcome page of the Mobyle portal (i.e., the contents of the "welcome" tab) can also be customized, to display the contents of an "external" HTML page or an ATOM news feed.

\textbf{WELCOME\_CONFIG} is a dictionary with 2 entries:
\begin{itemize}
  \item \textbf{'url'} : the url of the document to include in the portal
  welcome page,
  \item \textbf{'format'}: with 2 possible values:\begin{itemize}
        \item \textbf{'html'} if the url is the location of an html page,
        \item \textbf{'atom'} if the url is the location of an ATOM feed.
        \end{itemize}
\end{itemize}
An example file, \texttt{news\_ex.atom}, is available in the Example/Local directory.



\section{User authentication and management}

User actions in the portal, such as job submissions, can be configured to require some informations such as a valid email, to be able to contact the
 user if necessary.

\textbf{OPT\_EMAIL}: allow users to run jobs without  having to provide an email. ( default = False )\\
\textbf{PARTICULAR\_OPT\_EMAIL}: per-program override of the previous option.\\

Example: the email is mandatory to run any program, except for golden (a very short program).
\begin{verbatim}
OPT_EMAIL = False
PARTICULAR_OPT_EMAIL = { 'golden' : True }
\end{verbatim}

\subsection{General e-mail validation configuration}

User-provided emails can be validated beyond syntax check by performing a DNS resolution on the domain part of the user email, to ensure that a valid corresponding mail server exists, in order to limit fake user email addresses. This is done by setting \textbf{DNS\_RESOLVER} to True. In this case the \textbf{dnspython} library must be installed. By default, domain name resolution is not performed.

\subsection{User accounts management}

Each user has a space to store his data. This space can be temporary (during the working session) or persistent. The temporary space is created when a user connects to the portal and is accessible during the work session. These temporary accounts are refered to in the configuration as \textbf{anonymous sessions}. Persistent accounts are refered to as authenticated sessions, as they should require a higher level of validation for user informations (such as emails). Such accounts allow the user to access his workspace across several browser sessions, by signin in in the portal.

Administrators can enable/disable the use of anonymous and authenticated sessions by setting the \textbf{ANONYMOUS\_SESSION} and \textbf{AUTHENTICATED\_SESSION} values.

\textbf{ANONYMOUS\_SESSION} can take 3 values :
\begin{itemize} 
\item 'no'      : anonymous sessions are not allowed,
\item 'yes'     : anonymous sessions are allowed, without any verification,
\item 'captcha' : anonymous sessions are allowed, but with a captcha challenge (default value).
\end{itemize} 

\textbf{AUTHENTICATED\_SESSION} can also take 3 values :
\begin{itemize}
  \item 'no'    : the authenticated sessions are not allowed,
  \item 'yes'   : the authenticated sessions are allowed and activated, without any restriction,
  \item 'email' : the authenticated sessions are allowed but an email confirmation is needed to activate it (default value).
\end{itemize}

User workspaces are allocated a given amount of disk space. This disk space stores mainly their data bookmarks, which they can reuse across multiple tools. Note that this disk space \textbf{does not} include the job files, which are stored separately, and only ``linked'' from the user account. This size can be set using \textbf{SESSIONLIMIT}, which has a default value of 50Mib.

\textbf{OPENID} : activate OpenID authentication. 
When the OpenID identification system is enabled, users can connect without any registration: their authenticated session is automatically created and activated (default=False).\\

\subsection{Jobs management}

To limit disk space problems generated by excessively large jobs, or by programming bugs (e.g., execution loops which output ever-growing files), the \textbf{FILELIMIT} value allows to define the maximum size for a file generated by a job \textbf{if and only if the job is executed using the OS batch system, ``SYS``}. If you use a distributed resource management system such as SGE or Torque/PBS, please refer to its documentation.

Jobs can be configured to have a limited ``storage'' time, meaning that once they are finished, they are kept on the server for a limited number of days before being removed. Jobs should be removed using the mobclean tool (see Tools/mobclean and Tools/README), which can be started periodically from a cron. To configure this ``cleaning'' tasks, please set the \textbf{RESULT\_REMAIN} value which stores the lifetime of a finished job in days (default value=10 days).

Users are notified by email that their job is finished only if its execution time is superior to a given limit. This limit is set using \textbf{EMAIL\_DELAY} in seconds, and the default value is 20s.

\section{Portal settings}

The Mobyle portal includes a javascript-based polling mechanism which regularly updates the user workspace by getting its contents from the server. The frequency of these requests can be set with \textbf{REFRESH\_FREQUENCY}, in seconds. Its default value is 240 seconds, meaning the automatic workspace refresh is launched every 4 minutes.
User data (parameter values or job results) are displayed in textarea elements in the portal (if text-based). However, this preview/editing mechanism can be problematic if the data file to be displayed is too large. Hence, the portal does not display the contents of a data file if its size is above a given limit. The size limit can be set with \textbf{PREVIEW\_DATA\_LIMIT}, and the default size is 1048576 octets (1 Mib).

\section{Services management}

\subsection{Deployment}\label{structure}
Before deployment, the XML descriptions of the services used in Mobyle are located in two directories: 
\begin{itemize}
 \item \texttt{\$MOBYLEHOME/Services/} intended to store service definitions provided by Mobyle maintainers,
 \item \texttt{\$MOBYLEHOME/Local/Services/} intended to store your own service definitions.
\end{itemize}
The use of two directories allows to maintain a list of service definitions from an external source 
located in \texttt{\$MOBYLEHOME/Services}, without any risk to loose local modifications located  in \texttt{\$MOBYLEHOME/Local/Services}. 
Beware, if there are 2 services,
one in \texttt{\$MOBYLEHOME/Local/Services} and the other in \texttt{\$MOBYLEHOME/Services}, 
with the same name, \underline{only the description in \texttt{\$MOBYLEHOME/Local/Services} will be deployed}.

\texttt{\$MOBYLEHOME/Services/} contains 4 subdirectories: 
\begin{itemize}
\item \texttt{Programs}: to store program definitions,
\item \texttt{Workflows}: to store workflow definitions,
\item \texttt{Viewers}: to store viewer definitions and \underline{dependencies}. 
\item \texttt{Entities}: to store pieces of XML shared by several XML files, such as package information. (see \texttt{how\_to\_write\_a\_xml.pdf})
\end{itemize}

\texttt{\$MOBYLEHOME/Local/Services/} contains 5 subdirectories: 
\begin{itemize}
\item \texttt{Programs}: to store program definitions
\item \texttt{Workflows}: to store workflow definitions
\item \texttt{Viewers}: to store viewer definitions and dependencies
\item \texttt{Entities}: to store pieces of XML shared by several XML, like packages or nucleic databanks available
\item \texttt{Env}: to store some environment variables required by some programs (BLASTDB \ldots)
\end{itemize}

\subsubsection{Configuration}
There are 2 kinds of services:
\begin{description}
 \item[local services] corresponding to services executed on the local Mobyle server
 \item[imported services] corresponding to services appearing on the local Mobyle portal but executed on another Mobyle server
\end{description}

Whatever the service is, it must be deployed to be invoked in the Mobyle portal. 
This deployment operation will:
\begin{enumerate}
  \item get the XML files (on your computer or the distant server for
  imported services) and resolve the entities
  \item validate of the service descriptions
  \item transform the descriptions to make them ready to use by Mobyle (to lower the
  burden of the portal)
  \item copy the descriptions in the HTTP-accessible area for web publication
\end{enumerate}

The \texttt{mobdeploy} script located in \texttt{\$MOBYLEHOME/Tools} allows to deploy or undeploy a service. 
It needs the Mobyle configuration file \texttt{\$MOBYLEHOME/Local/Config/Config.py} to determine:
\begin{itemize}
  \item which services to deploy,
  \item which services are imported ones.
\end{itemize}

\subsubsection{Local services}\label{local_services}
The local service descriptions are stored in 2 directories (as described in
\ref{structure}). However Mobyle can be configured to deploy only a subset of these
descriptions by using the following 2 directives:
\begin{description}
 \item[LOCAL\_DEPLOY\_INCLUDE] indicating which descriptions must be deployed
 \item[LOCAL\_DEPLOY\_EXCLUDE] indicating which descriptions must \underline{not} be deployed
\end{description}
These 2 directives are python dictionaries with the 3
following keys: 'programs', 'workflows','viewers' and the values are lists of
corresponding services to deploy or not. Jokers can be used to defined a set of
services. By default all definitions found are deployed because of the use
of the wildcard \texttt{*} in each list of each entry of the
\textbf{LOCAL\_DEPLOY\_INCLUDE}  dictionary.

Beware, Mobyle evaluates \textbf{LOCAL\_DEPLOY\_INCLUDE} first meaning that if the same service
appears in the two lists, it will be excluded because \underline{\textbf{LOCAL\_DEPLOY\_EXCLUDE}
has the final say}. 

\begin{lstlisting}
LOCAL_DEPLOY_INCLUDE = { 'programs' : [ '*' ] , 
                         'workflows': [ '*' ] ,
                         'viewers'  : [ '*' ] 
                       }

LOCAL_DEPLOY_EXCLUDE = { 'programs' : [ '' ] ,
                         'workflows': [ '' ] ,
                         'viewers'  : [ '' ] 
                       }
\end{lstlisting}

To illustrate how these two directives work, let's imagine that a portal
proposes  \texttt{hmmalign}, \texttt{hmmconvert}, \texttt{hmmemit}, \texttt{hmmfetch} and \texttt{muscle} as
programs. The definitions of \texttt{hmmalign}, \texttt{hmmconvert}, \texttt{hmmemit}, \texttt{hmmfetch}
\texttt{hmmscan}, \texttt{hmmsim} and \texttt{hmmstat} are provided in the \texttt{pasteur-programs} package.
\begin{lstlisting}
LOCAL_DEPLOY_INCLUDE = { 'programs' : [ 'hmm*' , 'muscle' ] ,# will deploy all
                                                             # programs beginning 
                                                             # with hmm plus
                                                             # muscle
                          'workflows': [ '*' ] , 
                          'viewers'  : [ '*' ] }

LOCAL_DEPLOY_EXCLUDE = { 'programs' : [ 'hmms*' ] , # all programs beginning with
                                                    # hmms will be removed from
                                                    # the list of programs to
                                                    # deploy so 'hmmscan',
                                                    # 'hmmsim', 'hmmstat' won't
                                                    # appear in the portal
                         'workflows': [ '' ] , 
                         'viewers'  : [ '' ]  }
\end{lstlisting}


\subsubsection{Imported services}
Mobyle offers the possibility to import services executed on other portals. The detailed configuration is described in the MobyleNet (\ref{MobyleNet}) section.

\subsubsection{mobdeploy usage}

Before running \texttt{mobdeploy}, make sure that the current user (e.g. apache user) 
has permissions to read and write the files belonging to the Mobyle user. 
\begin{verbatim}
Usage: mobdeploy [options] cmd
\end{verbatim}
Here are the available commands:
\begin{itemize}
   \item \textbf{deploy}: deploy services available options [-s
           --server, -p --programs , -w --workflows , -v --viewers , -f --force , -V --verbose] 
   \item \textbf{clean}: clean the repository  available options [-s
           --server, -p --programs , -w --workflows , -v --viewers , -f --force , -V --verbose] 
    \item \textbf{index}: generate indexes      available options [ -V
    --verbose ]
\end{itemize}
And options:
\begin{itemize} 
   \item \texttt{-s --server} specifies the names (comma separated list) of the servers on which the command is applied.
       The keyword \texttt{all} means all servers as defined in the \textbf{PORTALS} dictionary in \texttt{\$MOBYLEHOME/Local/Config/Config.py}
   \item \texttt{-p --programs} specifies the names (comma separated list) of the programs the command is applied to.
       The keyword \texttt{all} means all programs as defined in \texttt{\$MOBYLEHOME/Local/Config/Config.py}
   \item \texttt{-w --workflows} specifies the names (comma separated list) of the workflows the command is applied to.
       The keyword \texttt{all} means all workflows as defined in \texttt{\$MOBYLEHOME/Local/Config/Config.py}
   \item \texttt{-v --viewers} specifies the names (comma separated list) of the viewers on which the command is applied.
       The keyword \texttt{all} means all viewers as defined in \texttt{\$MOBYLEHOME/Local/Config/Config.py}
   \item \texttt{-V --verbose} increases the verbosity. 3 levels are available: \texttt{warning}, \texttt{info} and \texttt{debug}. Default is \texttt{warning}.
\end{itemize}
                    
\noindent After deployment, the services are stored in a directory hierarchy with the following schema:
\begin{itemize}
 \item the root directory is \texttt{\$MOBYLE\_ROOT\_URL/HTDOC\_PREFIX/data/services/}
 \item inside the root directory, there is one directory for each server:
 \begin{itemize}
  \item your Mobyle server is called ``local''
  \item all other directories are named as specified in the \textbf{PORTALS} dictionary set in the \texttt{\$MOBYLEHOME/Local/Config/Config.py}
 \end{itemize}
 \item in each server directory, there are the following subdirectories: programs, workflows and viewers (local server only) if some
programs, workflows and viewers have respectively been deployed for this server. The descriptions of the corresponding services are inside those directories.
\end{itemize}

A full description of \texttt{mobdeploy} is available in \texttt{\$MOBYLEHOME/Tools/README}.                                  

\subsection{Disabling services execution}
It is sometimes necessary to disable jobs execution, for maintenance operations for instance. You can disable a given service by adding it in the \textbf{DISABLED\_SERVICES} list or disable all programs by setting \textbf{DISABLE\_ALL} to True to forbid the execution of any service. This mechanism prevents all new job submissions, portal-wide or service-specific, without any need to remove it from the portal. Conversely, new submissions can be enabled back by setting \textbf{DISABLE\_ALL} to False or updating the services listed in \textbf{DISABLED\_SERVICES}. By default, nothing is disabled (default DISABLE\_ALL=False, DISABLED\_SERVICES=[ ]). Please note that even though no job submission is accessible, the form itself can still be displayed.

\subsection{Services access restriction}\label{service_access_restriction}
The access to some services can be restricted based on their IP address. By default, all the programs available are usable by any user. If you need to filter this access, based on some license restrictions for example, you can use \textbf{AUTHORIZED\_SERVICES}. This dictionary defines the list of IPs and IP ``masks`` which can access specific services.

Example: here toppred can be accessed by any machine with IP 125.234.60.18 or any IP with ``mask'' 125.234.60.*
\begin{verbatim}
AUTHORIZED_SERVICES = {
   'http://myMobyle.mydomain.fr/data/programs/toppred.xml' : [ '125.234.60.18', '125.234.60.*']
                       } 
\end{verbatim}

Example: you have a new version of neighor program and you want to test it so you restrict the access to your IP 125.234.60.18 and set the debug level of the program to 2 doing this:
\begin{verbatim}
PARTICULAR_DEBUG={ 'neighbor' : 2 }

AUTHORIZED_SERVICES = {
   'http://myMobyle.mydomain.fr/data/programs/neighbor.xml' : [ '125.234.60.18']
                       } 
\end{verbatim} 

\section{\textit{MobyleNet} functionality\label{MobyleNet}}

Mobyle includes a set of functionalities allowing the use of services available on remote Mobyle servers. \textit{MobyleNet}-ed servers are connected using HTTP, to enable remote job submissions or results retrievals for instance.

\subsection{Motivation}

Administrators of systems such as Mobyle portals are often asked to integrate new services very quickly, to answer the need of a user. The installation and maintenance of such services can be problematic (is this program compatible with my architecture? how does it work? will it be often accessed by my users?).
The proposed solution is to provide Mobyle server administrators the possibility to share resources, exporting and importing services (programs, workflows) between servers: while users still access the same portal, the jobs corresponding to some programs can be remotely executed and stored.

\subsection{Mobyle Network jobs}

Mobyle Network job submissions are routed to an alternative endpoint, which does not require such an activation process in order to execute a job. This alternative endpoint, instead of checking for the existence of an activated session, only verifies that the invoked program is part of the list of exported services. Consequently, as opposed to the classical submission endpoint, all the submitted data values have to be provided, since no session is used to store data on the \textit{remote server}.

\subsection{Program imports, job submission, status poll and results display}

When a remote program is imported, its definition is downloaded from the \textit{remote server} to the \textit{portal server} to avoid network access overheads, and this definition is indexed during the same deployment process.

Once this task has been performed by the administrator, the \textit{remote server} will not be queried until a user submits a job. When this happens, the user request is first processed by the \textit{portal server}, which performs an initial set of controls and may replace user workspace parameter references by their values, since the user workspace cannot be directly accessed by the \textit{remote server}. The request for job submission is then forwarded to this last server.

Similarly to job submission, job status polls and job results display are forwarded by the portal to the \textit{remote server}, and the \textit{portal server} only handles the display of the data. The data corresponding to the remote job are not stored on the \textit{portal server}.

\subsection{Configuration directives}

The configuration of the MobyleNet functionality is set in the following values:
\begin{itemize}
 \item \textbf{PORTALS}: The Mobyle server from which you want to import services. To define a server, you need to specify these 6 fields: 
  \begin{itemize}
    \item name: the nickname you give to a Mobyle server. The services imported from this server will be labeled with 
  this name in the Services panels of the portal.
    \item url: the url of the portal (minus '/portal.py').
    \item help: the email address to which help requests should be sent (should probably be the email set in the \textbf{HELP} configuration value of the corresponding server).
    \item repository: the container url where the remote data (service descriptions, jobs) are stored.
    \item services: the dictionary of services you want to import from this server, defining two lists, one with the 'programs' key, the other with the 'workflows' key.
  \end{itemize}
 \item \textbf{EXPORTED\_SERVICES}: The list of local services you want to enable from other portals.
 \item \textbf{PORTAL\_NAME} is the name you want to give to your own portal, when seen from other MobyleNet nodes.
\end{itemize}

Example:
\begin{verbatim}
PORTALS={
         'mobyleA': {
            'url': 'http://mobyle.domain.ex/cgi-bin/MobylePortal',
            'help' : 'user@domain.ex',
            'repository': 'http://mobyle.domain.ex/data/programs/',
            'programs': ['clustalw-multialign'],
            'jobsBase': 'http://mobyle.domain.ex/data/jobs' 
            } ,
        'univB': {
            'url': 'http://bio.univB.fr/cgi-bin/',
            'help' : 'mobyle-help@univB.fr',
            'repository': 'http://bio.univB.fr/Mobyle/programs',
            'programs': ['muscle' , 'sspro' ],
            'jobsBase': 'http://bio.univB.fr/Mobyle/Results' 
            } 
         }

EXPORTED_SERVICES = [ 'protpars' , 'dnapars', 'neighbor' ]

PORTAL_NAME = 'me'
\end{verbatim}

\subsection{Important remarks}

\begin{itemize}
 \item Any change to the list of imported programs needs to be followed by a re-deployment of the impacted programs, using the \texttt{mobdeploy} tool. 
 \item Importing a program from a remote server is your responsibility: please ask for the permission of the server administrator. Please do not abuse either any remote resource, and try to comply with the terms of services that server administrators may specify.
\end{itemize}


\chapter{Mail Templates}

Mobyle uses emails in several circumstances. You can tune the contents of each mail by modifying the
corresponding template in \texttt{Local/mailTemplate.py}.

A template is composed of two parts, the header and the body. The header must contain the fields ``From'' and ``Subject'', and can contain the fields ``Cc'', ``Bcc'', ``Reply-To'' and ``Organization''. For each template, you can use some variables which will be expanded at runtime. All available variables are documented in \texttt{Local/mail.template.py}. 

\begin{itemize}
  \item \textbf{CONFIRM\_SESSION} : if \textbf{AUTHENTICATED\_SESSION} is set to 'email', an email is sent to the user to confirm his
  registration.
  \item \textbf{LONG\_JOB\_NOTIFICATION} : when a job is longer than \textbf{EMAIL\_DELAY}, we consider this job as a ``long`` job, and
  an email is sent to the user to notify him that his job keeps running.
  \item \textbf{RESULTS\_FILES} : once a ''long'' job is finished, all results are sent to the user as a zip archive (if an email was
  specified).
  \item \textbf{RESULTS\_TOOBIG} : once a ''long'' job is finished, if the size of the results is bigger than \textbf{MAXMAILSIZE}, the
  URL of the results is sent to the user.
  \item \textbf{RESULTS\_NOTIFICATION} : once a ''long'' job is finished, if the compression of the results cannot be performed, the
  URL of the results is sent to the user.
  \item \textbf{HELP\_REQUEST} : whereas the previous emails are sent by ``Mobyle'' \textbf{SENDER} to the user, this email is sent by the
  user to the email address set in the \textbf{HELP} directive. The user triggers this action by clicking on the ``ask for help'' button in the results page.
  \item \textbf{HELP\_REQUEST\_RECEIPT}: this is a receipt sent to the user who asks for help, containing a copy of his request.
  \item \textbf{NEW\_PASSWD}: text of the password modification notification sent by the \texttt{mobpasswd} tool.
\end{itemize}

\chapter{Local Policy}

The \texttt{Local/Policy.py} module is used to overload internal Mobyle functions, to adapt them to your administration needs.

\section{emailCheck}

This function checks if the email is valid, according to the local rules. This function takes one argument, the \textit{email} address, and must return one of the following values:
\begin{itemize}
  \item \textbf{Mobyle.Net.EmailAddress.VALID}    : if the email is valid,
  \item \textbf{Mobyle.Net.EmailAddress.INVALID}  : if the email is rejected,
  \item \textbf{Mobyle.Net.EmailAddress.CONTINUE} : to continue further the email validation process.
\end{itemize}

This function is called after the syntax checking, black-list filter and, if enabled by your configuration, before the ``DNS'' checking.

\section{authenticate}

This function overloads the Mobyle authentication session method. You can provide here any custom authentication mechanism. This function takes 2 arguments, \textit{login} and \textit{password}, and must return one of the following values:
\begin{itemize}
  \item \textbf{Mobyle.AuthenticatedSession.AuthenticatedSession.CONTINUE}: the method did not authenticate this
  login/passwd. The fall back method must be applied.
  \item \textbf{Mobyle.AuthenticatedSession.AuthenticatedSession.VALID}: the login/password has been authenticated and is valid.
  \item \textbf{Mobyle.AuthenticatedSession.AuthenticatedSession.REJECT}: the login/password is invalid.    
\end{itemize}

\chapter{Black list}
The file \texttt{Local/black\_list.py} allows to forbid job submission to ``non-desired'' users. In this file 2 structures are defined to store emails or IP addresses for which you want to forbid job submissions.

\section{users}
The first structure, \texttt{users}, is a list of emails which are not allowed to run a job on your Mobyle server.
You can use * as a joker to black list a set of email address in one time
Example:
\begin{verbatim}
users = [ 'foo@bar.com', 'spam@eggs.fr', '*@spam.com']
\end{verbatim} 

\section{host}
The second structure, \texttt{host}, is a list of IP addresses from which users are not allowed to run a job on your Mobyle server.
You can use * as a joker to black list a whole subnet.\
% 
Example:
\begin{verbatim}
host = [ '192.168.3.1' , '192.168.2.*' ]
\end{verbatim}

\chapter{Portal referencing in search engines}

If you manage a publicly accessible Mobyle portal, you may be interested in having it referenced by various search engines. To facilitate this task, the Mobyle portal includes a sitemap-generating CGI. You can ``declare'' your sitemap to different search engines (refer to their respective webmaster help pages). This sitemap will facilitate the crawling of your Mobyle portal, by declaring a different page for each available program. For more detailed information about sitemaps, refer to \url{http://www.sitemaps.org/}. In Mobyle, this sitemap, \texttt{sitemap.py} is located in the same directory as the other cgi scripts.


\chapter{Rotate and compress mobyle log files}

Due to some limitations in the python ``logging'' library (the library crashes when the log file exceeds 2Go) It's a good practice to rotate the logs.
However, the ``logging'' library can only rotate a file if it exceeds a given size or on a n days frequency, but is not able to rotate each month, which is the most
appropriate frequency to rotate mobyle logs. We therefore provide a small script (Tools/mob\_log\_rotate) which rotates the log files and compress them. This script
can be setup in a cron, so you can choose the frequency of the log rotation.

Example to rotate logs the first day of each mont at 0H5min am
\begin{verbatim}
#send statistics
0 0 1 * * apache /opt/bin/mob_stat -o /tmp/mobyle_stat.txt  /var/log/mobyle/access_log -m 
#rotate logs
0 5 1 * * apache /opt/bin/mob_log_rotate
\end{verbatim}

Note that in this example we send, every month, to the maintainers of the server some very basic statitics 
of mobyle usage just before to compress the log files.


\end{document}
