Loading...
Searching...
No Matches
FtpPoller::FtpPoller Class Reference

FtpPoller client class implementation. More...

## Public Member Functions

| constructor (hash< auto > nconf)
| creates the FtpPoller object from the configuration hash argument passed
| constructor (Qore::FtpClient n_ftp, hash< auto > nconf)
| creates the FtpPoller object from the FtpClient argument and configuration hash argument passed
| destructor ()
| stops the polling operation if in progress and destroys the object
*bool | fileEvent (list< hash< FtpPollerFileEventInfo > > l)
| called for each poll with a list of all files matched before transfer
binary | getFile (string path)
| retrieves a binary file and returns the file's contents
list< hash< FtpPollerFileEventInfo > > | getFiles (string subdir, int sort=FtpPoller::SortNone, int order=FtpPoller::OrderAsc)
| returns a list of regular file hashes matching any file name mask set for the object
int | getPollCount ()
| returns the current poll count
hash< FtpPollerFileEventInfo > | getRemoteFileData (hash< FtpPollerFileEventInfo > event)
| Retrieves remote file data and adds it to the event data.
| getStoreFile (string remote_path, string local_path)
| retrieves a remote file and stores it to a local path
string | getTextFile (string path)
| retrieves a text file and returns the file's contents
abstract | postSingleFileEvent (hash< FtpPollerFileEventInfo > event)
| called after singleFileEvent() for each matching file individually
| removeFile (string fn)
| deletes a file on the server
| rename (string old, string nnew)
| renames a file on the server
bool | runOnce ()
| runs a single poll (useful for checking for errors inline before starting a background thread)
abstract | singleFileEvent (hash< FtpPollerFileEventInfo > event)
| called for each matching file individually whenever matching files are polled
int | start ()
| starts polling in the background; returns the thread ID of the polling thread
| startInline ()
| starts the polling operation inline (not in a background thread)
| stop ()
| stops the polling operation, returns when the polling operation has been stopped
| stopNoWait ()
| stops the polling operation, returns immediately
| updateOptions (FtpClient ftp)
| Updates options.
| waitStop ()
| waits indefinitely for the polling operation to stop

## Static Public Member Functions

static | checkPath (string path, string type, bool write=False)
| checks a path on the local file system

## Public Attributes

const | AllKeys = RequiredKeysWithHost + keys Defaults + OptionalKeys
| all keys
const | Defaults = ...
| default values for constructor hash argument
const | ErrorDelay = 1m
| pause when Ftp errors are detected
const | OptionalKeys = ...
| optional constructor hash keys
const | OrderAsc = 0
| ascending sort order
const | OrderDesc = 1
| descending sort order
const | RequiredKeys = ...
| minimum required keys for all constructors
const | RequiredKeysWithHost = RequiredKeys + "host"
| RequiredKeys for the constructor(hash) without an Qore::FtpClient argument.
const | SortName = 1
| sort by name
const | SortNone = 0
| no sorting

## Protected Member Functions

| ftpSleep (softint secs)
| sleeps for the specificed number of seconds
| logDebug (string fmt,...)
| calls the `"log_debug"` closure or call reference with verbose debugging information
| logDetail (string fmt,...)
| calls the `"log_detail"` closure or call reference with detail information
| logInfo (string fmt,...)
| calls the `"log_info"` closure or call reference with important information
Mutex | m ()
| start mutex
hash< FtpPollerFileEventInfo > | retrieveFile (hash< FtpPollerFileEventInfo > event)
| Retrieves the remote file to local_dir directly.
hash< FtpPollerFileEventInfo > | retrieveTempFile (hash< FtpPollerFileEventInfo > event)
| Retrieves the remote file to local_dir using a temporary file.
| run ()
| starts the polling operation
Counter | sc ()
| stop counter
| setMask ()
| converts a glob mask into a regex

## Protected Attributes

bool | atomic_transfer
| Atomic transfer flag for use with `local_dir`.
bool | delete_remote_file
| Delete file on remote FTP server after processing?
bool | fatal = False
| internal fatal error flag
Qore::FtpClient | ftp
| FtpClient object.
bool | get_files
| internal "get files" flag
string | host
| host or address name
string | local_dir
| Local directory to transfer file.
*code | log_debug
| optional debug log closure
*code | log_detail
| optional detail log closure
*code | log_info
| optional info log closure
*string | mask
| file glob name mask (ignored if `"regex_mask"` also set)
*softint | minage
| minimum file age
*string | move_remote_file
| Remote target directory for moving the file on the remote FTP server after processing.
FtpClient | new_ftp
| Connection to update while polling.
*string | pass
| password
softlist< string > | path = "."
| path(s) to poll
int | poll_interval
| poll interval in seconds
int | pollcnt = 0
| internal poll counter
int | port
| port
string | protocol
| Protocol ("ftp" or "ftps")
int | reopts = 0
| file matching regex options
string | rootFtpPath
| path after connecting to Ftp server
bool | runflag = False
| run flag
*code | sleep
| optional sleep closure
*code | start_thread
| optional start thread closure
string | tempfile_template
| The temporary filename template when `local_dir` is set.
int | tid
| polling tid
timeout | timeout
| timeout in ms
string | url
| url
hash< UrlInfo > | urlh
| url hash
string | user
| user

## Detailed Description

FtpPoller client class implementation.

See the constructor options for information on how to configure this object; note that either

- `delete_remote_file` or
- `move_remote_file` must be set, or FTP files polled will remain where they are in the remote FTP server, unless they are removed or moved during polling processing.

Only one of these options may be set at one time; setting both will result in an error

## Member Function Documentation

## checkPath()

static FtpPoller::FtpPoller::checkPath | ( | string | path,
|  | string | type,
|  | bool | write = False )

static

checks a path on the local file system

Exceptions

  DIR-ERROR | this exception is thrown if the local path does not exist, is not readable, is not a directory, or should be writable and is not

## constructor() [1/2]

FtpPoller::FtpPoller::constructor | ( | hash< auto > | nconf | ) |

creates the FtpPoller object from the configuration hash argument passed

Parameters

  nconf

  a hash with the following keys:
  - `atomic_transfer:` if True and `local_dir` is set, then `tempfile_template` is used to write FTP files to a temporary filename during transfer
  - `delete_remote_file:` if True then the remote file is deleted on the FTP server after processing. Cannot be set if `move_remote_file` is also set
  - `host:` (required) the hostname or address to connect to
  - `local_dir:` a local directory to use to transfer remote files so large file data does not appear in events
  - `log_debug:` a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message
  - `log_detail:` a closure or call reference for logging detailed information; must accept a single string giving the log message
  - `log_info:` a closure or call reference for logging important information; must accept a single string giving the log message
  - `mask:` the file glob mask to use (default: `"*"`, ignored if `"regex_mask"` is also present)
  - `minage:` the minimum file age in seconds before a file will be acquired (default: 0)
  - `move_remote_file:` the remote target directory on the FTP server where files will be moved to after processing; cannot be set if `delete_remote_file` is also set
  - `pass:` the password to use for the connection
  - `path:` the remote path(s) for retrieving the files
  - `poll_interval:` the integer polling interval in seconds (default: 10 seconds; must be > 0 if given)
  - `port:` the integer port number to connect to (default 21; must be > 0 if given)
  - `protocol:` either `"ftp"` (the detault) or `"ftps"` for secure FTP
  - `regex_mask:` a regular expression to use as a mask (overrides any `"mask"` value)
  - `reopts:` regular expression match options (ex RE_Caseless for case-insensitive matches)
  - `start_thread:` (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then background will be used)
  - `sleep:` (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used)
  - `tempfile_template:` (used when `local_dir` and `atomic_transfer` are set) the temporary filename prefix to use when transferring files to the local directory specified by `local_dir`; use `"%s"` as the placeholder for the target filename
  - `user:` the username to use for the connection

Exceptions

  FTPPOLLER-CONSTRUCTOR-ARG-ERROR | missing required key, invalid port, poll_interval, or protocol given
  FTPCLIENT-PARAMETER-ERROR | empty hostname passed
  SOCKET-CONNECT-ERROR | error establishing socket connection (no listener, port blocked, etc)
  FTPCLIENT-CONNECT-ERROR | error initializing ftp session or getting remote path

## constructor() [2/2]

FtpPoller::FtpPoller::constructor | ( | Qore::FtpClient | n_ftp,
|  | hash< auto > | nconf )

creates the FtpPoller object from the FtpClient argument and configuration hash argument passed

Parameters

  n_ftp | the new FtpClient object

  nconf

  a hash with the following optional keys:
  - `atomic_transfer:` if True and `local_dir` is set, then `tempfile_template` is used to write FTP files to a temporary filename during transfer
  - `delete_remote_file:` if True then the remote file is deleted on the FTP server after processing. Cannot be set if `move_remote_file` is also set
  - `local_dir:` a local directory to use to transfer remote files so large file data does not appear in events
  - `log_debug:` a closure or call reference for logging verbose debgugging information; must accept a single string giving the log message
  - `log_detail:` a closure or call reference for logging detailed information; must accept a single string giving the log message
  - `log_info:` a closure or call reference for logging important information; must accept a single string giving the log message
  - `mask:` the file glob mask to use (default: `"*"`, ignored if `"regex_mask"` is also present)
  - `minage:` the minimum file age in seconds before a file will be acquired (default: 0)
  - `move_remote_file:` the remote target directory on the FTP server where files will be moved to after processing; cannot be set if `delete_remote_file` is also set
  - `path:` the remote path(s) for retrieving the files; if a list of strings is given then each path will be polled for matching files according to the `"mask"` or `"regex_mask"` option
  - `poll_interval:` the integer polling interval in seconds (default: 10 seconds; must be > 0 if given)
  - `regex_mask:` a regular expression to use as a mask (overrides any `"mask"` value)
  - `reopts:` regular expression match options (ex RE_Caseless for case-insensitive matches)
  - `sleep:` (required when imported into a context where Qore::PO_NO_PROCESS_CONTROL is set) a closure or call reference to use instead of Qore::sleep() (if not set then Qore::sleep() will be used)
  - `start_thread:` (required when imported into a context where Qore::PO_NO_THREAD_CONTROL is set) a closure or call reference for starting threads; must return the integer thread ID (if not set then background will be used)
  - `tempfile_template:` (used when `local_dir` and `atomic_transfer` are set) the temporary filename prefix to use when transferring files to the local directory specified by `local_dir`; use `"%s"` as the placeholder for the target filename

Note
  Either `delete_remote_file` or `move_remote_file` must be set, or FTP files polled will remain where they are in the remote FTP server, unless they are removed or moved during polling processing. Only one of these options may be set at one time; setting both will result in an error

Exceptions

  FTPPOLLER-CONSTRUCTOR-ARG-ERROR | error with poller options
  FTPCLIENT-PARAMETER-ERROR | empty hostname passed
  SOCKET-CONNECT-ERROR | error establishing socket connection (no listener, port blocked, etc); timeout establishing socket connection
  FTPCLIENT-CONNECT-ERROR | error initializing ftp session or getting remote path

## fileEvent()

*bool FtpPoller::FtpPoller::fileEvent | ( | list< hash< FtpPollerFileEventInfo > > | l | ) |

called for each poll with a list of all files matched before transfer

if this method returns False or nothing, then the singleFileEvent method is not called

## getFiles()

list< hash< FtpPollerFileEventInfo > > FtpPoller::FtpPoller::getFiles | ( | string | subdir,
|  | int | sort = FtpPoller::SortNone,
|  | int | order = FtpPoller::OrderAsc )

returns a list of regular file hashes matching any file name mask set for the object

Parameters

  subdir | the directory to check
  sort | the sort option for the list returned
  order | the ordering of sorted data returned

Returns
  a list of regular file hashes with the following keys in each list element:
  - `name:` the name of the file, link, or directory
  - `size:` the size of the file in bytes
  - `mtime:` the last modified date/time of the file

## getStoreFile()

FtpPoller::FtpPoller::getStoreFile | ( | string | remote_path,
|  | string | local_path )

retrieves a remote file and stores it to a local path

Parameters

  remote_path | the remote file path
  local_path | the local file path

## postSingleFileEvent()

abstract FtpPoller::FtpPoller::postSingleFileEvent | ( | hash< FtpPollerFileEventInfo > | event | ) |

called after singleFileEvent() for each matching file individually

If any error occurs here, the polling operation stops

This method would normally delete / rename / move files processed by singleFileEvent() so that they would not be polled a second time. If an error occurs in this operation, then the polling event will stop since continuing after failing to delete, rename, or move a file already processed would cause the file to be processed more than once.

Parameters

  event | a hash of file data

## singleFileEvent()

abstract FtpPoller::FtpPoller::singleFileEvent | ( | hash< FtpPollerFileEventInfo > | event | ) |

called for each matching file individually whenever matching files are polled

If any error occurs here, the error is logged and the polling operation is retried

Parameters

  event | a hash of file data

## start()

int FtpPoller::FtpPoller::start | ( |  | ) |

starts polling in the background; returns the thread ID of the polling thread

if polling had already been started, then the thread ID of the polling thread is returned immediately

## startInline()

FtpPoller::FtpPoller::startInline | ( |  | ) |

starts the polling operation inline (not in a background thread)

Exceptions

  FTPPOLLER-ERROR | this exception is thrown if polling is already in progress

## stop()

FtpPoller::FtpPoller::stop | ( |  | ) |

stops the polling operation, returns when the polling operation has been stopped

if polling was not in progress then this method returns immediately

Exceptions

  THREAD-ERROR | this exception is thrown if this method is called from the event thread since it would result in a deadlock

See also
  stopNoWait()

## stopNoWait()

FtpPoller::FtpPoller::stopNoWait | ( |  | ) |

stops the polling operation, returns immediately

See also
  stop()

## waitStop()

FtpPoller::FtpPoller::waitStop | ( |  | ) |

waits indefinitely for the polling operation to stop

If polling was not in progress then this method returns immediately

Exceptions

  THREAD-ERROR | this exception is thrown if this method is called from the event thread since it would result in a deadlock

## Member Data Documentation

## delete_remote_file

bool FtpPoller::FtpPoller::delete_remote_file

protected

Delete file on remote FTP server after processing?

This cannot be True while move_remote_file is set

## move_remote_file

*string FtpPoller::FtpPoller::move_remote_file

protected

Remote target directory for moving the file on the remote FTP server after processing.

This cannot be True while delete_remote_file is set