Loading...
Searching...
No Matches
DataProvider::DataProviderPipeline Class Reference

Defines a class for passing data through record processors. More...

`#include <DataProviderPipeline.qc.dox.h>`

## Public Member Functions

| abort (*bool ignore_exceptions)
| Aborts execution of a pipeline in progress. More...
bool | aborting ()
| Returns True if the object is aborting.
| append (AbstractDataProcessor processor)
| Appends a data processor to the default queue. More...
| append (int id, AbstractDataProcessor processor)
| Appends a data processor to a queue. More...
int | appendQueue (int id)
| Appends a new queue to an existing pipeline and returns the new queue ID. More...
| constructor (*hash< PipelineOptionInfo > opts)
| Creates the object with the given options. More...
| copy ()
| Copy constructor; creates an empty pipeline with the same configuration as the original.
| destructor ()
| Destroys the object. More...
hash< PipelineInfo > | getInfo ()
| Returns pipeline info. More...
string | getName ()
| Returns the pipeline name.
bool | hasQueue (int id)
| Returns True if the given queue exists, False if not.
bool | isProcessing ()
| Returns True if the pipeline is processing data.
| logDebug (string fmt)
| Logs to the debug log, if set.
| logError (string fmt)
| Logs to the error log, if set.
| logInfo (string fmt)
| Logs to the info log, if set.
| registerThread (PipelineQueue queue)
| Registers a new thread.
| reportError (PipelineQueue queue, hash< ExceptionInfo > ex)
| Called from a pipeline queue object to report a fatal error durring processing.
| reset ()
| Resets the pipeline. More...
bool | stopping ()
| Returns True if the object is stopping.
| submit (AbstractDataProviderBulkRecordInterface i)
| Submits data for processing.
| submit (AbstractDataProviderRecordIterator i)
| Submits data for processing.
| submit (auto _data)
| Submits data for processing.
| submitData (AbstractIterator i)
| Submits data for processing.
| waitDone ()
| Waits for all queues to have processed remaining data. More...

## Protected Member Functions

Counter | cnt ()
| Thread counter.
Mutex | lck ()
| Atomic lock.
Sequence | seq (1)
| Pipeline ID sequence generator.

## Protected Attributes

bool | abort_flag
| Abort flag.
*code | debug_log
| Debug log closure; takes a single format string and then arguments for format placeholders.
bool | do_bulk = True
| Bulk flag. More...
list< hash< ExceptionInfo > > | error_list
| list of exceptions in pipelines
*code | error_log
| Error log closure; takes a single format string and then arguments for format placeholders.
*code | info_log
| Info log closure; takes a single format string and then arguments for format placeholders.
bool | locked = False
| Locked flag.
string | name
| A descriptive name for logging purposes.
hash< string, PipelineQueue > | pmap
| Hash of queues keyed by queue ID.
int | record_count = 0
| Record count.
date | start_time
| run start time
bool | stop_flag
| Stop flag.
date | stop_time
| run stop time (set in waitDone())
*code | thread_callback
| a closure or call reference for setting thread-local data in new pipeline queue threads

## Private Member Functions

| checkLockedIntern ()
| Throws an exception if the pipeline is locked. More...
| checkSubmitIntern ()
| Throws an exception if the pipeline cannot be used; locks the pipeline for changes otherwise. More...
| checkUpdatePipelineIntern (int id)
| Check if the given queue exists.
PipelineQueue | copyPipeline (PipelineQueue old_queue)
| Called by the copy constructor to copy the queues.
| resetIntern ()
| Resets the pipeline. More...
| stopIntern ()
| Stops all background pipeline queues.
| stopInternUnlocked ()
| Stops all background pipeline queues; lock must be held.
| submitBulkIntern (AbstractDataProviderBulkRecordInterface i)
| Submits bulk data for processing. More...
| submitDataIntern (auto _data)
| Submits a single record for processing. More...
| submitIntern (auto _data)
| Submits data for processing. More...
| throwPipelineException ()
| Throws an exception if errors occured in background pipeline processing. More...

## Detailed Description

Defines a class for passing data through record processors.

Record processing pipelines run in background threads. Each queue has an integer queue ID; a queue with ID 0 is created by default as the initial queue.

Note
  Pipeline data can be of any type

## Member Function Documentation

## abort()

DataProvider::DataProviderPipeline::abort | ( | *bool | ignore_exceptions | ) |

Aborts execution of a pipeline in progress.

Parameters

  ignore_exceptions | if True then any processing exceptions are ignored

Exceptions

  PIPELINE-FAILED | thrown if ignore_exceptions is not True and there are any errors in pipeline processing, in which case the exception argument will have a list of exception arguments thrown by pipeline threads

Note
  The pipeline must be reset once aborted to be used again

## append() [1/2]

DataProvider::DataProviderPipeline::append | ( | AbstractDataProcessor | processor | ) |

Appends a data processor to the default queue.

Note
  The initial queue is queue 0

See also
  appendQueue()

## append() [2/2]

DataProvider::DataProviderPipeline::append | ( | int | id,
|  | AbstractDataProcessor | processor
| ) |  |

Appends a data processor to a queue.

Parameters

  id | the queue ID as returned from appendQueue()
  processor | the data processor to append to the pipeline queue

Exceptions

  PIPELINE-ERROR | invalid queue ID or the queue already terminates in additional queues; the pipeline is locked

Note
  The initial queue is queue 0

See also
  appendQueue()

## appendQueue()

int DataProvider::DataProviderPipeline::appendQueue | ( | int | id | ) |

Appends a new queue to an existing pipeline and returns the new queue ID.

Parameters

  id | the queue to which the new queue will be appended

Returns
  the new queue ID

Exceptions

  PIPELINE-ERROR | the pipeline is locked, or the given queue does not exist

Note
  The initial queue is queue 0

See also
  append(int, AbstractDataProcessor)

## checkLockedIntern()

DataProvider::DataProviderPipeline::checkLockedIntern | ( |  | ) |

private

Throws an exception if the pipeline is locked.

Must be called with the lock held

## checkSubmitIntern()

DataProvider::DataProviderPipeline::checkSubmitIntern | ( |  | ) |

private

Throws an exception if the pipeline cannot be used; locks the pipeline for changes otherwise.

Must be called with the lock held

## constructor()

DataProvider::DataProviderPipeline::constructor | ( | *hash< PipelineOptionInfo > | opts | ) |

Creates the object with the given options.

Parameters

  opts | any options for the pipeline; see PipelineOptionInfo for more information

Note
  The object is created with an initial queue with ID 0

## destructor()

DataProvider::DataProviderPipeline::destructor | ( |  | ) |

Destroys the object.

To ensure that the destructor does not throw a `PIPELINE-FAILED` exception, call run() or runAsync() and waitDone()

Exceptions

  PIPELINE-FAILED | thrown if there are any errors in pipeline processing, in which case the exception argument will have a list of exception arguments thrown by pipeline queue threads

## getInfo()

hash< PipelineInfo > DataProvider::DataProviderPipeline::getInfo | ( |  | ) |

Returns pipeline info.

Returns
  pipeline info; see PipelineInfo for more information

Note
  record count and performance intormation is only valid after the pipeline has completed processing

## reset()

DataProvider::DataProviderPipeline::reset | ( |  | ) |

Resets the pipeline.

Exceptions

  PIPELINE-ERROR | this method cannot be called while the pipeline is processing data; call abort() or waitDone() before resetting if the pipeline is processing data

## resetIntern()

DataProvider::DataProviderPipeline::resetIntern | ( |  | ) |

private

Resets the pipeline.

Must be called with the lock held

## submitBulkIntern()

DataProvider::DataProviderPipeline::submitBulkIntern | ( | AbstractDataProviderBulkRecordInterface | i | ) |

private

Submits bulk data for processing.

Must be called with the lock held

See also
  Data Provider Pipeline Bulk Processing

## submitDataIntern()

DataProvider::DataProviderPipeline::submitDataIntern | ( | auto | _data | ) |

private

Submits a single record for processing.

Must be called with the lock held

## submitIntern()

DataProvider::DataProviderPipeline::submitIntern | ( | auto | _data | ) |

private

Submits data for processing.

Must be called with the lock held

## throwPipelineException()

DataProvider::DataProviderPipeline::throwPipelineException | ( |  | ) |

private

Throws an exception if errors occured in background pipeline processing.

Must be called with the lock held

## waitDone()

DataProvider::DataProviderPipeline::waitDone | ( |  | ) |

Waits for all queues to have processed remaining data.

Exceptions

  PIPELINE-FAILED | thrown if there are any errors in pipeline processing, in which case the exception argument will have a list of exception arguments thrown by pipeline queue threads

## Member Data Documentation

## do_bulk

bool DataProvider::DataProviderPipeline::do_bulk = True

protected

Bulk flag.

See also
  Data Provider Pipeline Bulk Processing