DataProvider Module

# Introduction to the DataProvider Module

The DataProvider module provides APIs for hierarchical data structures from arbitrary sources to be described, queried, introspected, and updated. It also supports data providers with request-reply semantics such as REST schemas or with SOAP messaging.

The data provider module supports high-performance reading (searching) and writing as well as record creation and upserting and transaction management if supported by the underlying data provider implementation as well.

The Qore command-line program `qdp` provides a user-friendly interface to data provider functionality.

This module provides the following primary classes:

- AbstractDataField
- AbstractDataProcessor
- AbstractDataProvider
- AbstractDataProviderBulkOperation
- AbstractDataProviderBulkRecordInterface
- AbstractDataProviderFactory
- AbstractDataProviderRecordIterator
- DataProvider
- DataProviderBulkRecordIterator
- DataProviderPipeline
- DefaultBulkInserter
- DefaultBulkUpserter
- DefaultBulkRecordIterface
- DefaultRecordIterator
- QoreDataField

The following supporting type classes are also provided:

- AbstractDataProviderType
- DataProviderTypeCache
- HashDataType
- HashDeclDataType
- ListDataType
- NullDataProvider
- NullDataProviderFactory
- QoreBinaryDataType
- QoreBinaryDataTypeBase
- QoreBinaryOrNothingDataType
- QoreBoolDataType
- QoreBoolDataTypeBase
- QoreBoolOrNothingDataType
- QoreDataType
- QoreDateDataType
- QoreDateDataTypeBase
- QoreDateOrNothingDataType
- QoreFloatDataType
- QoreFloatDataTypeBase
- QoreFloatOrNothingDataType
- QoreHashDataType
- QoreIntDataType
- QoreIntDataTypeBase
- QoreIntOrNothingDataType
- QoreListDataType
- QoreNumberDataType
- QoreNumberDataTypeBase
- QoreNumberOrNothingDataType
- QoreSoftBinaryDataType
- QoreSoftBinaryNoNullDataType
- QoreSoftBinaryOrNothingDataType
- QoreSoftBoolDataType
- QoreSoftBoolNoNullDataType
- QoreSoftBoolOrNothingDataType
- QoreSoftDateDataType
- QoreSoftDateNoNullDataType
- QoreSoftDateOrNothingDataType
- QoreSoftFloatDataType
- QoreSoftFloatNoNullDataType
- QoreSoftFloatOrNothingDataType
- QoreSoftIntDataType
- QoreSoftIntNoNullDataType
- QoreSoftIntOrNothingDataType
- QoreSoftNumberDataType
- QoreSoftNumberNoNullDataType
- QoreSoftNumberOrNothingDataType
- QoreSoftStringDataType
- QoreSoftStringOrNothingDataType
- QoreStringDataType
- QoreStringDataTypeBase
- QoreStringOrNothingDataType

## Data Provider Modules

This module uses the `"QORE_DATA_PROVIDERS"` environment variable to register data provider modules. Each data provider registration module must provide one of the following two public functions.

### Data Provider Dynamic Discovery

Implement a public function with the following signature to support dynamic discovery of data providers:

# returns a hash of connection provider factory names to module names
public hash<string, string> sub get_data_provider_map() { ... }

### Data Provider Type Dynamic Discovery

Implement a public function with the following signature to support dynamic discovery of data provider types:

# returns a hash of type prefix paths (ex: \c "qore/sftp") to module names
public hash<string, string> sub get_type_provider_map() { ... }

Data provider registration modules declared in the `"QORE_DATA_PROVIDERS"` environment variable must be separated by the platform-specific PathSep character as in the following examples:

Unix Example

      export QORE_DATA_PROVIDERS=MyConnectionProvider:OtherConnectionProvider

Windows CMD.EXE Example

      set QORE_DATA_PROVIDERS=MyConnectionProvider;OtherConnectionProvider

Windows PowerShell Example

      $env:QORE_DATA_PROVIDERS="MyConnectionProvider;OtherConnectionProvider"

# Data Provider Pipelines

Data provider pipelines allow for efficient processing of record or other data in multiple streams and, if desired, in multiple threads.

Pipeline data can be any data type except a list, as list values are interpreted as multiple output values in pipeline procesor objects.

# Data Provider Pipeline Bulk Processing

Bulk processing is processing of record data that is in "hash of lists" form, so a single hash, where each key value is a list of values for that key. Records can be formed as by taking each hash key and then using each list value in order for the values of each record. In case a key is assigned a single value instead of a list, it's interpreted as constant value for all records. Note that the first key value for bulk data must be a list of values in order for the bulk data to be properly detected.

Each pipeline processor element must declare if it is compatible with bulk processing by implementing the AbstractDataProcessor::supportsBulkApiImpl() method.

If a processor does not support the bulk API, but bulk data is submitted, then the bulk data will be separately iterated, and each record will be passed separately to the processor with a significant performance penalty when processing large volumes of data.

# Release Notes

## DataProvider v1.1

- the DataProvider::DefaultRecordIterator class supports the `limit` search option

## DataProvider v1.0.6

- fixed a bug where the data type was not supported correctly as a data provider type (issue 4232)

## DataProvider v1.0.5

- fixed an issue where it was not possible for pipeline to flush any pending data or to perform reset or purge operations when pipeline processing is complete; for this the AbstractDataProcessor::flush() and AbstractDataProcessor::flushImpl() methods were implemented (issue 4107)

## DataProvider v1.0.4

- fixed type-handling bugs handling data provider options (issue 4062)

## DataProvider v1.0.3

- fixed bugs in the default return value for AbstractDataProcessor::getExpectedType() and AbstractDataProcessor::getReturnType() (issue 4043)

## DataProvider v1.0.2

- fixed a bug where HashDataType would not enforce fields or field types when checking for type and value compatibility (issue 4037)
- added new Qore base types to the data type hierarchy
- implemented the following classes:
  - AbstractDataProcessor
  - DataProviderBulkRecordIterator
  - DataProviderPipeline
    (issue 3876)
- added `DUPLICATE-RECORD` exception documentation for AbstractDataProvider::createRecord()
- updated to allow data provider type attributes to appear as children in the type hierarchy (issue 4015)

## DataProvider v1.0.1

- implemented callbacks to allow for dynamic elements of request-response data providers (such as URI paths) to be resolved at runtime (issue 3838)
- AbstractDataProviderType and AbstractDataField classes made serializable (issue 3809)
- implemented the DataProviderTypeCache class to encapsulate reusable type caching logic (issue 3810)

## DataProvider v1.0

- initial release of the module