| Qore Programming Language Reference Manual 1.0.8

Module Description

# Module Overview

Qore modules allow the Qore language to be extended at run-time.

To load a module at parse time (normally required for most modules), use the %requires or %try-module parse directive. If the named feature is not already present in Qore, Qore will try to load the module from its list of module directories (module path). This list contains the following directories:

1. directories listed in the environment variable `QORE_MODULE_DIR` (see Environment Variables)
2. `USER_MODULE_VER_DIR` - build variable set to directory where Qore user modules of the specific Qore version should be installed (ex: `/usr/share/qore-modules/0`.8.12)
3. `MODULE_VER_DIR` - build variable set to directory where Qore binary modules of the specific Qore version should be installed (ex: `/usr/lib/x86_64-linux-gnu/qore-modules/0`.8.12)
4. `USER_MODULE_DIR` - build variable set to directory where user modules of external binary modules should be installed (ex: `/usr/share/qore-modules`)
5. `MODULE_DIR` - build variable set to directory where external binary modules should be installed (ex: `/usr/lib/x86_64-linux-gnu/qore-modules`)
6. directories added using %append-module-path

Modules can also be loaded directly by using a path relative to the Qore script or module requiring them when using %requires or %try-module parse directives:

%requires ../../directory/Module.qm

Module implementation could be separated to multiple files and stored as a directory named with the same name as the module. To require them use %requires ModuleName without ".qm" extension. All files with the extenstion *qc, *ql (e.g. functions and classes) inside the directory will be automatically included at program parse time.

Examlpe of such separated module:

ModuleName/
|-ModuleName.qm # typical "qore" module file with the exception that implementation of classes and functions that it is used is stored separately
|-SomeModuleClass.qc
|-SomeModuleFunction.ql

Use the load_module() function to load Qore modules at run-time; however, note that any module providing parse support (classes, constants, functions, etc) must be loaded at parse time using the %requires or %try-module parse directive.

From Qore 0.7.1 onwards, you can specify a comparison operator (one of < <=, =, >=, or >) and version information after the module name as well. Version numbers are compared via integer comparisons of each element, where elements are separated by a '.'. If one of the versions does not have as many elements as another, the missing elements are assumed to be '0' (i.e. version `"1.0"` compared with version `"1.0.1"` will be extended to `"1.0.0"`).

Also note that DBI drivers are loaded on demand by the Qore::SQL::Datasource and Qore::SQL::DatasourcePool classes.

There are two types of modules: Binary Modules and User Modules.

At the time of writing this documentation, the following modules exist for Qore:

Modules Provided With Qore

Type | Module | Description
binary | astparser | Provides a language parser API to Qore
user | AwsRestClient | Provides a REST API for AWS services
user | BillwerkRestClient | providing APIs for communicating with billwerk.com's REST API
user | BulkSqlUtil | Provides APIs based on SqlUtil for executing bulk DML operations
user | CdsRestClient | providing APIs for communicating with the Microsoft Common Data Service REST API
user | CdsRestDataProvider | Provides a data provider API for the Microsoft Common Data Service REST API
user | ConnectionProvider | Provides an API for pluggable connection and URL providers
user | CsvUtil | Provides code to help parse CSV or other structured text files and also to easily generate such files
user | DataProvider | Provides a generic API for data providers
user | DatasourceProvider | Provides an API to translate datasource identifiers to datasource connection strings
user | DbDataProvider | Provides a data provider API for databases
user | DebugCmdLine | Provides an API to develop command line debugger clients
user | DebugLinenoiseCmdLine | Provides an API for command line debugger clients based on the linenoise module
user | DebugProgramControl | Provides an API to develop debug server
user | DebugHandler | Provides a websocket for debug server
user | DebugUtil | Provides a general stuff for debugger
user | FileLocationHandler | Provides an API for retrieving file data based on a URL-like location string
user | FilePoller | Provides an API for polling files on the filesystem
user | FixedLengthUtil | Provides code to help parse data in format of fixed length lines records
user | FreetdsSqlUtil | Provides a high-level DB-independent API for working with MS SQL Server and Sybase database objects; loaded automatically by the SqlUtil module when working with the freetds DBI driver for automated schema management, programmatic DB access, schema and data synchronization, and more
user | FsUtil | Provides a high level API for working with the filesystem
user | FtpPoller | Provides an API for polling files from a remote directory with the FTP protocol
user | FtpPollerUtil | Provides definitions for the FtpPoller module
user | HttpServer | Provides a multi-threaded HTTP server and request handler APIs; provides the infrastructure for server-side HTTP services
user | HttpServerUtil | Provides base definitions for handler classes integrating with the HttpServer module
user | Logger | Provides a flexible logging API (aka "log4q")
user | MailMessage | Provides supporting classes for the Pop3Client and SmtpClient modules; mail message serialization and deserialization, attachment handling
user | Mapper | Provides data mapping classes and iterators
user | MapperUtil | Provides common definitions for data mapping APIs
user | Mime | Provides MIME definitions and functions; MIME type lookups, MIME encoding and decoding functions, MIME multipart handling, etc
user | MysqlSqlUtil | Provides a high-level DB-independent API for working with MySQL database objects; loaded automatically by the SqlUtil module when working with MySQL databases for automated schema management, programmatic DB access, schema and data synchronization, and more
user | OracleSqlUtil | Provides a high-level DB-independent API for working with Oracle database objects; loaded automatically by the SqlUtil module when working with Oracle databases for automated schema management, programmatic DB access, schema and data synchronization, and more
user | Pop3Client | Provides POP3 client functionality; provides an API to retrieve email messages from a POP3 server
user | PgsqlSqlUtil | Provides a high-level DB-independent API for working with PostgreSQL database objects; loaded automatically by the SqlUtil module when working with PostgreSQL databases for automated schema management, programmatic DB access, schema and data synchronization, and more
user | Qdx | Provides support for documentation generation with doxygen from Qore sources
user | Qorize | Provides basic support for automatically-generating Qore code from data
user | QUnit | Provides an API for defining and executing tests and reporting results in various formats
binary | reflection | Provides a reflection API to Qore
user | RestClient | Provides a simple API for communicating with HTTP servers implementing REST services
user | RestHandler | Provides an easy to use interface to the Qore HttpServer module for implementing server-side REST services
user | RestSchemaValidator | Provides a REST schema validation API
user | SalesforceRestClient | Provides an API for communicating with Salesforce.com REST services (extends and reexports functionality provided by the RestClient module)
user | SalesforceRestDataProvider | Provides a data provider API for Salesforce REST connections
user | Sap4HanaRestClient | providing APIs for communicating with SAP S/4 Hana's REST API
user | Schema | Provides automatic schema management functionality as a meta-layer for SqlUtil's medium and low-level schema management functionality
user | ServiceNowRestClient | providing APIs for communicating with the ServiceNow REST API
user | ServiceNowRestDataProvider | Provides a data provider API for the ServiceNow REST API
user | SewioRestClient | providing APIs for communicating with Sewio.net's RTLS Studio REST API
user | SewioWebSocketClient | providing APIs for communicating with Sewio.net's RTLS Studio WebSocket API
user | SmtpClient | Provides SMTP client functionality; provides an API for sending emails via an SMTP server
user | SqlUtil | Provides a high-level DB-independent API for working with databases; for automated schema management, programmatic DB access, schema and data synchronization, and more
user | Swagger | Provides a Swagger 2.0 REST API validation API
user | SwaggerDataProvider | Provides a data provider API for Swagger schemas
user | TableMapper | Provides a data mapping functionality and iterator support using SqlUtil and Mapper to map arbitrary data to an SQL table target
user | TelnetClient | Provides Telnet client functionality
user | TextWrap | Provides text wrapping and filling functionality
user | Util | Provides a some miscellaneous generally useful routines; often used by other user modules for example
user | WebSocketClient | Provides an event-driven client API for connecting to WebSocket servers
user | WebSocketHandler | Provides an interface to the Qore HttpServer module for implementing server-side WebSocket services
user | WebSocketUtil | Provides common client and server code for implementing WebSocket protocol services in Qore
user | WebUtil | Provides higher-level classes for implementing more complex web services in Qore; works with the HttpServer module
user | ZeyosRestClient | providing APIs for communicating with Zeyos.com's REST API

Modules Provided Separately

Type | Module | Description
binary | freetds | Provides a FreeTDS-based DBI driver for communicating with Sybase and MS SQL Server databases
binary | fsevent | Provides an event-driven filesystem event API
binary | jni | Provides access to Java APIs in Qore
binary | json | Provides JSON and JSON-RPC client functionality, also provides the following user modules: - JsonRpcHandler: provides infrastructure for implementing JSON-RPC server-side services using the HttpServer module
binary | linenoise | Provides a readline-like API to Qore under a permissive license
binary | msgpack | Provides APIs for efficient data serialization in Qore using the MessagePack format
binary | mysql | Provides a MySQL / MariaDB / Percona DBI driver
binary | odbc | Provides an ODBC DBI driver to Qore
binary | oracle | Provides an Oracle DBI driver, providing many advanced features such as support for named types and collections, advanced queuing, etc, also provides the following user modules: - OracleExtensions: provides infrastructure for SQL tracing
binary | openldap | Provides an OpenLDAP API to Qore
binary | pgsql | Provides a PostgreSQL DBI driver
binary | process | Provides an API for controlling external processes
binary | sqlite3 | Provides an SQLite3 DBI driver
binary | ssh2 | Provides SSH2 and SFTP functionality, also provides the following user module: - SftpPoller: provides event-driven support for polling an SFTP server for new data
binary | sybase | Provides a Sybase DBI driver
binary | sysconf | Provides sysconf, pathconf, and confstr APIs
binary | tibae | Provides TIBCO ActiveEnterprise(TM) (TIBCO, Inc) functionality
binary | tibrv | Provides TIBCO Rendezvous(R) (TIBCO, Inc) functionality
binary | uuid | Provices an API for generating UUIDs
binary | xml | Provides XML (SAX and DOM parsers), XPath, XML-RPC, SOAP client and server, etc functionality, also provides the following user modules: - SoapClient: provides an easy to use API for making requests to SOAP servers - SalesforceSoapClient: provides an easy to use API for making requests to Salesforce.com SOAP servers - SoapHandler: provides infrastructure for implementing SOAP server-side services using the HttpServer module - WSDL: provides underlying web service/WSDL support for the SoapClient and SoapHandler modules - WebDavHandler: provides an API for serving WebDAV content using the HttpServer module

|binary|xmlsec|Provides xmldsig and xmlenc functionality |binary|yaml|Provides YAML functionality, also provides the following user modules:
- DataStreamClient: provides a DataStream client API extending the RestClient
- DataStreamRequestHandler: provides a DataStream server-side handler API extending the RestHandler
- DataStreamUtil: provides underlying DataStream client and server protocol support
- YamlRpcClient: provides an API for easily making YAML-RPC calls over the network
- YamlRpcHandler: provides infrastructure for implementing YAML-RPC server-side services using the HttpServer module |binary|zmq|Provides ZeroMQ distributed queue and clustering API support

# Binary Modules

Binary modules are written in C++ and delivered in binary form. They must conform to the Qore Module API and have the file extension `"*.qmod"`. Binary modules normally depend on other shared libraries and therefore can only be loaded if the libraries they require are present on the system and can be found in the library path.

Binary modules are merged into Program objects in the same way as the static system namespace objects are imported. It is possible to import a binary module and be able to use only part of its functionality, for example, if the importing Program cannot access the filesystem, and the module has functions that access the filesystem, only the functions that access the filesystem will not be available. In user modules, the functional domain is set on the module level and not on the individual function or method level as with builtin objects, so user modules are either completely imported or not at all.

Please note that as of version 0.7.0 onwards, the source code for binary modules has been split from the main Qore library source code into separate projects; see Qore's home page for more information.

# User Modules

User modules are written in Qore and delivered in source form. They must have the extension `"*.qm"`.

User modules will have their own dedicated Program object; the Program object is created automatically when the module is loaded and initialized. The module's Program object is created with locked parse options as follows:

- PO_NO_TOP_LEVEL_STATEMENTS: modules provide API services and are not designed to be executed at the top-level. Put any initialization code in the module's `init` closure
- PO_REQUIRE_PROTOTYPES: to ensure that module's APIs are transparent, all method and function declarations must include parameter and return types
- PO_REQUIRE_OUR: this parse option is meant to ensure that typos in variable names are caught at parse time and to ensure transparency of variable types in the module's source code
- PO_IN_MODULE: this parse option is set so that module code is recognized by the system when parsing modules

When parsing module code, the default warning mask is set to WARN_MODULES, and any warnings are treated as errors. Furthermore, any restrictions that the importing Program object has will be added to the module's Program object.

Any namespaces, classes, constants, functions, and global variables declared as public will be exported into the importing Program object; all other declarations will be private to the module's Program object.

Note
  The root namespace of a module Program is always public by default, however declarations in the root namespace must be marked public to be exported to importing Program objects.

If a user module imports other modules with the `requires(reexport)` form of the %requires parse directive, then any loaded definitions will also be loaded into the importing Program.

Note that global variables exported from a module's Program object are exported as references; each global variable declared in a module is unique, and they reside in the module's Program object.

Furthermore, when using an environment with multiple Program objects, if a user module has already been loaded and initialized, then it's functional domain mask is compared against any importing Program object's restrictions; if the module uses functionality that is not allowed in the importing Program object, then an exception is raised and the module is not imported.

Also note that the Qore::Program::constructor() applies a mask to the parse option mask option when Program objects are created in a user module; in this case the parse options passed to the child Program object are masked with the current parse options in the user module, and they are locked so that they cannot be made less restrictive. This is to prevent user modules from circumventing functional restrictions imposed by parse options.

## User Module Declarations

User modules are declared with a special syntax in Qore:

User Module Declaration Syntax
`module` name `{`
`version = "`version string`";`
`desc = "`description string`";`
`author = "`author string`";`
`[url = "`URL string`";]`
`[license = "`license string`";]`
`[init =` initialization closure`;]`
`[del =` deletion closure`;]`
`}`

Module properties are as follows:

- `version:` (required) must be assigned to a string giving the version of the module
- `desc:` (required) must be assigned to a string giving a text description of the module
- `author:` (required) must be assigned to a string giving the module's author's name
- `url:` (optional) if present, must be a string giving the URL of the module
- `license:` (optional) if present, must be a string giving the license of the module
- `init:` (optional) if present, must be a closure that will be executed when the module is loaded; this can be used to initialize the module, for example
- `del:` (optional) if present, must be a closure that will be executed when the module is unloaded; this can be used to stop running services or free resources managed and still allocated by the module, for example

Note
  - any unhandled exceptions in the `init` closure will cause the module to fail to load
  - unhandled exceptions in the `init` and `del` closures are displayed on stdout

## The "public" Keyword

Only objects defined with the `public` keyword are made available in Program objects importing the user module. All other declarations and definitions are private to the module.

The `public` keyword also affects inheritance in child Program objects as well as controlling exported symbols from user modules.

The `public` keyword must be used with the following declarations in the module to export them:

- namespaces: namespaces themselves must be declared `public` in order for any of their contents also to be exported (it is an error to try to declare public members of a module-private namespace). ex:
  public namespace MyNamespace { ... }
- classes: classes not declared `public` will not be exported; there is no way to export part of a class; either the entire class is exported or it is not. ex:
  public MyNamespace {
  public MyClass { ... }
  }
- constants: constants must be declared `public` to be exported; ex:
  public namespace MyNamespace {
  public const MyConst = 100;
  }
- functions: function variants must be declared `public` in order to be exported. ex:
  public MyNamespace {
  public int sub my_func() { ... }
  }
- global variables: only global variables declared `public` will be exported. ex:
  public namespace MyNamespace {
  public our int OurInt;
  }

Note
  Global variable declarations in a namespace declaration cannot be initialized at the point they are declared, also, since Qore::PO_NO_TOP_LEVEL_STATEMENTS is set for user module Program objects, global variables also cannot be initialized at the top-level. Use the `init` closure to initialize the module and any global variables requiring initialization.

User modules are only imported into a Program if the importing Program can use all of the capabilities used in the user module. It is not possible to partially import a user module (in contrast with binary modules, which can be imported even if they expose functionality not allowed in the importing Program, however that functionality will not be available in that case). User module Program objects have a functional domain attribute set on the Program level, so either a user module is imported in its entirety or not at all.

## Module Example

Here is an example declaring user module `"foo"` version `"1.0"`:

module foo {
version = "1.0";
desc = "test module";
author = "Foobar Industries, inc";
url = "http://example.com";
license = "MIT";
init = sub () {
Foo::initialized = True;
Bar::OurBool = False;
};
del = sub () {
print("goodbye, cruel world!\n");
}
}
# do not use "$", assume local scope for variables
%new-style
# nothing in namespace ::Foo is exported
namespace Foo {
# inline global variable declarations cannot be initialized when declared
our bool initialized;
class SuperClass {
}
class NotSoGreatClass {
}
class UnstableClass {
}
}
# public members of namespace ::Bar are exported
public namespace Bar {
# Bar::SomeClass is exported
public class SomeClass {
}
# Bar::add(int, int) is exported
public int sub add(int x, int y) { return x + y; }
# Bar::OurBool is exported
public our bool OurBool;
# Bar::PrivateClass is not exported
class PrivateClass {
}
}

Since
  Qore 0.8.4 user modules are supported
  Qore 0.9.0 user modules implementation could be separated to multiple files

## Program Scope in Object-Oriented Programs Using User Modules Providing Their Own Threads

When using the %exec-class parse directive, the application object will go out of scope as soon as the constructor terminates (unless there are valid scope-extending references to the application object, such as making an assignment of `self` to a global variable in the contructor).

Therefore when using a module that provides services in its own threads (such as, for example, the HttpServer module), it's important to make sure that the application object does not go out of scope while non-static method call references to the application object are needed by active threads in the user module (for example, non-static method call references passed as callbacks to the HttpServer module, etc).

This also applies to call references to non-static methods passed to set_signal_handler().

If a module thread tries to use a callback or closure in a Program that has already gone out of scope, an `OBJECT-ALREADY-DELETED` exception (`"attempt to access member of an already-deleted object"`) is thrown.

Therefore in such cases it's best to wait for all threads in any modules to terminate before allowing the application object's constructor to terminate.

For an example of this, see the example program `examples/httpserver.q` where the main Program calls HttpServer::waitStop() before exiting the application object's constructor for this reason.