# How to add a new CPD language

How to add a new language module with CPD support.
Table of Contents

## Adding support for a CPD language

CPD works generically on the tokens produced by a `CpdLexer`. To add support for a new language, the crucial piece is writing a CpdLexer that splits the source file into the tokens specific to your language. Thankfully you can use a stock Antlr grammar or JavaCC grammar to generate a lexer for you. If you cannot use a lexer generator, for instance because you are wrapping a lexer from another library, it is still relatively easy to implement the Tokenizer interface.

Use the following guide to set up a new language module that supports CPD.

1. Create a new Maven module for your language. You can take the Golang module as an example.
   - Make sure to add your new module to the parent pom as `<module>` entry, so that it is built alongside the other languages.
   - Also add your new module to the dependencies list in “pmd-languages-deps/pom.xml”, so that the new language is automatically available in the binary distribution (pmd-dist).
2. Implement a `CpdLexer`.

   - For Antlr grammars you can take the grammar from antlr/grammars-v4 and place it in `src/main/antlr4` followed by the package name of the language. You then need to call the antlr4 plugin and the appropriate ant wrapper with target `cpd-language` to generate the lexer from the grammar. To do so, edit `pom.xml` (eg like the Golang module). Once that is done, `mvn generate-sources` should generate the lexer sources for you.

     You can now implement a CpdLexer, for instance by extending `AntlrCpdLexer`. The following reproduces the Go implementation:

        // mind the package convention if you are going to make a PR
        package net.sourceforge.pmd.lang.go.cpd;

        public class GoCpdLexer extends AntlrCpdLexer {

            @Override
            protected Lexer getLexerForSource(CharStream charStream) {
                return new GolangLexer(charStream);
            }
        }

   - If your language is case-insensitive, then you might want to overwrite `getImage(AntlrToken)`. There you can change each token e.g. into uppercase, so that CPD sees the same strings and can find duplicates even when the casing differs. See `TSqlCpdLexer` for an example. You will also need a “CaseChangingCharStream”, so that antlr itself is case-insensitive.
   - For JavaCC grammars, place your grammar in `etc/grammar` and edit the `pom.xml` like the Python implementation does. You can then subclass `JavaccCpdLexer` instead of AntlrCpdLexer.
   - If your JavaCC based language is case-insensitive (option `IGNORE_CASE=true`), then you need to implement `JavaccTokenDocument.TokenDocumentBehavior`, which can change each token e.g. into uppercase. See `PLSQLParser` for an example.
   - For any other scenario just implement the interface however you can. Look at the Scala or Apex module for existing implementations.

3. Create a `Language` implementation, and make it implement `CpdCapableLanguage`. If your language only supports CPD, then you can subclass `CpdOnlyLanguageModuleBase` to get going:

        // mind the package convention if you are going to make a PR
        package net.sourceforge.pmd.lang.go;

        public class GoLanguageModule extends CpdOnlyLanguageModuleBase {

            // A public noarg constructor is required.
            public GoLanguageModule() {
                super(LanguageMetadata.withId("go").name("Go").extensions("go"));
            }

            @Override
            public Tokenizer createCpdLexer(LanguagePropertyBundle bundle) {
                // This method should return an instance of the CpdLexer you created.
                return new GoCpdLexer();
            }
        }

   To make PMD find the language module at runtime, write the fully-qualified name of your language class into the file `src/main/resources/META-INF/services/net.sourceforge.pmd.lang.Language`.

   At this point the new language module should be available in `CPD` and usable by CPD like any other language.

4. Update the test that asserts the list of supported languages by updating the `SUPPORTED_LANGUAGES` constant in BinaryDistributionIT.

5. Add some tests for your CpdLexer by following the section below.

6. Add a page in the documentation. Create a new markdown file `<langId>.md` in `docs/pages/pmd/languages/`. This file should have the following frontmatter:

       ---
       title: <Language Name>
       permalink: pmd_languages_<langId>.html
       last_updated: <Month> <Year> (<PMD Version>)
       tags: [languages, CpdCapableLanguage]
       ---

   On this page, language specifics can be documented, e.g. when the language was first supported by PMD. There is also the following Jekyll Include, that creates summary box for the language:

       {% include language_info.html name='<Language Name>' id='<langId>' implementation='<langId>::lang.<langId>.<langId>LanguageModule' supports_cpd=true %}

7. Finishing up your new language module by adding a menu entry in PMD Sidebar. ```
   - title: url: /pmd_languages_.html output: web, pdf ```

### Declaring CpdLexer options

To make the CpdLexer configurable, first define some property descriptors using `PropertyFactory`. Look at `CpdLanguageProperties` for some predefined ones which you can reuse (prefer reusing property descriptors if you can). You need to override `newPropertyBundle` and call `definePropertyDescriptor` to register the descriptors. After that you can access the values of the properties from the parameter of `createCpdTokenizer`.

To implement simple token filtering, you can use `BaseTokenFilter` as a base class, or another base class in `net.sourceforge.pmd.cpd.impl`. Take a look at the Kotlin token filter implementation, or the Java one.

### Testing your implementation

Add a Maven dependency on `pmd-lang-test` (scope `test`) in your `pom.xml`. This contains utilities to test your CpdLexer.

Create a test class extending from `CpdTextComparisonTest`. To add tests, you need to write regular JUnit `@Test`-annotated methods, and call the method `doTest` with the name of the test file.

For example, for the Dart language:

    package net.sourceforge.pmd.lang.dart.cpd;

    public class DartTokenizerTest extends CpdTextComparisonTest {

        /**********************************
          Implementation of the superclass
        ***********************************/

        public DartTokenizerTest() {
            super("dart", ".dart"); // the ID of the language, then the file extension used by test files
        }

        @Override
        protected String getResourcePrefix() {
            // "testdata" is the default value, you don't need to override.
            // This specifies that you should place the test files in
            // src/test/resources/net/sourceforge/pmd/lang/dart/cpd/testdata
            return "testdata";
        }

        /**************
          Test methods
        ***************/

        @Test  // don't forget the JUnit annotation
        public void testLiterals() {
            // This will look for a file named literals.dart
            // in the directory identified by getResourcePrefix,
            // tokenize it, then compare the result against a baseline
            // literals.txt file in the same directory

            // If the baseline file does not exist, it is created automatically
            doTest("literals");
        }

    }

Tags: devdocs extending