Go to page content

Building Xtext plugins with Maven and Tycho: a step-by-step guide

Highly prescriptive, Step-by-Step instructions for building xtext projects from the command line with Maven & Tycho. Requires no prior knowledge of Maven or Tycho.

Out of the box, Xtext generates much of the infrastructure required to create working editors for a Domain Specific Language. At some point however, it’s likely you’ll want to run your generated editor separate from the eclipse instance you’re developing it in - for example to make it available to users. That means generating an update site. Automating this process makes the task easily repeatable. It’s also good practice to run all tests as part of the build.

Note: this article is long (very long...).  If you'd rather skip the narrative, the source files for everything covered are available here.

The goals covered below are:

  1. To enable an Xtext-derived DSL editor to be installed into another eclipse instance (using the Help->Install New Software menu). This is known as Building and Deploying an Update Site
  2. To be able to run all JUnit tests from the command line by issuing a single command
  3. To only build & deploy the update site if the tests pass.

There are a few options for achieving these goals. Perhaps surprisingly, step-by-step documentation appears to be thin on the ground. For someone not already familiar with the tools involved it can be quite frustrating.

This article provides one way to achieve the goals. It is highly prescriptive and makes very little assumption about prior experience with any of the tools & techniques involved. I make no apologies for including all the information, on the basis it’s easier to skip detail you know than find something that’s been omitted.

Note this tutorial uses Tycho to automate the build. Other options are available (e.g. Buckminster).

Overview of the Process

There are essentially three steps involved in meeting the goals above:

  1. Creating the Extra Projects. Create some extra projects required to generate the update site
  2. Automating the Test and Build Process. Defining the build process and running it.
  3. Testing the Result. Try to install the plugin from the generated update site into a fresh eclipse instance.

Precursor: the DSL project

First we need a DSL project to work with. If you already have one (quite likely) you can skip ahead to Creating the Extra Projects. If not, you need to create one using the xtext project wizard. Detailed instructions for this are provided in the xtext documentation so are not covered here. Note this tutorial uses the Juno release (eclipse 4.2 / Xtext 2.3.1).

The xtext project wizard creates 3 eclipse projects for each dsl. For the purposes of this tutorial, it’s assumed the name of the project is org.xtext.example.mydsl (as per the standard xtext example). As output, the xtext project creation wizard will create the following projects:

  • org.xtext.example.mydsl
  • org.xtext.example.mydsl.ui
  • org.xtext.example.mydsl.tests

Depending on the version of xtext you are using, the wizard may also generate an additional project org.xtext.example.mydsl.sdk. (see The Feature Project below).

The purpose of each is also described in the Xtext Documentation so again isn’t covered further here.

We don’t need to change the DSL infrastructure - but do need to have xtext generate the DSL code for the sample. If you haven’t done so, generate it now:

  1. Expand the org.xtext.example.mydsl project
  2. Expand the src folder
  3. Expand the org.xtext.example.mydsl package
  4. Open MyDsl.xtext
  5. Right click in the newly-opened editor window for MyDsl.xtext and select Run As->1 Generate XText Artifacts

This will generate the DSL implementation files (note: accept download of the antlr parser generator if prompted).

Creating a Unit Test

We’ll need this later to ensure the tests are run as part of the build. If you already have tests in your project skip ahead to Creating the Extra Projects. If not here’s how to create a simple one. It doesn’t matter what the test does, just that we have one:

  1. Open the org.xtext.example.mydsl.tests project
  2. Right click on the src folder and select New->Package
  3. Enter org.xtext.example.mydsl.tests as the name and click Finish
  4. Right click on the newly created package in the project explorer and select New->Other...->Java->JUnit->JUnit Test Case then click Next
  5. Enter MyDslTest as the Class Name then click Finish
  6. When the editor opens, replace the generated source with the following:
    package org.xtext.example.mydsl.tests;
    
    import static org.junit.Assert.*;
    import org.junit.Test;
    
    public class MyDslTest {
    
        @Test
        public void test() {
                assertTrue(true);
        }
    }

Now test the test. Right click in the editor and select Run As->2 JUnit Test. The JUnit test runner will start and should show a green bar (i.e. the test passed).

OK. We now have a test language to automate building.

Creating the extra projects

Why are extra projects needed? To fit with the eclipse architecture. The projects - and their purposes - are as follows:

  • A Feature project. Xtext generates two plugin projects that, combined, deliver the DSL editing functionality (org.xtext.example.mydsl and org.xtext.example.mydsl.ui). Neither is much use on their own; they need to be combined. Such a wrapper is called a Feature; the Feature project defines the wrapper.
  • An Update Site project. Collects the Feature with its plugins and dependencies and creates a repository that allows the feature to be installed from Eclipse via the Help->Install New Software menu.

The Feature Project

Later versions of the Xtext wizard create the Feature project automatically. If you have a project with an sdk suffix (org.xtext.example.mydsl.sdk in the example) then skip forward to The Update Site Project. If not, create the Feature project as follows:

  1. File->New->Other...->Plug-in Development->Feature Project & click Next.
  2. Set the ID as org.xtext.example.mydsl.sdk
  3. Enter a name for the feature, e.g. MyDsl SDK Feature. This is the name users will see when they install the feature. Click Next
  4. Add the two plugins that comprise the feature from the list: org.xtext.example.mydsl and org.xtext.example.mydsl.ui. Click Finish.

That’s it. Note you don’t need to include the tests project plugin in the feature definition - it’s not required for the feature to run.

The Update Site Project

For a Feature to be installable by a User it must be published on an Update Site. The update site provides some info about the feature - name, version, description etc. - as well as the plugins that implement it. Create the Update Site Project as follows:

  1. File->New->Other...->General->Project and click Next.
  2. Set the project name to e.g. org.xtext.example.mydsl.updatesite & click Finish

We now need to create a Category. Categories are used to classify plugins and make it easier for Users to search for and find your feature. To add the Category:

  1. Select main folder of update site project and add a category:
  2. File->New->Other...->Plug-in Development->Category Definition & click Finish
  3. In the created category.xml, click New Category and create a new category with ID org.xtext.example.mydsl and give it a name (e.g. MyDsl Example Category).
  4. Click Add Feature and add the org.xtext.example.mydsl.sdk feature to the category
  5. Save the Category definition (File->Save).

That’s it. We now have the extra projects required to generate the update site.

Automating the test and build process

This article uses Tycho to automate the build process. Other options are available, e.g. Buckminster.

Tycho, in turn, is built on Maven. It’s fair to say Maven polarises opinion: it has equally as many ardent fans as it does vocifeous detractors. And t’s perhaps not the best documented tool on the planet [1].

However: the good news is you don’t need to know much about Maven to achieve the goal of automated builds.

Overview

There are three steps to automating the build with Tycho:

  1. Install Maven
  2. Define the Build Instructions: Tell Maven/Tycho how to build each project
  3. Build.

Installing Maven

  1. Download & install from here. Note you need at least version 3.x.
  2. Ensure maven is on your path (follow instructions on the download site).
  3. Ensure it’s working: run mvn --version. The result should be something like this:
    $ mvn --version
    Apache Maven 3.0.4 (r1232337; 2012-01-17 08:44:56+0000)
    Maven home: /apps/mvn
    Java version: 1.6.0_35, vendor: Apple Inc.
    Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
    Default locale: en_GB, platform encoding: MacRoman
    OS name: "mac os x", version: "10.7.5", arch: "x86_64", family: "mac"

Defining the Build Instructions

Maven is configured using a Project Object Model (POM)[2]. This is usually stored in a file named pom.xml in the root directory of the project. To automate an Xtext DSL infrastructure build therefore requires a pom.xml for:

  • Both plugin projects (org.xtext.example.mydsl & org.example.mydsl.ui)
  • The feature project (org.xtext.example.mydsl.sdk)
  • The update site project (org.xtext.example.mydsl.updatesite)
  • The test project (org.xtext.example.mydsl.tests)

Finally, we need a master POM to link them all together.

The POMs are created as follows.

Plugin Project POMs

  1. Select the main plugin project in the project explorer (org.xtext.example.mydsl)
  2. Right Click->New->File
  3. Name the file pom.xml & Click Finish
  4. The pom editor will open (if not, right click on the newly created file and click Open).
  5. At the bottom of the editor window click on the tab named pom.xml (note: don’t select the one named Effective POM).
  6. Paste in the following contents & save the file:
<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <relativePath>../org.xtext.example.mydsl.releng/pom.xml</relativePath>
    <groupId>org.xtext.example.mydsl</groupId>
    <artifactId>parent</artifactId>
    <version>1.0.0-SNAPSHOT</version>
  </parent>

  <artifactId>org.xtext.example.mydsl</artifactId>
  <packaging>eclipse-plugin</packaging>

  <name>MyDSL core language services</name>
</project>

Don’t worry about what the <parent> entry means for now, it’s covered later.

Now repeat as above for the UI plugin (org.xtext.example.mydsl.ui). Create a pom.xml in the root directory of the project and copy in the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <relativePath>../org.xtext.example.mydsl.releng/pom.xml</relativePath>
        <groupId>org.xtext.example.mydsl</groupId>
        <artifactId>parent</artifactId>
        <version>1.0.0-SNAPSHOT</version>
  </parent>

  <artifactId>org.xtext.example.mydsl.ui</artifactId>
  <packaging>eclipse-plugin</packaging>

  <name>MyDSL core language User Interface</name>
</project>

Feature Project POM

Similar drill as above. Right click on the Feature project (org.xtext.example.mydsl.sdk) and create the pom.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <relativePath>../org.xtext.example.mydsl.releng/pom.xml</relativePath>
    <groupId>org.xtext.example.mydsl</groupId>
    <artifactId>parent</artifactId>
    <version>1.0.0-SNAPSHOT</version>
  </parent>

  <artifactId>org.xtext.example.mydsl.sdk</artifactId>
  <packaging>eclipse-feature</packaging>

  <name>MyDSL Example Feature</name>
</project>

While we’re in the Feature project, it’s a convenient time to make sure the contents of the feature.xml are correct. Open the file and ensure it appears as follows. In particular, ensure the version of the plugins matches that used in the POM (1.0.0.qualifier to match the POM above):

<?xml version="1.0" encoding="UTF-8"?>
<feature id="org.xtext.example.mydsl.sdk"
        label="MyDsl SDK Feature "
        version="1.0.0.qualifier">
        <plugin
                        id="org.xtext.example.mydsl"
                        download-size="0"
                        install-size="0"
                        version="1.0.0.qualifier"
                        unpack="false"/>
        <plugin
                        id="org.xtext.example.mydsl.ui"
                        download-size="0"
                        install-size="0"
                        version="1.0.0.qualifier"
                        unpack="false"/>
</feature>

Update Site Project POM

As above with pom.xml contents as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project>
        <modelVersion>4.0.0</modelVersion>

        <parent>
          <relativePath>../org.xtext.example.mydsl.releng/pom.xml</relativePath>
          <groupId>org.xtext.example.mydsl</groupId>
          <artifactId>parent</artifactId>
          <version>1.0.0-SNAPSHOT</version>
        </parent>

        <artifactId>org.xtext.example.mydsl.updatesite</artifactId>
        <packaging>eclipse-repository</packaging>

        <name>MyDSL Example Update Site</name>
</project>

Test Project POM

And again for the test project:

<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <relativePath>../org.xtext.example.mydsl.releng/pom.xml</relativePath>
    <groupId>org.xtext.example.mydsl</groupId>
    <artifactId>parent</artifactId>
    <version>1.0.0-SNAPSHOT</version>
  </parent>

  <artifactId>org.xtext.example.mydsl.tests</artifactId>
  <packaging>eclipse-test-plugin</packaging>

  <name>MyDSL Example tests</name>

  <build>
    <plugins>
      <plugin>
        <groupId>org.eclipse.tycho</groupId>
        <artifactId>tycho-surefire-plugin</artifactId>
        <version>${tycho-version}</version>
        <configuration>
          <useUIHarness>false</useUIHarness>
          <useUIThread>false</useUIThread>
        </configuration>
      </plugin>
        </plugins>
  </build>
</project>

While we’re in the test project, it’s worth checking the contents of the build.properties file. It should read as follows:

source.. = src/,\
           src-gen/
bin.includes = META-INF/,\
               .

If you have anything else - notably a final entry that reads plugin.xml - remove it. For some reason it breaks the maven build, whilst removing it doesn’t appear to affect the eclipse project in any way.

Tying it all together - the parent POM

We now have POMs to build each of the components; the final step is to tie them all together. This is parent POM referenced in each of those above. It’s possible to put this in the eclipse workspace directory outside of an eclipse project: that will work for maven but means the POM can’t be edited from within eclipse. So instead we’ll create another project - the release engineering project - and put the master POM in there. Here’s how:

  1. File->New->Other...->General->Project & click Next.
  2. Name the project org.xtext.example.mydsl.releng
  3. Create pom.xml in the root directory of the releng project (as per above) with contents as follows:

 

<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.xtext.example.mydsl</groupId>
  <artifactId>parent</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <modules>
    <module>../org.xtext.example.mydsl</module>
    <module>../org.xtext.example.mydsl.ui</module>
    <module>../org.xtext.example.mydsl.tests</module>
    <module>../org.xtext.example.mydsl.sdk</module>
    <module>../org.xtext.example.mydsl.updatesite</module>
  </modules>

  <properties>
    <tycho-version>0.15.0</tycho-version>
    <!-- this is only required if you use the eclipselabs unittesting plugin; remove otherwise -->
    <eclipselabs-site>http://xtext-utils.eclipselabs.org.codespot.com/git.distribution/releases/unittesting-0.9.x</eclipselabs-site>
  </properties>

  <repositories>
    <repository>
      <id>juno</id>
      <layout>p2</layout>
      <url>http://download.eclipse.org/releases/juno</url>
    </repository>

    <!-- this is only required if you use the eclipselabs unittesting plugin; remove otherwise -->
    <repository>
      <id>eclipselabs</id>
      <layout>p2</layout>
      <url>${eclipselabs-site}</url>
    </repository>
  </repositories>

  <build>
    <plugins>
                <plugin>
                  <groupId>org.eclipse.tycho</groupId>
                  <artifactId>tycho-maven-plugin</artifactId>
                  <version>${tycho-version}</version>
                  <extensions>true</extensions>
                </plugin>

                <plugin>
                  <groupId>org.eclipse.tycho</groupId>
                  <artifactId>tycho-p2-director-plugin</artifactId>
                  <version>${tycho-version}</version>
                  <executions>
                    <execution>
                      <id>materialize-products</id>
                      <goals>
                        <goal>materialize-products</goal>
                      </goals>
                    </execution>
                    <execution>
                      <id>archive-products</id>
                      <goals>
                        <goal>archive-products</goal>
                      </goals>
                    </execution>
                  </executions>
                </plugin>
    </plugins>
  </build>
</project>

Running the build

That’s everything finally set up, time to run the build. Open a terminal, change to the releng project directory then run the build:

$ cd projects/xtext/org.xtext.example.mydsl.releng
$ mvn clean install

The output is pretty verbose. And it may take some time on first run since maven has to download the dependencies. Eventually though you should see something like:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] parent ............................................ SUCCESS [0.187s]
[INFO] MyDSL core language services ...................... SUCCESS [3.499s]
[INFO] MyDSL core language User Interface ................ SUCCESS [1.597s]
[INFO] MyDSL Example tests ............................... SUCCESS [8.606s]
[INFO] MyDSL Example Feature ............................. SUCCESS [0.329s]
[INFO] MyDSL Example Update Site ......................... SUCCESS [8.189s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 54.083s
[INFO] Finished at: Fri Oct 12 22:30:44 BST 2012
[INFO] Final Memory: 56M/117M
[INFO] ------------------------------------------------------------------------

If you look further back in the build output you’ll see status updates for each of the build items. For example, the test output should look like this:

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.xtext.example.mydsl.tests.MyDslTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.029 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] All tests passed!

That’s it - build done! The generated update site is now located in org.xtext.example.mydsl.updatesite/target/repository.

Other maven/tycho targets

Maven supports various other targets, a couple of the more useful being:

$ mvn integration-test                 //build & run tests
$ mvn -o clean install                 //don't download dependencies, use local cache

Test the Result

You should check that the generated plugin can be installed into a separate eclipse instance. To do that:

  1. Download & install a fresh eclipse distribution. The Modeling Tools version is a good place to start since it already includes the core EMF infrastructure.
  2. Start up the new eclipse instance (create/select a workspace as required)
  3. Select Help->Install New Software...
  4. Click Add to add a new software site
  5. Click the Local button and select the org.xtext.example.mydsl.updatesite/target/repository/ directory, click Open then OK
  6. Your feature should now be named in the panel below, e.g. MyDsl SDK Feature under the category MyDsl Example Category. Select the feature and follow the wizard to install.

Summary

Maven and Tycho perhaps aren’t the most user-friendly tools. The documentation could be better, the POM structure is a little verbose and the build output can be somewhat intimidating. However, with relatively little work, they enable command line building and testing of an xtext language update site. The result can be used to automate builds using a Continuous Integration Server (e.g. Jenkins) if required.

Troubleshooting

What’s presented above is a distillation of my journey. I certainly don’t claim any expertise in maven or tycho, so can’t offer much advice over and above what’s here. However the Tycho user community is a friendly and helpful place so is a good first port of call.

Acknowledgements

Getting this working would have been considerably more difficult without the help of the Tycho user community and Lars Vogel’s excellent Tycho tutorial. Brian de Alwis also has a really nice overview of the concepts and background here. Much appreciation all round!

Comments & Suggestions

Greatfully received...

[1]as Brian de Alwis says, “to understand the Maven documentation, you need to understand Maven”.
[2]a POM in Maven is analogous to a Makefile or Ant build Script.

Comments

  • avatar

    Udo

    Posted 11 months, 11 days ago.

    The title of this tutorial is misleading. You describe the building of Eclipse plug-ins with Maven/Tycho.

    You do not describe the automatic generating and building of Xtext plug-ins. The generation step from the DSL grammar to source code in your example is manually.

  • avatar

    Site Administrator

    Posted 10 months, 30 days ago.

    Udo, thanks for the comment. You're right: it doesn't cover the initial step; getting to this point took a lot more effort than I expected. However there's a nice summary here of how to automate the initial step with just a few extra entries in the POMs created above: http://fsteeg.com/2011/07/15/run-mwe2-workflows-for-xtext-2-0-in-a-tycho-build/. Hope that helps.

  • avatar

    Michael

    Posted 3 months, 25 days ago.

    You can also use this Maven Archetype to create an Xtext project with a multi module Maven layout and Tycho (manifest-first approach):

    https://github.com/fuinorg/emt-xtext-archetype

  • avatar

    Christian

    Posted 17 days, 1 hour ago.

    Hi, I'm trying to let Tycho Surefire use the instrumented classes from Cobertura -> I want to add them to the Tycho Surefire classpath. Do you know wheather it's possible or how? I also aked on stackoverflow but with no answer yet: https://stackoverflow.com/questions/22541368/run-cobertura-on-xtext-project-together-with-maven-and-tycho-surefire
    Thanks for your nice work!
    Christian