Add generator for Eclipse JDT project files

[thahnen]

Summary

This pull requests provides an additional generator for the Mill Build Tool that is targeted towards working with the Eclipse IDE. There is currently no direct support for the Mill Build Tool inside the Eclipse IDE nor any maintained plug-in for Scala (as the language the build files are written in) or Kotlin, and therefore this generator is only targeting Java projects (for now?).

This references #3372 to enable users of the Eclipse IDE to generate projects based on the Java Development Tools (JDT). The goal is to have a working generator, aggregating the Java Mill Modules (and the corresponding Test Modules) into logical Eclipse projects as good as possible.
This does not provide support for the Mill build files itself (or support for Scala) since it relies on the Build Server Protocol (BSP) for which no supporting plug-in yet exists in the Eclipse IDE.

In-depth information

In order to generate Eclipse project files, this generator searches for all Java (Test) Modules inside the build/project structure and tries to aggregate them together. This is contrary to the existing IntelliJ IDEA Project generator that results in one big project - Eclipse has one workspace with multiple projects created based on the modules.

The aggregation follow the approach of having a module with production code and its direct corresponding test code that will become one Eclipse project. If there is no connection between production and test code found by the generate, a separate Eclipse project is created.
Dependencies between modules are correctly translated to inter-project dependencies, source folders are correctly resolved (and marked whether they have production or test code in them) and other external dependencies (e.g. libraries from Maven or local ones) added to the Eclipse project.

To make the generator more user-friendly when invoked from the command line, logs are provided showing all the steps as well as every Eclipse project created with its name and location - so it is easier later for users to locate them and import them into the workspace.
A synthetic non-Java Eclipse project is created for the root Mill build file (build.mill) in case no module was located in this folder. This way users can also access the build script using a project inside the Eclipse workspace if they like despite the fact that the Eclipse IDE, as mentioned above, provides no active support for the build tool, the build scripts and/or the Scala programming language. The file will be handled like a plain text file for now.

As this generator is not 100% failsafe for every corner case, these generated files should be usable and if not, then they can be adjusted from inside the Eclipse IDE - they are not the holy grail and users might want to tweak them as they wish anyway.
This also relates to logic from the Mill build that is not translated, like launcher configurations for applications or tests, these need to be configured manually from inside the Eclipse IDE (these configurations rely on separate files not linked to the ones generated by this generator).

Testing

The generator was tested against complying example Java projects already present inside this repository. Additionally, integration tests were added for the generator to cover the supported (corner) cases and to point out the not supported cases (e.g. Scala / Kotlin projects). This way upcoming breaking changes to the API could be found as early as possible.

Production testing

Additionally to automatic tests running on the CI and manual tests against example projects and simple, created ones, this was also tested with a non-trivial open source project. I choose swaldman/c3p0 due to the fact that its module structure and build file are a bit more complex with the following caveats:

• generated sources outside of a module directory (will become a linked resource inside Eclipse)
• test modules relying on Junit 5 (tests should start from within the Eclipse IDE)
• arguments for the test modules (that won't be picked up by the generator intentionally)
• source folders shared between modules (src-proxy-interface)

Test plan

1. Checkout this branch on the latest commit locally
2. Download the repository on disk
3. From this local repository, run the following command

./mill dist.run <path to swaldman/c3p0 repository> mill.eclipse/

4. Validate the output is correct and similar to the one in the screenshot
5. Open Eclipse IDE for Java Developers with a new workspace, import the projects as shown below (should be 4)

6. After importing the project, await the projects to be build automatically by the IDE (happens in the background) and then check that there is no problem shown, both in the Package Explorer and Problems view:

7. To make sure that everything works as expected, we run one of the Junit tests in the project c3p0 at the location test/src -> com.mchange.v2.c3p0.test.junit -> MarshallUnmarshallDataSourcesJUnitTestCase.java -> testRefDerefRoundTrip() via the context menu

8. The JUnit view should open somewhere in the workbench and show that the test was successful, the Console view might also come to the foreground showing output. I have no idea what this tests tho. Not all tests run successfully from the IDE due to configurations for the test arguments in the build.mill - see this limitation above!

9. One last check: Making sure that re-generating the Eclipse project files work after applying changes to the Mill build script. For that, modify this line to another Java version, e.g. 17 and run the command again

./mill dist.run <path to swaldman/c3p0 repository> mill.eclipse/

10. We have to refresh the projects inside the Eclipse workspace, as seen in the screenshots below. After that, they should show that they now rely on Java 17:

This should conclude the testing on a non-trivial open source project relying on the Mill Build Tool successfully!

Further improvements

There might be further improvements to this generator that are out of the scope of this pull requests. These include:

• Creating a rudimentary Eclipse plug-in providing support for importing Mill builds (by calling the generator in the back) and supporting at least syntax highlighting to the build files, maybe context menu options to "re-generate" the files etc.
• Finetuning the Eclipse project Java source / target versions by also looking at the JavaModuleApi#javacOptions or something similar
• creating project-based filters to hide nested projects inside the IDE in the Project Explorer or Package Explorer (this is not that trivial sadly)

I'm open to contribute any of these if possible and time (and life) permits 😄

[thahnen]

@lihaoyi may I ask for a first round of review from you?

This is currently in good shape. I will add integration tests and documentation in the next couple of days.

I have one question, tho: Since this should only be enabled for Java Modules, I was thinking about implementing something similar to this for IntelliJ. But I'm not sure how to plug it in so that it is disabled for Scala / Kotlin, etc.
For now, it will not fail when it tries to generate Eclipse JDT Project files for content other than Java. Either this could be solved by implementing something similar to IntelliJ but more restrictive, or adding this to the docs.

I'm also not sure, but for now I would discard THIS failure. Of course this is a new API and I guess it fails therefore.

[lihaoyi]

For documentation, a section under Installation and IDE Setup / IDE Support for Eclipse with instructions for CLI usage and screenshots of how to import the project would work. I assume this implementation won't provide assistance in build.mill files since those are written in Scala, which eclipse doesn't support, so we should mention that in the docs that it only works on JavaModules and .java source files

For automated testing, an integration test in the same style as ./mill integration.ide[gen-idea].packaged.daemon' should suffice. It would run mill mill.eclipse/ on a few example JavaModules and make sure their source folders, internal dependencies, and external dependencies are all wired up correctly in the generated files.

The PR description should also include screenshots of manual testing, with you using mill.eclipse/ on some non-trivial open-source Java project, loading it into eclipse, and showing that navigation etc. works. I'll run through that workflow myself before merging just as a sanity check.

Of course, you should also test this on the project you want to use it on to make sure it satisfies your own requirements

[thahnen]

@lihaoyi the user facing documentation I added alongside screenshots, I will add integration tests next.

For now I relied mostly on the examples when manually testing, can I copy them (in the IT build, not on disk) into the integration tests and re-use them there?

[lihaoyi]

@thahnen for integration tests, you should try and minimize a small example project to use in your test. Maybe example/javalib/basic/3-multi-module and example/javalib/basic/4-compat-modules so we exercise both projects with the inter-module dependencies as well as projects with the MavenModule source layout

[lihaoyi]

I tried this out, seems to work!

@thahnen one thing I noticed was that the build.mill file is not visible within Eclipse at all. Is there some way to make eclipse display it in the project explorer, even as an un-highlighted text file? I imagine being able to edit the build.mill config within an IDE would be a common ask from users. Maybe we could generate a synthetic project at the root of the repository that contains that file?

[lihaoyi]

Also, I noticed that source files get duplicated in the project view: once under the source root, and once under the generic filesystem tree. Is that expected for how the Eclipse UI works? Or is there some way to avoid the duplication?

[thahnen]

Hi @lihaoyi,

regarding the point of duplication: This is expected since this is due to the Eclipse UI for that view (Project Explorer).
It is possible to filter out these duplicated files by configuring it manually or to use the Java-specific "Package Explorer" view that does not duplicate the resources. It can happen that it shows the folders twice, I'm not sure what is the reason for this then happening once or not at all.

The duplication is due to the fact that that in the Project Explorer the directory is once kept as a source set (that is why there is this little icon above the folder) and once as a lame folder. I think this is a useless duplication but hey, who am I to judge ^^
In case you just have a src folder and not the nested Maven structure, it is not shown twice in most cases as Eclipse seems to be smart enough in that case to not show the source set again as a separate folder.

Regarding the point of having the projects either link to the build.mill file with a linked resource or having a separate synthetic project for it, I think the latter one makes more sense with the following cases:

• in case there will be a Eclipse project created in that folder anyway (e.g. for the integration/ide/gen-eclipse/resources/simple-java-project, don't create one.
• in case there will be no project created in that folder, create a synthetic one with only a simplified .project file - as it won't contain Java sources most of it does not need to be created

[lihaoyi]

Your approach to include the build.mill sounds reasonable

[thahnen]

Hey @lihaoyi,

I'm done. Feel free to validate the PR and provide one last round of feedback!
The instructions in the PR description should be sufficient I hope.

Best,
Tobias

[lihaoyi]

@thahnen I have merged this and tagged 1.0.2-23-d3f2ba for you to try out