Use @snippet javadoc tag for snippets

[jpountz]

Description

Now that main requires Java 21, we can start using the @snippet javadoc tag, which is quite more convenient to use than the <pre class="prettyprint"></pre> HTML tags we are currently using.

One problem I noticed is that gradlew tidy messes up with formatting when @snippet tags are used.

[dweiss]

google/google-java-format#789
google/google-java-format#886

Unfortunately, I think these make this issue a no-go to unless you want to resign from using google java formatter... which I don't think we should do.

[rmuir]

Related: if we bump main to java 23+ (#14229) then we could do snippets in javadoc with the typical markdown mechanisms.

[rmuir]

The markdown would have a huge advantage for the many analyzers with xml-style prettyprint docs today. I don't think @snippet works with languages other than java.

For me, editor displays the current "prettyprint" of analyzers XML docs without any syntax highlighting:

On the other hand, with markdown you'd get this:

<fieldType name="text_arnormal" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.ArabicNormalizationFilterFactory"/>
  </analyzer>
</fieldType>

[rmuir]

@dweiss what do you think about exploring the option to swap out spotless java.GoogleJavaFormatStep with java.EclipseJdtFormatterStep but using google-java format config: https://github.com/google/styleguide/blob/gh-pages/eclipse-java-google-style.xml

I know the history that google says nobody else's formatter is good enough, they say you need "their" formatter for the google style and that others eclipse/intellij can't do it right, but their formatter can't support @snippet nor markdown comments (it will mess them up), so they fail technically, they don't keep up with java standard.

[rmuir]

The Palantir one is no better off from what I can tell, it is a fork of the google one with some cosmetic tweaks. I know eclipse JDT doesn't choke on these constructs, although I don't have much experience with their formatter, their compiler has been pretty good and they are generally responsive about new java features.

[dweiss]

I have been using google's tool mostly because it's been fairly solid in terms of formatting between releases (so no surprise changes on updates). I also like (or have slowly grown accustomed to) what they do with the indents and more complex code blocks. There are quirks, sure, but they'll be present with any tool.

The beauty of having a single standard should be that the tool used to format the code is irrelevant as long, as it ends up with the same output. If we can make eclipse do the same thing (or even better, format snippets reliably) then it's a transparent change for the better - I don't have a problem with that at all.

[dweiss]

It's close but there are many differences. See the commit above and try running it on one of the modules, for example suggest (./gradlew -p lucene/suggest spotlessApply).

Maybe it can be made closer, I haven't tried.

[rmuir]

@dweiss thank you so much for that starter commit for evaluation. I will try it tonight and fire up eclipse and see what our options are.

I finally finished parser (https://github.com/rmuir/tree-sitter-javadoc) and am now looking at options to (ab)use it, to convert our docs to markdown automatically, when I discovered that google-java-format will mess up markdown comments, e.g. too-long /// will wrap to another line with // and break everything.

So currently, there is no way to escape from the prettier because neither @snippet nor markdown can work with the formatter, so no way to make a patch that passes build.

[dweiss]

Nice!

So... google java format has this option, at least in the cmd line version:

If nothing else works, we could just make a multipass and format javadocs using a different tool than
the rest of the code... Or fork gjf and implement proper javadoc formatting, which should be a fun project to work on.

[rmuir]

I think in the markdown case, the bug I saw was that it didn't treat /// as javadoc but as an ordinary inline comment. But I can experiment with the option still.

[dweiss]

google/google-java-format#1193

Disabling Javadoc formatting doesn't prevent either issue.

So it seems it's broken entirely. Argh.

[rmuir]

I played with this a bit and reduced noise in two ways:

Original file:

113 files changed, 3656 insertions(+), 5216 deletions(-)

1. Disable reformatting of Apache License header:

<setting id="org.eclipse.jdt.core.formatter.comment.format_header" value="false"/>

93 files changed, 2525 insertions(+), 3861 deletions(-)

2. Disable reformatting of javadocs comments
   We could fight it, but my goal is to move them all to markdown format which google doesn't support anyway. Most differences are in things such as indentation of html tags.
   So the differences are just not relevant and create noise when doing comparison:

<setting id="org.eclipse.jdt.core.formatter.comment.format_javadoc_comments" value="false"/>

75 files changed, 1922 insertions(+), 3380 deletions(-)

Eclipse has a lot of options and I think this file is just out of date. With the unnecessary noise out of the way, it is a matter of exploring all the settings and trying to reduce the differences more. I'm guessing there would always be differences, but it would be nice to minimize them.

[rmuir]

For the record those diffstats were based on ./gradlew -p lucene/suggest spotlessApply and include the changes of the patch/formatter XML itself