Got error when running scala-cli in a directory with .scala-build with previous scala-compiler version

[dragove]

Version(s)
Scala CLI version: 1.11.0
Scala version (default): 3.7.4

Describe the bug
got error when running scala-cli

[E] Scala instance jar /home/dove/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.7.1/scala3-compiler_3-3.7.1.jar doesn't exist!

To Reproduce

mkdir re && cd re
printf "//> using jvm 21\n//> using scala 3.7.1" > project.scala
echo 'println("hi")' > demo.sc
scala-cli .
# after running finished
jps | grep BloopServer | cut -d " " -f 1 | xargs kill
sed -i 's/3.7.1/3.7.4/' project.scala
rm ~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.7.1/scala3-compiler_3-3.7.1.jar
scala-cli .

Expected behaviour

There should not report an error since the compiler version has already changed.

Additional Information

This problem can not be reproduced without kill the bloop server.

[SethTisue]

Thanks for the finding the reproduction — I see this a lot, and I'd been meaning to report it.

I've never been sure under what conditions it happens.

[Gedochao]

So, a couple things.

This is caused by the Coursier cache being in an inconsistent state (as in, broken) - and not by fault of the software, but the user.
If you remove ~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.7.1/scala3-compiler_3-3.7.1.jar, but not ~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.7.1/scala3-compiler_3-3.7.1.pom, then Coursier gets confused and throws and error (b/c smh the POM is there, but the JAR isn't, and that's assumed to never be the case, so an error is thrown).

Note that removing the POM fixes this reproduction.

And so:

• I'd advise to always remove entire directories from the coursier cache, rather than individual files, otherwise this sort of error is expected. You mess with the app's internal files, you might break the app if not careful.
• for the broken cache to be actually accepted as a valid state and handled by Coursier, that's something that could be raised upstream in https://github.com/coursier/coursier/issues
• all that being said, Scala CLI shouldn't require 3.7.1 here, so investigation is needed for the CLI to recover from this on its own (or perhaps not call for it in the first place... suspecting Bloop could be involved, I need to look into it some more)
• we could try to catch this error from Coursier in the CLI and show a meaningful message about the cache being in a broken state, so that the user can go and fix it

[Gedochao]

Slightly simpler reproduction:

• start with repro.scala (no .sc script or multiple files necessary):

//> using scala 3.7.1
@main def main = println("Hello, World!")

• and then:

# no need to run, just compile
scala-cli compile repro.scala 
# does the same as jps | grep BloopServer | cut -d " " -f 1 | xargs kill
scala-cli bloop exit --power 
sed -i 's/3.7.1/3.7.4/' repro.scala # or just make the change in an editor
# this is platform dependent, as for where Coursier's cache is located, as per https://get-coursier.io/docs/cache.html#location
rm ~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.7.1/scala3-compiler_3-3.7.1.jar 
# re-compiling triggers the error, but still succeeds
scala-cli compile repro.scala
# [E] Scala instance jar ~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.7.1/scala3-compiler_3-3.7.1.jar doesn't exist!
# Compiling project (Scala 3.7.4, JVM (21))
# Compiled project (Scala 3.7.4, JVM (21))

[Gedochao]

Note: while the error is being printed, the compilation then proceeds and succeeds, eventually.
So it's really about a failing check for something we shouldn't even be checking at this point.
The cause is that Bloop checks all existing builds before running something (and it doesn't know what to run before).
While we could clean the past build with different Scala version from the CLI, identifying it isn't simple.
There is an easy workaround in removing the POM (or just generally not breaking your Coursier cache manually).
And even without the workaround, all you get is an annoying log about a non-issue.
Treating it as low priority, contributions are welcome.

[SethTisue]

You mess with the app's internal files, you might break the app if not careful.

But I promise I'm not messing with the cache myself...!

If something is messing with it, I really don't know what that thing might be, I'm afraid.

[Gedochao]

You mess with the app's internal files, you might break the app if not careful.

But I promise I'm not messing with the cache myself...!

If something is messing with it, I really don't know what that thing might be, I'm afraid.

Then there might be another reproduction of this somewhere in the wild 🤔

[SethTisue]

I saw this again today:

[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/compiler-interface/1.10.7/compiler-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/compiler-interface/1.10.7/compiler-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/compiler-interface/1.10.7/compiler-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/compiler-interface/1.10.7/compiler-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/compiler-interface/1.10.7/compiler-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/util-interface/1.10.7/util-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/util-interface/1.10.7/util-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/util-interface/1.10.7/util-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/util-interface/1.10.7/util-interface-1.10.7.jar doesn't exist!
[E] Scala instance jar /opt/homebrew/Cellar/scala/3.7.4/libexec/maven2/org/scala-sbt/util-interface/1.10.7/util-interface-1.10.7.jar doesn't exist!

and I'm almost certain I know what the exact circumstances are: it was the first time I ran a scala shebang script after brew upgrade scala from 3.7.4 to 3.8.1

the script in question has //> using scala 3.8.0, so it doesn't seem to matter what the script's Scala version is, the key thing is simply that /opt/homebrew/Cellar/scala/3.7.4 ceased to exist because I upgraded using Homebrew

I'm almost entirely certain didn't do anything else that might have messed with any cache or anything. since upgrading I know I used the scala command a number of times to fire up a REPL; I just hadn't used to run a shebang script yet, I don't think

running the same script a second time does not cause the errors to be printed a second time

[SethTisue]

And again on my other laptop. Even more sure now that it's simply as I described.

[Gedochao]

Interesting.
So the likely cause is that the cache gets corrupted by Homebrew, during the upgrade.
I'll see if I can reproduce it later.