Bootstrapping Kotlin from scratch
Hi all,
I've attempted a different approach for bootstrapping Kotlin and I'd
like to share with you the result. For the reminder, Kotlin is extremely
difficult to build from sources because it is written itself in Kotlin
and its build system is based on Gradle which also depends on Kotlin,
that's the perfect scenario for a serious headache.
Instead of starting with a relatively recent release of Kotlin and a
binary bootstrap compiler and standard library that are subsequently
rebuilt, I tried to start further back in time with a release that
doesn't depend on Kotlin and then build up a path toward the recent
releases.
TL;DR it didn't work, but I'd like to document the process if someone
plans to follow this path in the future.
The journey starts with locating the latest version of the Kotlin
compiler that doesn't have .kt source files, and that's the version
0.6.786 (tagged as build-0.6.786). It goes back to October 2013. At this
time Ant was the build system (the Ant build was dropped in the version
build-1.2.30-dev-715 tagged in January 2018, so Gradle isn't an
immediate concern). Java 6 was the version targeted by the built but I
assume Java 7 (released in 2011) was used. For this bootstrapping
process I'm using Java 8, which is the lowest version available in
Debian unstable and the baseline for the current ant package.
Kotlin 0.6.786 requires the IntelliJ SDK 132.27, the issue is the
ancient SDK builds are not tagged in the Git repository. There are only
branches for the stable releases (131, 133, 135...). Fortunately, the
code on the branch 131 is good enough to build Kotlin 0.6.786. The
branch 133 is fine too with a minor tweak (the class StubBuildingVisitor
requires an extra constructor to preserve the backward compatibility
with Kotlin < 0.6.1203). The SDK 133 can be used to build Kotlin up to
the version 0.6.2122.
A side note on the SDK: its dependencies are checked into the lib
directory of the repository as binary jar files, and some of them were
modified. We can replace them with Debian libraries, except one, ASM4,
which was relocated to the org.jetbrains.asm4 package and got a few
undocumented changes. Theoretically the lib/src directory contains the
source archives for these modifications, but for ASM the sources don't
compile and some modifications are missing (the jar was modified several
times but the source archive was never updated). Without these
modifications the Kotlin build fails with Java 8. After some
investigation, it appears that the required modification is a change to
the ClassReader class that ensures that parsing unsupported class file
formats doesn't trigger an exception. This is a modification that's is
still carried today by the Jetbrains variant of ASM
(https://github.com/JetBrains/intellij-deps-asm/commit/db1b691).
So Kotlin 0.6.786 + SDK 133 is able to build Kotlin up to the version
0.6.895. It fails on the version 0.6.900 due to a new annotation
(jet.KotlinTraitImpl) added to Kotlin 0.6.855 and now used by the
compiler. I opted to add the annotation to Kotlin 0.6.786 and continue.
It works up to Kotlin 0.6.1364.
With Kotlin 0.6.1379 the build fails due to the "externalannotations"
attribute added to the <withKotlin> Ant task in Kotlin 0.6.1130 and
first used in this version. So Kotlin 0.6.1364 now replaces 0.6.786 as
the bootstrap compiler.
Kotlin 0.6.1364 + SDK 133 builds Kotlin up to the version 0.6.1475 (note
that 0.6.1379 fails due to the use of closure-compiler for the Kotlin to
JavaScript compiler, I've simply disabled this part of the build). The
version 0.6.1485 failed with an error I couldn't explain:
[javac2] ERROR:
/home/ebourg/kotlin/compiler/frontend/src/org/jetbrains/jet/lang/psi/psiUtil/jetPsiUtil.kt:
(84, 58) Type mismatch: inferred type is (???) -> jet.Boolean but
com.google.common.base.Predicate<org.jetbrains.jet.lang.psi.JetElement>
was expected
[javac2] ERROR:
/home/ebourg/kotlin/compiler/frontend/src/org/jetbrains/jet/lang/psi/psiUtil/jetPsiUtil.kt:
(84, 60) Cannot infer a type for this parameter. To specify it
explicitly use the {(p : Type) => ...} notation
It looks like the problematic function
JetPsiUtil.getOutermostLastBlockElement() is neither used by the
compiler nor the standard library, so I've removed it.
The next failure is with Kotlin 0.6.1507 due to the new
org.jetbrains.annotations.ReadOnly annotation which was moved to the
runtime but used by the compiler before the runtime is built. Copying
the annotation into the compiler source solves the issue.
The next stop is Kotlin 0.6.1799 where the compilation breaks due to a
bytecode verification error (java.lang.VerifyError: Bad type on operand
stack). I suspect the bytecode verification got stricter in Java 8 and
the bytecode generated by Kotlin back then is now considered invalid. I
worked around the issue by adding the -noverify JVM argument (a trick
shared by the Kotlin CLI script, so I assume it's ok).
In Kotlin 0.6.1862 the Java to Kotlin converter (j2kConverter) fails to
build, I'm not sure why but since it is optional I disabled it too.
In Kotlin 0.6.1910 the following error appears:
[javac2] ERROR:
/home/ebourg/kotlin-from-scratch/kotlin.upstream/compiler/frontend/src/org/jetbrains/jet/lang/evaluate/OperationsMapGenerated.kt:
(104, 64) Unresolved reference: compareTo
I worked around this by commenting the offending line.
The process continues up to Kotlin 0.6.2052 and this is the end of the
road for Kotlin 0.6.1364 as the bootstrap compiler. The next version
0.6.2063 has unresolved references in
compiler/builtins-serializer/src/run.kt. Kotlin 0.6.2052 becomes the new
bootstrap compiler and we continue.
Kotlin 0.6.2052 + SDK 133 builds Kotlin up to the version 0.6.2107. We
switch again the bootstrap compiler.
Kotlin 0.6.2107 + SDK 133 works up to Kotlin 0.6.2338.
And now start the real troubles with Kotlin 0.6.2350. It breaks because
this is the first version using the SDK 134, but there is no tag nor
branch for the SDK 134. The 135 branch doesn't work due to the switch to
ASM 5 (and Kotlin didn't switch to ASM 5 until the version 0.7.391).
I've located where the version of the SDK changed from 134 and 135
(commit 4bb7296c2747fa73898247da03e7c9923935cc61) and I've built the SDK
just before this point. The Kotlin build fails with this error:
runtime:
[echo] Cleaning
/home/ebourg/kotlin-from-scratch/kotlin.upstream/dist/classes/runtime
[mkdir] Created dir:
/home/ebourg/kotlin-from-scratch/kotlin.upstream/dist/classes/runtime
[java] exec() finished with INTERNAL_ERROR return codeEXCEPTION:
java.lang.IllegalArgumentException: Missing extension point:
com.intellij.psi.classFileDecompiler in area null
[java] at
com.intellij.openapi.extensions.impl.ExtensionsAreaImpl.getExtensionPoint(ExtensionsAreaImpl.java:373)
[java] at
com.intellij.openapi.extensions.impl.ExtensionsAreaImpl.getExtensionPoint(ExtensionsAreaImpl.java:37)
[java] at
com.intellij.openapi.extensions.Extensions.getExtensions(Extensions.java:110)
[java] at
com.intellij.openapi.extensions.Extensions.getExtensions(Extensions.java:98)
[java] at
com.intellij.openapi.extensions.ExtensionPointName.getExtensions(ExtensionPointName.java:48)
[java] at
com.intellij.psi.compiled.ClassFileDecompilers.find(ClassFileDecompilers.java:88)
[java] at
com.intellij.psi.ClassFileViewProviderFactory.createFileViewProvider(ClassFileViewProviderFactory.java:33)
[java] at
com.intellij.psi.impl.file.impl.FileManagerImpl.createFileViewProvider(FileManagerImpl.java:202)
[java] at
com.intellij.psi.impl.file.impl.FileManagerImpl.findViewProvider(FileManagerImpl.java:160)
[java] at
com.intellij.psi.impl.file.impl.FileManagerImpl.findFile(FileManagerImpl.java:328)
[java] at
com.intellij.psi.impl.PsiManagerImpl.findFile(PsiManagerImpl.java:188)
I have no idea how to fix this.
In a last attempt I tried to build Kotlin 0.7.391 with Kotlin 0.6.2338 +
SDK 135 but it failed.
This is where the story halts.
To summarize:
Step 1: build Kotlin 0.6.786 with a patched SDK 133
Step 2: build Kotlin 0.6.1364
Step 3: build Kotlin 0.6.2052
Step 4: build Kotlin 0.6.2107 (good up to Kotlin 0.6.2338).
Step 5: how to build Kotlin 0.6.2350 ???
If someone figures out the next step, or has any hint about the last
error, please ping me :)
Emmanuel Bourg
Reply to: