Hello, Habr! I bring to your attention a translation of a wonderful article from a series of articles by the
notorious Jake Worton about how Android 8 is supported by Java.
The original article is hereI worked from home for several years, and I often heard my colleagues complain about Android supporting different versions of Java.
This is a rather complicated topic. First you need to decide what we mean by “Java support in Android”, because in one version of the language there can be a lot of things: features (lambdas, for example), bytecode, tools, APIs, JVM and so on.
When they talk about Java 8 support in Android, they usually mean support for language features. So, let's start with them.
Lambdas
One of the main innovations of Java 8 was lambdas.
The code has become more concise and simple, lambdas have saved us from writing cumbersome anonymous classes using an interface with a single method inside.
class Java8 { interface Logger { void log(String s); } public static void main(String... args) { sayHi(s -> System.out.println(s)); } private static void sayHi(Logger logger) { logger.log("Hello!"); } }
After compiling this using javac and legacy
dx tool , we get the following error:
$ javac *.java $ ls Java8.java Java8.class Java8$Logger.class $ $ANDROID_HOME/build-tools/28.0.2/dx --dex --output . *.class Uncaught translation error: com.android.dx.cf.code.SimException: ERROR in Java8.main:([Ljava/lang/String;)V: invalid opcode ba - invokedynamic requires --min-sdk-version >= 26 (currently 13) 1 error; aborting
This error occurs due to the fact that lambdas use the new instruction in the bytecode -
invokedynamic , which was added in Java 7. From the error text, you can see that Android only supports it starting with the 26 API (Android 8).
It doesn’t sound very good, because hardly anyone will release an application with 26 minApi. To get around this, the so-called
desugaring process is
used , which makes lambda support possible on all versions of the API.
History of Desaccharization
She's pretty colorful in the Android world. The goal of desaccharization is always the same - to allow new language features to work on all devices.
Initially, for example, to support lambdas in Android, developers connected the
Retrolambda plugin. He used the same built-in mechanism as the JVM, converting lambdas to classes, but he did it in runtime, and not at compile time. The generated classes were very expensive in terms of the number of methods, but over time, after refinements and improvements, this indicator dropped to something more or less reasonable.
Then, the Android team
announced a new compiler that supported all Java 8 features and was more productive. It was built on top of the Eclipse Java compiler, but instead of generating a Java bytecode, it generated a Dalvik bytecode. However, its performance is still poor.
When the new compiler (fortunately) was abandoned, the Java bytecode transformer in the Java bytecode, which did the juggling,
was integrated into the Android Gradle Plugin from
Bazel , Google’s build system. And its performance was still low, so in parallel the search continued for a better solution.
And now we were
dexer -
D8 , which was supposed to replace the
dx tool . Desaccharization was now performed during the conversion of compiled JAR files to
.dex (dexing). The D8 is much better in performance compared to
dx , and since Android Gradle Plugin 3.1 it has become the default dexer.
D8
Now, using D8, we can compile the code above.
$ java -jar d8.jar \ --lib $ANDROID_HOME/platforms/android-28/android.jar \ --release \ --output . \ *.class $ ls Java8.java Java8.class Java8$Logger.class classes.dex
To see how the D8 converted lambda, you can use the
dexdump tool , which is included in the Android SDK. It will display quite a lot of everything, but we will focus only on this:
$ $ANDROID_HOME/build-tools/28.0.2/dexdump -d classes.dex [0002d8] Java8.main:([Ljava/lang/String;)V 0000: sget-object v0, LJava8$1;.INSTANCE:LJava8$1; 0002: invoke-static {v0}, LJava8;.sayHi:(LJava8$Logger;)V 0005: return-void [0002a8] Java8.sayHi:(LJava8$Logger;)V 0000: const-string v0, "Hello" 0002: invoke-interface {v1, v0}, LJava8$Logger;.log:(Ljava/lang/String;)V 0005: return-void …
If you haven’t read the bytecode yet, don’t worry: a lot of what is written here can be understood intuitively.
In the first block, our
main method with index
0000 gets a reference from the
INSTANCE field to the
INSTANCE class
Java8$1 . This class was generated during
. The main method bytecode also does not mention the body of our lambda anywhere, so most likely it is associated with the
Java8$1 class. Index
0002 then calls the
sayHi static method using the link to
INSTANCE . The
sayHi requires
Java8$Logger , so it seems
Java8$1 implements this interface. We can verify this here:
Class #2 - Class descriptor : 'LJava8$1;' Access flags : 0x1011 (PUBLIC FINAL SYNTHETIC) Superclass : 'Ljava/lang/Object;' Interfaces - #0 : 'LJava8$Logger;'
The
SYNTHETIC flag means that the
Java8$1 class
Java8$1 been generated and the list of interfaces that it includes contains the
Java8$Logger .
This class represents our lambda. If you look at the implementation of the
log method, you will not see the body of the lambda.
… [00026c] Java8$1.log:(Ljava/lang/String;)V 0000: invoke-static {v1}, LJava8;.lambda$main$0:(Ljava/lang/String;)V 0003: return-void …
Instead, the
static method of the
Java8 class is
Java8 -
lambda$main$0 . Again, this method is presented only in bytecode.
… #1 : (in LJava8;) name : 'lambda$main$0' type : '(Ljava/lang/String;)V' access : 0x1008 (STATIC SYNTHETIC) [0002a0] Java8.lambda$main$0:(Ljava/lang/String;)V 0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream; 0002: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V 0005: return-void
The
SYNTHETIC flag again tells us that this method was generated, and its bytecode just contains the lambda body: a call to
System.out.println . The reason the lambda body is inside
Java8.class is simple - it may need to access
private members of the class, which the generated class will not have access to.
Everything you need to understand how
desaccharization works is described above. However, looking at it in the Dalvik bytecode, you can see that everything is much more complicated and frightening there.
Source Transformation
To better understand how
desaccharization occurs, let's try step by step to convert our class into something that will work on all versions of the API.
Take the same class with lambda as a basis:
class Java8 { interface Logger { void log(String s); } public static void main(String... args) { sayHi(s -> System.out.println(s)); } private static void sayHi(Logger logger) { logger.log("Hello!"); } }
First, the lambda body is moved to the
package private method.
public static void main(String... args) { - sayHi(s -> System.out.println(s)); + sayHi(s -> lambda$main$0(s)); } + + static void lambda$main$0(String s) { + System.out.println(s); + }
Then a class is implemented that implements the
Logger interface, inside which a block of code from the lambda body is executed.
public static void main(String... args) { - sayHi(s -> lambda$main$0(s)); + sayHi(new Java8$1()); } @@ } + +class Java8$1 implements Java8.Logger { + @Override public void log(String s) { + Java8.lambda$main$0(s); + } +}
Next, a singleton instance of
Java8$1 , which is stored in the
static variable
INSTANCE .
public static void main(String... args) { - sayHi(new Java8$1()); + sayHi(Java8$1.INSTANCE); } @@ class Java8$1 implements Java8.Logger { + static final Java8$1 INSTANCE = new Java8$1(); + @Override public void log(String s) {
Here is the final
dubbed class that can be used on all versions of the API:
class Java8 { interface Logger { void log(String s); } public static void main(String... args) { sayHi(Java8$1.INSTANCE); } static void lambda$main$0(String s) { System.out.println(s); } private static void sayHi(Logger logger) { logger.log("Hello!"); } } class Java8$1 implements Java8.Logger { static final Java8$1 INSTANCE = new Java8$1(); @Override public void log(String s) { Java8.lambda$main$0(s); } }
If you look at the generated class in the Dalvik bytecode, you will not find names like Java8 $ 1 - there will be something like
-$$Lambda$Java8$QkyWJ8jlAksLjYziID4cZLvHwoY . The reason why such naming is generated for the class, and what are its advantages, draws to a separate article.
Native lambda support
When we used the
dx tool to compile a class containing lambdas, an error message said that this would only work with 26 APIs.
$ $ANDROID_HOME/build-tools/28.0.2/dx --dex --output . *.class Uncaught translation error: com.android.dx.cf.code.SimException: ERROR in Java8.main:([Ljava/lang/String;)V: invalid opcode ba - invokedynamic requires --min-sdk-version >= 26 (currently 13) 1 error; aborting
Therefore, it seems logical that if we try to compile this with the
—min-api 26 flag, then desaccharization will not occur.
$ java -jar d8.jar \ --lib $ANDROID_HOME/platforms/android-28/android.jar \ --release \ --min-api 26 \ --output . \ *.class
However, if we dump the
.dex file, then it can still be found in it
-$$Lambda$Java8$QkyWJ8jlAksLjYziID4cZLvHwoY . Why is that? Is this a D8 bug?
To answer this question, and also why desaccharization
always occurs , we need to look inside the Java bytecode of the
Java8 class.
$ javap -v Java8.class class Java8 { public static void main(java.lang.String...); Code: 0: invokedynamic
Inside the
main method, we again see
invokedynamic at index
0 . The second argument in the call is
0 - the index of the
bootstrap method associated with it.
Here is a list of
bootstrap methods:
… BootstrapMethods: 0: #27 invokestatic java/lang/invoke/LambdaMetafactory.metafactory:( Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String; Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType; Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;) Ljava/lang/invoke/CallSite; Method arguments: #28 (Ljava/lang/String;)V #29 invokestatic Java8.lambda$main$0:(Ljava/lang/String;)V #28 (Ljava/lang/String;)V
Here the
bootstrap method is called
metafactory in the
java.lang.invoke.LambdaMetafactory class. He
lives in the JDK and creates anonymous on-the-fly classes in runtime for lambdas, just like D8 generates them in compute time.
If you look at the
Android java.lang.invokeor to the
AOSP java.lang.invoke , we see that this class is not in the runtime. That's why de-juggling always happens at compile time, no matter what minApi you have. The VM supports bytecode instructions similar to
invokedynamic , but the
invokedynamic built-in to the JDK
LambdaMetafactory not available for use.
Method references
Along with lambdas, Java 8 added method references - this is an effective way to create a lambda whose body references an existing method.
Our
Logger interface is just such an example. The lambda body referred to
System.out.println . Let's turn the lambda into a reference method:
public static void main(String... args) { - sayHi(s -> System.out.println(s)); + sayHi(System.out::println); }
When we compile it and take a look at the bytecode, we will see one difference with the previous version:
[000268] -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM.log:(Ljava/lang/String;)V 0000: iget-object v0, v1, L-$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM;.f$0:Ljava/io/PrintStream; 0002: invoke-virtual {v0, v2}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V 0005: return-void
Instead of calling the generated
Java8.lambda$main$0 , which contains a call to
System.out.println , now
System.out.println is called directly.
A class with a lambda is no longer a
static singleton, but by the
0000 index in the bytecode, we see that we get a link to
PrintStream -
System.out , which is then used to call
println on it.
As a result, our class turned into this:
public static void main(String... args) { - sayHi(System.out::println); + sayHi(new -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM(System.out)); } @@ } + +class -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM implements Java8.Logger { + private final PrintStream ps; + + -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM(PrintStream ps) { + this.ps = ps; + } + + @Override public void log(String s) { + ps.println(s); + } +}
Default and static methods in interfaces
Another important and major change that Java 8 brought was the ability to declare
default and
static methods in interfaces.
interface Logger { void log(String s); default void log(String tag, String s) { log(tag + ": " + s); } static Logger systemOut() { return System.out::println; } }
All this is also supported by D8. Using the same tools as before, it is easy to see a logged-in version of Logger with
default and
static methods. One of the differences with lambdas and
method references is that the default and static methods are implemented in the Android VM and, starting with the 24 API, D8 will not
decouple them.
Maybe just use Kotlin?
Reading the article, most of you probably thought about Kotlin. Yes, it supports all Java 8 features, but they are implemented by
kotlinc in the same way as D8, with the exception of some details.
Therefore, Android support for new versions of Java is still very important, even if your project is 100% written in Kotlin.
It is possible that in the future Kotlin will cease to support Java 6 and Java 7 bytecode.
IntelliJ IDEA , Gradle 5.0 switched to Java 8. The number of platforms running on older JVMs is decreasing.
Desugaring APIs
All this time I talked about Java 8 features, but did not say anything about the new APIs - streams,
CompletableFuture , date / time and so on.
Returning to the Logger example, we can use the new date / time API to find out when messages were sent.
import java.time.*; class Java8 { interface Logger { void log(LocalDateTime time, String s); } public static void main(String... args) { sayHi((time, s) -> System.out.println(time + " " + s)); } private static void sayHi(Logger logger) { logger.log(LocalDateTime.now(), "Hello!"); } }
Compile it again with
javac and convert it to the Dalvik bytecode with D8, which
decouples it for support on all versions of the API.
$ javac *.java $ java -jar d8.jar \ --lib $ANDROID_HOME/platforms/android-28/android.jar \ --release \ --output . \ *.class
You can even run this on your device to make sure that it works.
$ adb push classes.dex /sdcard classes.dex: 1 file pushed. 0.5 MB/s (1620 bytes in 0.003s) $ adb shell dalvikvm -cp /sdcard/classes.dex Java8 2018-11-19T21:38:23.761 Hello
If API 26 and above is on this device, the Hello message will appear. If not, we will see the following:
java.lang.NoClassDefFoundError: Failed resolution of: Ljava/time/LocalDateTime; at Java8.sayHi(Java8.java:13) at Java8.main(Java8.java:9)
D8 dealt with lambdas, the reference method, but did nothing to work with
LocalDateTime , and this is very sad.
Developers have to use their own implementations or wrappers on date / time api, or use libraries like
ThreeTenBP to work with time, but why can't you do D8 with your hands?
Epilogue
Lack of support for all the new Java 8 APIs remains a big problem in the Android ecosystem. Indeed, it is unlikely that each of us can allow us to specify the 26 min API in our project. Libraries supporting both Android and JVM cannot afford to use the API introduced to us 5 years ago!
And even though Java 8 support is now part of D8, every developer should still explicitly specify source and target compatibility in Java 8. If you write your own libraries, you can strengthen this trend by laying out libraries that use Java 8 bytecode (even if you are not using new language features).
A lot of work is being done on D8, so it seems that everything will be ok in the future with support for language features. Even if you write only on Kotlin, it is very important to force the Android development team to support all new versions of Java, improve bytecode and new APIs.
This post is a written version of my talk
Digging into D8 and R8 .