A Concurrent Affair » xajavac

ANTLRWorks and OpenJDK Compiler Grammar

Mathias — Tue, 18 Nov 2008 05:19:18 +0000

Now that my qualifying exam is behind me, I’ve got more time to focus on other things. I have to admit that the exam pretty much occupied all of my thoughts and hogged lots of time, to the degree that I didn’t prepare as well as I used to for my COMP 202 class. I think that has changed again now.

I’m also starting to look at the Java multi-stage programming project again, and I discovered that the OpenJDK people have added a Compiler Grammar project. This project is trying to replace the hand-coded parser for javac by a parser generated from an ANTLR grammar file. I haven’t fully checked how far this has progressed, but it is interesting. Modifying the grammar to allow annotations on statements and expressions should be easier than rewriting the interpreter by hand. It seems like there is a Java 5 grammar that integrates into javac, but of course there is no JSR 308 grammar, so I’d have to develop that first.

When I looked at ANTLR and the Compiler Grammar project, I also noticed an interesting IDE that has been built to make developing grammars easier: ANTLRWorks. It integrates a several checking tools and a debugger and looks tremendously helpful. For example, it can explain how a grammar is ambiguous. In my case, an annotation on a parenthetical expression could be parsed in two different ways, so I’ll have to be careful there.

ANTLR Ambiguous Path Visualization

On MacOS, I’m still struggling a bit to get the Compiler Grammar project and ANTLRWorks running. It seems like Compiler Grammar needs SoyLatte to work, and I have managed to build the Compiler Grammar javac from the command line, but when I try to run it inside ANTLRWorks, then it seems like it is using the default Apple Java version 1.6.0-dp, which doesn’t work. Maybe I’ll take a look at it on Windows first.

I also need to start thinking about the COMP 202 final exam, and about a lecture schedule and programming tasks for COMP 402 next semester.

Java Multi-Methods Using Proxies and Annotations

Mathias — Sat, 04 Oct 2008 02:00:04 +0000

After working a little bit with Java proxy classes for my yet incomplete support library for xajavac, which introduces subtyping to annotations, I started to think about what else I could do with them.

Proxies are essentially a wrapper around another object. Instead of calling a method of the wrapped object, a single invoke method is called and provided the Method object of the method that was to be called, along with the parameters:

1
2
3
4
5
6

public class MyProxy implements java.lang.reflect.InvocationHandler {
public Object invoke(Object proxy, Method m, Object[] args) throws Throwable {
// ...
return result;
}
}

Inside this method, the programmer can now do string comparisons on the method name, for example, and dispatch different methods, or perform some kind of pre- and post-processing. I used this, together with a very simple @MultiMethod marker annotation, to create multi-methods in Java.

Java normally only performs static dispatch and single dispatch. In static dispatch, the class of the method is decided at compile time already; in single (dynamic) dispatch, the class containing the method depends on the run-time type of the object. Static dispatch applies to static methods. Single dispatch, on the other hand, works with non-static methods and is the foundation of polymorphism. To demonstrate single dispatch, let’s consider the classic example of polymorphism:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

interface IAnimal {
public void speak();
}

class Cat implements IAnimal {
public void speak() { System.out.println("Meow"); }
}

class Dog implements IAnimal {
public void speak() { System.out.println("Woof"); }
}

// ...
IAnimal a = new Cat();
a.speak(); // prints "Meow"
a = new Dog();
a.speak(); // prints "Woof"

At the call site, we just have an IAnimal, and we call its speak() method. Depending on whether the run-time type is Cat or Dog, the appropriate method is called. The dispatch is based on the receiver, object whose method is called.

However, the dispatch only depends on the receiver; the run-time types of the arguments do not matter. For example, if we create a feed(IFood food) method in IAnimal and then provide a specialized feed(Catnip food) method in the Cat class, the general Cat.feed(IFood food) method will be called even when the argument is of type Catnip. It is single dispatch, not multiple dispatch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

interface IFood { }
public Catnip implements IFood { }
public PetFood implements IFood { }

interface IAnimal {
public void feed(IFood food);
}

class Cat implements IAnimal {
public void feed(IFood food) { System.out.println("Num num"); }
public void feed(Catnip food) { System.out.println("Meow, my favorite!"); }
}

class Dog implements IAnimal {
public void feed(IFood food) { System.out.println("Gobble gobble"); }
}

// ...
IAnimal a = new Cat();
a.feed(new PetFood()); // prints "Num num"
a.feed(new Catnip()); // also prints "Num num", not "Meow, my favorite!"
a = new Dog();
a.feed(new PetFood); // prints "Gobble gobble"

If Java were to call the most appropriate method, where the class is determined by the receiver run-time type, and the actual overloaded method by the best match of the argument run-time types, then Java would be performing multiple dispatch. This, however, is not part of the Java Language Specification.

Using proxies, however, it is possible to intercept the call to the feed(IFood food) methods, look through the overloaded methods and compare their parameter types to the provided argument run-time types, and then call the method that is the closest to the run-time types, i.e. where all argument types are supertypes of the argument run-time types, and where the types are most closely related (defined as the number of edges separating the nodes in the type hierarchy).

To get a behavior like the one above, Java programmers (and programmers of most languages) usually employ the Visitor design pattern. Multi-methods (methods with multiple dispatch) make this unnecessary. Here is a more useful example of an abstract syntax tree (AST) consisting of integer and variable leaves and addition and multiplication interior nodes. First the code using the Visitor pattern to achieve double dispatch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

interface IAST {
public Object execute(IASTVisitor v);
}

class Num implements IAST {
private int _num;

public Num(int n) {
_num = n;
}

public int getNum() { return _num; }
public String execute(IASTVisitor v) { return v.numCase(this); }
}

class Var implements IAST {
private String _var;

public Var(String v) {
_var = v;
}

public String getVar() { return _var; }
public String execute(IASTVisitor v) { return v.varCase(this); }
}

class Add implements IAST {
private IAST _left;
private IAST _right;

public Add(IAST l, IAST r) {
_left = l;
_right = r;
}

public IAST getLeft() { return _left; }
public IAST getRight() { return _right; }
public String execute(IASTVisitor v) { return v.addCase(this); }
}

class Mul implements IAST {
private IAST _left;
private IAST _right;

public Mul(IAST l, IAST r) {
_left = l;
_right = r;
}

public IAST getLeft() { return _left; }
public IAST getRight() { return _right; }
public String execute(IASTVisitor v) { return v.mulCase(this); }
}

interface IASTVisitor {
public String numCase(Num ast);
public String varCase(Var ast);
public String addCase(Add ast);
public String mulCase(Mul ast);
}

class ToStringVisitor implements IASTVisitor {
public String numCase(Num ast) {
return String.valueOf(ast.getNum());
}
public String varCase(Var ast) {
return ast.getVar();
}
public String addCase(Add ast) {
return "("+ast.getLeft().execute(this)+" + "+ast.getRight().execute(this)+")";
}
public String apply(Mul ast) {
return "("+ast.getLeft().execute(this)+" * "+ast.getRight().execute(this)+")";
}
}

//...
IAST a = new Mul(new Add(new Var("x"), new Num(5)), new Num(-2));
a.execute(new ToStringVisitor());

The interesting point here is that each subclass of IAST has to have an execute(IASTVisitor v) method that accepts the visitor. The method implementations in the subclasses then call the corresponding case method in the IASTVisitor. When we execute the visitor on the AST (the host), we achieve double dispatch because there are two instances of single dispatch, once depending on the run-time type of the host, a, and again inside the execute method depending on the run-time type of the visitor, v.

With multi-methods, we can now remove the execute methods in the IAST hierarchy. We now just have an overloaded apply method in IASTVisitor where the type of the parameter changes and also determines which method gets invoked:

interface IAST { }

class Num implements IAST {
private int _num;

public Num(int n) {
_num = n;
}

public int getNum() { return _num; }
}

class Var implements IAST {
private String _var;

public Var(String v) {
_var = v;
}

public String getVar() { return _var; }
}

class Add implements IAST {
private IAST _left;
private IAST _right;

public Add(IAST l, IAST r) {
_left = l;
_right = r;
}

public IAST getLeft() { return _left; }
public IAST getRight() { return _right; }
}

class Mul implements IAST {
private IAST _left;
private IAST _right;

public Mul(IAST l, IAST r) {
_left = l;
_right = r;
}

public IAST getLeft() { return _left; }
public IAST getRight() { return _right; }
}

@MultiMethod
public interface IASTVisitor {
public String apply(IAST ast);
public String apply(Num ast);
public String apply(Var ast);
public String apply(Add ast);
public String apply(Mul ast);
}

class ToStringVisitor implements IASTVisitor {
public static IASTVisitor create() {
IASTVisitor v = new ToStringVisitor();
v.proxy = MMProxy.newInstance(v);
return v.proxy;
}
private IASTVisitor proxy;

public String apply(IAST ast) {
throw new AssertionError("Should never happen.");
}
public String apply(Num ast) {
return String.valueOf(ast.getNum());
}
public String apply(Var ast) {
return ast.getVar();
}
public String apply(Add ast) {
return "("+proxy.apply(ast.getLeft())+" + "+proxy.apply(ast.getRight())+")";
}
public String apply(Mul ast) {
return "("+proxy.apply(ast.getLeft())+" * "+proxy.apply(ast.getRight())+")";
}
}

//...
IAST a = new Mul(new Add(new Var("x"), new Num(5)), new Num(-2));
ToStringVisitor().create().apply(a);

The @MultiMethod marker annotation states that all methods in the IASTVisitor interface and all its subclasses should use multiple dispatch. To minimize overhead, this annotation can also be placed just in front of individual methods, then only those methods will use multiple dispatch.

There needs to be a method that takes the supertype of the AST classes, IAST, even though that method should never be called, because there are no objects of that type (unless you create anonymous inner classes, or add another subclass, e.g. Sub, for which there is no overloaded method). We therefore throw an exception there. Then there are the method that handle the concrete subclasses. This time, they all have the same name and overload the general method above that accepts the supertype.

Since we need to create the proxy that wraps around the object with the multi-methods, i.e. around the visitor here, we cannot just create the visitor using the new keyword. Instead, we have a static create() method that creates a new object and creates the proxy around it, and then returns it typed as interface. This is important! It must be returned as an IASTVisitor, not as a ToStringVisitor, because as it turns out, the proxies can only simulate interfaces, not classes. The created proxy is an IASTVisitor, but it is not a ToStringVisitor, even though it wraps around one.

Another very important difference is apparent in the methods for the composites Add and Mul, where recursion is necessary. Using the visitor, we could recurse into the subtrees using ast.getLeft().execute(this), where this was the visitor itself. Since we want the multiple dispatch behavior in the recursion too, we need to use the proxy object for the recursion, not the visitor itself. That means we cannot use this, because this is the visitor, not the proxy. Instead, we use the proxy field, where the create() method stored the created proxy. This is really inelegant and can cause numerous problems: If we accidentally use this, we lose the multiple dispatch; we could also make a mistake and overwrite the proxy field, with null for example. But I haven’t found a better way.

Finally, there is a huge limitation: As stated, proxies can only simulate interfaces, not classes. That means that the classes around which we wrap a proxy can only implement interfaces, but not extend other classes. You can never use the extends keyword! That doesn’t for good code reuse. For example, even though pretty much every visitor needs the proxy field, I cannot hoist it into an abstract class. The following example will cause a ClassCastException:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

@MultiMethod
public interface IASTVisitor {
public String apply(IAST ast);
public String apply(Num ast);
public String apply(Var ast);
public String apply(Add ast);
public String apply(Mul ast);
}

abstract class AASTVisitor implements IASTVisitor {
protected IASTVisitor proxy;

public String apply(IAST ast) {
throw new AssertionError("Should never happen.");
}
}

class ToStringVisitor extends AASTVisitor {
public static IASTVisitor create() {
IASTVisitor v = new ToStringVisitor();
v.proxy = MMProxy.newInstance(v);
return v.proxy;
}

public String apply(Num ast) {
return String.valueOf(ast.getNum());
}
public String apply(Var ast) {
return ast.getVar();
}
public String apply(Add ast) {
return "("+proxy.apply(ast.getLeft())+" + "+proxy.apply(ast.getRight())+")";
}
public String apply(Mul ast) {
return "("+proxy.apply(ast.getLeft())+" * "+proxy.apply(ast.getRight())+")";
}
}

That’s what we would ideally want to do, but we can’t. This is really a huge impediment, but right now, without heavier machinery such as a bytecode-rewriting class loader, I can’t find a way around it. Proxies are promising, but not as powerful as I would like them to be.

I’m nonetheless making the source code for Java multi-methods using proxies and annotations available, even though it’s definitely a work in progress.

And Another Week Gone

Mathias — Wed, 24 Sep 2008 14:45:44 +0000

The last week was still spent catching up from Ike. I got lucky, I got power back early, but until pretty much today, the area of Houston that I see on my bike commute to work had been without power. There are still lots of uprooted trees that need to be cleared, and the trash bags are piled high along the streets. Traffic lights seem to be working a bit better, and grocery stores are stocked again.

I was supposed to give a presentation right after Ike, but that day Rice was closed, and not only I was affected, so my talk got pushed back by a week. I finally gave it this Monday, on September 22, in the RAP seminar. It was titled Java Annotations for Invariant Specification and was about Java annotations in general, how I used them to specify and check program invariants, and how much easier that got once I had extended Java to support subtyping for annotations as well. The talk was generally well received, but many of the comments said I should have given a clear definition of what annotations are at the beginning, not interspersed throughout the lecture. I didn’t want to make it so syntax-heavy at the beginning, but I guess I was wrong. I also ran out of time towards the end.

After that, I spent the last two days configuring the JSR 308 prototype compiler on my Mac and my PC at home. It’s not as easy as it sounds, but I finally got it to work, except for the tests. Yesterday I also transferred the changes from xajavac that allow subtyping for annotations into the JSR 308 compiler. But that’s still a work in progress.

Is It Safe?

Mathias — Fri, 08 Aug 2008 17:42:17 +0000

For the last couple of days I have been contemplating buying another fire-proof security file. I currently have two Sentry KS-4100 files.

I keep a lot of records: Some are important, like identification documents for three countries, my last will and testament, lease agreements, insurance policies, deeds of ownership, computer backups, and, yes, even tax records. Others hopefully aren’t so important: Receipts for many of the items I have purchased. My credit card has a nice benefit that extends the manufacturer’s warranty by up to a year, and to claim that benefit, I need the receipt. Also, in case of a disaster that destroys my property, my renters insurance will want to examine the receipts so I can be reimbursed for the purchase or replacement costs, not just what the lost items were worth at the time.

Unfortunately, I’m running out of room for these receipts. While looking at buying another KS-4100, I more closely inspected its UL-72 certification. This particular Sentry file is manufactured to keep the interior at or below 350°F (177°C) for a fire that rages at 1550°F (855°C) for half an hour, which is apparently typical for an apartment fire.

As we all know, paper burns at 451°F (233°C) and it singes a bit sooner, at about 405°F (207°C), so keeping the interior of the file at or below 350°F keeps paper safe. Sentry also states that “CDs, DVDs, USB drives and memory sticks” are protected.

CDs and DVDs are mostly made of polycarbonate, with a melting point 513°F (267°C); jewel cases are mostly polystyrene that melts at 464°F (240°C), so CDs, DVDs and their cases are protected. I didn’t check the material of memory sticks and USB drives, although I’m a little worried about my portable hard drives now.

However, I also keep credit cards and ID cards in there, and they are usually made of polyvinyl chloride (PVC)¹^,², and I organize the receipts in nice colorful transparent polypropylene (PP) draw-string envelopes. PVC products, and therefore nearly all ID and credit cards, melt at a mere 176°F (80°F), and my PP envelopes melt at 338°F (170°F), just below the temperature the safe could reach while working as certified.

I’m really experiencing mixed up feelings right now, and I am considering switching to paper envelopes for organization, but the PP envelopes are so much more resilient and practical, and the temperature difference from 338°F to 350°F isn’t that great. That’s perhaps a risk I’m willing to take.

With IDs and credit cards, however, that’s a completely different matter. I’m fairly certain that in an apartment fire, the inside of the safe could easily reach 176°F and melt my passports, my international driver’s licenses and ID cards, and my credit cards (oh shucks, I’ll just get new ones, Washington Mutual willing, but that’s another rant for another hour).

I could get a data-storage security file like the Sentry 6720, which keeps the interior at or below 125°F (52°C). Apparently that’s good enough for floppy disks, and assumeably for hard drives as well, but unless I’m mistaken, it’s still too hot for credit and ID cards! And more importantly, it retails more than ten times the cost of the KS-4100.

How do protect your documents? Am I paranoid? Am I a “filing rat”, and you simply don’t have documents to protect? Please send me your ideas, if you like.

Footnotes:

I have since been corrected that credit cards are usually produced from polyvinyl chloride acetate, or PVCA, but I haven’t found the melting point of that yet. [back]
Apparently, the properties of PVCA can vary a lot, but one low figure that was provided to me was 257°F (125°C), which is still too low to be protected in the KS-4100 security file; PVCA cards would probably be safe in the expensive Sentry 6720 media file mentioned below. [back]

JSR 308 Animosity

Mathias — Wed, 14 May 2008 07:47:04 +0000

I’ve come across two blog posts that leave me perplexed: First there was “When Should You Jump? JSR 308. That’s When.”, and now I just found “What Hath Java Wrought”.

I don’t know anything about the authors and I have only read the two posts, but statements like “Every language has a complexity budget. Java blew through it with generics in Java 5.” and “First we got Generics. We all know how that went. While it’s passably better than what was originally there, it’s still not worth a damn.” make me furrow my brow. As for generics, I gladly trade in the more complicated syntax for better static typing. Annotation processors can do the same for non-null values, mutability, problems with equality and identity. They can bring linear types into the language, and make pretty much every kind of program analysis I can think of easier.

I don’t understand why there is such much animosity towards JSR 308. It is much less intrusive than generics were; unless you plan to use raw types, the updated collections libraries virtually forced you to adopt generics to a certain degree. That’s not the case at all with annotations. If you like you can easily ignore the entire idea.

Now, perhaps I’m just not a good programmer, and that’s why I prefer to ride with training wheels like generics, more precise type systems and better warnings, but this kind of humility has never hurt me so far. It does not make sense to be so frugal on the syntax side; the semantic benefits by far outweigh the costs.

I’m definitely not a fan of how generics were implemented using erasure, and I believe there are shortcomings both in the annotations provided with Java 1.5 and JSR-308 (see LAPT-javac, xajavac and blog posts), but I’m a firm believer that programmers have benefited a lot from generics, and that they will benefit in a similar way from annotations.

The beautiful thing about annotations is that dealing with them works on two different difficulty levels: Implementing annotations is not perfectly straight-forward, they use a syntax that is a bit different than the one used for regular Java classes, and that can be a bit confusing. Using annotations that have already been implemented, however, is easy; it’s an absolute piece of cake. This means that the annotations framework, especially with the annotation processors in JSR 308, can be used by more seasoned programmers to provide a lot of value even to programmers who do not know all of the details.

As a programming language researcher, I’m excited about annotations because they allow me to extend and modify the Java language in a relatively safe, convenient and compatible way. The resulting experimental languages may not be the most elegant ones, but they do the trick. Considering the two different levels of difficulty, and the possibility to completely ignore annotations, I have no idea why programmers would dislike annotations so much that they would abandon a language that can still be used exactly the same as without annotations.

Semi-Weekly Log from 2/18/08 to 2/21/08

Mathias — Fri, 22 Feb 2008 04:04:01 +0000

Again, another (semi-)weekly log from the DrJava blog. I’m really trying to crank out the basic design and most of the features quickly, and this is my major focus right now, because that way I can hopefully focus on a SIGCSE paper I plan to write (deadline: March 19) and the Java extension that allows subtyping for annotation. But here we go:

Changed keyboard shortcut for Debug mode to MASK + D, instead of just D.

Added DrJavaProperties and PropertyMaps instead of a Map of Properties. These DrJavaProperties can be lazy and delay computationally intensive tasks like that of finding all compiled class files until actually necessary. In the “Preview”, they may display stale values.

Two basic categories existed before, “Java” with the System.getProperties, and “Config” with the values of the DrJava configuration. The “Config” category was previously named “DrJava”, but I thought “Config” was a better name. All the entries in “Config” start with “config.”. The entries in “Java” have the names as specified by Sun.

A new category has been added, “DrJava”, with just a few sample variables. One of them is “drjava.current.time.millis”, which contains the current time in milliseconds since the beginning of the epoche — useless, but a good first test.

“drjava.current.file” contains the currently open document as an absolute path.

“drjava.all.files” contains all open files, separated by File.pathSeparator. “drjava.project.files”, “drjava.included.files” and “drjava.external.files” do the same for project files, external files that are saved with the project, and files that are open but not part of the project.

The same “drjava.*.files” properties exist as “drjava.*.files.spaces”, but here, a space is used to separate the files.

Properties can listen to each other to find out when their values have been invalidated. There is a debug mechanism in place that prevents infinite loops.

These are only some very basic sample properties, and we need lots more, and we need to make sure that we invalidate the values of the properties in all the right places, but it’s a good start.

On a Linux system, for example, try opening a project and then run an external process with the command line:

tar cfvz test.tar.gz ${drjava.all.files.spaces}

It will create a tar file with all the source files currently open.

Not bad, eh? Please help me figure out what other properties are needed, and how we can easily make them relative to other directories, etc.

This is a massive commit, with far-reaching changes. Please update your working copy of DrJava to the newest revision and test the changes, the new features, and also the old ones to make sure I haven’t broken anything.

Thanks!

M    src/edu/rice/cs/drjava/config/OptionConstants.java
A    src/edu/rice/cs/drjava/config/DrJavaProperty.java
A    src/edu/rice/cs/drjava/config/JavaSystemProperty.java
A    src/edu/rice/cs/drjava/config/PropertyMaps.java
A    src/edu/rice/cs/drjava/config/ConfigProperty.java
A    src/edu/rice/cs/drjava/config/EagerProperty.java
M    src/edu/rice/cs/drjava/ui/MainFrame.java
M    src/edu/rice/cs/drjava/ui/InsertVariableDialog.java
M    src/edu/rice/cs/drjava/ui/ExecuteExternalDialog.java
M    src/edu/rice/cs/util/JVMProcessCreator.java
M    src/edu/rice/cs/util/StringOps.java
M    src/edu/rice/cs/util/StringOpsTest.java
M    src/edu/rice/cs/util/ProcessCreator.java

Still Working on Subtyped Annotation API

Mathias — Sun, 23 Dec 2007 19:59:54 +0000

Just when I thought I had a complete working API for my annotations with subtyping, and after I had written a special version of my invariant checker that uses annotations with subtyping, I ran into some trouble with more complicated annotations that contain other annotations. For example:

1
2
3
4
5
6

@Not(@And({@Or({@ThreadWithName("foo"),
@ThreadWithName("bar")}),
@And({@EventThread,@SynchronizedThis})}))
public void testNotAndOrTwoAndTwo() {
System.out.println("testNotAndOrTwoAndTwo!");
}

Internally, the JVM instantiates these annotation classes using proxies, just like I do. These proxies are created at runtime and therefore, of course, do not have any associated class files. My API was still trying to find them and failing. I think I’ve got most of those issues fixed now, but the holidays have taken over, and I was also busy with DrJava for a few days.

I still need to write a few more tests. I’m already very happy with the way these invariant annotations look like. The annotation expressed above would probably have been several hundred lines of code before, just because there was no general way to perform Boolean AND and OR operations on any kind of invariant annotation. They pretty much had to be custom-written.

Now that I have been working a bit with proxies, I remembered the idea about the @DelegatesTo annotation that I had a while back. I think proxies would make the implementation of that really simple. I still have to generate the interface, and the type checker of the Java compiler is still a problem, as explained in the original posting, but proxies would probably make all bytecode rewriting unnecessary. I could do it all using proxies and reflection. Again, I think it’s worth investigating.

@Inherited for Invariant Annotations?

Mathias — Mon, 03 Dec 2007 21:15:00 +0000

When I wrote the API code for xajavac, my Java compiler variant that allows subtyping for annotations, I discovered the @Inherited annotation for the first time. This annotation is used on another annotation and indicates that the other annotation should be inherited from superclasses to subclasses. Specifically, annotations with @Inherited also appear in the array of annotations returned by Class.getAnnotations() if they annotated any of the superclasses of the class in question; in contrast, the array returned by Class.getDeclaredAnnotations only returns the annotations that are actually annotation the class itself.

When I wrote the implementation of the invariant checker outlined in my Master’s thesis, I implemented and described an inheritance mechanism for these invariant annotations that didn’t rely on the @Inherited annotation. I introduced separates meta-annotation, @PredicateLink and @Combine, which I needed anyway and which also incorporated the meaning of @Inherited.

Now that I have discovered @Inherited, I’m wondering whether I could or even should use it. That would mean making non-inheritance the default behavior, and letting @Inherited imply the inheriting case. I don’t think that’s the behavior I want. A @NotInherited meta-annotation would be better.

Small Things

Mathias — Thu, 29 Nov 2007 01:12:31 +0000

In the last few days, I’ve worked on a few small things: I have created a compiler adapter for Ant that allows me to use my xajavac compiler — once you have annotations with subtyping it’s hard to let go of them again. On Monday, I made a bugfix for the remote control feature of DrJava, and today I tried to address another issue that came up in a bug report.

I’ve also started on a version of the invariant checker that uses annotations with subtyping, but that hasn’t progressed too far yet; I first had to solve the Ant problem.

Multiple Annotations of the Same Type

Mathias — Thu, 22 Nov 2007 01:34:06 +0000

Since I already made one small modification today, I also looked at allowing multiple annotations of the same type on an annotated element. Again, it was a very simple change in the compiler.

Both the modified compiler and the extended annotation framework now allow you to repeat the same annotation multiple times. Unfortunately, if the original reflection API is used, Java throws a java.lang.annotation.AnnotationFormatError: Duplicate annotation for class: interface TestMulti$MultiAnnotation: @TestMulti$MultiAnnotation(value=bar) exception (or similar), so make sure to use the new API in subannot.jar if you use this feature.

Since I couldn’t rely on Java’s reflection API anymore (because of the exception), I had to rework my framework a bit. The most complicated change involved the distinction between getAnnotations() and getDeclaredAnnotations(). I now have to inherit annotations to subclasses myself.

All that is done now, though. I still need to make a more thorough write-up and announce xajavac on several discussion boards. I don’t know when I’ll do that, though; I’ve had an incredibly productive day, but I’ve been working for 12 hours straight already. On the other hand, I’m rained in at the office and everyone else has left for Thanksgiving…

Final Modifier for Annotations with Subtyping

Mathias — Wed, 21 Nov 2007 19:46:33 +0000

The JSR 308 proposal mentioned that there might be trust issues for annotations that can be subclassed outside a trusted framework. I contend that this is not a severe problem since it occurs with normal classes already.

To still address this issue, I decided to allow the final modifier for annotations: An annotation that is final cannot be subclassed, just like a final class cannot be extended.

1
2
3
4
5
6
7
8

final @interface FinalAnnotation {
String value();
}

// ERROR: cannot inherit from final FinalAnnotation
@interface SubAnnotation extends FinalAnnotation {
int i();
}

Again, the change involved only half a dozen lines.

Referenced in JSR 308 Proposal

Mathias — Mon, 19 Nov 2007 18:53:34 +0000

I just noticed that the Related Work section of the JSR 308 proposal references my LAPT-javac modification for local variable annotations.

At last! I was a little bit disappointed by the original feedback I received for LAPT-javac.

Added Formal Grammar Changes and API Description to xajavac Website

Mathias — Mon, 19 Nov 2007 18:07:57 +0000

I finally finished modifying the Java grammar to faithfully represent my changes that enable subtyping for annotations. I also added a description of the extended API I had to develop to allow reflection for annotations with subtyping. I have also decided to remove the treatment for local variable annotations, originally implemented in LAPT-javac, from xajavac now that the JSR 308 proposal provides a different format.

All of this information, along with the binaries for the modified compiler and the API, is available on the xajavac website.

Now I should write a more formal tech report about this, I should contact Michael Ernst of the JSR 308: Annotations on Types project, and I should modify the code I wrote to for my MS thesis to take advantage of annotations with subtyping.

JSR 308 Proposal Available

Mathias — Tue, 13 Nov 2007 23:39:07 +0000

The proposal for JSR 308, “Annotations on Java Types”, is available now. I’ve read it diagonally once, and it looks pretty nice. They’re finally adding class and runtime retention for local variables, something my LAPT-javac already did in October 2006. xajavac also incorporates my own non-standard representation for local variable annotations. Now that a standard representation is emerging, I’ll either have to change my implementation to make it adhere to the JSR 308 proposal, or I’ll have to drop it. That’s not a surprise, though; when I wrote LAPT-javac, I fully expected that it would become obsolete soon.

Subtyping for annotations is also mentioned, but to my delight only under “Out-of-scope issues”, a.k.a. suggestions. So at least in the near future, xajavac seems to be the only project to incorporate annotations with subtyping into Java.

Work Website Redesign and Miscellaneous To Do

Mathias — Tue, 13 Nov 2007 07:32:34 +0000

I just finished redesigning my work website. It’s now in the same style as the DrJava and the Concutest websites. I liked the other design, but for lengthy text documents, the new design is just more practical.

While all tests of annotations with subtyping that I have tried so far have succeeded, I have realized that I need to test all my programs, not only the new subannot software, with primitive types. It’s embarrassing, but until a couple of days ago, I didn’t know that int.class existed. I think now I have to put a switch statement in front of every line that uses a type name so I can filter out 'I's and 'J's, etc. and replace them properly with int.class and long.class.

And I need to start writing that tech report about annotations with subtyping.

Subtyping for Array Annotations Now Working

Mathias — Mon, 12 Nov 2007 22:47:19 +0000

I’m creating more and more tests for annotations with subtyping, and while most of it seems to work, I ran into one very important problem: Right now, my arrays do not support polymorphism yet. Consider the following:

1
2
3
4
5
6
7
8
9
10
11
12

@interface TopLevel { }
@interface Leaf extends TopLevel {
String value();
}
@interface Interior extends TopLevel {
TopLevel[] value();
}

@Interior({@Leaf("foo"),
@Interior({@Leaf("bar"),@Leaf("fee")}),
@Leaf("fum")})
class Something { ... }

I definitely want polymorphic arrays. In the example above, the @Interior declares an array of @TopLevel interfaces, but since @Interior and @Leaf are both subclasses of @TopLevel, I should be able to store them in the array as well.

The type checker and compilers don’t have a problem with this, and everything gets encoded correctly in the class file, but my runtime library has a bug: It creates an array of type TopLevel, which is correct, but then also imposes the type TopLevel for all its members! That means all members get “sliced” down to type TopLevel, so they don’t have any of their additional members present.

I have to use the type from the array declaration and use it to create the array. Then I have to use the type of each annotation for the annotation instance. Unfortunately, this information isn’t conveniently present. But I’ll work it out.

Update

I just fixed the problem with subtyping in arrays as described above.

Automatic Delegation Would Be Nice

Mathias — Sat, 10 Nov 2007 02:34:02 +0000

Last night I finished the core of the extended Class implementation supporting annotations with subtyping. I load default values, but I punted on the order of the members in the toString() method. As far as I know, it is implementation-specific anyway.

Today I decided not to go to the office, but instead work from home. My workstation here has a better setup for coding, and I felt I was at the office long enough yesterday and that my presence wasn’t necessary for grading anymore, because as far as I know the missing grad TA had returned.

So I spent most of today writing extended versions of the Constructor, Method, Field and Package classes. I was able to factor most common code out, so implementing the core functionality wasn’t hard. Now I’m working on the supporting methods, e.g. those that allow you to get a Method from a Class object, without having to refer back to Java’s original Class object.

In essence, I’m creating gigantic wrappers for Java’s classes, and I’m forwarding each call to the corresponding method in the original class. It would be nice if this could be automated, and in fact, I think that could easily be done, again through annotations and bytecode rewriting. Consider the following listing:

1
2
3
4
5
6
7
8
9
10
11
12

class A {
public void foo() { }
public void bar() { }
}

@DelegateTo(A.class)
class B { }

@DelegateTo(A.class)
class C {
public void bar() { }
}

The A class defines two methods, foo() and bar(). Classes B and C are then annotated with @DelegateTo(A.class). When the bytecode rewriting framework sees this annotation, it automatically adds all of A‘s methods to B and C and generates bytecode to forward the calls. In the case of C, only calls to foo() are automatically forwarded, because an implementation of bar() has been provided.



The problem with this is that many times some processing has to be done before and after the delegation. For example, in the public MethodEx getMethod(String name, ClassEx... parameterTypes)() method of my extended Class class, I have to extract the original java.lang.Class objects from the elements in the paramTypes array, then pass the new array to the original method, and then finally create a new extended MethodEx instance using the return value from the forwarded call:

1
2
3
4
5
6
7
8
9
10
11
12
public class ClassEx<T> {

  // ...

  public MethodEx getMethod(String name, ClassEx... parameterTypes)

    throws NoSuchMethodException, SecurityException {

    Class[] pt = new Class[parameterTypes.length];

    for(int i=0; i<pt.length; ++i) {

      pt[i] = parameterTypes[i].java;

    }

    return new MethodEx(java.getMethod(name, pt));

  }

  // ...

}

This kind of processing is very common. Even though I'm writing several dozen methods that delegate, there are only about half a dozen ways to process data. This made me think of C++'s conversion operators (e.g. operator int() { return ...; }), which are basically user-designed casts with the ability to perform processing.

In the above example, the two processing methods I need are from ClassEx[] to Class[], and from Method to MethodEx. I could write a class with static methods to perform these conversions:

1
2
3
4
5
6
7
8
9
10
11
12
class Conversions {

  public static Class[] convertToClassArray(ClassEx[] input) {

    Class[] output = new Class[input.length];

    for(int i=0; i<pt.length; ++i) {

      output[i] = input[i].java;

    }

    return output;

  }

  public static MethodEx convertToMethodEx(Method input) {

    return new MethodEx(input);

  }

}

The names aren't really important, because the bytecode rewriting framework would look for a method with the correct input and output types for the conversion it has to perform. Some care needs to be taken, though, because Java doesn't allow overloading if only the return type differs. That's why I have suffixed the methods above with a description of the return type.

Now I can change the @DelegateTo annotation to specify a class that contains the conversion methods:



@DelegateTo(target=java.lang.Class.class, conversions=Conversions.class)

public class ClassEx { }


The bytecode rewriting framework will add all methods from java.lang.Class to ClassEx, but all occurrences of Class[] will be replaced by ClassEx[], and all occurrences of Method will be replaced by MethodEx. Instead of having a method public Method getMethod(String name, Class... parameterTypes)() like java.lang.Class, ClassEx will have a method public MethodEx getMethod(String name, ClassEx... parameterTypes)(). The String name is not processed in any way, because there is no conversion method from String to String, but Class... parameterTypes will be changed to ClassEx... parameterTypes, and the Class[] convertToClassArray(ClassEx[] input) will be used for the conversion. The return value will be changed from Method to MethodEx, and the MethodEx convertToMethodEx(Method input) method will be used for the conversion.

It probably makes sense to allow an array of classes with conversion methods, since the conversion methods may be spread out across several classes. The framework will have to check that there is no ambiguity as to which conversion method is used, i.e. there may not be two or more methods in all specified classes with the same input and output types. It may also be a good idea to allow different conversion methods for input and output. That would lead us to a syntax like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@DelegateTo(target=java.lang.Class.class,

            input={InputConversions.class},

            output={OutputConversions.class})

public class ClassEx<T> { }



class InputConversions {

  public static Class[] convertToClassArray(ClassEx[] input) {

    Class[] output = new Class[input.length];

    for(int i=0; i<pt.length; ++i) {

      output[i] = input[i].java;

    }

    return output;

  }

}



class OutputConversions {

  public static MethodEx convertToMethodEx(Method input) {

    return new MethodEx(input);

  }

}

Considering that this will handle the conversions for more than one method, I think the effort would well be worth it.

There's still a problem with type checking, of course: Since classes for which the forwarding methods should be generated automatically don't actually contain the methods in the source code, the Java compiler will emit errors whenever the methods are used. Compiling should therefore probably happen in three phases: First, all classes except the ones to be generated automatically are compiled; second, the automatic bytecode rewriting is done; third, all classes except the ones generated automatically are compiled again.



Long Day
Mathias — Fri, 09 Nov 2007 00:06:44 +0000
This was a long but productive day so far: I got to the office at 10 AM to grade COMP 311 midterm exams because one of the graduate TAs was out of town. Due to scheduling conflicts, we didn’t actually grade, but that was fine with me, instead I had nothing better to do than work on subtyping for annotations.

I still don’t load default values, but annotations and arrays as members are now implemented. There’s still one small problem: I don’t quite know the order in which annotation members appear in the return value of a toString() call. At first I thought the members were sorted alphabetically, so I implemented my toString() method that way, but now that I have a few more test cases, I see both @TestTwoAnnotation(s="foo", i=123) and @TestTwoAnnotation(i=123, s="foo") as output, and so far I can’t a difference in their contexts.



From the Trenches
Mathias — Thu, 08 Nov 2007 22:12:14 +0000
I’m still working on the API improvements to support annotations with subtyping. It turned out there is much more to do than I thought: Reading the class files, parsing the annotations and then creating the corresponding runtime objects is quite a bit of work, not difficult but tedious. I still have to deal with annotations as members, and with arrays of annotations and arrays of arrays. Additionally, I have to read out the default values of annotation members; right now, if a member isn’t specified, its value is null.

The high level parts of my framework make extensive use of the Visitor and Template Method design patterns. Unfortunately, I made a mistake again that I’ve made often, and that I’ve pointed out very often too. Consider the visitor interface and the abstract class implementing it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
interface IVisitor<R,P> {

  public R intCase(IntLeaf host, P param);

  public R addCase(AddNode host, P param);

  public R mulCase(MulNode host, P param);

}

abstract class ADefaultVisitor<R,P> {

  public abstract R defaultCase(INode host, P param);

  public R intCase(IntLeaf host, P param) {

    return defaultCase(host, param);

  }

  public R addCase(AddNode host, P param) {

    return defaultCase(host, param);

  }

  public R mulCase(MulNode host, P param) {

    return defaultCase(host, param);

  }

}

The abstract class implements a very simple Template Method pattern: It adds an abstract method that needs to be implemented, and then implements all the methods from the interface so that they call the new method. A default visitor like this is useful when most cases of a visitor are treated the same, e.g. during error checking when only one case is acceptable and all other cases should throw an exception:

1
2
3
4
5
6
7
8
9
class LeafOnlyVisitor<Void,Void> {

  public Void defaultCase(INode host, Void param) {

    // called by addCase and mulCase

    throw new RuntimeException("leaf expected");

  }

  public Void intCase(IntLeaf host, Void param) {

    return Void.TYPE;

  }

}

Unfortunately, it’s very easy to accidentally change the signature of the method that we’re trying to override, and therefore not override the method but add another method, one that is never called. In the first of the two examples below, the method name is misspelled; in the second, the type of the first parameter is different.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class LeafOnlyVisitorFlawed<Void,Void> {

  public Void defaultCase(INode host, Void param) {

    // called by addCase and mulCase

    throw new RuntimeException("leaf expected");

  }

  public Void untCase(IntLeaf host, Void param) {

    return Void.TYPE;

  }

}



class LeafOnlyVisitorFlawed2<Void,Void> {

  public Void defaultCase(INode host, Void param) {

    // called by addCase and mulCase

    throw new RuntimeException("leaf expected");

  }

  public Void intCase(INode host, Void param) {

    return Void.TYPE;

  }

}

In both cases, there is no compiler error, because all methods from the interface and the method from the abstract class have been implemented, so it’s often difficult to discover this problem. I just realized how nice it would be to specify that subclasses of a class may only override methods, never add new methods. Using annotations, the syntax could look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
@OverrideOnly

abstract class ADefaultVisitor<R,P> {

  public abstract R defaultCase(INode host, P param);

  public R intCase(IntLeaf host, P param) {

    return defaultCase(host, param);

  }

  public R addCase(AddNode host, P param) {

    return defaultCase(host, param);

  }

  public R mulCase(MulNode host, P param) {

    return defaultCase(host, param);

  }

}

When the classes LeafOnlyVisitorFlawed and LeafOnlyVisitorFlawed2 accidentally add a method instead of overriding one, an annotation-based checker could emit errors.

To allow better code reuse, it probably makes sense to allow private methods to be added to a class marked @OverrideOnly.



Subtyping for Annotations
Mathias — Mon, 05 Nov 2007 22:06:55 +0000
Last night, I managed to change the Java compiler to parse and encode annotations with inheritance. I’ve complained about this shortcoming many times already: Even though subtyping (inheritance) is one of the core concepts in Java, and even though annotations are represented using (slightly enhanced) interfaces, which support inheritance, Java disallows inheritance for annotations. I never quite understood why.

I’ve now created a modified compiler, based on the LAPT-javac compiler I wrote a while back. The new compiler is called xajavac, for “eXtended Annotation javac”, and supports annotations on local variables, like LAPT-javac, and subtyping for annotations.

Now I can write the following:

1
2
3
4
5
6
7
8
9
10
11
12
@interface InvariantAnnotation { }

@interface OnlyThreadWithName extends InvariantAnnotation {

  String value();

}

@interface OnlyEventThread extends InvariantAnnotation { }

@interface Or extends InvariantAnnotation {

  InvariantAnnotation value();

}



@Or({@OnlyThreadWithName("main"),

     @OnlyEventThread})

void foo() { ... }

I’m now working on providing an API that supports access to these annotations at runtime.