A Concurrent Affair » Mint

Automatic Generation of Optimized Domain-Specific Operations, by Jason Eckhardt

Mathias — Mon, 22 Nov 2010 19:09:04 +0000

Jason Eckhardt, a student of Keith Cooper, had an interesting COMP 600 talk about StencilDSL and simplifying writing certain stencil-based array operations:

Automatic Generation of Optimized Domain-Specific Operations

In previous work, we introduced a compiler-based algorithm for eliminating inter-iteration redundancies in array-based loop computations. This algorithm detects and transforms loops from ordinary programs using sophisticated analysis techniques. From that work we discovered that many of the codes amenable to our technique were concentrated in a few application-specific domains such as PDE solvers and image processing. Moreover, the loops in these particular codes exhibit a regularity or simplicity which can simplify the task of analyzing and optimizing them. We also discovered that these computations are often specified in the domain literature not as elaborate loop nests, but rather with simple matrices or other compact forms. In this talk, we discuss a prototype automatic program generation tool which allows the domain expert to specify computations in a simple and familiar way, while letting the tool generate the actual source code. By utilizing domain knowledge encoded in the specification, the tool generates optimized code by suppressing as many inter-iteration redundancies as possible. The result is a highly efficient program with little effort from the domain expert, and with no need for a programming or computer architecture expert to optimize the code.

I wonder how this would fit into Mint, especially since we used one of those stencil-based operations, a simple Gaussian blur, as benchmark example. Using eta-expansion as “the trick”, the staged generator might look exactly like the unstaged program.

But I definitely don’t have time to look at this right now.

New Mint Release: r15952

Mathias — Wed, 20 Oct 2010 22:05:29 +0000

I made a new release of Mint and DrJava with Mint yesterday: October 19, 2010 (r15952). The latest release is, as always, available from the Mint implementation page:

There were several bug fixes regarding assignment to variables declared as CodeFree, the places where SafeCode is required, treatment of type variables with bounds in escapes, and array types, which were erroneously printed as ‘[Lfoo/bar;’ or ‘[B’.

The version of DrJava with Mint is based on the current trunk (and therefore is newer than the updated stable release of DrJava that was recently made available).

(Re-posted from The Java Mint Blog.)

Back from GPCE 2010

Mathias — Sun, 17 Oct 2010 14:43:10 +0000

After a rather long return journey and more than enough time at the Amsterdam-Schiphol airport, I’m back from my trip to GPCE 2010 in Eindhoven. It was fun.

Eddy and I presented a tutorial on DSL implementation in Mint. Of course, I don’t have a picture of that, but Eelco might. But here’s Sukyoung Ryu presenting her tutorial on Fortress.

Sukyoung Ryu's GPCE 2010 Tutorial on Fortress

The auditorium for GPCE was nice, but it didn’t have power outlets. After 10 years in Texas, I guess that’s how I generally feel about Europe: quaint, a bit cramped and underpowered ;-)

GPCE Auditorium in Eindhoven

Also very European: Train, bike, landscape with canals.

Train, bike, landscape with canals

Some more pictures from the trip:

Eindhoven train station, for a quick escape

Landscape from the train

Ready to leave Amsterdam-Schiphol airport

And here are some more pictures that Eddy took. Proof that I exist!

At the Amsterdam-Schiphol airport train station

"Looking very European" with my scarf

Benoît Mandelbrot Died at Age 85

Mathias — Sun, 17 Oct 2010 01:46:44 +0000

Just after Eddy and I used the Mandelbrot set as benchmark in our DSL tutorial, Benoît Mandelbrot died. The Franco-American mathematician, who is most well-known for popularizing fractal geometry, died at the age of 85.

Thank you for giving us such a visually appealing way of demonstrating our optimization for embarrassingly parallel loops.

Another Image Processing Example in Our Staged DSL

Mathias — Sun, 10 Oct 2010 12:56:49 +0000

Last night, we implemented a Mandelbrot example in the image processing DSL that we created for our GPCE tutorial. Here’s a screenshot.

Screen Capture of Mandelbrot in our Staged DSL

GPCE’10 Tutorial Slides

Mathias — Sun, 10 Oct 2010 12:44:55 +0000

The slides for our GPCE 2010 tutorial presentation “Agile and Efficient Domain-Specific Languages using Multi-Stage Programming in Java Mint” are available now as PowerPoint and PDF file.

The zip file with the source code for our GPCE 2010 tutorial is available too.

The simplest way to experiment with Mint is to download the latest version of DrJava with Mint.

(Re-posted from The Java Mint Blog.)

GPCE’10 Tutorial Source

Mathias — Sun, 10 Oct 2010 10:06:03 +0000

Here is a zip file with the source code for our GPCE 2010 tutorial.

The simplest way to experiment with Mint is to download the latest version of DrJava with Mint.

(Re-posted from The Java Mint Blog.)

Gaussian Blur Implemented in Our DSL

Mathias — Tue, 05 Oct 2010 21:28:04 +0000

Sexy!

Gaussian Blur in DSL Screenshot

A screenshot showing an original image of random grayscale values, and a blurred version of the image, with the source code in our example DSL for the GPCE tutorial in the background.

Power_Let Was Wrong

Mathias — Mon, 04 Oct 2010 19:44:00 +0000

I just did some experiments with Mint, because we ran into some problems with using HJ inside DrJava, and I wanted to make sure we didn’t have those problems. That’s when I noticed that our Power_Let example had been wrong. One of our undergrads had written it as , and no one had noticed…

Embarrassing… But at least the unstaged code was equally wrong, so the benchmark is valid ;)

Demo of DrJava/HJ

Mathias — Mon, 04 Oct 2010 19:29:48 +0000

In today’s DrJava/Habanero meeting, Vincent, Jarred and I presented a demo of DrJava/HJ to Jack Dennis of MIT. Generally, we were quite pleased how we could just re-build Habanero Java and DrJava, and get a working version with the newest features from both packages.

We also noticed a few problems, of course. Perhaps most critically, when using the HJ compiler from within DrJava, the compiler uses the wrong directories when it generates class files for classes not in the default package.

I fixed part of that bug. The compiler adapter was ignoring the destination directory passed to the HJ compiler adapter. Now, if you create a project and set a build directory, the class files will end up in the right place. I have committed this fix into the DrJava trunk and also into the drjava-hj branch.

I didn’t fix the problem (a) if there is no build directory set, or (b) if we are not in project mode. This is something that I think should be fixed in the HJ compiler, because it doesn’t behave the same way javac behaves: If you invoke javac and don’t give it a destination directory, it will generate the class files in the same directory where the source files are. The HJ compiler generates the class files in the current directory.

Example:

javac src/HelloWorld.java generates the class file src/HelloWorld.class. hjc src/HelloWorld.hj generates the class files HelloWorld.class, HelloWorld$0.class, and HelloWorld$Main.class. in the current directory, not in src.

I hope the HJ team will agree this is a change that should be made in the HJ compiler.

Furthermore, we noticed again that compiler error messages aren’t displayed in DrJava’s Compiler Output pane. We also noted that we cannot call methods directly from the Interactions pane: We have to use the hj or run commands, otherwise the class loader isn’t properly set up. We should at least come up with a better error message than throwing an exception:

Welcome to DrJava. Working directory is /Users/mgricken/bin/hj.release/examples
> new FibFuture().fib(10)
java.lang.NullPointerException
at hj.lang.Runtime.here(Runtime.java:317)
at hj.lang.Object.(Object.java:44)
at FibFuture.(FibFuture.hj)

Interestingly, we can call Mint methods from the Interactions pane, we just cannot use the new syntax elements in Mint, brackets and escape.

Welcome to DrJava. Working directory is /Users/mgricken/Documents/Research/Mint/java-mint/trunk/langtools/mintTest
> import edu.rice.cs.mint.runtime.*
> Code c = DrJavaInteractions.makeCodeInt(5)
> c
<| ((5)) |>
> Code cTo10 = DrJavaInteractions.spower(c, 10)
> cTo10
<| (((5)) * (((5)) * (((5)) * (((5)) * (((5)) * (((5)) * (((5)) * (((5)) * (((5)) * (((5)) * (1))))))))))) |>
> cTo10.run()
9765625

New Mint Release: r15858

Mathias — Fri, 01 Oct 2010 04:37:06 +0000

I made a new release of Mint and DrJava with Mint yesterday: September 29, 2010 (r15858). The latest release is, as always, available from the Mint implementation page:

We have added the edu.rice.cs.mint.runtime.CodeFree interface and the edu.rice.cs.mint.runtime.CheckedExceptionInCode exception class.

The CodeFree interface provides a different, more transparent way of specifying separable classes that can be used for non-local assignments and CSP. When a class is declared to implement the CodeFree interface, the class is guaranteed to be code-free, provided a number of static checks are passed:

All field types and method return types are code-free, either using the old definition that required the types to be final, or by themselves implementing the CodeFree interface.
The same applies to all subtypes of types implementing the CodeFree interface. It is an error to extend a class declared to be code-free using the CodeFree interface and then to add a field containing or a method returning code.

This new rule makes it simpler to work with non-primitive types since the classes do not have to be final anymore. In the past, lifting had to be done instead of using CSP; that is not necessary anymore. Consider the old code with the required lift method and its application instead of straight-forward CSP:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

public abstract class Value {
public separable int intValue() { throw new WrongTypeException(); }
public separable boolean booleanValue() { throw new WrongTypeException(); }
public separable abstract boolean valueEq(Value other);
public separable abstract Code<Value> lift();
}

public class IntValue extends Value {
private int _data;
// ...
public separable Code<Value> lift() {
final int lfData = _data; // hack to help the Mint compiler
return <| (Value) new IntValue(lfData) |>;
}
}

public class BooleanValue extends Value {
private boolean _data;
// ...
public separable Code<Value> lift() {
final boolean lfData = _data; // hack to help the Mint compiler
return <| (Value) new BooleanValue(lfData) |>;
}
}

// ...

public class Val implements Exp {
private Code<Value> _value;
public Val(final Value value) {
_value = value.lift(); // lifting here
}
// ...
}

and the new code that doesn’t require the lift method anymore:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

public abstract class Value implements CodeFree, Serializable {
public separable int intValue() { throw new WrongTypeException(); }
public separable boolean booleanValue() { throw new WrongTypeException(); }
public separable abstract boolean valueEq(Value other);
}

public class IntValue extends Value {
private int _data;
// ...
}

public class BooleanValue extends Value {
private boolean _data;
// ...
}

// ...

public class Val implements Exp {
private Code<Value> _value;
public Val(final Value value) {
_value = <| value |>; // straight-forward CSP
}
// ...
}

Please note that a class used in CSP still needs to be serializable if the code object it is used in is to be saved using MintSerializer.save. That means that whenever you implement CodeFree, you should consider also implementing the java.io.Serializable interface.

The second change involves the way checked exceptions are thrown inside a code object. In the past, checked exceptions were not allowed, because the Code.run() method did not have a throws Throwable clause.

We still didn’t add such a clause, because it would essentially require a try-catch construct around every call to Code.run(). Instead, checked exceptions are caught and rethrown wrapped in a edu.rice.cs.mint.runtime.CheckedExceptionInCode unchecked exception. Here is an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

public static separable void m() throws IOException {
new File("/this/is/a/bad/path").createNewFile(); // throws IOException
}

// ...

SafeCode<Void> c = <| { m(); } |>;
try {
c.run();
}
catch(CheckedExceptionInCode ce) {
Throwable cause = ce.getCause();
// ...
}

We also improved error checking for calls to separable methods and fixed a bug that required SafeCode<T> in too many places.

The version of DrJava with Mint is based on the current trunk (and therefore is newer than the updated stable release of DrJava that was recently made available).

(Re-posted from The Java Mint Blog.)

New Mint Release: r15772

Mathias — Thu, 16 Sep 2010 21:00:24 +0000

We have just made a new release of Mint and DrJava with Mint: September 16, 2010 (r15772). The latest release is, as always, available from the Mint implementation page:

We have added the edu.rice.cs.mint.runtime.MintSerializer class to it that can write code object, including CSP data, to a jar file (or whatever stream you like), and then restore it again. Here is a very simple example:

1
2
3
4
5
6

Code<Integer> c = <|123|>;
File f = new File(dir, "IntegerCode1.jar");
MintSerializer.save(c, "IntegerCode1", f);

Integer s = MintSerializer.load(f);
System.out.println(s);

The version of DrJava with Mint is based on the current trunk (and not on updated stable release of DrJava that was made available this week).

(Re-posted from The Java Mint Blog.)

Mint on the Mac

Mathias — Wed, 01 Sep 2010 18:16:11 +0000

I guess I’m a bit behind the technology curve. The MacBook that I’m using as one of my development machines is one of the original white Intel MacBooks with a Core Duo CPU (not Core 2 Duo). It’s a 32-bit machine, and Apple doesn’t offer Java 6 for 32-bit computers.

Mint, however, requires a version of Java 6, which is why we recommended using SoyLatte to run Mint. That comes with all kinds of inconveniences because SoyLatte uses X11 for its GUIs, which means DrJava needs to run under X11.

Yesterday I tested Mint and DrJava with Mint using Apple’s Java SE 6.0 Release 1 Developer Preview 6 (file name: javase6release1dp6.dmg), which I still had floating around and which is the last version of Apple’s Java 6 that still runs on 32-bit Macs. It works! SoyLatte is only required if you don’t have a version of Java 6 (either Apple’s official version on 64-bit Macs, or Apple’s Java SE 6.0 Release 1 Developer Preview 6 on 32-bit Macs) on your Mac.

I have updated the instructions on how to install Java Mint on the Mac and how to run DrJava with Mint accordingly.

(Re-posted from The Java Mint Blog.)

Reflection-Based S-Expression Parser

Mathias — Wed, 01 Sep 2010 05:43:54 +0000

I’m really quite proud of this little reflection-based S-expression parser that I wrote for our GPCE Mint tutorial.

We wanted to have a parser so we don’t have to construct our ASTs using Java code. The problem was that we’ll probably have ten different mini-languages, and we didn’t want to write a parser for each language. Reflection was the answer, of course.

For this parser to work, you need to define the base class of the class hierarchy (e.g. Exp) and the subclasses as static nested classes inside the same outer class (e.g. all inside UnstagedLint).

You then simply specify the base class, and the parser will search for the subclasses. The names of the S-expressions are the simple names of the subclasses (e.g. Add and Int if there are Add and Int subclasses).

When encountering an S-expression, the parser will try all of the constructors of the corresponding class. As arguments for a constructor, the parser can process all primitive types, the boxed types, strings, recursive occurrences of the base class (e.g. Add containing two Exp instances), as well as other class hierarchies, which are parsed using subparsers.

These subparsers work the same way the main parser works. For example, if a Val constructor needs a Value argument, and there is a subparser registered for the Val hierarchy, the main parser will delegate to that subparser. Here’s an example:

1
2
3
4

DataTypes.Exp e =
new LintParser<DataTypes.Exp>(DataTypes.Exp.class)
.addSubParser(new LintParser<DataTypes.Value>(DataTypes.Value.class))
.parse("(Add (Val (IntValue 1)) (Val (IntValue 2)))");

This creates a parser for expressions of class DataTypes.Exp, and it registers a subparser for expressions of class DataTypes.Value. Then it parses the string (Add (Val (IntValue 1)) (Val (IntValue 2))). The (IntValue 1) and (IntValue 2) substrings are parsed using the subparser.

It’s really quite elegant. Especially the ease of integrating the subparsers surprised me. Here’s the parser in full.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337

package parser;

import java.lang.reflect.*;
import java.util.*;

/**
* A reflection-based S-expression parser returning an expression of type E.
* Written by Mathias Ricken and Edwin Westbrook.
*/
public class LintParser<E> {
/** Convenience method to parse string s into an expression of class c,
* using the specified subparsers as help, if the parser encounters non-primitive,
* non-string arguments to S-expressions not within the class hierarcy under class c.
* @param c class of the expression
* @param s string to parse
* @param subParsers subparsers for non-primitive, no-string values
* @return expression of class c */
public static <T> T parse(Class<T> c, String s, LintParser... subParsers) {
LintParser<T> p = new LintParser<T>(c);
for(LintParser sp: subParsers) {
p.addSubParser(sp);
}
return p.parse(s);
}

/** Exception being thrown if the string cannot be parsed. */
public static class ParseException extends RuntimeException { }

/** The class of the expression (e.g. lint1_arithmetic.UnstagedLint.Exp). */
private Class<E> _expClass;

/** The class enclosing the expression classes (e.g. lint1_arithmetic.UnstagedLint). */
private Class _enclosingClass;

/** A map from S-expression names to classes (e.g. Int -> lint1_arithmetic.UnstagedLint.Int). */
private HashMap<String,Class extends E>> _classes =
new HashMap<String,Class extends E>>();

/** A map from classes for which subparsers have been registered to the subparsers. */
private HashMap<Class,LintParser> _subParsers =
new HashMap<Class,LintParser>();

/** Create a new parser for expressions of class expClass.
* @param expClass the class of the expressions to be parsed */
public LintParser(Class<E> expClass) {
_expClass = expClass;
_enclosingClass = _expClass.getEnclosingClass();
// find all concrete, static subclasses of expClass contained in the same enclosing class
for(Class c: _enclosingClass.getDeclaredClasses()) {
if (((c.getModifiers() & Modifier.ABSTRACT)==0) &&
((c.getModifiers() & Modifier.STATIC)!=0)) {
try {
Class extends E> contained = c.asSubclass(_expClass);
_classes.put(contained.getSimpleName(), (Class extends E>)contained);
println(contained.getSimpleName());
}
catch(ClassCastException cce) { /* skip */ }
}
}
}

/** Add a subparser for expressions of type S. Return this parser (not the subparser!)
* for method chaining.
* @param p subparser for expressions of type S
* @return this parser (for expressions of type E) */
public <S> LintParser<E> addSubParser(LintParser<S> p) {
_subParsers.put(p._expClass, p);
return this;
}

/** Parse the string.
* @param s the string to be parsed
* @return the expression of type E */
public E parse(String s) {
println("parse '"+s+"'");
RestoreStringTokenizer st = new RestoreStringTokenizer(s.trim());
return parse(st);
}

/** Parse the tokens in the tokenizer.
* @param s the tokenizer with the tokens
* @return the expression of type E */
protected E parse(RestoreStringTokenizer s) {
// position to restore everything
final int sPos = s.getPosition();

// read (
parseWhitespace(s);
String word = s.nextToken();
println("word: '"+word+"'");
if (!word.equals("(")) {
s.restorePosition(sPos);
throw new ParseException();
}

// read S-expression name
parseWhitespace(s);
word = s.nextToken();
println("word: '"+word+"'");
if (!_classes.containsKey(word)) {
// don't know what that is, restore
s.restorePosition(sPos);
throw new ParseException();
}

// known subclass of E
Class extends E> c = _classes.get(word);
println("Class: "+c.getSimpleName());
parseWhitespace(s);

// position to restore after trying out a constructor
final int sTryCtorPos = s.getPosition();

Constructor extends E> ctor;
for(Constructor ctor2: c.getDeclaredConstructors()) {
println("\ttrying ctor "+ctor2);

// this is necessary since c.getDeclaredConstructors() returns Constructor
// but we want Constructor
try { ctor = c.getConstructor(ctor2.getParameterTypes()); }
catch(NoSuchMethodException nsme) { throw new AssertionError("Should never happen."); }

// try to parse the arguments for this constructor
Object[] args = new Object[ctor.getParameterTypes().length];
int argIndex = 0;
try {
for(Class paramC: ctor.getParameterTypes()) {
// primitive types and their boxed types
if ((paramC==int.class) || (paramC==Integer.class)) {
args[argIndex] = parseLeaf(s,Integer.class);
}
else if ((paramC==boolean.class) || (paramC==Boolean.class)) {
args[argIndex] = parseLeaf(s,Boolean.class);
}
else if ((paramC==long.class) || (paramC==Long.class)) {
args[argIndex] = parseLeaf(s,Long.class);
}
else if ((paramC==double.class) || (paramC==Double.class)) {
args[argIndex] = parseLeaf(s,Double.class);
}
else if ((paramC==float.class) || (paramC==Float.class)) {
args[argIndex] = parseLeaf(s,Float.class);
}
else if ((paramC==byte.class) || (paramC==Byte.class)) {
args[argIndex] = parseLeaf(s,Byte.class);
}
else if ((paramC==short.class) || (paramC==Short.class)) {
args[argIndex] = parseLeaf(s,Short.class);
}
// char or Character
else if ((paramC==char.class) || (paramC==Character.class)) {
// parse as string
Object temp = parseLeaf(s,String.class);
// must be exactly one character
if (temp.toString().length()!=1) throw new ParseException();
args[argIndex] = new Character(temp.toString().charAt(0));
}
// strings
else if (paramC==String.class) {
args[argIndex] = parseLeaf(s,String.class);
}
// recursively parse expressions of type E
else if (_expClass.equals(paramC)) {
args[argIndex] = parse(s);
}
// try to use a subparser
else if (_subParsers.containsKey(paramC)) {
// try one of the subparsers
args[argIndex] = _subParsers.get(paramC).parse(s);
}
else {
// don't know what that is
// restore happens below
throw new ParseException();
}
++argIndex;
}
}
catch(ParseException pe) {
// this constructor didn't work out, we need to restore and try another constructor
s.restorePosition(sTryCtorPos);
// continue with next constructor
continue;
}
try {
// we read all values required for this constructor
// make sure that the next token is )
parseWhitespace(s);
word = s.nextToken();
println("word: '"+word+"'");
if (!word.equals(")")) {
// it wasn't ), we need to restore and try another constructor
s.restorePosition(sTryCtorPos);
// continue with next constructor
continue;
}
parseWhitespace(s);

// successfully used this constructor
println("new "+ctor.getDeclaringClass().getSimpleName()+" "+
java.util.Arrays.toString(args));
E e = (E)ctor.newInstance(args);
return e;
}
catch(Exception e) {
// something went wrong using this constructor, we need to restore and try another
s.restorePosition(sTryCtorPos);
// continue with next constructor
continue;
}
}

s.restorePosition(sPos);
throw new ParseException();
}

/** Parse a leaf of class c, which must have a unary constructor taking a String, from
* the tokenizer s.
* @param s tokenizer from which to parse
* @param c the class, which has a unary constructor taking a String, for that we want to create a value
* @return parsed value of class c */
protected Object parseLeaf(RestoreStringTokenizer s, Class c) {
// position so we can undo this attempt to parse
final int sPos = s.getPosition();

String word = null;
try {
// get the unary constructor taking String
Constructor ctor = c.getDeclaredConstructor(String.class);

// skip whitespace, get the string, skip whitespace
parseWhitespace(s);
word = s.nextToken();
println("word: '"+word+"'");
Object o = ctor.newInstance(word);
parseWhitespace(s);

return o;
}
catch(NoSuchMethodException nsme) {
// something went wrong, restore and abort
s.restorePosition(sPos);
throw new ParseException();
}
catch(InstantiationException ie) {
// something went wrong, restore and abort
s.restorePosition(sPos);
throw new ParseException();
}
catch(IllegalAccessException iae) {
// something went wrong, restore and abort
s.restorePosition(sPos);
throw new ParseException();
}
catch(InvocationTargetException ite) {
// something went wrong, restore and abort
s.restorePosition(sPos);
throw new ParseException();
}
}

/** Parse whitespace. Stop before the next non-whitespace token, or when there are no
* more tokens.
* @param s tokenizer from which to parse whitespace */
protected void parseWhitespace(RestoreStringTokenizer s) {
String word;
int sPrevPos;
while(s.hasMoreTokens()) {
sPrevPos = s.getPosition();
word = s.nextToken();
if (!word.trim().equals("")) {
s.restorePosition(sPrevPos);
break;
}
}
}

/** Debug method to print, comment System.out.println call out to disable debug printing */
public static void println(Object o) {
// System.out.println(o.toString());
}

/** This tokenizer stores all read tokens in a stack so they can easily be
* restored. This isn't as memory as it could be, but it's good enough for
* small examples. */
public static class RestoreStringTokenizer extends StringTokenizer {
/** Stack of read tokens. */
private Stack<String> _read = new Stack<String>();
/** Stack of restored tokens. */
private Stack<String> _restored = new Stack<String>();
/** Create a new string tokenizer that can restore positions. */
public RestoreStringTokenizer(String s) { super(s," ()",true); }
/** Return the current position. */
public int getPosition() { return _read.size(); }
/** Rewind to a previous position. This only goes backward, not forward. */
public void restorePosition(int pos) {
while(_read.size()>pos) {
_restored.push(_read.pop());
}
}
// overridden methods from StringTokenizer
public int countTokens() { return _restored.size() + super.countTokens(); }
public boolean hasMoreTokens() { return (_restored.size()>0) || super.hasMoreTokens(); }
public boolean hasMoreElements() { return hasMoreTokens(); }
public String nextToken() {
String token = (_restored.size()>0) ? _restored.pop() : super.nextToken();
_read.push(token);
return token;
}
public Object nextElement() { return nextToken(); }
public String nextToken(String delim) {
if (_restored.size()>0) return _restored.pop();
return super.nextToken(delim);
}
}

public static void main(String[] args) {
lint2_cond.UnstagedLint.Exp e = parse
(lint2_cond.UnstagedLint.Exp.class,"(Ifz (Int 1) (Int 10) (Mul(Int 3)(Add(Int 5)(Int 10))))");
System.out.println(new lint2_cond.UnstagedLint.Program(e).peval(lint2_cond.UnstagedLint.env0));

lint.DataTypes.Value v = parse(lint.DataTypes.Value.class, "(IntValue 1)");
System.out.println(v);

lint.DataTypes.Exp e2 =
new LintParser<lint.DataTypes.Exp>(lint.DataTypes.Exp.class)
.addSubParser(new LintParser<lint.DataTypes.Value>(lint.DataTypes.Value.class))
.parse("(Add (Val (IntValue 1)) (Val (IntValue 2)))");
System.out.println(new lint.DataTypes.Program(e2).peval(lint.DataTypes.env0, lint.DataTypes.fenv0));

lint.DataTypes.Exp e3 =
LintParser.parse(lint.DataTypes.Exp.class, "(Add (Val (IntValue 1)) (Val (IntValue 2)))",
new LintParser<lint.DataTypes.Value>(lint.DataTypes.Value.class));
System.out.println(new lint.DataTypes.Program(e3).peval(lint.DataTypes.env0, lint.DataTypes.fenv0));

}
}

Here is the definition of a simple language for arithmetic expressions that uses the parser:

package lint1_arithmetic;

public class UnstagedLint {

/*
type exp = Int of int
| Var of string
| Add of exp * exp
| Sub of exp * exp
| Mul of exp * exp
| Div of exp * exp
type def = Definition of string * int
type prog = Program of def list * exp
*/

public static interface Exp {
public int eval(Env e);
}

public static class Int implements Exp {
private int _value;
public Int(int value) {
_value = value;
}
public int eval(Env e) {
return _value;
}
}

public static class Var implements Exp {
private String _s;
public Var(String s) {
_s = s;
}
public int eval(Env e) {
return e.get(_s);
}
}

public static abstract class BinOp implements Exp {
protected Exp _left, _right;
public BinOp(Exp left, Exp right) {
_left = left;
_right = right;
}
}

public static class Add extends BinOp {
public Add(Exp left, Exp right) {
super(left, right);
}
public int eval(Env e) {
return _left.eval(e) + _right.eval(e);
}
}

public static class Sub extends BinOp {
public Sub(Exp left, Exp right) {
super(left, right);
}
public int eval(Env e) {
return _left.eval(e) - _right.eval(e);
}
}

public static class Mul extends BinOp {
public Mul(Exp left, Exp right) {
super(left, right);
}
public int eval(Env e) {
return _left.eval(e) * _right.eval(e);
}
}

public static class Div extends BinOp {
public Div(Exp left, Exp right) {
super(left, right);
}
public int eval(Env e) {
return _left.eval(e) / _right.eval(e);
}
}

/*
exception EnvironmentException
let env0 = fun x -> raise EnvironmentException
let fenv0 = env0
let ext env x v = fun y -> if x=y then v else env y
*/

public static class EnvironmentException extends RuntimeException { }

// environment; implemented as a function object from String to Exp
public static interface Env {
public int get(String y);
}
public static final Env env0 = new Env() {
public int get(String s) { throw new EnvironmentException(); }
};

public static Env ext(final Env env, final String x, final int v) {
return new Env() {
public int get(String y) {
if (x.equals(y)) return v; else return env.get(y);
}
};
}

public static class Definition {
private String _name;
private int _v;
public Definition(String name, int v) {
_name = name;
_v = v;
}
public String name() { return _name; }
public int value() { return _v; }
}

public static class Program {
public Definition[] _defs;
public Exp _body;
public Program(Exp body, Definition... defs) {
_defs = defs;
_body = body;
}

public int peval(Env env) {
return peval(env,0);
}

private int peval(final Env env, int defIndex) {
// match p with
if (_defs.length<=defIndex) {
// Program ([],e) -> eval e env
return _body.eval(env);
}
else {
// |Program (Definition (n,v)::tl,e) -> peval (Program(tl,e)) (ext env n v)
final Definition d = _defs[defIndex];
return peval(ext(env, d.name(), d.value()),defIndex+1);
}
}
}

/*
let rec eval e env fenv =
match e with
Int i -> i
| Var s -> env s
| Add (e1,e2) -> (eval e1 env fenv)+(eval e2 env fenv)
| Sub (e1,e2) -> (eval e1 env fenv)-(eval e2 env fenv)
| Mul (e1,e2) -> (eval e1 env fenv)*(eval e2 env fenv)
| Div (e1,e2) -> (eval e1 env fenv)/(eval e2 env fenv)

let rec peval p env =
match p with
Program ([],e) -> eval e env
|Program (Definition (s1,i1)::tl,e) -> peval (Program(tl,e)) (ext env s1 i1)

*/

public static Program term10times20plus30 = new Program
(new Add(new Mul(new Var("x"), new Var("y")), new Int(30)),
new Definition("x", 10),
new Definition("y", 20));

public static Program parsedTerm10times20plus30 = new Program
(parser.LintParser.parse(Exp.class, "(Add (Mul (Var x) (Var y)) (Int 30))"),
new Definition("x", 10),
new Definition("y", 20));

public static Program termWhatX = new Program
(new Var("x"));

public static Program parsedTermWhatX = new Program
(parser.LintParser.parse(Exp.class, "(Var x)"));

public static void main(String[] args) {
int i = term10times20plus30.peval(env0);
System.out.println("term10times20plus30 = "+i);

i = parsedTerm10times20plus30.peval(env0);
System.out.println("parsedTerm10times20plus30 = "+i);

try {
i = termWhatX.peval(env0);
System.out.println("termWhatX = "+i);
throw new AssertionError("This should throw an EnvironmentException");
}
catch(EnvironmentException ee) { System.out.println("termWhatX threw "+ee); /* expected */ }

try {
i = parsedTermWhatX.peval(env0);
System.out.println("parsedTermWhatX = "+i);
throw new AssertionError("This should throw an EnvironmentException");
}
catch(EnvironmentException ee) { System.out.println("parsedTermWhatX threw "+ee); /* expected */ }
}
}

String Pool Interning Saves the Weakly Separable Day

Mathias — Tue, 31 Aug 2010 06:03:52 +0000

I just took a swim in the string pool. Who would have thought that interning strings would be so useful for weak separability in Mint?

I knew before that we had some problems calling String.equals in a separable method, like the Env.get(String y) method that does an environment look-up, because String.equals wasn’t declared separable itself. That’s why I used the == operator in the initial Lint interpreter I wrote for the PLDI 2010 paper:

1
2
3
4
5
6
7
8
9
10
11
12
13

public static separable Env ext(final Env env,
final String x,
final Code<Integer> v) {
return new Env() {
public separable Code<Integer> get(String y) {
// error: if (x.equals(y))
if (x==y)
return v;
else
return env.get(y);
}
};
}

This wasn’t a problem because we didn’t have a parser and were creating our ASTs using hand-written Java code. The variable names are string constants, which are always interned:

1
2
3
4

public static Program term10times20plus30 = new Program
(new Add(new Mul(new Var("x"), new Var("y")), new Int(30)),
new Definition("x", <|10|>),
new Definition("y", <|20|>));

Tonight, I wrote a reflection-based S-expression parser, and now the variable names are substrings of a longer string:

1
2
3
4

public static Program parsedTerm10times20plus30 = new Program
(parser.LintParser.parse(Exp.class, "(Add (Mul (Var x) (Var y)) (Int 30))"),
new Definition("x", <|10|>),
new Definition("y", <|20|>));

Substrings aren’t interned automatically, and the comparison using == fails. Fortunately, I can simply intern the strings when I create the Var AST nodes and the Definition variable bindings:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

public static class Var implements Exp {
private String _s;
public Var(String s) {
_s = s.intern(); // must intern here, so we can use == in the ext method below
}
public separable Code<Integer> eval(Env e) {
return e.get(_s);
}
}
public static class Definition {
private String _name;
private Code<Integer> _v;
public Definition(String name, Code<Integer> v) {
_name = name.intern(); // must intern here, so we can use == in the ext method above
_v = v;
}
public separable String name() { return _name; }
public separable Code<Integer> value() { return _v; }
}

This was a whole lot simpler and cleaner than making some kind of exception for String.equals.

New Mint Release: r15716

Mathias — Mon, 30 Aug 2010 18:17:24 +0000

As mentioned before, Eddy and I discovered a problem with type variables in code generated by the Mint compiler. We have now fixed this problem in the new release of Mint and DrJava with Mint: August 30, 2010 (r15716). The latest release is, as always, available from the Mint implementation page:

The problem occurred when the programmer used a type variable inside a bracket. An example of this would be a generic method like this:

1
2
3
4

public separable <X> Code<X> fun(Code<X> c1, Code<X> c2) {
return <| ( `(lfTest.eval(e,f).booleanCodeValue()) ?
`c2 : `c1 ) |>;
}

This caused Mint to generate 2nd stage code that contained the type variable X unbound. Unfortunately, the types are erased, and there is no way to find out what type X actually refers to at runtime, e.g. by writing something like X.class (this generates the Java compiler error “cannot select from a type variable”).

To get around this problem, we now require a final instance of type Class<X> to be in scope for every type variable X. For example, in the generic method above, there is a type variable X that is being used in the bracket. Therefore, we now have to have a final instance of Class<X> somewhere in scope, for example as a method parameter.

1
2
3
4
5

public separable <X> Code<X> fun(Code<X> c1, Code<X> c2,
final Class<X> xc) {
return <| ( `(lfTest.eval(e,f).booleanCodeValue()) ?
`c2 : `c1 ) |>;
}

That’s all that is necessary. The variable xc is not actually used in the code, it just needs to be in scope.

(Re-posted from The Java Mint Blog.)

Passing a Class for Every Type Variable T

Mathias — Mon, 30 Aug 2010 18:12:54 +0000

When working on our GPCE Mint tutorial, Eddy and I realized that there is a problem when programmers use type variables inside brackets. A method like

1
2
3
4

public separable <X> Code<X> fun(Code<X> c1, Code<X> c2) {
return <| ( `(lfTest.eval(e,f).booleanCodeValue()) ?
`c2 : `c1 ) |>;
}

causes Mint to generate code containing the free type variable X. There is no way to get any type information from a type variable at runtime, e.g. by doing X.class, so instead we decided to require the user to have a final instance of Class<X> in scope somewhere.

This was a pragmatic solution that seems to work pretty well. The method above can be rewritten as

1
2
3
4
5

public separable <X> Code<X> fun(Code<X> c1, Code<X> c2,
final Class<X> xc) {
return <| ( `(lfTest.eval(e,f).booleanCodeValue()) ?
`c2 : `c1 ) |>;
}

To implement this, we had to find all variables of type Class<X> that were in scope at the time a bracket was generated. This was surprisingly confusing to do with the javac environment and scope classes. In the end, I decided to adapt the Resolver.findVar and Resolver.findField methods.

/** This method returns a list of all VarSymbols of type Class in scope where X is a type variable. */
List<VarSymbol> getClassVarsInScope() {
// System.out.println("env = "+env);
List<VarSymbol> ret = List.nil();
Scope current = env.info.scope;
// System.out.println("Scope: "+current);
while(current != null) { // also go into the outer (enclosing) scopes
// for(Symbol s: current.elems) {
for (Scope.Entry e = current.elems; e != null; e = e.sibling) {
ret = processPotentialClassVarInScope(e.sym, ret);
}
current = current.next;
}
// System.out.println("Asking for vars in "+env.enclClass.sym);
ret = _getClassVarsInScope(env, ret);
return ret.reverse();
}

List<VarSymbol> processPotentialClassVarInScope(Symbol s, List<VarSymbol> ret) {
// System.out.println("in scope: "+s+" : "+s.getClass().getSimpleName());
if (s instanceof VarSymbol) {
VarSymbol vs = (VarSymbol)s;
if ((vs.type.tsym == syms.classType.tsym) &&
((vs.flags() & FINAL) != 0) &&
(vs.type.getTypeArguments().nonEmpty()) &&
(vs.type.getTypeArguments().head instanceof TypeVar)) {
// vs is of type Class where X is a type variable
ret = ret.prepend(vs);
}
}
return ret;
}

List<VarSymbol> _getClassVarsInScope(Env<AttrContext> env, List<VarSymbol> ret) {
Env<AttrContext> env1 = env;
boolean staticOnly = false;
while (env1.outer != null) {
if (rs.isStatic(env1)) staticOnly = true;
Scope.Entry e = env1.info.scope.elems;
while (e != null) {
Symbol sym = e.sym;
if (!(staticOnly &&
sym.kind == VAR &&
sym.owner.kind == TYP &&
(sym.flags() & STATIC) == 0)) {
ret = processPotentialClassVarInScope(sym, ret);
}
ret = _getClassFieldsInScope(env1, env1.enclClass.sym.type, env1.enclClass.sym, ret);

e = e.sibling;
}
ret = _getClassFieldsInScope(env1, env1.enclClass.sym.type, env1.enclClass.sym, ret);

if ((env1.enclClass.sym.flags() & STATIC) != 0) staticOnly = true;
env1 = env1.outer;
}

ret = _getClassFieldsInScope(env, syms.predefClass.type, syms.predefClass, ret);

return ret;
}

List<VarSymbol> _getClassFieldsInScope(Env<AttrContext> env,
Type site,
TypeSymbol c,
List<VarSymbol> ret) {
while (c.type.tag == TYPEVAR)
c = c.type.getUpperBound().tsym;
Scope.Entry e = c.members().elems;
while (e != null) {
if (e.sym.kind == VAR && (e.sym.flags_field & SYNTHETIC) == 0) {
if (rs.isAccessible(env, site, e.sym)) {
ret = processPotentialClassVarInScope(e.sym, ret);
}
}
e = e.sibling;
}
Type st = types.supertype(c.type);
if (st != null && (st.tag == CLASS || st.tag == TYPEVAR)) {
ret = _getClassFieldsInScope(env, site, st.tsym, ret);
}
for (Type it: types.interfaces(c.type)) {
ret = _getClassFieldsInScope(env, site, it.tsym, ret);
}
return ret;
}

A new Mint release can be expected shortly.

New Mint Release: r15707

Mathias — Tue, 24 Aug 2010 22:06:20 +0000

When working on our GPCE tutorial, Eddy and I discovered a small bug in the Mint compiler which I have now fixed in the new release of Mint and DrJava with Mint: August 24, 2010 (r15707). The latest release is, as always, available from the Mint implementation page:

The only change I made was to fix the let expression pretty printer when the expression contained more than one declaration.

The problem never came up before because we only had one declaration per let expression, and it was always possible to rewrite

1	let int x = 1, y = 2; x+y

1	let int x = 1; let int y = 2; x+y

We are pushing the Mint compiler more, and we have discovered another problem with generic methods. Currently, Mint generates a class extending Code<X> with X occuring unbound when this method is called:

1	public separable abstract <X> Code<X> fun(Code<X> c1, Code<X> c2, Object... param);

We are looking at making another bugfix soon.

(Re-posted from The Java Mint Blog.)

New Mint Release: r15700

Mathias — Sat, 21 Aug 2010 05:32:28 +0000

On Friday, I created a new release of Mint and DrJava with Mint: August 20, 2010 (r15700). The latest release is, as always, available from the Mint implementation page:

The only changes that we made were a small change to the build process on Mac OS, and the addition of the Range and Lift utility classes.

The Lift class allows the user to manually lift primitive values and strings when the compiler did not do that already, for example when working with arrays.

The Range class provides methods that allow many for loops to be written as foreach loops with a final loop variable that can be used across bracket boundaries immediately. Consider this example of a staged sparse matrix multiplication:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

public static separable
Code<Void> smmult(final double[][] a,
Code<double[][]> b,
Code<double[][]> output,
int l, int m, int n) {
Code<Void> stats = <| { } |>;
for(final int i: range(0, l)) {
for(final int j: range(0, m)) {
Code<Double> c = <| 0.0 |>;
for(final int k: range(0, n)) {
if(a[i][k] == 0.0)
continue;
else if(a[i][k] == 1.0)
c = <| `c + (`b)[k][j] |>;
else
c = <| `c + (`(lift(a[i][k])) * (`b)[k][j]) |>;
}
stats = <| { `stats; (`output)[i][j] = `c; } |>;
}
}
return stats;
}

is a lot cleaner than

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

public static separable
Code<Void> smmult(final double[][] a,
Code<double[][]> b,
Code<double[][]> output,
int l, int m, int n) {
Code<Void> stats = <| { } |>;
for(int i = 0; i < l; i++) {
for(int j = 0; j < m; j++) {
final int ii = i;
final int jj = j;
Code<Double> c = <| 0.0 |>;
for(int k = 0; k < n; k++) {
final int kk = k;
if(a[i][k] == 0.0)
continue;
else if(a[i][k] == 1.0)
c = <| `c + (`b)[kk][jj] |>;
else
c = <| `c + (`(lift(a[ii][kk])) * (`b)[kk][jj]) |>;
}
stats = <| { `stats; (`output)[ii][jj] = `c; } |>;
}
}
return stats;
}

The ii, jj, and kk variables aren’t necessary anymore.

(Re-posted from The Java Mint Blog.)

GPCE’10 Tutorial Lecture: Agile and Efficient Domain-Specific Languages using Multi-stage Programming in Java Mint

Mathias — Mon, 16 Aug 2010 19:50:14 +0000

Eddy, Walid, and I proposed a tutorial lecture for GPCE’10, and we’re delighted to report that it has been accepted for presentation on Sunday, October 10, 2010.

Agile and Efficient Domain-Specific Languages using Multi-stage Programming in Java Mint

Domain-specific languages (DSLs) are a powerful productivity tool because they allow domain experts, who are not necessarily programming experts, to quickly develop programs. DSL implementations have unique constraints for programming languages because they must be efficient, in order to ensure high productivity, but they must also be agile, in order to meet the rapidly changing demands of their domains. In this tutorial we show how multi-stage programming (MSP) can be used to build staged interpreters, which combine the agility of interpreters with the efficiency of compilers. The tutorial is conducted in Java Mint, an multi-stage Java based on recent work incorporating MSP into imperative object-oriented languages. In the first half of the tutorial, we introduce MSP by demonstrating how to write a staged interpreter for a number of basic language constructs, such as recursive functions, conditionals, and let expressions. In the second half, we extend our staged interpreter to take advantage of several well-known compiler optimizations, including type inference, constant folding, and static parallel loop scheduling. We highlight the opportunities afforded by using MSP with object-oriented design to quickly create efficient DSL implementations.

Brief Outline of Tutorial

This tutorial uses Java Mint to introduce MSP in imperative, object-oriented languages. We will progress as follows:

Introduction to MSP, the three language constructs “bracket”, “escape” and “run”, and their use in Java Mint.
Brief overview of MSP applications other than interpreters.
Development of a staged interpreter for a small DSL. We incrementally add:
1. Arithmetic expressions
2. Conditionals
3. Recursive functions
4. Let expressions
Application of compiler optimizations in the interpreter for our DSL. We discuss:
1. Extending the interpreter with multiple data types
2. Type inference
3. Constant folding
4. Static parallel loop scheduling

Biographies

Mathias Ricken is a doctoral candidate in the Programming Languages Team at Rice University and one of the principal developers of the DrJava integrated development environment. His research interests include concurrent programming, extending the Java language, and computer science education. He is the developer of the Concutest concurrent unit testing framework and has created various experimental extensions of Java to address, for instance, programming with meta-data. Currently, Mathias is contributing to Java Mint, a multi-stage extension of Java that allows safe and expressive statically typed program generation and specialization in an imperative language setting.

Edwin Westbrook is a post-doctoral researcher at Rice University. His primary interests are in developing techniques for implementing and verifying properties of domain-specific languages (DSLs). He has worked on a number of projects in this area, including: Cinic, a type theory for building machine-checked proofs of properties of DSLs using a new approach to higher-order abstract syntax; Java Mint, a multi-stage version of Java used for efficient implementations of DSLs; and Acumen, a DSL for designing cyber-physical systems.

Walid Taha is an professor at Halmstad University. His current interest is in modeling and simulation of cyberphysical systems. He was the principal investigator on a number of research awards and contracts from the National Science Foundation (NSF), Semi-conductor Research Consortium (SRC), and Texas Advanced Technology Program (ATP). He received an NSF CAREER award to develop Java Mint. He is the principle designer Java Mint, Acumen, MetaOCaml, and the Verilog Preprocessor. He founded the ACM Conference on Generative Programming and Component Engineering (GPCE), the IFIP Working Group on Program Generation (WG 2.11), and the Middle Earth Programming Languages Seminar (MEPLS). In 2009, he chaired the IFIP Working Conference on Domain Specific Languages (DSLs).

(Re-posted from The Java Mint Blog.)