Towards a Compact Deadlock Detector

On the way to a version of the deadlock detector that uses the compact representation, there are a few things I have to do:

  1. Make sure that timed pulls still work. If the master application cannot pull data whenever it wants to, then the debugged application can go into deadlock and I cannot extract the last few sync points anymore.
  2. Right now, there are only sync points generated just after a monitorenter and just before a monitorexit. In order to recognize a deadlock, I don’t only have to know what locks a thread is holding, I also have to know what locks a threadwants. That requires a sync point before the monitorenter instruction.

This information is enough to recussitate the deadlock detector. There are a few more issues.

  1. I should probably check that the PC value that is reported is still correct, now that I have inserted code before the monitorenter instruction.
  2. It might be nice to know where an object was created. I could add a special sync point that gets recorded when the object ID gets assigned. That may make it easier to identify what objects are involved in the deadlock.
  3. It would be nice to be able to get the line number from a PC value reported as part of a sync point, and in the case a deadlock is detected, a call stack for each thread would be nice. That was possible with the old deadlock detector with fat objects, but here it may be a little more complicated. We’ll see.

It seems like I have done tasks 1-3, and the code to report a sync point before a monitorenter instruction works, and the PC values are correct again now, too. Now I just have to rewrite the deadlock detector to work with the new compact data.

I also really need headless versions of all the apps so they can be run over X-less SSH.

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

Compact Object IDs Again

I have begun to reimplement the deadlock detector using the compact scheme. I already had such a system a long time ago, but that was with the fat objects, not with the compact array. For this to work with the compact scheme, I have to assign a unique ID to each object, and I’ve tried that before and run into nothing but trouble.

I think I’ve finally figured out how to solve this problem at least for a substantial subset of the classes: Instead of adding a field to java.lang.Object, I add a method public long $$$getObjectID$$$() that by default returns -1, meaning “object ID not available”. All subclasses of Object (except for java.lang.String, for which adding a field also didn’t work) then add the long $$$objectID$$$ field, initialize it in their constructors, and override the method to return the value.

So far, this seems to work for a lot of classes. So far, I haven’t checked the numbers or actually tried to detect a deadlock, but at least I’m getting numbers that seem sensible. I’m getting a few zeros in constructors, but that is to be expected, because there are monitorenter instructions before the object ID gets assigned. I’m getting -1 for Strings and plain Objects, but curiously also for a class’ java.lang.Class instance (e.g. synchronized(Integer.class) { }) and arrays (e.g. Integer[] ia = new Integer[5]; synchronized(ia) { }).

The following Java program

public class ObjectIDTest {
public static void main(String[] args) {
// string array
synchronized(args) { }
// integer array
Integer[] ia = new Integer[5];
synchronized(ia) { }
// class
synchronized(ObjectIDTest.class) { }
// plain object
Object o = new Object();
synchronized(o) { }
// subclass of Object
Integer i = new Integer(5);
Integer j = new Integer(6);
synchronized(i) {
synchronized(j) { }
}
}
}

generates the following trace, starting from where it enters the main method:

0 1 // 000030e4 0003 0001        357 sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang
0 2 // 000030e4 0003 0052        357 sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang
0 1 // 000031d7 0002 0003         -1 ObjectIDTest.main([Ljava/lang/String;)V PC=3
0 2 // 000031d7 0002 002d         -1 ObjectIDTest.main([Ljava/lang/String;)V PC=45
0 1 // 000030e4 0003 0001        357 sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang
0 2 // 000030e4 0003 0052        357 sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang
0 1 // 000031d7 0002 0058         -1 ObjectIDTest.main([Ljava/lang/String;)V PC=88
0 2 // 000031d7 0002 0082         -1 ObjectIDTest.main([Ljava/lang/String;)V PC=130
0 1 // 000031d7 0002 00aa          0 ObjectIDTest.main([Ljava/lang/String;)V PC=170
0 2 // 000031d7 0002 00d4          0 ObjectIDTest.main([Ljava/lang/String;)V PC=212
0 1 // 000031d7 0002 0104         -1 ObjectIDTest.main([Ljava/lang/String;)V PC=260
0 2 // 000031d7 0002 012e         -1 ObjectIDTest.main([Ljava/lang/String;)V PC=302
0 1 // 00003187 0001 0003          0 java.lang.Number.()V PC=3
0 2 // 00003187 0001 003c          0 java.lang.Number.()V PC=60
0 1 // 00003187 0001 0003          0 java.lang.Number.()V PC=3
0 2 // 00003187 0001 003c          0 java.lang.Number.()V PC=60
0 1 // 000031d7 0002 016b        543 ObjectIDTest.main([Ljava/lang/String;)V PC=363
0 1 // 000031d7 0002 0187        544 ObjectIDTest.main([Ljava/lang/String;)V PC=391
0 2 // 000031d7 0002 01b2        544 ObjectIDTest.main([Ljava/lang/String;)V PC=434
0 2 // 000031d7 0002 01ec        543 ObjectIDTest.main([Ljava/lang/String;)V PC=492
0 4 // 000031aa 0015 0032         -1 java.lang.Thread.exit()V PC=50

The new thing here now is the 6th column of numbers, the one after the wide space and before the class name: It displays the object ID.

The AppClassLoader instance has object ID 357, and it occurs multiple times, indicating that the same instance is used. The object ID now also allows me to match monitorenter and monitorexit instructions more easily. The first two, for example, belong together since they both have object ID 357.

The next two lines with -1 as object ID correspond to the synchronization on the array of String. Then the class loader is invoked again, probably for the Integer class. The next two object IDs of -1 correspond to the Integer array. As stated before, array objects apparently don’t give me object IDs because they somehow don’t overload the $$$getObjectID$$$ method, and therefore the original method in java.lang.Object is executed. Off hand, I don’t even know what class in the rt.jar file is used for array instances. It might be built-in, in which case I can’t do much except try some magic in java.lang.Object‘s default implementation of the method.

The next two lines with zero as object ID corresponds to the synchronization on a class’ Class instance, in this case ObjectIDTest.class. Clearly the $$$getObjectID$$$ method is overridden here because the object ID is not -1. However, the constructor has not been executed and no value has been assigned to the object’s $$$objectID$$$ field. 0 is just the default value that Java puts in a long variable. I have written my program so that valid object IDs start at 1. I might be able to write a special implementation of $$$getObjectID$$$ just for java.lang.Class here.

The next two lines with -1 are using a plain java.lang.Object for synchronization, so the default implementation is invoked directly. No surprise here.

The four next lines with zeros occur in the constructors of the java.lang.Integer class, apparently before the object ID has been assigned.

The next four lines finally are the juicy ones that I have wanted: Two nested synchronized blocks, locking two different Integer instances created back to back. You can see that they have object IDs that are neither -1 nor zero, differ by just one and are nested. Nice.

I’ll have to look at how to deal with these special cases of zeros and -1s and also look at Strings again. Maybe I can get it to work now. Maybe there’s even some general workaround that I could put in the java.lang.Object.$$$getObjectID$$$ method. But even if I can’t, this should cover a major portion of the classes programs actually use, and many of the uses that aren’t covered can probably be easily substituted with something that is.

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

The Gap

I’m bad. I guess I have to quickly summarize what I did the last two months

On the DrJava side, I improved multi-screen behavior and discovered and filed an AWT/Swing bug. Supposedly it has been fixed for 6.0, but I haven’t tested that yet. I added two big new features, “Find All” and “Browser History”, brought the user documentation and quickstart guide up to date, and also committed numerous bugfixes from the SourceForge website.

“Find All” allows the user to collect all matches of a “Find” operation in a region tree panel (like bookmarks). The user can maintain an arbitrary number of these “Find All” result panels at the same time. This turns out to be a tremendous help when searching, because typically you discover another thing you need to search for while you’re already searching for something.

The “Browser History” collects all changes of the current document and puts them in a list with a cursor. Using Alt-Left/Right, the user can browser forward and backward, just like in a web browser.

Dr. Nguyen and I turned the Temperature Calculator from our SIGCSE 2006 workshop into an assignment that will probably be used for Comp 202/212 at some point. We also submitted it to the OOPSLA 2006 Educators Symposium for review. Most of the work for this had already been done, but writing the assignment specs, solution and putting the finishing touches on everything nonetheless took two days.

Now that I have the Linux PC that Corky, my advisor, bought for my office almost working (almost: subversion and audio still doesn’t work), I have started to make everything Linux-compatible. Most things already were, just a few path-related issues had to be straightened out. At the same time, I made the launchers a whole lot more flexible: They now support Java properties as variables, so you can write %user.dir% or %java.class.path% and these strings will get replaced with the values of the properties. The launchers also set a few properties themselves, so the name of the log file can be based on the current directory, the name of the rt.jar file and the class, for example: %user.dir%/%rt.file%-%class.name%.log.

I also converted the entire build process to an Ant script. I was just sick of forgetting to compile or re-instrument before running something. The Ant script’s dependencies do that for me now. The Java properties and variables I described above were tremendously helpful here, because now I could write a general XML configuration file and then fill in the actual values from the Ant script. The launchers also do more error checking and highlight invalid entries in red. They even check if the main class to be executed can be found on the class path.

Share
Posted in Concurrent Unit Testing, DrJava, Research | Leave a comment

Print This Post Print This Post  

Paper: Programming for Change

Programming for Change – Nifty Assignment

OOPSLA 2006 Educators Symposium

The development of any piece of software begins with a set of specifications describing in some detail the problems to be solved. Almost invariably, the specifications change during the development of the software. Trying to anticipate all possible changes and write a program that does everything is a futile undertaking. However, it is not unreasonable to expect that programs be written in such a way that an “epsilon” (read “small”) change in the specifications will only necessitate a “delta” (read “manageable”) change in code.

Programming for change is a continual process in which software is designed over many iterations to capture the problem’s essence and express. At the heart of this process is the effort to identify those elements that can vary (variants) and delineate them from those that do not – the invariants. A properly designed software system should strive to decouple the variants from the invariants in order to facilitate the re-use of the invariants and allow modifications to the variants with minimal perturbation to the existing code.

To illustrate the importance of programming for change to students, we guide them through the development of a program that converts temperature measurements. The assignment consists of a series of small exercises, each of which imposes a small change in the requirements and forces appropriate modifications of the code. To promote code re-use, we apply the Janus Principle and require that the programs must be written in a way that supports multiple distinct user interfaces. For certain specification changes, we ask students to identify the variants and the invariants and make appropriate modifications to the code. In several situations, we require the students to modify their code in more than one way and discuss the pros and cons.

Share
Posted in OOP Book, Publications | Leave a comment

Print This Post Print This Post  

The Lab, DrJava and Javadoc

Last Friday we upgraded the equipment in our research group’s lab: We added a second 20″ TFT to each machine, which involved adding a second graphics card, and doubled the RAM from 512 MB to 1 GB. Initially everyone was excited to tear open boxes and take stuff out, but when the real work began and we had to open cases, attach cables, most people quickly disappeared, and Eric Cheng and I were left with monitors, modules, cables, boxes, and styrofoam everywhere. Eric managed to rewrite the Linux display configuration file, something I couldn’t have done, but in the end, it was just I who cleaned up. I need to figure out when it’s time to leave.

We couldn’t work on one of the computers because we didn’t have the right keys, and were confused by the memory in another machine, so on Monday I tried to finish the job. Five of the six machines are now upgraded. I still didn’t get into the sixth. We also have to open all of them up again because we got the wrong graphics cards, and replacement cards should be here some time this week.

While I waited for our IT guy to come back from a meeting so I could get the keys, I started working on the last “minor” feature that I had in mind to introduce: A quick way to open up Java API javadocs for a class. I used the predictive input dialog again and added all links on the allclasses-frame.html document. By default, this action is bound to Shift-F6. With Ctrl-F6, DrJava takes the word under the cursor and tries to use it as Java API class name. If there’s a unique match, the Javadoc webpage is opened immediately; otherwise a window with candidates is displayed.

Today I renamed a few actions to make their intentions clearer and provided tooltips for some of the actions that I had forgotten to add.

Share
Posted in DrJava, Uncategorized | Leave a comment

Print This Post Print This Post  

Even Better Notion of Program End

I found another problem with the way I determined if a program had ended: Sometimes a program was deemed dead even though it was still alive. This was because I sent the “thread started” sync point just before Thread.start() was exited, but after calling Thread.start0(), the native method that actually starts the thread. That made it possible for the thread to start executing before the recorder was aware of it being alive. If another thread died before the “thread start” sync point reached the recorder, the count count erroneously drop to zero. I fixed that by sending the “thread start” sync point just before the call to Thread.start0().

To do that, I had to refactor the AInsertBeforeReturnStrategy again. Now it wasn’t inserting before a return all the time, insertion was governed by another predicate — now there are three predicates, one each to decide if it’s the right class, right method, and right opcode.

For the opcode predicate, I needed more than one parameter, though, just the InstructionList wasn’t enough, I needed at least the ClassFile with the constant pool as well. The good old unary ILambda wasn’t going to do it anymore.

Unfortunately, writing a generic N-ary lambda isn’t all that easy. Now I added different versions: ILambda<R,P> is still a unary lambda, ILambda.Binary<R,P,Q> is a binary lambda, ILambda.Ternary<R,P,Q,S> is a ternary lambda, and ILambda.Nary<R,P> is an N-ary lambda that uses variable argument list and that therefore is restricted to having parameters of the same type.

Writing these lambdas was fun, the generics just looked pretty, so I wrote decorators for the lambdas that turn a binary lambda to a unary lambda by binding a constant to one of its parameters (ILambda.Binary.Bind1st<R,P,Q> is a unary lambda that binds a constant value to the first parameter of a binary lambda). Ternary lambdas can be turned into binary or unary lambdas that way (ILambda.Ternary.Bind3rd<R,P,Q,S> is a binary lambda that binds a constant to the third parameter of a ternary lambda; ILambda.Ternary.Bind1st3rd<R,P,Q,S> is a unary lambda binding constants to the first and third parameter of a ternary lambda). There are also adaptors to turn N-ary lambdas into unary, binary or ternary lambdas, and vice versa.

For N-ary lambdas, ILambda.Nary.Bind<R,P> is an (N-1)-ary lambda that binds a constant to an index-specified parameter of an N-ary lambda. To avoid thick wrapping, ILambda.Nary.BindK<R,P> is an (N-K)-ary lambda that binds constants to K index-specified parameters of an N-ary lambda.

For predicates and some operations dealing with Comparables, I have already created lambdas: there are different flavors of And, Or, Xor, Less, Max, and so on…

All of that was just an enjoyable detour, but it allowed me to refactor the AInsertBeforeReturnStrategy into a very general AInsertBeforeOpcodeStrategy very nicely.

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

Better Notion of Program End

When I started my tests with several threads the last time, I noticed that sync points in a spawned secondary thread were commented out and treated as sync points “during VM termination”. It was clear why: I declared that VM termination began as soon as the main method was exited.

That hardly corresponds to the end of most real programs, particularly GUI programs. I am now instrumenting Thread.start and Thread.exit again to keep track of the number of live user threads. Since thread 0, the main thread, never explicitly gets created, the count starts at 1 and gets incremented at every Thread.start and decremented at every Thread.exit. Leaving Thread.exit triggers a push of the sync points to make sure they are received. If the count of live user threads reaches 0 on leaving Thread.exit, immediate transfers are enabled.

Regardless of the number of threads spawned, all synchronization points after entering the main method are now labelled as active except for exactly the eight synchronization points that occur in the ominous “DestroyJavaVM”: The thread ID of that thread varies, of course, it’s not always 9, as in earlier tests, but the sequence is always the same: “1 tid; 2 tid; 1 tid; 1 tid; 2 tid; 1 tid; 2 tid; 2 tid”.

// 1 18 // 000030af 000b 0005 java.lang.Shutdown.shutdown()V PC=5
// 2 18 // 000030af 000b 004e java.lang.Shutdown.shutdown()V PC=78
// 1 18 // 000030af 000b 006e java.lang.Shutdown.shutdown()V PC=110
// 1 18 // 000030af 0009 0005 java.lang.Shutdown.sequence()V PC=5
// 2 18 // 000030af 0009 0047 java.lang.Shutdown.sequence()V PC=71
// 1 18 // 000030af 0009 006a java.lang.Shutdown.sequence()V PC=106
// 2 18 // 000030af 0009 0098 java.lang.Shutdown.sequence()V PC=152
// 2 18 // 000030af 000b 0097 java.lang.Shutdown.shutdown()V PC=151

Thread.start and Thread.exit are logged as sync points 4 and 5. I haven’t decided if I’ll treat them as comments and skip them during replay, or if I’ll enforce their existence. Probably the latter.

I haven’t yet added more parts of the replay algorithm back, but at least I’m on my way.

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

Probably Not My Last DrJava Commit

Ok, so that certainly wasn’t my last DrJava commit. On the following Monday, 4/24, I found a few bugs that I fixed: The color of breakpoint and bookmark highlights wasn’t updated when the color was changed in the preferences, and breakpoint color wasn’t toggled when a breakpoint was enabled or disabled. The right OptionListeners just weren’t installed.

On Wednesday I found another bug while I was preparing for a presentation: When the user goes to a bookmark in a yet unseen document, the cursor is not placed at the right offset. For documents that have been seen yet, it worked. The problem was that the DefinitionsPane hadn’t been created yet, so all I had to do was defer setting the cursor using invokeLater.

I also implemented a crufty version of the clipboard history: Whenever something is placed in the clipboard, it is added to a list of the latest N entries. The user can then bring up that list and select from it, putting the selected entry back into the clipboard, moving it to the front of the list, and pasting it into the document.

On Thursday I compiled a list of sections in the DrJava user documentation that need to be updated. I’m still hoping for volunteers.

While grading programming projects on Friday, I noticed and diagnosed a problem with “Open Folder Recursively”. Corky, however, fixed it before I had a chance to commit my fix. His fix turned out to be less work anyway.

On Tuesday I also held my last Comp 212 lab and on Wednesday my last scheduled Comp 212 office hours. After the grading from Friday, now I only have one project milestone and the final exam left to grade. I’m feeling a weird mixture of relief and sadness. Comp 212 has been a part of my life since Fall 2001… That’s a long time. I’m sure it will still be a part of my life to some degree, and I’m also convinced the course will do fine without me (even though this year I’m not impressed with the other TAs), but I will definitely be less involved. After this, I have one semester of teaching requirement left, and I will TA Comp 311 again, a task that I have found more stimulating lately anyway. It is time for a change.

On Friday and on Saturday, I tried to figure out the cause of another bug that we had noticed, but failed: On some tablet PCs, DrJava hangs and does not close correctly if the application is closed with a newly created project open. On my tablet, resetting the interactions pane even on a quit solved the problem, but on others it didn’t. This bug remains a mystery, but it seems to be a concurrency problem.

Today (or rather, yesterday, on Sunday) I tried to stimulate a discussion on one of the DrJava mailing lists as to what goals we have achieved and where DrJava development should go next. Quite frankly, I’m a little disappointed that so many of the little tasks did not get accomplished. On the other hand, several big things on my wishlist got done: relative paths, “Go to File”, improved debugger and saved breakpoints, bookmarks, and even a limited version of code completion. Aside from full-blown code completion and code navigation (which requires light-weight parsing), the biggest things not accomplished are asynchronous finds and the navigation history that includes document switches, finds, and navigation to errors, breakpoints, and bookmarks.

I have already limited my engagement in DrJava, and I’m trying to stick to just correcting bugs that I have introduced or that affect “my” code. I would still like to get the navigation history done, but the biggest, baddest burning for big participation in the DrJava development is fortunately gone. This semester has made it so much more usable.

Share
Posted in DrJava, Uncategorized | Leave a comment

Print This Post Print This Post  

My Last DrJava Commits?

In the realm of DrJava, I have added a new error handling dialog. The first time an error occurs, a small notification box pops up with “More Information” and “Close” buttons. There is also a “Keep showing this notification” checkbox that allows the user to completely disable the notification.

In addition to the notification box, a red “DrJava Errors” button appears in the toolbar. That button or the “More Information” button lead to a dialog that lets the user browse through all exceptions that have occurred since the last reset, to copy stack traces and environment information, and to dismiss all errors that have occurred.

The advantage of this new system is that it is less distracting. Many of the errors that do occur are not fatal to the execution of DrJava, and the user can quite easily continue, though perhaps with limited functionality. More importantly, serious errors in the event thread cannot, as in the past, bury the user under more and more error dialogs, thus effectively deadlocking the program. I have also fixed a good dozen bugs. My DrJava involvement should come to an end for this semester, but I’m not so sure it will… DrJava has become a guilty pleasure of mine.

Share
Posted in DrJava | Leave a comment

Print This Post Print This Post  

Wrong Insertion Method

I finally found the problem why the boolean $$$oldThread$$$ flag wasn’t sticking: I seemed to set it to false, but the next time I looked at it, it was true again.

The Java code basically is this:

public static synchronized void compactWait(long code, long tid) {
if (!_replaying) {
return;
}

monitorEnter(_waitAlgoObj);

if (isOldThread()) {
// If this is NOT the first sync point for this thread...
...
}
else {
// Otherwise, if it was the first time... set flag.
setOldThread();
...
}

// More code...
...

monitorEnter(), isOldThread() and setOldThread() are dummy marker methods and get replaced with bytecode during instrumentation:

  • monitorEnter() becomes just a monitorenter
  • isOldThread() becomes invokestatic java/lang/Thread.currentThread()Ljava/lang/Thread;
    getfield java/lang/Thread.$$$oldThread$$$Z
  • setOldThread() becomes invokestatic java/lang/Thread.currentThread()Ljava/lang/Thread;
    iconst_1;
    putfield java/lang/Thread.$$$oldThread$$$Z

The Java code above roughly corresponds to this bytecode:

getstatic edu/rice/cs/cunit/SyncPointBuffer._replayingZ
ifne 4
return
getstatic edu/rice/cs/cunit/SyncPointBuffer._waitAlgoObjLjava/lang/Object;
invokestatic edu/rice/cs/cunit/SyncPointBuffer.monitorEnter(Ljava/lang/Object;)V
invokestatic edu/rice/cs/cunit/SyncPointBuffer.isOldThread()Z
ifeq 13

// If this is NOT the first sync point for this thread...
...
goto 17

// Otherwise, if it was the first time... set flag.
invokestatic edu/rice/cs/cunit/SyncPointBuffer.setOldThread()V
...

// More code...
...

My faulty instrumentation of the marker methods, however, generated this bytecode:

getstatic edu/rice/cs/cunit/SyncPointBuffer._replayingZ
ifne 4
return
getstatic edu/rice/cs/cunit/SyncPointBuffer._waitAlgoObjLjava/lang/Object;
monitorEnter
invokestatic java/lang/Thread.currentThread()Ljava/lang/Thread;
getfield java/lang/Thread.$$$oldThread$$$Z
ifeq 18

// If this is NOT the first sync point for this thread...
...
goto 20

// Otherwise, if it was the first time... set flag.
invokestatic java/lang/Thread.currentThread()Ljava/lang/Thread;
iconst_1;
putfield java/lang/Thread.$$$oldThread$$$Z
...

// More code...
...

I was inserting the correct bytecode, but I was modifying call targets it in a way such that calls to the insertion location would go to the opcode just PAST the inserted code, whereas it should have gone right TO the inserted code. Now I can move on and face new, exciting problems ;-)

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

Shame on Me

I’ve been avoiding posting to the blog, because that would tell me exactly when the last time was that I’ve really done any work on my research project. It’s been 10 weeks. I really didn’t think it was that much… My estimate would have been six. I guess the whole month around the SIGCSE conference kind of disappeared from my memory — the 212 exam, the conference, catching up afterwards. I was very busy — I did a tremendous amound of work on DrJava — but that’s still no excuse.

On DrJava, I’ve been rewriting the “Create Jar” dialog to work when both class and source files were supposed to be archived, to save its state, and to clean up the interface. After that, I added a “Go to File” dialog that narrows the list of file names down as you type. I also use this dialog for some primitive code completion that can auto-complete all file and class names. It completely ignores the context, but it can be helpful anyway.

After that, I realized that recent changes rendered the debugger pretty much unusable: Because the interactions pane is now being reset before the main document is run — which involves turning off the debugger — and because traditionally all breakpoints were lost when the debugger was turned off, you couldn’t really do any debugging. I made breakpoints and watches persistent, they’re actually saved in the project. Breakpoints can even be disabled without removing them. And if a breakpoint is on a non-executable line, it will be disabled with a warning message. Unfortunately, because the code to detect that is very slow, I only do that if the document is shorter than 500 lines.

In the last few days, I fixed a lot of bugs since we’re gearing up to make a new beta release. I changed find/replace from a recursive implementation to an iterative one — when searching within the DrJava codebase, we were actually running out of stack or heap space.

I’ve been insanely productive. Unfortunately, in the grand scheme of things, I’m just working on the wrong project. For about two weeks now I’ve been saying “I’m going to do this one thing, and then I’m done with DrJava”. But there are always more things to do, and quite frankly I enjoy working on DrJava a lot. This is the first time in a while that I just get to code away. Development work is so much easier than research.

That “one thing” that I still have on my list now is a change of the way DrJava handles exceptions. The exceptions dialog is too obtrusive — quite often, the program can gracefully continue, so it’s not necessary to thrust the exception in the user’s face. Sometimes, the dialog can even completely lock down the program. I intend to turn the exceptions dialog into a log that the user can browse if so desired.

After that, I really need to focus on my research again. Otherwise I will have constructively wasted a whole semester. If I remember correctly, I was trying to implement the algorithm I proved correct using SPIN in Java, and I was running into a problem with setting a boolean flag to true… it just wouldn’t stick, and I have no idea why.

Share
Posted in DrJava, Research | Leave a comment

Print This Post Print This Post  

Workshop: Object-Oriented Design Festival

Object-Oriented Design Festival

SIGCSE 2006

Object-oriented (OO) programming begins with analysis and design that produce a model describing the objects in the problem domain,
their relationships, creation and interactions. The workshop covers fundamentals of OO analysis and design such as abstraction, separation of variants from invariants and decoupling of system components, via appropriate applications of composition, inheritance, polymorphism, and design patterns. The workshop will progress from a small design example illustrating the principles to a larger design problem to be solved by small teams of participants. Their solutions will be discussed in terms of design goals and compared against a solution provided by the presenters.

Share
Posted in OOP Book, Publications | Leave a comment

Print This Post Print This Post  

Lots of DrJava

It’s really been two weeks since I’ve posted here. It’s also been two weeks since I’ve worked on my concurrent unit testing project. Scary.

Of course, it’s not that I’ve done nothing, far from it. The first week Corky and I started working on adding unit tests for the Javadoc component, but then got sidetracked by a request from University of Washington. They requested that the current directory (the “working directory”, basically new File(".")) should be configurable, something I’ve requested numerous times before, particularly in connection with the Rice MBS.

On Wednesday of that week, we got the initial sketch for that change done, at least outside of project mode. For projects, the working directory should be configurable on a per-project basis and stored with the project file. Doing that took a few more days. Corky did most of the work.

When Corky submitted his changes at the beginning of the second week, I noticed that they were failing on my Linux configuration, but only there. I looked at what was happening, and finally on Wednesday realized that the changes we made don’t work if the working directory was set to a location that didn’t exist anymore. I made the necessary changes.

On Thursday, I wanted to write a unit test for this condition (because technically, this was a bug, so there should be a unit test exhibiting it), but found that the majority of the unit tests for DrJava were set up incorrectly. The setUp() and tearDown() methods didn’t call super.setUp() and super.tearDown(), so there was no one place where I could put a line of code that should be executed for all of the tests. With a lot of busy hand work, I changed that and did a massive commit on Friday.

I guess now I’ve been promoted to an “experienced person” in the DrJava team, so Corky and I won’t be working as a pair anymore, now Tim and I will handle small projects together. We’ve started to look at adding a “Go to file” dialog that does some sort of prediction based on what the user inputs.

Share
Posted in DrJava | Leave a comment

Print This Post Print This Post  

Added Instrumentor for Inlining monitorEnter, monitorExit, isOldThread, and setOldThread

Now I’ve written an instrumentor, MarkerInlineStrategy that inlines the respective bytecode snippets for monitorEnter, monitorExit, isOldThread, and setOldThread. I didn’t limit it to just SyncPointBuffer, instead I’m currently employing it with a ConditionalStrategy as decorator that only invokes the decoree if the class to be instrumented happens to be SyncPointBuffer.

I also did a lot of house cleaning. I removed the parameters of type ChangeCallTargetStrategy and CreateDecoratorStrategy from instrumentors that can be invoked from FileInstrumentor‘s command line interface, and now I also allow zeroary constructors, even though the constructor accepting a MethodHashTableStrategy is still favored.

I improved the @DoNotInstrument annotation. It is now possible to specify which instrumentation strategies should not be run. The default still is that no instrumentation is performed at all:

@DoNotInstrument
public class Foo {
...
}

However, the SyncPointBuffer class now has to be instrumented by the MarkerInlineStrategy, but may not be instrumented by any of the *Synchronized*, ObjectCallStrategy, or AssignObjectIDStrategy instrumentors. I specify that this way:

@DoNotInstrument(instrumentors =
"\*\*\*.\*Synchronized\*;\*\*\*.ObjectCallStrategy;\*\*\*.AssignObjectIDStrategy")
public class Foo {
...
}

The string is a semicolon-separated list of class name patterns for ClassFileTools.classNameMatches().

Furthermore, I’ve just split the “method index” long transmitted in the CompactSynchronizedBlockDebugStrategy in a “method index & program counter” value. The method index occupies the upper 32 bit, the program counter where the monitorenter or monitorexit instruction was found resides in the lower 32 bit. This fits very well with the JVM’s own limitations. Now it’s possible to exactly pinpoint a monitorenter or monitorexit in the modified class files. I have to note that this leads to quite a bit more “bloating”, since every pair of method index-program counter that occurs in a class file will now have to be put in the constant pool (i.e. 0x00010000 and 0x00010001 for a monitorenter; monitorexit; return method instead of just 0x00000001), but since this is just for debugging, that’s probably acceptable.

I’m running a few tests now to make sure that the recording stage still works after these relatively extensive modifications, then I have to focus on the replay stage again.

One major item still missing is the compact treatment of wait, sleep, notify, and notifyAll. That might cause some more headache when it comes to the replay algorithm.

Update

I also need to write an instrumentor that inserts a call to SyncPointBuffer.compactThreadExit at the end of java.lang.Thread.exit.

Update

Done.

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

Ported Back to Java

I’ve ported the Promela algorithm back to Java. The unit tests that I have outside the system all pass, but of course that doesn’t mean anything without a unit testing framework that takes concurrency into account.

Now I’m putting the algorithm into the system, but to make that work, I first have to write the instrumentation strategy that replaces invokestatic instructions to monitorEnter, monitorExit, isOldThread, and setOldThread with inlined bytecode, as previously explained.

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

Synchronization During Java VM Initialization and Termination

For my 590 research class, Corky asked me to write a short report about the synchronization points during Java VM initialization and termination. That’s what I worked on last weekend. It’s available on my research page now: Synchronization During Java VM Initialization and Termination.

Since then, I’ve had to deal with an upset stomach and with moving the DrJava site from Sourceforge to CS.net. Sourceforge is limiting our quota to 100 MB, and we were using over 600 MB. I was able to cut it down to approximately 180 MB, but without sacrificing the Javadocs and Klover, we can’t get below 100 MB. So we moved.

On the concurrent unit testing project, I’ve tried to somehow record the sync points during termination in a better way. That’s taken care off now: When the main method is exited, I enable a flag to transfer every sync point immediately. There’s one caveat: With an empty main method, Java never enters it, so it’s never exited. That means that for a program with an empty main method, no sync points during termination are recorded. I can live with that, though.

I’ve also renamed a number of important classes: The former edu.rice.cs.cunit.tm package (tm stands for “Thread Monitor”) is now called edu.rice.cs.cunit.record. The old class ThreadMonJPDA in that package is now called Record. That fits better with the edu.rice.cs.cunit.replay package and the Replay class.

I’ve been wanting to port the verified Promela algorithm back to Java, but after throwing up multiple times, I’ve been too weak for that.

Share
Posted in Concurrent Unit Testing, DrJava, Graduate School, xajavac | Leave a comment

Print This Post Print This Post  

Kooprey Release

I’ve compiled my source code again to make sure it works, improved the user interface a bit, and written a readme file, but here’s the first public release of Kooprey, our generator for object-oriented parsers.

There’s still a lot to be done, especially in the area of error detection. If you have any suggestions, please let me know. I’ll try to accommodate.

There’s also no license as of yet. Something in the spirit of the GPL is what I have in mind, basically fair use. If it’s useful in an educational way, go have a ball. I would really appreciate it if you kept me up-to-date, though, both about good and bad aspects of it.

Share
Posted in Research | Leave a comment

Print This Post Print This Post  

Message Channels

I picked up my search for a way to correctly implement notifyAll in Promela again, and just before I had to teach Comp 212 lab, I found out about message chanels.

Channels provide two operations, channel!value and channel?variable, that send a value into a and receive a value from a channel, respectively. This is exactly what I needed! And I don’t even care about the contents, I just care about blocking. Both block until the value is received or sent, though, so I have to send exactly the right number of times, i.e. not at all if no thread is waiting.

Now I was able to implement primitives that very closely mimic the Java synchronization operations:

typedef WAITOBJ {
bool used = 0;
int lock = 0;
int count = 0;
chan signal = [0] of {bit}
}

#define monitorenter(obj) \
atomic { \
if \
:: (obj.lock==0) -> obj.lock = (_pid + 1); \
:: (obj.lock==_pid+1) -> skip; \
fi; \
}

#define monitorexit(obj) \
obj.lock = 0;

#define wait(obj) \
atomic { \
monitorexit(obj); \
obj.count = obj.count + 1; \
obj.signal?0; \
monitorenter(obj); \
}

#define notify(obj) \
atomic { \
if \
:: (obj.count > 0) -> { \
obj.signal!0; \
obj.count = obj.count - 1; \
} \
:: else -> skip; \
fi; \
}

#define notify\_all(obj) \
atomic { \
do \
:: (obj.count > 0) -> { \
obj.signal!0; \
obj.count = obj.count - 1; \
} \
:: else -> break; \
od; \
}

With these macros used in place of the synchronization code I had before, the code passes all four tests I have written, and it looks a whole lot cleaner. I have to admit, though, that the tests are very simple still…

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

Advance Only During First Iteration

After I had got up, I had realized that the problem involving advancing the index was caused by a really stupid mistake. The counter should only ever advanced when the compactWait algorithm is entered, i.e. during the first but not during subsequent iterations of the loop. This fixed the problem. The algorithm now looks like this:

compactWait:

  1. If scheduled replay is enabled…
    1. monitorenter _waitAlgoObj (beginning of main synchronized block)
    2. If this is NOT the first sync point for this thread…
      1. Advance the indices.
    3. Repeat this… (“RETRY” loop)
      1. If the buffer has never been loaded, or if the index is at the end of the buffer…
        1. Load the buffer and reset the indices.
        2. Wake up all threads waiting for a buffer update.
      2. If the current sync point marks the end of the schedule…
        1. Disable scheduled replay.
        2. Wake up all threads waiting for a buffer update.
        3. monitorexit _waitAlgoObj
        4. Break out of the “RETRY” loop.
      3. If there’s a thread waiting for this wait array entry…
        1. Notify it
        2. Set a flag to scan ahead for the current sync point (“SCAN”).
        3. Do not advance indices.
      4. Otherwise…
        1. If the current sync point in the array is the right one…
          1. Do not advance indices.
          2. Allow the thread to exit the “RETRY” loop.
        2. Otherwise…
          1. Set a flag to scan ahead (“SCAN”).
      5. If the “SCAN” flag is set (i.e. either a thread was woken up or the current sync point did not match)…
        1. Look for the sync points in the remaining part of the array.
        2. If it could be found…
          1. Insert a new Object() into corresponding slot of the wait array.
          2. Set that Object() as “wait object” for this method.
          3. Allow the thread to exit the “RETRY” loop.
        3. Otherwise…
          1. Set the new buffer wait object as “wait object” for this method.
          2. Do not allow the thread to exit the “RETRY” loop.
        4. monitorenter waitObj (beginning of wait object synchronized block)
      6. monitorexit _waitAlgoObj (end of main synchronized block)
      7. If a “wait object” has been set for this method…
        1. Call wait() on that object (make sure to continue waiting if interrupted).
        2. monitorexit waitObj (end of wait object synchronized block)
        3. Do not advance indices.
        4. If the thread is NOT allowed to exit the loop (i.e. it was waiting for a new buffer) AND scheduled replay is enabled…
          1. monitorenter _waitAlgoObj (beginning of main synchronized block for next iteration)
    4. …while the thread is not allowed to exit the loop and scheduled replay is enabled (end “RETRY” loop).

compactThreadExit:

  1. If scheduled replay is enabled…
    1. monitorenter _waitAlgoObj (beginning of main synchronized block)
      1. Advance the indices.
      2. If the index is at the end of the buffer…
        1. Load the buffer and reset the indices.
        2. Wake up all threads waiting for a buffer update.
      3. Otherwise…
        1. monitorenter _waitArray[index] (beginning of wait object synchronized block)
        2. If there’s a thread waiting for this wait array entry…
          1. Notify it.
        3. Otherwise, if the current sync point marks the end of the schedule…
          1. Disable scheduled replay.
          2. Wake up all threads waiting for a buffer update.
        4. Otherwise…
          1. Do nothing, because there’s still a thread awake that will reach this sync point.
        5. monitorexit _waitArray[index] (end of wait object synchronized block)
    2. monitorexit _waitAlgoObj (end of main synchronized block)

This let three of the four simple tests I’ve written pass, but a more complicated one failed. I tracked the error down and realized that I currently don’t guarantee that all threads are actually woken up when a new buffer is loaded. Thread A and B were waiting for a new buffer, thread C loads it and notifies A and B. Threads A wakes up, but thread B doesn’t. Threads A and C run, and one of them requests another new buffer. Thread B hasn’t woken up yet, so it waits for the next buffer as well. If there happens to be a sync point for B in the current buffer, then the program has deadlocked.

I’ve tried several attempts so somehow make sure that all of them are woken up, like using a signal bit that is enabled and that wakes the threads up, which then decrement the number of threads waiting one by one, and all threads involved in this operation wait until that number has reached zero. However, I couldn’t get this to work.

I can probably use arrays and one entry per thread or some kind of ring buffer, but that would complicate the code a lot and obscure the important details of the algorithm. There has to be an easier solution in Promela that allows me to ascertain this. This problem is really annoying, because I have the feeling that it’s not actually a problem with my algorithm, but just a lack of skill in mapping the algorithm from Java to Promela.

I’m tabling this for the day, though, to eat pizza and bum around. It’s MLK day, after all. Wait… that’s not politically correct!

Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post  

After a Buffer Reload

After a buffer reload, the thread that caused the reload is awake (thread A), as well as all threads that were woken up. The thread that caused the reload has the lock, so it will go first. The order of the other threads is nondeterministic, though. Additionally, if thread A finds a sync point match, it is allowed to execute and reenter compactWait, so the number of threads that can run might actually stay the same and not decrease.

In some cases, I’ve found that a thread that has been woken up after a buffer update advances the indices, but it really shouldn’t. In others, though, they clearly need to be advanced. I haven’t yet found the rule that governs when the indices have to be changed and when not.

Here’s an example. Assume the schedule contains these events, where dashes mark a buffer reload:

1 1
1 2
1 3
1 1
---
1 1
1 2
1 3
1 1

Further assume that the threads initially run in the order 3, 2, 1. This is what happens:

  1. 3 enters, does not advance. Waits at index 2…
  2. 2 enters, does not advance. Waits at index 1…
  3. 1 enters, does not advance. Executes.
  4. 1 enters, advances (index=1). Wake up 2. Waits at 3…
  5. …2 wakes up, executes.
  6. 2 enters, advances (index=2). Wake up 3. Waits for new buffer…
  7. …3 wakes up, executes.
  8. 3 enters, advances (index=3). Wake up 1. Waits for new buffer…
  9. …1 wakes up, executes.
  10. 1 enters, advances (index=4). Reload buffer, reset indices (index=0). Executes.
  11. Note: Threads 1, 2, and 3 are all running nondeterministically.
  12. …2 wakes up after reload, advances (index=1). Executes.
  13. Note: Still, threads 1, 2, and 3 are all running nondeterministically.
  14. 1 enters, advances (index=2). Waits at 3…
  15. Note: Threads 2 and 3 still running nondeterministically.
  16. …3 wakes up after reload, advances (index=3). Index 2 was skipped! Wake up 1. Waits for new buffer…
  17. …1 wakes up, executes. Assertions violated, sync point “1 1” executed, but “1 3” was expected.
  18. The confusing thing is that in step 12, thread 2 clearly has to advance the indices, yet in step 16, thread 3 clearly may not advance them. It has something to do with whether the previous thread executed immediately or not (for thread 2, the previous thread did; for thread 3, it did not).

    I guess I’ll try to sleep over it again.

    Share
Posted in Concurrent Unit Testing | Leave a comment

Print This Post Print This Post