Luckily, we are getting help. The London Java Community has done a great job of pushing along information about how to do builds and providing very constructive and timely feedback on these topics. And there’s a developer from SAP named Volker Simonis who has done a fantastic job at documenting his own experiences building OpenJDK on a number of platforms. There are lots of stars in the community, but we’re always looking for more!

I’m very excited to say that as of today I’m a Mozillian! I’ll be working with the fine folks on the WebAPI team. This is an entirely new set of technologies for me so I’ll be learning a tonne. I’m looking forward to meeting and working with all the excellent people in the Mozilla and broader web communities. I’ll be working out of the beautiful Toronto office.

As for Eclipse and Fedora stuff, I’m really looking forward to the release of Linux Tools 1.0 and the rest of Juno in June, 2012. Before then, Fedora 17 will come out and is going to be an outstanding release, especially from the Eclipse point of view. As I’ll be consuming Eclipse tools for my new work, I’ll be sure to file and fix bugs and spread the good word about the power of Eclipse for C++ developers. When I have some good HOWTO content for developing Mozilla stuff with Eclipse, I’ll post it here and be sure to tag it so Planet Eclipse picks it up.

With apologies to Neil Young: keep on rockin’ in the Free and Open Source world!

The IcedTea project provides a harness to build the source code from OpenJDK6 using Free Software build tools, along with additional features such as a PulseAudio sound driver and support for alternative virtual machines.

A new set of releases is now available for IcedTea6, which uses OpenJDK6 as its base:

• IcedTea6 1.10.7 (based on OpenJDK6 b22)
• IcedTea6 1.11.2 (based on OpenJDK6 b24)

Full details of each release can be found below.

PLEASE NOTE: With this release, the 1.9 series is now NO LONGER SUPPORTED, in a general push to reduce the release support burden to two releases per JDK release (6, 7, 8). We strongly recommend that you upgrade to a new release series; either of the two above for OpenJDK6. Alternatively, make the jump to OpenJDK7 with 2.0.1, 2.1.0 or 2.2.0 (to be released shortly).

What’s New?

New in release 1.10.7 (2012-05-11)

• Fixed build with GCC 4.7
• Bug fixes
  • PR732: Use xsltproc for bootstrap xslt in place of Xerces/Xalan
  • PR881: Sign tests (wsse.policy.basic) failures with OpenJDK6
  • Specify both source and target in IT_GET_DTDTYPE_CHECK.
  • PR758: [regression] javah from 6hg/b23 generates `jlong’ from `private int’
  • Install nss.cfg into j2re-image too.
• Backports
  • S6792400: Avoid loading of Normalizer resources for simple uses
  • S7103224: collision between __LEAF define in interfaceSupport.hpp and /usr/include/sys/cdefs.h with gcc
  • S7140882: Don’t return booleans from methods returning pointers

New in release 1.11.2 (2012-05-11)

• Bug fixes
  • RH789154: javac error messages no longer contain the full path to the offending file:
  • PR797: Compiler error message does not display entire file name and path
  • PR881: Sign tests (wsse.policy.basic) failures with OpenJDK6
  • PR886: 6-1.11.1 fails to build CACAO on ppc
  • Specify both source and target in IT_GET_DTDTYPE_CHECK.
  • Install nss.cfg into j2re-image too.
  • PR584: Don’t use shared Eden in incremental mode.
• Backports
  • S6792400: Avoid loading of Normalizer resources for simple uses

The tarballs can be downloaded from:

• http://icedtea.classpath.org/download/source/icedtea6-1.10.7.tar.gz (signature)
• http://icedtea.classpath.org/download/source/icedtea6-1.11.2.tar.gz (signature)

SHA256 checksums

• aefa76073ee5c44b9173554d8c50610a394f3123df40b0060ad796fb6d36788e icedtea6-1.10.7.tar.gz
• 078637dc8323951b18cbb2aac56fd2f24baaa81f0757391aaa17e1b7577e9ce5 icedtea6-1.11.2.tar.gz

Each tarball is accompanied by a digital signature (links above). This is produced using my public key.

• PGP Key: 248BDC07 (http://keys.indymedia.org)
• Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07

The following people helped with these releases:

• Deepak Bhole (GCC 4.7 fixes, including S7140882)
• Danesh Dadachanji (backport of PR732)
• Andrew Haley (fix for PR584)
• Andrew John Hughes (all other fixes and release management)
• Denis Lila (fix for PR758)
• Omair Majid (backport of S7103224, patch reviews)
• Xerxes Ranby (review of CACAO/JamVM arch fix)
• Pavel Tisnovsky (backport ofS6792400, fix for PR881 & RH789154))

We would also like to thank the bug reporters and testers!

To get started:

$ tar xzf <tarball name>
$ cd <tarball name minus .tar.gz suffix>

Full build requirements and instructions are in INSTALL:

$ ./configure [--with-parallel-jobs[=x] --enable-pulse-java --enable-systemtap ...]
$ make

IKVM.NET has always had a class granularity JIT. Whenever a type is first "used" the CLR fires the AppDomain.TypeResolve event and at that point the IKVM.NET runtime compiles the Java bytecode to CIL for all of the methods in the class.

I don't recall my exact thought process, but I assume that when I started on IKVM.NET I looked at MethodRental.SwapMethodBody and was scared away by the lack of documentation and the fact that it requires full trust and manually constructing a method body blob.

Later on, I focussed more on static compilation and didn't care too deeply about dynamic performance. So I never revisited this decision. However, recently I have been thinking about dynamic performance, triggered by potential invokedynamic optimizations.

To get reacquainted with MethodRental.SwapMethodBody I wrote a small program that dynamically creates the following class:

class Frob {

               public Frob(int i) { Console.WriteLine(i); } public static void M(int i) { Console.WriteLine();
               Console.WriteLine(new Frob(i)); } }

The code is available here: MethodRentalDemo.cs

When the constructor and the M method are first created, the method body is defined, using MethodBuilder.CreateMethodBody, as a simple trampoline that calls Program.JIT to just-in-time generate the actual CIL for the method.

Here's the managed JIT trampoline CIL code for Frob.M:

ldtoken    method void Frob::M(int32))
call       void Program::JIT(valuetype System.RuntimeMethodHandle)
jmp        void Frob::M(int32)

The jmp instruction is interesting, it transfers control to a method that takes the same arguments as the current method and passes the current argument values. Here it is used to jump to a new version the same method, where the method body has been replaced with the actual CIL code.

The native code that is generated for the trampoline is:

push   ebp 
mov    ebp,esp 
sub    esp,8 
xor    eax,eax 
mov    dword ptr [ebp-8],eax 
mov    dword ptr [ebp-4],ecx 
lea    ecx,[ebp-8] 
mov    edx,6231A0h 
call   680065F0 
lea    eax,[ebp-8] 
push   dword ptr [eax] 
call   FFE39B70 
mov    ecx,dword ptr [ebp-4] 
mov    esp,ebp 
pop    ebp 
jmp    dword ptr ds:[006231A8h] 

push   rbx 
sub    rsp,20h 
mov    ebx,ecx 
lea    rcx,[00257330h] 
call   FFFFFFFFF35036D0 
mov    rcx,rax 
call   00000000001C84C0 
mov    ecx,ebx 
lea    rax,[00247D80h] 
add    rsp,20h 
pop    rbx 
jmp    rax
add
               rsp,20h pop rbx ret

(Note that the x64 JIT generates three unreachable instructions at the end. Shown in gray.)

When this code is invoked, it loads the method handle and calls the Program.JIT method. After that returns, it invokes the new method body that the JIT method installed using MethodRental.SwapMethodBody.

When run on the CLR* this all appears to work as you'd hope, but unfortunately that's no guarantee that it will work in the real world. Googling (and experience) suggests that there aren't many** users of MethodRental.SwapMethodBody, so it is quite possible that there are some showstoppers lurking somewhere.

* It does not work on Mono at the moment.
** I found one reference to it in a paper on RubySharp from 2004.

• First version released by GAP
• Interface redone in Gorm
• Extensive fixes to BSD tar support
• Bug fixes in the handling of archiver outputs, options and dates
• Portability fixes and crash fixes
• Updated infrastructure to current GNUstep make and runtime
• Macintosh port (to prove portability)

Were it grounded in reality, Oracle’s claim that copyright law gives them proprietary control over any software that uses a particular functional API would be terrible for free software and programmers everywhere. It is an unethical and greedy interpretation created with the express purpose of subjugating as many computer users as possible, and is particularly bad in this context because it comes at a time when the sun has barely set on the free software community’s celebration of Java as a language newly suitable for use in the free world. Fortunately, the claim is not yet reality, and we hope Judge Alsup will keep it that way.

John Sullivan, executive director of the Free Software Foundation

Has been a release week for almost all of our projects (no surprisingly, since Fedora 17 is around the corner!), and I'm happy to announce that Thermostat 0.2 "The Dharma Bums" has been released, and is ready for download.

New and noteworthy in this release:

• New GC tool monitoring GC frequency and time spent in GC.
• Basic history mode for saving and offline analysis of monitoring data.
• Command line interface providing one-shot commands as well as shell mode.
• GUI client now displays CPU load information for each process.
  

From a personal point of view, I'm very happy because we had a chance to introduce many design concepts and development techniques (yes, mvc, ttd, scrum, agile, all the funny things), which are not only speeding up development, but also ensure that the code is well tested and well designed and organized.

We went from very minimal to almost complete code coverage in only two months, and the coverage includes UI as well, very neat.

This proves that is possible and cost effective to deploy a full TTD model and that the model scales well after the initial obvious impact, and it turned critical in at least few opportunities were we could just trust the code coverage to backup our refactoring efforts without introducing new bugs. This stuff works, if you ever had any doubt!! :)

Anyway, I'm digressing now. You can read the original announcement here.

A big thanks to the rest of the Thermostat community, Jon, Omair and Roman, which are doing a tremendous work all the time, and to Andrew Haley and Deepak because of all the external testing and great feedback they keep giving us, and because they maintain their expectations all the time very high forcing us to deliver quality code ;)

At Devoxx France 2012, I attended to conference titled Java Caching with Guava presented by Charles Fry.

The Guava Cache library works one level above ConcurrentHashmap but doesn’t offer as much features as Ehcache. Guava caches aren’t pre-loaded by default and are all thread safe. The loading of a cache is done with an implementation of the abstract class CacheLoader.

The build of the cache is done progressively, starting by a call to the static method CacheBuilder.newBuilder(), then chaining calls to methods from class CacheBuilder. This allows to specify its configuration and especially its eviction strategy for values : value accessed least recently with expireAfterAccess(long, TimeUnit), value modified least recently modified with expireAfterWrite(long, TimeUnit) … Besides time goes, eviction of values can be activated by specifying the maximum total weight of the cache with a call to maximumWeight(long), requiring also a call to weigher(Weighter) to indicate how to weigh a cache entry.
The build of the cache terminates by calling build() or build(CacheLoader), which returns a new Cache instance.

Besides classical features, cache statistics can be activated by calling the recordStats() method. Usage of the removalListener(RemovalListener) method allows to know when a value has just been deleted from the cache. By default, this notification is synchronous, which can slow down cache operations. To process notifications asynchronously, you must wrap your RemovalListener by calling RemovalListeners.asynchronous(RemovalListener, Executor). While building the cache, a call to refreshAfterWrite(long, TimeUnit) will schedule an automatic refresh of a value some time after its creation or its last modification. In some cases, it’s more interesting to load all cache values once by overloading the CacheLoader.loadAll(Iterable) method.

Among features, there is Cache.get(K, Callable) to get a value from the cache by specifying how to add it when it’s missing. In this case, a unique call to Callable is done by the cache, even if multiple threads are asking the value for that key. Finally, it’s possible to get a view of the cache as a ConcurrentMap by calling asMap().

Bookmark and Share

More »

Powered by Bookmarkify™

Now that Cacio is finally released and promoted to Maven central repositories, I want to provide you a quick HOWTO about testing user interfaces.

For writing GUI tests, I am using FEST. However, one problem with running GUI tests is that they need to create windows, grab keyboard focus, and do all sorts of interaction with the screen. This has several disadvantages:

• When running on a developer machine, it requires to not touch anything while running the test suite, otherwise tests may fail or get blocked, e.g. because keyboard focus cannot be acquired or a test window suddenly goes into the background when it should not, etc. This can be quite annoying, especially with large test suites.
• When running on a Unix-based continuous integration server, it’s possible to utilize xvfb driver to create a virtual framebuffer, but I still often noticed problems that seem to come from asynchronous behaviour of the X architecture. For example, a test would draw a border in RED, and when checking some pixels to be red, they are not. They would only turn red after some unknown time, when the drawing commands have been processed by the X server and graphics card.
• When running on Windows based CI server, this sort of asynchronous problems don’t occur, but to make up for it, Windows has its own share of problems. First of all, on Windows you need a real screen/graphics card to be present (bad on servers). Even worse, on many Windows servers, you need a user to be logged in, and stay logged in, and the CI server running in that session to be able to access the screen. On other servers, multiple concurrent logons are possible, but not sharing a screen, e.g. when some guy logs into the CI server to do some admin work, it would grab the screen from the CI user, etc. All very complicated and time consuming to setup.

This is where Cacio-tta comes into play. It provides a graphics stack for the Java VM, that is completely independent from the environment. It renders everything into a virtual screen, which is simply a BufferedImage, and is driven solely by java.awt.Robot events. This makes it a perfect fit for GUI testing environments. And using it is very easy. There are two ways do it, and I usually combine them.

1. Include Cacio in your Maven dependencies

Simply add the following in your pom.xml:

<dependency>
  <groupId>net.java.openjdk.cacio</groupId>
  <artifactId>cacio-tta</artifactId>
  <scope>test</scope>
</dependency>

2. Run your test with CacioTestRunner

Add the following annotation to your test class:

@RunWith(CacioTestRunner.class)

If you are using FEST and want to take advantage of its additional capabilities (create screenshots on failure, run tests in EDT, etc), use:

@RunWith(CacioFESTRunner.class)

instead. Those annotations will make your test run in a Cacio-tta virtual screen environment

3. Optionally, run the whole test suite in Cacio

In some cases, it may be necessary to run the whole test suite in Cacio-tta. In order to do so, add the following to your pom:

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.12</version>
      <configuration>
        <systemPropertyVariables>
          <java.awt.headless>false</java.awt.headless>
          <awt.toolkit>net.java.openjdk.cacio.ctc.CTCToolkit</awt.toolkit>
          <java.awt.graphicsenv>net.java.openjdk.cacio.ctc.CTCGraphicsEnvironment</java.awt.graphicsenv>
        </systemPropertyVariables>
      </configuration>
    </plugin>

This makes sure that Cacio is loaded instead of the default toolkit. This may be necessary, if any of your tests load any AWT, Java2d or Swing class, and are not annotated with the above annotation. This is because Java only allows to set the toolkit once, and it cannot be unloaded or unset. When you load any GUI class before loading the CacioTestRunner, the default toolkit will be loaded, and tests will not run in Cacio.

4. For non-Maven based builds

Of course, Cacio can also be used in non-Maven builds (ant, make, ..). Just download the Cacio sources (http://ladybug-studio.com/downloads/cacio/) build them, include the JARs in your project somehow, and add the annotations as described above, or set the system properties awt.toolkit, java.awt.graphicsenv and java.awt.headless as described in the Maven snippet above.

Last but not least, if you encounter any problems with Cacio or the above instructions, let us know!

During Devoxx France 2012, I attended the conference titled CRaSH a shell to extend the Java platform, presented by Julien Viet (his github account).

CRaSH (Common ReusAble SHell) is a shell in the form of a command line interpreter allowing access to the Java virtual machine running it. This shell, under the LGPL open source license, has just been released as version 1.0 and can be downloaded as a zip file or as an installer. It works in standalone mode (like the Bash shell) or in attachment mode (a software agent is attaching itself to a running Java virtual machine).

To take benefit of the hot deploy of new CRaSH commands, you must develop commands with groovy by extending the CRaSHCommand class. It’s also possible to write commands in Java by implementing the ShellCommand interface, but without getting benefit of the hot deploy. A CRaSH command can also be developed as a groovy script. Besides addition of new commands, a plugin mechanism allows to add new features by extending the CRaSHPlugin class.

Among available standard commands, the jdbc command allows to connect to a database through a DataSource defined in a JNDI directory or directly by using a JDBC connection string. It’s then possible to directly execute SQL queries. Among other available commands, there is the thread command to control threads and the log command to manage logs.

By default, you can connect to a CRaSH instance by SSH or by Telnet but it can be disabled. For more informations, have a look at the documentation.

Thanks to CRaSH, you can for example write scripts to test a Swing application by using the java.awt.Robot class.

Bookmark and Share

More »

Powered by Bookmarkify™

This should hopefully be the last snapshot based on OpenJDK 7 FCS as 7u4 has shipped and the bundle should be available soon. After that I'll start work on integrating 7u4 and working towards the IKVM 7.1 release.

Changes:

• Bug fix. When adding certificates to virtual cacerts file make sure that the aliases that are generated from the certificate subject are unique.
• Bug fix. If a class file UTF-8 string ends with an incomplete multi byte char, we should throw the appropriate exception.
• IKVM.Reflection: Added ModuleBuilder.__GetAssemblyToken() API.
• IKVM.Reflection: Removed Mono CryptoConvert code and use .NET 2.0 API instead.
• IKVM.Reflection: Rewrote StrongNameKeyPair to remove dependency on System.Reflection.StrongNameKeyPair.
• IKVM.Reflection: Improved assembly title and description.
• IKVM.Reflection: Metadata is now sorted with a stable sort algorithm.

Binaries available here: ikvmbin-7.1.4507.zip

After 4 years of development we are proud to finally announce the release of Caciocavallo, as Roman noted on his blog, the 1.1 due to issues while uploading the 1.0 version! That is, after 4 years, we even skip the 1.0 milestone!

The release is available in maven repositories, but you can get a copy directly from mercurial or from the ladybug website. It will only work on OpenJDK 7 onward (and I've not tried on closed JDK either). This is for once not a broken thing, Cacio is an OpenJDK plugin, simply as it is...

So many, many memories that go deep inside my hearth and that made a great part of my own career are part of this project!

Perhaps one day I should sum up all those memories and write them down, because the story of the development of a piece of software is inevitably filled up with personal interludes, it's linked with both drama and joy, some mistakes and many glorious days, both personal and professional. We have seen companies going up and down, we've taken part of the Java liberation directly and from the inside of it, and then we've seen Sun falling down to the oblivion and captured (and possibly saved, but at the time this thought was outrageous!) by Oracle.

I still remember the day we heard about the OpenJDK Challenge, it was just a couple of days before FOSDEM 2008, and before its deadline, and as usual we realized we could actually try this out. Then we broke the knife with the cheese, in this famous under exposed and reddish photo I never really fixed:

I told the guys we could never cut it with that small knife, but anyway!

We were mostly kicked out from the pub (was the BXL, but this year I found it closed, sadly... another piece of memory that goes away, float, changes... that's how's life I guess) because they didn't like that we were eating our own stuff in there (despite the fact we were probably like 30 or so, and we were eating and drinking all the day long in there!). We ended up at "La Morte Subit" and finally eat it!

That was a very incredible Fosdem, we presented the cool stuff, with Roman mainly complaining with the fact that it was taking ages to contribute even simple fixes to OpenJDK, and that nobody seemed to even reply our mails! How thing have changed, now they reply all our mails... Ok, it still take ages to get contributions in though ;)

It's with Cacio that I learned, above all, how to be a real developer.

We presented our project just one day or so before deadline finally, and I was amazed to learn how much internal discussion went on about us. I hope I don't disclose too much, but I've learn that actually many people wanted us to do this project, and so we got accepted in the final round.

Actually doing it was a different story, we even got sidetracked so many times by our employer at the time, we barely managed to push the code the very last day (and after a really long weekend of work, which I think matched with my birthday, so I even had a birthday coding party, how geeky!).

I think aicas at first didn't believe in it. After all, we tried something similar in GNU Classpath, and it really didn't work that well... But we learned out of it, and we knew what things we had to avoid in order to make it work this time. 

After we managed (somehow) to finish the first part of the project, we realized that people really expected more from us. The project at the beginning was really just about making the code portable. I wrote the peers with the GNU Classpath code, while Roman did the port with Escher if I remember correctly, and by doing this, we shared a good deal of refactoring that eventually went into the JDK (actually, as of OpenJDK 7, everything is in!).

That meant we had two peers that were not OpenJDK original code and they were working, very exciting! But that wasn't enough. After the code was submitted, people started to file bugs report about the hocked code, not the refactoring work! Obviously, they were testing a proof of concept implementation that served as a test case for the refactoring work, but they wanted the proof of concept to work well. Kind of: "well, yeah, I know this proves that the refactoring works, but you know, this widget is not that nice looking!"

After all, who wanted another full Peer? Maybe me and Roman were happy to reuse GNU Classpath peer code, and show that some other random company could do the same, without depending on SunSurfaceManager and SunGraphicsEnvironment etc... But really, seriously, who was going to do this?

What was at the time the Cacio project repository (a set of patches and a specially crafted OpenJDK tree with those patches applied) became the ng repository. Cacio-ng was the start of an effort to actually make AWT on Swing possible, it didn't belong to the original version as some people think, but it started right after the Challenge ended. It took us a little bit of thinking that we wanted AWT on top of Swing, though, wasn't clear right at the beginning. Then the idea. Why not? Swing is pure java, and in fact, even the XPeer do this Swing magic every now and then, even if it's not as refined and way more messed up (I feel this like a kind of evolution, XPeer/GNU Classpath which is completely different code of course, but has a similar basic ideas, then Cacio which expands on both, and finally the OSX port, which does the same as Cacio, but with more people and more money in it :)

This is how Cacio started, and It has been a long journey, really. But the rest of the story is for another time.

Few people contributed during the year to Cacio, and they really deserve to share the honor with us, quoting in toto from Roman's blog (yes, I even copy-paste a thank you to myself! how lame, is it?!):

• First of all, Sun Microsystems, which sparked the original idea(s) through the OpenJDK Innovators Challenge. And the folks in Sun’s graphics team, Phil Race, Jim Graham, Dmitri Trembovetski, Andrei Dmitriev and a bunch of invisible folks for following and supporting us.
• Mario Torre, my friend and compagnon on this journey. It would not have been possible without you.
• Of course, Clemens Eisserer for Cacio-Web and for revitalizing Cacio.
• Andrei Dmitriev, he supported us a lot by initial testing, reporting issues, running the TCK and some then-Sun-internal test suites over it. And he presented at JavaOne 2009 about Cacio with us.
• Aicas, who provided me and Mario with time and support on the initial OpenJDK Challenge project, and keep contributing to it, in particular:
• Ralf Heini and Ingo Proetel, who keep submitting bug reports and fixes.
• Glen Schrader who joined Cacio just some weeks ago to submit some important bugfixes.
• A whole bunch of individuals who give Cacio a try, who write encouraging words in our blog comments, who lobby their development teams to use Cacio, etc etc.

I am very proud to announce that after 4 years of development, we publish the first release of Caciocavallo. Due to some funny problems with Maven and my network, it is numbered 1.1.

Cacio started out in 2008 as one of ten projects to take on the OpenJDK Innovators Challenge. This is how it all began, at FOSDEM 2008:

There, Rich Sand and Tom Marble told us about the OpenJDK Innovators Challenge, and Mario brought this fine cheese (on which Steph broke the above knife), and thus the idea to do *something* was born. The original idea was to improve OpenJDK’s GUI APIs implementation (AWT, Java2D and Swing) to make it easier to port the stack to new platforms and plug them in as separate modules. As a proof of concept, we provided a very simple implementation, it was based on Escher if I remember correctly. Back then, Cacio was nominated bronze for the challenge, which made us very proud.

Following that, the focus shifted from fixing OpenJDK to actually porting the GUI stack (although merging the patches into JDK7 would keep us busy for a while to come, I have some special memories of the FontManager in particular). At this time, Cacio evolved from a simple proof of concept implementation into a fully fledged framework for porting Java’s graphics stack. The idea was to make it as easy as possible to do it. And because a large part of that porting work would consist of implementing all those AWT widgets that almost nobody’s actually using (and which are not even available native on many interesting platforms), we thought out a little trick: why not let Swing do all the widget magic in AWT? I.e. from the standpoint of the programmer, you would code AWT, but under the hood, Swing would be used for rendering, event handling, etc. The only thing that a Java GUI port would need to provide is an implementation of Java2D (which is fairly easy to do if you don’t shoot for the fastest ever implementation), and some event handling. Using that, I was able to get a working prototype (working as in, can run Java2Demo) of a DirectFB backend working in 1 week!

After I left aicas and joined Sun in June 2009, things got a bit quiet around Cacio for a while. I was still merging back some of the FontManager fixes into OpenJDK7 mainline, but that was basically it. Until Clemens Eisserer (the guy who took the gold award in the OpenJDK Challenge in 2008 with his XRender work) hit the stage with his Google Summer of Code 2011 proposal to implement a Java GUI layer that would render a Swing application running on the server (in the cloud if you will) directly to a browser, using HTML5. What followed was a time of very intense development, and the outcome was (and still is) absolutely remarkable. We could get a number of Swing applications to run ‘in the cloud’ and talking to browsers running on all sorts of platforms without Java installed locally, including iPad/iPhone, Android phones, embedded devices and of course the usual desktop OSes. It was a huge success which sparked new life in Caciocavallo land.

By this time, I already left Sun and worked for JP Morgan, and there I got really hooked up with test driven development and all other sorts of agile practices. One of the problems that we hit there was running GUI tests on continuous integration servers, because they would either need access to a desktop (when running on a Windows server) and get disturbed whenever somebody logged on the machine, or hit problems with X servers (on Unix systems) due to asynchronicity and other funny stuff. At some point I realized that Cacio could help us here as well, and this gave birth to another side project of Cacio, called Cacio-TTA (pronounced Caciotta which is one special type of Caciocavallo in terms of cheese). This is basically a virtual GUI stack, made for running GUI tests in an isolated environment. It would render to a BufferedImage only, and process events solely through java.awt.Robot. Relatively simple to implement (took me 1 evening, thanks to the already existing Cacio framework), and enourmously useful: each GUI test would now run completely independent from any platform GUI stack, be it X server or Windows desktop or whatever. You can run tests in parallel if you like (just spawn enough JVMs), and every test would get its own GUI sandbox. This finally makes Java GUI testing reliable and predictable.

That was a long journey, and now we are here and proudly releasing Caciocavallo 1.1! Let me take this opportunity to say thanks to a couple of companies and individuals:

• First of all, Sun Microsystems, which sparked the original idea(s) through the OpenJDK Innovators Challenge. And the folks in Sun’s graphics team, Phil Race, Jim Graham, Dmitri Trembovetski, Andrei Dmitriev and a bunch of invisible folks for following and supporting us.
• Mario Torre, my friend and compagnon on this journey. It would not have been possible without you.
• Of course, Clemens Eisserer for Cacio-Web and for revitalizing Cacio.
• Andrei Dmitriev, he supported us a lot by initial testing, reporting issues, running the TCK and some then-Sun-internal test suites over it. And he presented at JavaOne 2009 about Cacio with us.
• Aicas, who provided me and Mario with time and support on the initial OpenJDK Challenge project, and keep contributing to it, in particular:
• Ralf Heini and Ingo Proetel, who keep submitting bug reports and fixes.
• Glen Schrader who joined Cacio just some weeks ago to submit some important bugfixes.
• A whole bunch of individuals who give Cacio a try, who write encouraging words in our blog comments, who lobby their development teams to use Cacio, etc etc.

THANK YOU ALL!

Life doesn’t stop here of course. We are currently using and improving Cacio for our Thermostat at Red Hat. And we still have a long list of plans:

• Finally get WebJDK project going. The idea is to build on top of Cacio-Web a distribution of OpenJDK that truly runs ‘in the cloud’ (i.e. on a stupid webserver farm in a boring server room). Swing applications running on WebJDK would talk directly to the client’s browser, would have (limited) access to the client filesystem and printer, will have their session managed properly, it will be scalable and easy to just throw more servers at it should resources become a problem, etc etc. I still have that filesystem support lying around on my harddisk, uncommitted.
• Another working prototype that is uncommitted is the DirectFB port, to run Java on embedded systems, using DirectFB as graphics layer.
• Finish a bunch of unfinished backends, in particular SDL.
• Package Cacio for Fedora 17.
• Plus all sorts of interesting things that we don’t know about yet, but which will surely happen.

In case you are interested in the code (*cough*), you can find source tarballs here. Or in the Cacio Mercurial repository. Or in Maven central (should appear soonish):

<dependency>
<groupId>net.java.openjdk.cacio</groupId>
<artifactId>cacio-tta</artifactId>
<version>1.1</version>
</dependency>

Please notice that Cacio only works on OpenJDK7 and most likely Oracle JDK 7  (no JDK6, no OpenJDK6, OpenJDK8 will be supported once it’s out, no other JVM runtimes like Classpath based ones are supported).

The latest release of Ubuntu, version 12.04 aka Precise, has a lot of updates we’ve been waiting on for a while — GNOME 3.4, Haskell 7.4.1, and a huge stack of bugfixes. On the desktop side, quite a number of Linux kernel vs X video modes vs suspend glitches have gone away. That’s fantastic. During most of Oneiric, my laptop was freezing and needing a hard reset at least once a day. Tedious. So I’m quite pleased to report that running Precise, Linux 3.2, gdm, and GNOME 3.4, things are vastly more stable.

Getting upgraded to Precise, however, has not been a pleasant experience.

First we’ve had unattended-upgrades overwriting any configuration stating “no automatic upgrades”. The number of non-technical friends who were set to “security updates only” calling in wondering why a “big upgrade” happened and now their computers don’t work has been staggering. Needless to say we nuked unattended-upgrades from all of our systems a hurry, but for those people it was already too late.

Several desktop upgrades failed half-way through because dpkg suddenly had unresolved symbol errors. Fortunately I was able to work out the missing library binary and manually copy it in from another machine, which was enough to get package system working. Hardly auspicious.

Server side was fraught with difficulty. You cannot yet upgrade from Lucid to Precise. It breaks horribly.

E: Could not perform immediate configuration on 'python-minimal'. Please
see man 5 apt.conf under APT::Immediate-Configure for details. (2)

Brutal. I tried working around it on one system by manually using dpkg, but that just led me into recursive dependency hell:

# cd /var/cache/apt/archvies
# dpkg -r libc6-i686
# dpkg -i libc6_2.15-0ubuntu10_i386.deb
# dpkg -i libc-bin_2.15-0ubuntu10_i386.deb
# dpkg -i multiarch-support_2.15-0ubuntu10_i386.deb
# dpkg -i xz-utils_5.1.1alpha+20110809-3_i386.deb
# dpkg -i liblzma5_5.1.1alpha+20110809-3_i386.deb
# dpkg -i dpkg_1.16.1.2ubuntu7_i386.deb
# apt-get dist-upgrade

Huh. That actually worked on one system. But not on another. Still slammed into the python-minimal failure. For that machine I couldn’t mess around, so I had to give up and did a re-install from scratch. That’s not always feasible and certainly isn’t desirable; if I wanted to be blowing systems away all the time and re-installing them I’d be running Red Hat.

Anyway, I then located this bug about being unable to upgrade (what the hell kind of QA did these people do before “releasing”?) where, very helpfully, Stefano Rivera suggested a magic incantation that gets you past this:

# apt-get install -o APT::Immediate-Configure=false -f apt python-minimal
# apt-get dist-upgrade

(I had tried something very close to this, but didn’t think of doing both apt and python-minimal. Also, it hadn’t occurred to me to use -f. Ahh. For some reason one always sees apt-get -f install not apt-get -f install whatever-package-name).

Ta-da.

AfC

Two weeks shy of ten years ago, I joined Red Hat as an intern on the Red Hat Database team. I was incredibly lucky to have gotten that internship and then to have been hired on full time 2 years later. I’ve had a great time working at Red Hat but have made the very difficult decision to leave. My last day is tomorrow, Monday April 30th.

Red Hat has provided me with many things: innumerable learning opportunities, career growth, exposure to open source communities, public speaking opportunities at conferences, and more. Most importantly, though, I’ve been given the chance to work with a lot of really amazing people both inside Red Hat and in a variety of vibrant Free Software and Open Source communities.

The Eclipse community has been my home for a few years now. It’s full of some of the most talented and passionate software developers I’ve ever met. I plan to remain involved in the Eclipse world and am looking forward to getting Linux Tools 1.0 released as a part of Juno and what comes after that!

I’ve excited for my new job but have a few weeks of down time in between where I hope to largely be AFK. I’ll write a blog entry here when I start my new job. I’ll still be around the open source world and if you’d like to contact me I’m reachable here on my blog or via various social networks. I wish everyone continued success both personally and professionally. Thanks for everything and I’ll see you around.

Funny to see Tap the Waterdroplet (the GNU Classpath mascot) used in court to explain what Java is:

Tap makes a couple more cameo appearances in the documents. It is a fun read.

Here are some somewhat random musings on building the jdk and various build system observations. It might be observed that some of these may sound like whining, I can assure you that whining is not allowed in my blog, only constructive criticism, it's everyone else that is whining. :^) Apologies for the length.

Build and test of the JDK has multiple dimensions, and I cannot say that I have them all covered here, it is after all, just a blog, and I am not endowed with any supernatural guidance here.

• Continuous Build & Smoke Test
  Every component, every integration area, every junction point or merge point should be constantly built and smoke tested. I use the term smoke test because completely testing a product can take many weeks, and many tests do not lead themselves to being fully and reliably automated. The job of complete testing belongs to the testing organization. Smoke tests should provide a reasonable assurance that a product is not brain dead and had no major flaws to prevent further testing later on down the road. These smoke tests should be solid reliable tests, and any failure of these tests should signify a flaw in the changes to the product and raise major red flags for the individuals or teams that integrated any recent changes. Over the last year or more we have been trying to identify these tests for the jdk and it's not an easy task.
  Everyone cuts corners for the sake of productivity, it's just important to cut those corners with eyes wide open. The ideal would be that every changeset was known to have passed the build and smoke test, the reality is far less than that, but we know where we want to be.
• Build and Test Machines
  The hardware/machine resources for a build and test system is cheap, and a bargain if it keeps all developers shielded from bad changes, finding issues as early in the development process as possible. But it is also true that hardware/machine resources do not manage themselves, so there is also an expense to managing the systems, some of it can be automated but not everything. Virtual machines can provide benefits here, but they also introduce complications.
• Continuous Integration
  Depending on who you talk to, this can mean a variety of things. If it includes building and smoke testing before the changes are integrated, this is a fantastic thing. If people consider this to mean that developers should continuously integrate changes without any verifications whatsoever that the changes work and don't cause regressions, that could be a disaster. Some so called 'Wild West' projects purposely want frequent integrations with little or no build and test verifications. Granted, for a small tight team, going 'Wild West' can work very well, but not when the lives of innocent civilians are at risk. Wild West projects must be contained, all members must agile, wear armor, and be willing to accept the consequences of arbitrary changes sending a projectile through your foot.
• Multiple Platforms
  The JDK is not a pure Java project and builds must be done on a set of different platforms. When I first started working on the JDK, it became obvious to me that this creates a major cost to the project, or any project. Expecting all developers to have access to all types of machines, be experienced with all platforms, and to take the time to build and test manually on all of them is silly, you need some kind of build and test system to help them with this. Building on multiple platforms (OS releases or architectures) is hard to setup, regardless of the CI system used, this is a significant issue that is often underestimated. Typically the CI system wants to try and treat all systems the same and the fact of the matter is, they are not, and somewhere these differences have to be handled very carefully. Pure Linux projects, or pure Windows projects will quickly become tethered to that OS and the various tools on them. Sometimes that tethering is good, sometimes not.
• Multiple Languages
  Again, the JDK is not a pure Java project, many build tools try and focus on one language or set of languages. Building a product that requires multiple languages, where the components are tightly integrated, is difficult. Pure Java projects and pure C/C++ projects have a long list of tools and build assists in creating the resulting binaries. Less so for things like the JDK, where not only do we have C/C++ code in the JVM, but C/C++ code in the various JNI libraries, and C/C++ code in JVM agents (very customized). The GNU make tool is great for native code, the Ant tool is great for small Java projects, but there aren't many that work well in all cases.
  Picking the right tools for the JDK build is not a simple selection.
• Multiple Compilers
  Using different C/C++ compilers requires a developer to be well aware of the limitations of all the compilers, and to some degree if the Java code is also being compiled by different Java compilers, the same awareness is needed. This is one of the reasons that builds and tests on all platforms is so important and also why changing compilers, even just new versions of the same compiler can make people paranoid.
• Partial Builds
  With the JDK we have a history of doing what we call partial builds. The hotspot team rarely builds the entire jdk, but instead just builds hotspot (because that is the only thing they changed) and then places their hotspot in a vetted jdk image that was built by the Release Engineering team at the last build promotion. Dito for the jdk teams that don't work on hotspot, they rarely build hotspot. This was and still is considered a developer optimization, but is really only possible because of the way the JVM interfaces to the rest of the jdk, it rarely changes. To some degree, successful partial builds can indicate that the changes have not created an interface issue and can be considered somewhat 'compatible'.
  These partial builds create issues when there are changes in both hotspot and the rest of the jdk, where both changes need to be integrated at the same time, or more likely, in a particular order, e.g. hotspot integrates a new extern interface, later the jdk team integrates a change that uses or requires that interface, ideally after the hotspot changes have been integrated into a promoted build so everyone's partial builds have a chance of working.
  The partial builds came about mostly because of build time, but also because of the time and space needed to hold all the sources of parts of the product you never really needed. I also think there is a comfort effect by a developer not having to even see the sources to everything he or she doesn't care about. I'm not convinced that the space and time of getting the sources is that significant anymore, although I'm sure I would get arguments on that. The build speed could also become less of an issue as the new build infrastructure speeds up building and makes incremental builds work properly. But stay tuned on this subject, partial builds are not going away, but it's clear that life would be less complicated without them.
• Build Flavors
  Similar to many native code projects we can build a product, or a debug, or a fastdebug version. My term for these is build flavors. My goal in the past is to make sure that the build process stays the same, and it's just the flavor that changes. Just like ice cream. ;^) (fastdebug == -O -g + asserts).
• Plug and Play
  Relates to build flavors, it has been my feeling that regardless of the build flavor, the API's should not change. This allows for someone to take an existing product build, replace a few libraries with their debug versions, and run tests that will run with the best performance possible, and be able to debug in the area of interest. This cannot happen if the debug or fastdebug versions have different APIs, like MSVCRTD.DLL and MSVCRT.DLL.
• Mercurial
  Probably applies to Git or any distributed Source Code Management system too.
  Face it, DSCM's are different. They provide some extremely powerful abilities over single repository model SCM's, but they also create unique issues. The CI systems typically want to treat these SCM systems just like SVN or CVS, in my opinion that is a mistake. I don't have any golden answers here, but anyone that has or does work with a distributed SCM, will struggle with CI systems that treat Mercurial like subversion.
  The CI systems are not the only ones. Many tools seem to have this concept of a single repository that holds all the changes, when in reality with a DSCM, the changes can be anywhere, and may or may not become part of any master repository.
• Nested Repositories
  Not many projects have cut up the sources like the OpenJDK. There were multiple reasons for it, but it often creates issues for tools that either don't understand the concept of nested repositories, or just cannot handle them. It is not clear at this time how this will change in the future, but I doubt they will go away. But it has been observed by many that the lack of bookkeeping with regards to the state of all the repositories can be an issue. The build promotion tags may not be enough to track how all the repository changeset states line up with built together.
• Managing Build and Test Dependencies
  Some build and test dependencies are just packages or products installed on a system, I've often called those "system dependencies". But many are just tarballs or zip bundles that needs to be placed somewhere and referred to. In my opinion, this is a mess, we need better organization here. Yeah yeah, I know someone will suggest Maven or Ivy, but it may not be that easy.
  We will be trying to address this better in the future, no detailed plans yet, but we must fix this and fix it soon.
• Resolved Bugs and Changesets
  Having a quick connection between a resolved bug and the actual changes that fixed it is so extremely helpful that you cannot be without this. The connection needs to be both ways too. It may be possible to do this completely in the DSCM (Mercurial hooks), but in any case it is really critical to have that easy path between changes and bug reports. And if the build and test system has any kind of archival capability, also to that job data.
• Automated Testing
  Some tests cannot be automated, some tests should not be automated, some automated tests should never be run as smoke tests, some smoke tests should never have been used as smoke tests, some tests can seriously mess up automation and even the system being used, ... No matter what, automating testing is not easy. You cannot treat testing like building, it has unique differences that cannot be ignored. If you want the test runs to be of the most benefit to a developer, you cannot stop on the first failure, you need to find all the failures. That failure list may be the evidence that links the failures to the change causing the failures, e.g. only tests using -server fail, or only tests on X64 systems fail, etc. At the same time, it is critical to drive home the fact that the smoke tests "should never fail", it is a slippery slope to start allowing smoke tests to fail. Sometimes, you need hard and fast rules on test failures, the smoke tests are those. If accepting failing smoke tests is a policy, that same policy needs to exclude the failing smoke test for everyone else so that life can go on for everyone else.
  In an automated build and test system, you have to protect yourself from the tests polluting the environment or the system and impacting the testing that follows it. Redirection of the user.home and java.io.tmpdir properties can help, or at least making sure these areas are consistent freom test run to test run. Creating a separate and unique DISPLAY for X11 systems can also protect your test system from being impacted by automated GUI tests that can change settings of the DISPLAY.
• Distributed Builds
  Unless you can guarantee that all systems used are producing the exact same binary bits, in my opinion, distributed builds are unpredictable and therefore unreliable. A developer might be willing to accept this potential risk, but a build and test system cannot, unless it has extremely tight control over the systems in use. It has been my experience that parallel compilations (GNU make -j N) on systems with many CPUs is a much preferred and more reliable way to speed up builds.
  However, if there are logically separate builds or sub-builds that can be distributed to different systems, that makes a great deal of sense. Having the debug, fastdebug, and product builds done on separate machines is a big win. Cutting up the product build can create difficult logistics in terms of pulling it all together into the final build image.
• Distributed Tests
  Having one large testbase, and one large testlist, and requiring one very long testrun that can take multiple hours is not ideal. Generally, you want to make the testbase easy to install anywhere, and create batches of tests, so that using multiple machines of the same OS/arch can allow for the tests to be run in a distributed way. Getting the same testing done in a fraction of the time of running one large batch. If the batches are too small, you spend more time on test setup than running the test. The goal should be to run the smoke tests as fast as possible in the most efficient way possible, and more systems to test with should translate into the tests getting done sooner. This also allows for new smoke test additions as the tests run faster and faster. Unlike distributed building, the testing is not creating the product bits, and even if the various machines used are slightly different, that comes closer to matching the real world anyway. Ideally you would want to test all possible real world configurations, but we all know how impractical that is.
• Killing Builds and Tests
  At some point, you need to be able to kill off a build or test, probably many builds and many tests on many different systems. This can be easy on some systems, and hard with others. Using Virtual Machines or ghosting of disk images provides a chance of just system shutdowns and restarts with a pristine state, but that's not simple logic to get right for all systems.
• Automated System Updates
  Having systems do automatic updates while builds and tests are running is insane. The key to a good build and test system is reliability, you cannot have that if the system you are using is in a constant state of flux. System updates must be contained and done on a schedule that prevents any disturbance to the build and tests going on in the systems. It is completely unacceptable to change the system during a build or test.
• AV Software
  AV software can be extremely disturbing to the performance of a build and test system. It is important, but must be done in a way that preserves the stability and reliability of the build and test system. The dynamic AV scanning is a great invention, but has the potential to disturb build and test processes in very negative ways.

Hopefully this blabbering provided some insights into the world of JDK build and test.

-kto

I unintentionally posted this before I verified everything, so once I have verified it all works, I'll updated this post. But this is what should work...

Most Interesting Builder in the World:

"I don't always build the jdk, but when I do, I prefer The New JDK8 Build Infrastructure. Stay built, my friends."

So the new Build Infrastructure changes have been integrated into the jdk8/build forest along side the older Makefiles (newer in makefiles/ and older ones in make/). The default is still the older makefiles.

Instructions can be found in the Build-Infra Project User Guide. The Build-Infra project's goal is to create the fastest build possible and correct many of the build issues we have been carrying around for years. I cannot take credit for much of this work, and wish to recognize the people who do so much work on this (and will probably still do more), see the New Build Infrastructure Changeset for a list of these talented and hard working JDK engineers. A big "THANK YOU" from me.

Of course, every OS and system is different, and the focus has been on Linux X64 to start, Ubuntu 11.10 X64 in particular. So there are at least a base set of system packages you need. I decided to use Ubuntu 12.04, hot off the press, you should run the following after getting into a root permissions situation (e.g. have run "sudo bash"):

1. apt-get install aptitude
2. aptitude update
3. apt-get purge openjdk-6*
4. aptitude install mercurial openjdk-7-jdk rpm ssh expect tcsh csh ksh gawk g++ ccache build-essential lesstif2-dev

Then get the jdk8/build sources:

1. hg clone http://hg.openjdk.java.net/jdk8/build jdk8-build
2. cd jdk8-build
3. sh ./get_source.sh

Then do your build:

1. chmod a+x common/bin/*
2. cd common/makefiles
3. bash ../autoconf/configure
4. make

We still have lots to do, but this is a tremendous start.

-kto

Work on JDK 8 is well-underway, but we thought this late-breaking JEP for another language change for the platform couldn't wait another day before being published.

Summary

Provide the benefits of the time-testing goto control structure to Java programs. The Java language has a history of adding new control structures over time, the assert statement in 1.4, the enhanced for-loop in 1.5,and try-with-resources in 7. Having support for goto is long-overdue and simple to implement since the JVM already has goto instructions.

Success Metrics

The goto statement will allow inefficient and verbose recursive algorithms and explicit loops to be replaced with more compact code.

The effort will be a success if at least twenty five percent of the JDK's explicit loops are replaced with goto's. Coordination with IDE vendors is expected to help facilitate this goal.

Motivation

The goto construct offers numerous benefits to the Java platform, from increased expressiveness, to more compact code, to providing new programming paradigms to appeal to a broader demographic.

In JDK 8, there is a renewed focus on using the Java platform on embedded devices with more modest resources than desktop or server environments. In such contexts, static and dynamic memory footprint is a concern. One significant component of footprint is the code attribute of class files and certain classes of important algorithms can be expressed more compactly using goto than using other constructs, saving footprint. For example, to implement state machines recursively, some parties have asked for the JVM to support tail calls, that is, to perform a complex transformation with security implications to turn a method call into a goto. Such complicated machinery should not be assumed for an embedded context. A better solution is just to expose to the programmer the desired functionality, goto.

The web has familiarized users with a model of traversing links among different HTML pages in a free-form fashion with some state being maintained on the side, such as login credentials, to effect behavior. This is exactly the programming model of goto and code. While in the past this has been derided as leading to "spaghetti code," spaghetti is a tasty and nutritious meal for programmers, unlike quiche.

The invokedynamic instruction added by JSR 292 exposes the JVM's linkage operation to programmers. This is a low-level operation that can be leveraged by sophisticated programmers. Likewise, goto is a also a low-level operation that should not be hidden from programmers who can use more efficient idioms.

Some may object that goto was consciously excluded from the original design of Java as one of the removed feature from C and C++. However, the designers of the Java programming languages have revisited these removals before. The enum construct was also left out only to be added in JDK 5 and multiple inheritance was left out, only to be added back by the virtual extension method methods of Project Lambda.

As a living language, the needs of the growing Java community today should be used to judge what features are needed in the platform tomorrow; the language should not be forever bound by the decisions of the past.

Description

From its initial version, the JVM has had two instructions for unconditional transfer of control within a method, goto (0xa7) and goto_w (0xc8). The goto_w instruction is used for larger jumps. All versions of the Java language have supported labeled statements; however, only the break and continue statements were able to specify a particular label as a target with the onerous restriction that the label must be lexically enclosing.

The grammar addition for the goto statement is:

GotoStatement:
    goto Identifier ;

The new goto statement similar to break except that the target label can be anywhere inside the method and the identifier is mandatory. The compiler simply translates the goto statement into one of the JVM goto instructions targeting the right offset in the method. Therefore, adding the goto statement to the platform is only a small effort since existing compiler and JVM functionality is reused.

Other language changes to support goto include obvious updates to definite assignment analysis, reachability analysis, and exception analysis.

Possible future extensions include a computed goto as found in gcc, which would replace the identifier in the goto statement with an expression having the type of a label.

Testing

Since goto will be implemented using largely existing facilities, only light levels of testing are needed.

Impact

• Compatibility: Since goto is already a keyword, there are no source compatibility implications.
• Performance/scalability: Performance will improve with more compact code. JVMs already need to handle irreducible flow graphs since goto is a VM instruction.

I’ve been meaning to write some introductory articles to GDB ever since back in February at FOSDEM, where I was surprised by just how many people do not use GDB–or even know what it is! It’s taken me a while to figure out a nice example, but I finally found them.

Last week I wanted to extend ogg123, a tool for playing music files on the commandline. It supports several formats–Ogg Vorbis, FLAC and Speex–and the way it abstracts that is by passing around a struct format_t full of function pointers to various callbacks. I was adding a new format, and wanted to know what kind of arguments it was calling its callbacks with. I could have tried to figure it out by reading the code, but it was easier to just run it in GDB and see what was happening. Here’s how:

1. Build a copy of ogg123 with debugging support:

   wget http://downloads.xiph.org/releases/vorbis/vorbis-tools-1.4.0.tar.gz
   tar xf vorbis-tools-1.4.0.tar.gz
   cd vorbis-tools-1.4.0
   CFLAGS="-g -O0" ./configure
   make

   The important bit here is the CFLAGS="-g -O0". This causes configure to pass these extra options to GCC in the makefiles it generates. -g instructs GCC to include debugging information in the files it generates to allow GDB to relate the executable files you’re debugging to the source files they were generated from. -O0 instructs GCC not to optimise the code, which makes for easier debugging.

2. Start up GDB on the ogg123 you just built:

   gdb -args ogg123/ogg123 ~/music/Various/Disco\ Demands/4-08\ -\ Give\ It\ Up.flac

   Everything after the -args option is the command you’d normally type to run the program. GDB will print out some stuff as it starts up, then present you with a (gdb) prompt and wait for you to enter some commands:

   GNU gdb (GDB) 7.4.50.20120313-cvs
   Copyright (C) 2012 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later 
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
   and "show warranty" for details.
   This GDB was configured as "x86_64-unknown-linux-gnu".
   For bug reporting instructions, please see:
   <http://www.gnu.org/software/gdb/bugs/>...
   Reading symbols from /home/gary/vorbis-tools/ogg123/ogg123...done.
   (gdb) 

3. Ok, the callback I’m interested in is the “read” callback, and we’re playing a FLAC file so the callback is called flac_read. I’d like to run the program until flac_read is called, so I set a breakpoint on flac_read by entering break flac_read and then using the run command to set ogg123 going:

   (gdb) break flac_read
   Breakpoint 1 at 0x40dde5: file flac_format.c, line 253.
   (gdb) run
   Starting program: /home/gary/vorbis-tools/ogg123/ogg123 /home/gary/music/Various/Disco\ Demands/4-08\ -\ Give\ It\ Up.flac
   [Thread debugging using libthread_db enabled]
   Using host libthread_db library "/lib64/libthread_db.so.1".
   [New Thread 0x7fffeb863700 (LWP 10312)]
   [Thread 0x7fffeb863700 (LWP 10312) exited]
   
   Audio Device:   PulseAudio Output
   
   Playing: /home/gary/music/Various/Disco Demands/4-08 - Give It Up.flac
   
   Breakpoint 1, flac_read (decoder=0x637560, ptr=0x6182a0, nbytes=30240,
       eos=0x7fffffffe37c, audio_fmt=0x7fffffffe380) at flac_format.c:253
   253	  flac_private_t *priv = decoder->private;
   (gdb)

4. GDB now has a running copy of ogg123 stopped at the very start of flac_read. Pretty jazzy huh? You can see the values of the arguments in the second-to-last line it printed, so for example you can see that the caller has requested to read 30240 bytes of data. The various structures (decoder=0x637560 etc) aren’t so useful as they’re pointers, but we can deference them with the print command:

   (gdb) print *decoder
   $1 = {source = 0x637060, request_fmt = {big_endian = 0, word_size = 2,
       signed_sample = 1, rate = 0, channels = 0, matrix = 0x0}, actual_fmt = {
       big_endian = 0, word_size = 2, signed_sample = 1, rate = 0, channels = 0,
       matrix = 0x0}, format = 0x617ec0, callbacks = 0x7fffffffe3a0,
     callback_arg = 0x0, private = 0x6375d0}

   That’s pretty nice, we can see for example what format it wants the data in: little endian, signed, etc. Also, notice the line of code it stopped on:

   253	  flac_private_t *priv = decoder->private;

   That’s the next line of code GDB will execute if we set the program going again in some way. We can advance over just that line with the next command:

   (gdb) next
   254	  decoder_callbacks_t *cb = decoder->callbacks;

   Now it’s initialised the local variable priv, which we can also print out:

   (gdb) p *priv
   $2 = {decoder = 0x637000, is_oggflac = 0, channels = 2, rate = 44100,
     bits_per_sample = 16, totalsamples = 13337016, currentsample = 0,
     samples_decoded = 4096, samples_decoded_previous = 0, bytes_read = 24576,
     bytes_read_previous = 0, comments = 0x638240, bos = 1, eos = 0,
     buf = 0x637020, buf_len = 4096, buf_start = 0, buf_fill = 4096, stats = {
       total_time = 0, current_time = 0, instant_bitrate = 0, avg_bitrate = 0}}

   Did you see what I did there? Most GDB commands have abbreviated forms, and the abbreviation for print is p. The commands I’ve introduced in this article are some of the most commonly used, and their abbreviated forms are all simply their first letter: b for breakpoint, r for run and n for next. You can also repeat the previous command by pressing Return, which is handy for doing a load of next commands one after the other, for instance.

Ok, that’s all for today. Go and have a play!

I have posted a small JogAmp JOGL OpenGL ES 2.0 Vertex and Fragment shader introduction at:
https://github.com/xranby/jogl/blob/master/src/test/com/jogamp/opengl/test/junit/jogl/demos/es2/RawGL2ES2demo.java#L54

The nice thing about this introduction are that it will run unmodified on both Desktop OpenGL GL2 systems and Mobile OpenGL ES2 systems.
It uses the GL2ES2 GLProfile that use the common subset of OpenGL calls found in both desktop GL2 and mobile ES2.

wget http://jogamp.org/deployment/jogamp-test/archive/jogamp-all-platforms.7z
7z x jogamp-all-platforms.7z
cd jogamp-all-platforms
wget https://raw.github.com/xranby/jogl/master/src/test/com/jogamp/opengl/test/junit/jogl/demos/es2/RawGL2ES2demo.java
javac -cp jar/jogl.all.jar:jar/gluegen-rt.jar RawGL2ES2demo.java
java -cp jar/jogl.all.jar:jar/gluegen-rt.jar:. RawGL2ES2demo

There is a new update for jtreg 4.1, b04, available. The primary changes have been to support faster and more reliable test runs, especially for tests in the jdk/ repository.

jtreg can be downloaded from the OpenJDK jtreg page: http://openjdk.java.net/jtreg/.

Scratch directories

On platforms like Windows, if a test leaves a file open when the test is over, that can cause a problem for downstream tests, because the scratch directory cannot be emptied beforehand. This is addressed in agentvm mode by discarding any agents using that scratch directory and starting new agents using a new empty scratch directory. Successive directives use suffices _1, _2, etc. If you see such directories appearing in the work directory, that is an indication that files were left open in the preceding directory in the series.

Locking support

Some tests use shared system resources such as fixed port numbers. This causes a problem when running tests concurrently. So, you can now mark a directory such that all the tests within all such directories will be run sequentially, even if you use -concurrency:N on the command line to run the rest of the tests in parallel. This is seen as a short term solution: it is recommended that tests not use shared system resources whenever possible.

If you are running multiple instances of jtreg on the same machine at the same time, you can use a new option -lock:file to specify a file to be used for file locking; otherwise, the locking will just be within the JVM used to run jtreg.

"autovm mode"

By default, if no options to the contrary are given on the command line, tests will be run in othervm mode. Now, a test suite can be marked so that the default execution mode is "agentvm" mode.

In conjunction with this, you can now mark a directory such that all the tests within that directory will be run in "othervm" mode. Conceptually, this is equivalent to putting /othervm on every appropriate action on every test in that directory and any subdirectories. This is seen as a short term solution: it is recommended tests be adapted to use agentvm mode, or use "@run main/othervm" explicitly.

Info in test result files

The user name and jtreg version info are now stored in the properties near the beginning of the .jtr file.

Build

The makefiles used to build and test jtreg have been reorganized and simplified. jtreg is now using JT Harness version 4.4.

Other

• jtreg provides access to GNOME_DESKTOP_SESSION_ID when set.
• jtreg ensures that shell tests are given an absolute path for the JDK under test.
• jtreg now honors the "first sentence rule" for the description given by @summary.
• jtreg saves the default locale before executing a test in samevm or agentvm mode, and restores it afterwards.

Bug fixes

• jtreg tried to execute a test even if the compilation failed in agentvm mode because of a JVM crash.
• jtreg did not correctly handle the -compilejdk option.

Acknowledgements

Thanks to Alan, Amy, Andrey, Brad, Christine, Dima, Max, Mike, Sherman, Steve and others for their help, suggestions, bug reports and for testing this latest version.

Thanks to great work by Chris Phillips and others, we now have a Zero based OpenJDK7 RPM building on Fedora ARM

Peter Robinson (Fedora project volunteer) then started an icedtea-web build on ARM based on the above, which went fine.

I decided to try icedtea-web with OpenJDK7 on ARM. I only had access to an F15 machine so I had to force install the F17 OpenJDK RPM (it works other than where -lpng would be needed). Here is the result — the IcedTea-Web plug-in running with Midori on Fedora ARM!

IcedTea-Web plug-in working on ARM

Now to wait for Andrew Haley’s JIT work to go in, to make it run faster

We are proud to announce the release of GNU Classpath 0.99!

GNU Classpath, essential libraries for java, is a project to create free core class libraries for use with runtimes, compilers and tools for the java programming language.

The GNU Classpath developer snapshot releases are not directly aimed at the end user but are meant to be integrated into larger development platforms. For example JamVM, CACAO and Kaffe can make use of an
installed copy of GNU Classpath 0.99, while GCC (gcj) will use the developer snapshots as a base for future versions. For more projects based on GNU Classpath, see http://www.gnu.org/software/classpath/stories.html.

This release brings with it a number of interesting new features:

• Addition of java.util.regex.Pattern.quote.
• Addition of java.io.IOError.
• Addition of java.io.Console.

There have also been many bugfixes over the past year. Those relevant to 0.99 can be found in our bug database.

With the 0.95 release, we switched fully towards the 1.5 generics work that we previously released separately as classpath-generics. All this work is now fully integrated in the main release and various runtimes
(gcj, cacao, jamvm, ikvm, etc) have been extended to take advantage of the new generics, annotations and enumeration support in the core library. As a consequence, only 1.5 capable compilers (currently the
Eclipse Compiler for Java (ecj) and Sun’s javac) may be used to build Classpath.

The GNU Classpath developers site provides detailed information on how to start with helping the GNU Classpath project and gives an overview of the core class library
packages currently provided.

For each snapshot release generated documentation is provided through the gjdoc tool, which is part of GNU Classpath 0.99. Full documentation on the currently implementated packages and classes can
be found at: http://developer.classpath.org/doc/ We are looking into how to extend the documentation experience in the future. Please contact the mailinglist if you would like to help with this effort.

For more information about the project see also:

• GNU Classpath home page
• Developer information (wiki)
• Full class documentation
• GNU Classpath hacker blogs
• Autobuilder, current build status, build snapshots
• Application test pages (wiki): applets, AWT apps, Swing apps, SWT apps.
• GNU Classpath hacking with Eclipse (wiki)
• GNU Classpath promotion banners

GNU Classpath 0.99 can be downloaded from ftp://ftp.gnu.org/pub/gnu/classpath/ or one of the ftp.gnu.org mirrors

• File: classpath-0.99.tar.gz
• SHA256sum: f929297f8ae9b613a1a167e231566861893260651d913ad9b6c11933895fecc8

New in release 0.99 (Mar 07, 2012)

• Addition of java.util.regex.Pattern.quote.
• Addition of java.io.IOError.
• Addition of java.io.Console.
• Bug fixes:
  • PR39408: gjavah doesn’t generate constants in header files where they occur in a superclass
  • PR40590: namespace namespace broken in CNI
  • PR40630: java.util.Scanner fails when used for charset generation by the OpenJDK build
  • PR40653: Issue with XML stream writer and namespaces
  • PR40663: Support Stax API 1.0.1
  • PR39177: trunk revision 144128 – jar: internal error: java.lang.NullPointerException
  • PR41696: java.util.zip.Inflater:finished () returns false when it should return true
  • PR43536: CopyOnWriteArrayList bug in delete() when empty
  • PR36560: java.util.zip: Error parsing zip file with larger files in it
  • Restrict access to VM classes.
  • Cleanup use of message resources.
  • Throw exception when encrypted zip file entries are encountered.
  • Fix infinite recursion in javax.print.attribute.standard.JobStateReasons.add.
  • Native code cleanups in GtkToolkit.c and jcl.c.
  • Only log when debugging is on.
  • PR44411: System.nanoTime() is not independent of wall-clock time
  • PR46775: Calling Policy.setPolicy with a new Policy object has no effect on the DefaultSecurityManager
  • Use implementation of VMClass.getSimpleName from gcj.
  • Simplify security determination in ProtectionDomain in situations where all permissions are available.
  • PR42390: Missing Security Manager checks in classpath apis
  • Throw NullPointerExceptions appropriately for compatibility with OpenJDK.
  • Use Integer.parseInt in preference to Integer.decode in java.util.Formatter.
  • Use same default capacity in java.util.HashMap as documented in OpenJDK.
  • Check for hashcode equality before calling equals in java.util.HashMap.put
  • Make sure match is within input data limits in java.util.regex.Matcher.find.
  • Fix misuse of ArrayList.set in javax.swing.text.html.StyleSheet.resolveStyle.
  • PR48131: java.util.zip.ZipException: incomplete dynamic bit lengths tree
  • Check for negative capacity in VMDirectByteBuffer’s native code.
  • PR42823: tcp/ip sockets read/write operations causes memory leak
  • Generate META-INF/INDEX.LST for glibj.zip
  • Fix issues when building with -Werror and gcc 4.6.
  • PR40188: javah creates constants using name of superclass
  • PR45527: gjavah encodes $ as used in inner classes as 00024 where Oracle’s javah does not
  • PR45526: gjavah does not implicitly produce header files for inner classes
  • Fix NullPointerException for null keys in java.util.HashMap.put.
  • Fix BEAST security issue in gnu.javax.net.ssl.provider.
  • RH712013: pdftk crashes with java.lang.ArrayIndexOutOfBoundsException
• Updated to libtool 2.x.
• Lots of warning fixes / addition of generics.
• Fix license headers in tools.
• Normalise whitespace.
• Maintenance work on javac detection.
• Mark plugin as unmaintained and disable by default.

The following people helped with this release:

• Roland Brand
• Gert Brettlecker
• Chris Burdess
• Ludovic Claude
• Pekka Enberg
• Jeroen Frijters
• Richard Guenther
• Andrew Haley
• Andrew John Hughes
• Ivan Maidanski
• Daniel Noll
• Rainer Orth
• Mike Stump
• Tom Tromey
• Mark Wielaard
• Ralf Wildenhues

We would also like to thank the numerous bug reporters and testers! In addition, we’d like to extend our thanks to all those who’ve contributed over the years and have helped in building a thriving and friendly community around the GNU Classpath project.

I just committed a patch that makes GDB able to set breakpoints on inlined functions by name.

Work has commenced on the repeating annotations feature discussed in JEP 120: Repeating Annotations.

First, the meta-annotation type used to declare the containing annotation has been added to java.lang.annotation. Updates to the AnnotatedElement interface will be made later. Next, with feedback from other members of the compiler team, I've written a draft specification of the needed language changes:

If annotation type T is annotated with an annotation a of type ContainerAnnotation then:

• The value of the @ContainerAnnotation annotation is called the containing annotation type TC.
• TC must declare a value() method whose return type is T[], or a compile-time error occurs. (This implies that TC can be the containing annotation type for only a single annotation type.) [Note: Since an annotation type T cannot declare a method that returns T or T[] as a value, that implies that it is a compile time error for an annotation type to specify itself as a container annotation type.]
• TC may declare methods other than value(). Any such methods must have a default clause, or a compile-time errors occurs. [Note: this is a bit weird "action at a distance" in terms of compiler errors. The error message here is given at the site of the T since the containing annotation might only be present as a class file.]

• It is a compile time error if the retention of T is not a subset of the retention of TC. This comparison is made taking default retention into consideration.

  In more detail, if the retention of the TC is SOURCE, the retention of T must be SOURCE. If the retention of TC is CLASS, the retention of T may be either CLASS or SOURCE. if the retention of the TC is RUNTIME, the retention of T may be SOURCE, CLASS, or RUNTIME.

• It is a compile time error if T is @Documented and TC is not @Documented.

  [Note that it is permissible TC to be @Documented and for T to not be @Documented.]

• It is a compile time error if T is @Inherited and TC is not @Inherited.

  [Note that it is permissible TC to be @Inherited and for T to not be @Inherited.]

• It is a compile time error if T has target values that are not included in the target set of TC, taking default target values into account.

Note that an annotation type TC may be a container type for annotation type T while also having its own container type TCC.

[However, two annotation types cannot be containers for each other because that would create an illegal cyclic annotation type situation. For example, the following program is rejected by javac:

public @interface Mutual {
    Omaha[] value() default {};
}

@interface Omaha {
    Mutual[] value() default {};
}

@Mutual({@Omaha, @Omaha})
@Omaha({@Mutual, @Mutual})
class Foo {}

with the message

Mutual.java:2: error: cyclic annotation element type
    Omaha[] value() default {};
            ^
1 error]

"It is a compile-time error if a declaration is annotated with more than one annotation for a given annotation type."

Change to "unless the annotation type is annotated with an @ContainerAnnotation."

Annotation types with an @ContainerAnnotation are known as repeatable annotation types. The presence of an @ContainerAnnotation indicates multiple annotations of that annotation type will be logically replaced with a single annotation synthesized by the compiler. The type of the synthesized annotation, TC, is the type given by the value of the ContainerAnnotation annotation on T. The order of the elements of the value() method of the synthesized TC annotation matches the left to right order of the base repeatable annotations.

If an element only has a single repeatable annotation, the annotation is not replaced.

It is conventional to place the repeatable annotations of a given type contiguously on an element, but this is not required.

It is a compile time error if an element is annotated with both multiple base annotations of a repeatable annotation type T and with one or more annotations of the containing type TC.
[In other words, the collapsing of repeatable annotations to their containers is not recursive.]

IcedTea-Web 1.1.5 is now out. It contains a couple of important bug fixes for those who wish to remain on the 1.1 line:

• Fixes:
• PR820: Firefox 10 and above crashes when LiveConnect is heavily used
• PR838: IcedTea-Web plugin crashes with chrome browser when javascript is executed

Full notes with bug ids are available in the NEWS file:
http://icedtea.classpath.org/hg/release/icedtea-web-1.1/file/ab7e8272d45d/NEWS

Available for download here:
http://icedtea.classpath.org/download/source/icedtea-web-1.1.5.tar.gz

Build instructions are here:
http://icedtea.classpath.org/wiki/IcedTea-Web#Building_IcedTea-Web

SHA256 sum:
ab5c34a9dc6bff48baf1f1d1a34bf54bfb954ad93ee9e7e44d642fa991bcc919 icedtea-web-1.1.5.tar.gz

Thanks to everyone who helped with this release:
Matthias Klose
Denis Lila
Omair Majid
Thomas Meyer
Jiri Vanek