Climbing the Python hill

Finally I have started working on new year's resolution. As a part of it, I'm learning a dynamic language called Python.

There are several reasons why I'm a new Python convert. One of them is not because blogosphere keeps intimidating Java programmers that they are writing crap. And I've honest sympathy for those who hold the assumption that Java is dead and other strangling naive opinions like that.

I am rather going ahead with some mature opinion like programming in a language with different philosophy gives you another way of solving the same problem, And sometimes the other way is better or at least, it will bring out interesting techniques which can be useful in general. Since I have very limited experience writing sizable software using them, I am expecting a lot from dynamically typed languages, Python in particular.

Python has been particularly interesting to me because I am curious about it since academics. I've played with it in past but that was just a stab that didn't bleed my nerve. I have no hard feelings for Ruby (or "hoopla oriented" Rails), because I'm not a huge fan of Web applications - web applications are boring.

Python is actually fun to program, programming in Python is like kidding, no it's not as frightening as it's name implies. For example, Methods in Java you know? they can't be moved around with out puzzlers and pain. But in Python, methods are also treated as types, called function types, which you move around like variables - no wait believe me it's as simple as passing around variables. Look at the closure example below and tell me how difficult it is to understand?

x = lambda: a,b: a**b

x(10,12)

Like you do in a regular mathematical equation, assign an expression to a variable or symbol and bind that symbol with value as shorthand like this.

a = x(10,2) * x(6, 8) / x(10, 2)

Also, any method which is not using 'lambda; keyword is also treated in same way, no partiality. Other obvious thing to like about Python is - it forces correct indentation in source code, that makes Python program much more readable in a consistent style. No need for one thousand formatters and coding standards.

I'm not comparing Java with Python, that's not my Job. I'm just trying to realize that how apt it is to solve the same problem aptly in a different way.

Python community, like Python itself, looks great. It isn't impossible to realize that - hangout on Python google group sometime and you will know. I am glad that they are very accepting and responsive to the n00bs like me.

I have started with SQLAlchemy ORM and db api in general to learn python because I'm curious on how they play with tuples and all. Later on I'll write a bit of parser as I've heard Python has some neat and funky stuff for parsing.

And oh, I'm having little trouble getting out of Java shoes lately - hmm, there is nothing like private fields or methods. But I guess that's better than writing getter and setter for that private variable which does nothing but to add overhead for dynamic compiler of hotspot VM to inline it after 3000 iterations, gosh!! But hey, Python is missing the package convention or let me say it prefers configuration over convention here, __init__.py is not as intuitive, is it?

Closed development instance in Eclipse ecosystem

I've been reading Bjorn's posts on open source and open development. I can't disagree with his view on open source being more than just making code available. Getting project participation, in greater capacities than just mailing lists, from community is a good thing, and that should definitely be taken into consideration with proper feedback. The very purpose of being "open" is defeated when a project starts ignoring community contribution without sensible reasons.

I've reason to write this, although I'm not strictly against closely developed open source projects.

I've been working with a modeling toolkit for quite sometime and had customized it to suit my requirements. The contribution was in the core part of the tool (GEF based clipboard, motility and other ergonomic support for better modeling experience). I sent a mail to project admin asking if they would be interested in taking this contribution, after unknown number of days I got the reply asking for contribution review. I sent it as plug-in source code, not as patch, listing changes in the packages and configs. Most of the features offered in the contribution are actually in the project road-map as enhancement list.

But, after more than two months now, I'm still without reply(positive or negative), without comments, without acknowledgment. Now, I've almost lost interest in contributing it.

I hope those required features will be implemented (in anyway project team wishes) in next major release of the project, but I can only hope because the project is not really being developed openly. How sad...

Don't get me wrong. I've nothing against the tool, the tool is excellent and remarkably well designed and implemented; Nor with the team developing it, it's their job. But they should definitely mention somewhere that they are not open to external contributions in development.

Escape Analysis in Mustang - II

EDIT: As commented by Brian, these Escape Analysis optimizations were dropped before final release and deferred to Java 7. The debug version of VM can no longer be found at http://download.java.net/download/jdk6/6u1/promoted/b03/binaries/jdk-6u1-ea-bin-b03-windows-i586-debug-19_jan_2007.jar

In my previous post, I mentioned that escape analysis is available in mustang releases and debug flags available for it in HotSpot(tm) VM.

I ran some micro benchmarks yesterday, they are fairly trivial but good enough to identify the performance gains, and here is what I found:

import java.io.ByteArrayOutputStream;
import java.io.OutputStream;
import java.io.PrintStream;

public class TestEA {
   public static class MyPrintStream extends PrintStream {
       public MyPrintStream(OutputStream out) {
           super(out);
       }
      
       @Override
       public void println(int cnt) {/* deep inline candidate */
       }
      
   }
  
   private static final int COUNT = 10000000;
  
   public static void main(String[] args) throws Exception {
       System.setOut(new MyPrintStream(new ByteArrayOutputStream()));
       for (int i = 0; i <= 10; i++)
           test();
      
   }
  
   private static void test() {
       int cnt = 0;
       Object objLock = new Object();
       long start = System.currentTimeMillis();
       for (int i = 0; i < COUNT; i++) {
           synchronized (objLock) {
               cnt++;
           }
           consume(cnt);
       }
       long end = System.currentTimeMillis();
       System.err.println(cnt + ", time=" + (end - start) / 1000.0);
   }
  
   private static void consume(int cnt) {
       System.out.println(cnt);
   }
  
} 

The results
VMArgs: -server

10000000, time=0.907
10000000, time=0.938
10000000, time=0.969
10000000, time=0.906
10000000, time=0.906

With
VMArgs: -server -XX+DoEscapeAnalysis
10000000, time=0.89
10000000, time=0.922
10000000, time=0.016
10000000, time=0.016
10000000, time=0.015
10000000, time=0.016

JVMOut:
28 JavaObject NoEscape [[ 54F]] 28 Allocate ....
40 LocalVar NoEscape [[ 28P]] 40 Proj ....
90 LocalVar NoEscape [[ 28P]] 90 Phi ....
213 JavaObject NoEscape [[]] 213 Allocate ....
225 LocalVar NoEscape [[ 213P]] 225 Proj ....
======== Connection graph for TestEscapeAnalysis::test ....
172 JavaObject NoEscape [[]] 172 Allocate ....
184 LocalVar NoEscape [[ 172P]] 184 Proj ....

I ran the loop in main for few times because server VM does a deep inline and it might optimize the execution considering cnt is not significantly used anywhere.

Well, It can be seen that escape analysis successfully eliminated synchronization on objLock (lock elision) . As I posted earlier, synchronization has significant impact on execution speed, elimination of this heavy operation improved the speed substantially. Consider the effect of it on a highly concurrent web server (JAWS) handling hundreds of simultaneous requests. Of course, it happened because objLock is allocated locally and VM identified that it can't be shared between multiple threads and its safe to remove the sync overhead.

I also tried making objLock a static field of the class and found that escape analysis has no positive impact on execution speed as can be seen here:

VMArgs: -server -XX:+DoEscapeAnalysis (objLock as static field in above program)
10000000, time=0.922
10000000, time=0.922
10000000, time=0.906
10000000, time=0.891
10000000, time=0.921

As expected, objLock being a shared field, VM has no way of identifying whether lock on it can be eliminated or not.

Unfortunately, stack allocation seems to be absent in current builds of mustang (or there were no hints in debug output as when it was done. Also, its hard to decipher the VM output and I found no explanation for it). Here is an excellent presentation on new optimization in HotSpot(tm) VM. And for lock elimination enhancements refer to this.

So, there you go, one more optimization for the managed runtime; for those who are still in the illusion that native static compiler optimized programs are the fastest, think again, your programs are static at runtime can't organize itself, managed runtime is metamorphic, it can adapt and substantially optimize itself at runtime. Man, that's what I call programming.

Escape Analysis in Mustang - I

EDIT: As commented by Brian, these Escape Analysis optimizations were dropped before final release and deferred to Java 7. The debug version of VM can no longer be found at http://download.java.net/download/jdk6/6u1/promoted/b03/binaries/jdk-6u1-ea-bin-b03-windows-i586-debug-19_jan_2007.jar. Production VM here means production release type, as compared to debug, not the final release.

Java 6 brings a lot of improvements in (much ignored) synchronization; like lock elision, adaptive locking etc., under, the awaited, escape analysis hood. I've been reading quite a bit on stack allocation and lock detection techniques to reduce GC overhead. (as stack comes for free; and sometimes, on full speed core).

Escape analysis is one of the interesting techniques in compiler optimization, read more here.

Few days back, I posted an off-topic curiosity on lang.java google group, It's not officially mentioned anywhere that escape analysis is implemented in mustang releases. Brian Goetz wrote an article stating EA is available in mustang builds, that article is pretty old and someone questioned whether it's really available or not...

A web search revealed that escape analysis *is* available in mustang (not well marketed?), and production VM binary (I've b-03) supports it out of the box in server mode; debug binaries supports few more switches to see the analysis. I'm trying to benchmark it with few programs playing around large number of object creation, not a big deal.

Well, if you are interested, try these switches with mustang (in server mode)
-XX:+PrintEscapeAnalysis,
-XX:+PrintOptoAssembly,
-XX:+DoEscapeAnalysis (works on production VM only).

May be this weekend, I'll post the results of the benchmarks and other findings on it.
Java's performance is getting better and better..

The Pune Eclipse Developer Group

Sriram and Ketan initiated The Pune Eclipse Developers' group to promote collaboration between eclipse developers in Pune. I came to know about good strength working on eclipse plug-ins and related technologies in Pune, its good idea to get them together, share ideas-techniques and socialize with like minded people around town.

Recently at GNUnify '07 Ketan and Sriram spoke about Eclipse Rich Client Application Development. Eclipse Enthusiast can find the presentation here and workshop code here. The presentation is interesting, explains Eclipse architecture; frameworks around it; benefits etc.

I wish Pune Eclipse Developers' group existed before few months, I would have found few more good guys for my team :).