Real Refactoring

A lot of really good material has been written about refactoring. However, just like a lot of other things in software development, refactoring is also victimized by becoming the buzzword implying 'rewrites' and every other thing that is not refactoring. A lot of people really don't understand what they mean when they refer to 'refactoring', this post aims to provide a systematic explanation of refactoring for such audience.

Refactoring is merely a process of restructuring existing code so that its end result and meaning remains intact. However, this restructuring has a great potential to become far better design than the original one. (It is implied that the refactored design is easier to refactor and hence better) In rest of this post I will try to summarize how to approach and refactor a code in real life application development.

1. Get familiar with code

I am assuming here that we are almost always refactoring code written fully or partially by others, refactoring your own code should be much more simple exercise. Primarily you would be dealing with Legacy code or testable code. If you are lucky enough to get the code which is almost covered with unit tests, your job becomes substantially easy and you can start refactoring very fast without a lot of trouble. However if you are working with existing code which just exists and works somehow, you will have to start slowly. The first step is to understand the incomprehensible. There are several ways you can get familiar with code:

• Read the code

Reading code is one of the fundamental activity we developers do most of the time, if any employer is in the illusion that she is paying developers just to write code then they are paying 1/3 of the salary. Reading code is an art which develops over the time, reading good code can make you better developer (with a precondition that one possesses the ability to judge good and bad code for the language). The more you read bad code the more you familiarize yourself with the repeating mistakes in the code (duplication, absolute hostility towards testability, extremely fearful defensive checks and other hilarities). The goal is to grasp the mindset of the original developer(s) who wrote it. You will know it immediately how deeply thought out the code is in a few passes. Use any modern IDEs to navigate through files, ignore the comments which don't make sense.

• Write test

Once you have some understanding of inputs/outputs and/or interactions in code's life cycle you can write some tests just so that you can identify if you made some mistake in next step (Step 2). These tests might not be unit-tests (unless the code itself is unit testable). Writing test is incredibly useful way to get familiar with the code base.

2. Isolate the refactoring hot-spot (or "The Inflection points" as Michael Feathers calls it)

There has to be a really good reason why you are refactoring and it certainly can't be a full 'rewrite'. Irony will kill itself when it finds out that the problems in a bad code manifests itself in popular hot-spots. By isolating these hot-spots you should be able to mock out the uninteresting and untestable (or very slow) dependencies. If you have multiple inflection points; deal with them one by one in isolation. Isolation may become tricky because you may have to introduce some inevitable changes before you actually have any unit tests. For example, you might want to eliminate inherently untestable 'statics', extract methods/interfaces which you can stub with NullObjects/mocks. The tests created in earlier step will be valuable to identify any problems you introduced while isolating the code. The goal here is to not concern yourself with trivialities other than the refactoring hot-spots.

3. Write unit tests

Now that you have code which can be isolated, start writing unit tests. If you are working on Java sources I highly recommend using Mockito to mock the dependencies. Don't waste time on accidental complexities of other mock libraries. A reasonable analogy would be: if Agile software development is a good thing than other mocking tools are worse than Waterfall - Mockito is Agile.

Your primary goal is to cover code under refactoring as much as possible, there are no hard numbers on code/branch coverages (but anything beyond 90% should be good enough :)), use common sense and quality metrics like CRAP. The more unit tests you will write, more you will learn about the code base. There may be totally unused code for anti-YAGNI stronghold, or there might be bugs, Your unit test will reveal these naturally. Depending on the code in question, A good test suit generally reaches the size of the code under test.

4. Refactor

Refactoring is particularly painful when it has to be done in an environment with highly volatile code base and for the parts of code which are critical to overall application functionality. Make small changes; make sure all tests pass.

If you are using Eclipse - turn off auto build and add a ANT or Maven builder so that it runs your tests after each compile - you will be running tests a lot.

Use a real source control system, it might be much harder for you to revert the changes otherwise. You might consider using something like git in between upstream and your local repository if your primary SCM is not good enough. In practice, this becomes a lot more important than what IDE you use, you will need the ability to identify and rollback a commit specifically in code base with many hands. Commit after each change, the smaller the commit the lesser chance of breaking a lot of things together by a long shot.

With each commit going in repository without external complains (like "You broke my build!"), you will gain confidence to make bigger changes. You should continue writing tests while refactoring because if at this point you are making changes without violating functional contracts you are evolving a new design and doing something good. If you have broken something and your justification involves the word 'refactoring', you are doing it wrong! you either don't have enough tests or you failed to understand the code entrails.

So this is refactoring in real life. For the primary audience, please consider following these simple steps before tossing the word 'refactoring' around. If you disagree, then invent the new word and let me know :).

Pain of Sourceless Development for an Open source developer

I'm proudly an open source developer, working on Java, And I just realized I can't live without source code of every framework or library I work with. I know many of you are like me, we know we speak code, we write it, we hear it and we communicate in it. Not having source means we are impaired.

Before it happened to me, Neither could I imagine nor perceive myself working with a piece of software without source code, It was so common to have code before, and hey shouldn't that be included in programmer's bill of rights? But, First time in last few years, luck had it and I am working with a closed propriety framework, its painful for all. If you're in the great world of open-source software, you might have started thinking how painful it can be to program without code.

Not having source code means lot of things, debugging becomes nightmare which not only makes you feel stupid to debug but also it frustrates as you feel back in blind environments like the one in M$'. If there's a bug you have no way to look what's going on in the library or framework, you would wait till their (software vendor's) "support team" identifies or guesses what's the problem and gives a patch, which would take like weeks.

In Java, You can try decompiling the binaries, it's easy with Java unless binaries are obfuscated, this can be helpful to an extent. But most decompilers are not very debug-friendly, it would generate source in arbitrary assumptions, so unless you recompile the source back, you're stuck. But decompiling, recompiling and then looking around the weird identifier and method names and reordered statements will drive you crazy, but for Java developers there is a cool eclipse plug-in which would feel like a medic, it's name is JadeClipse. Though, it's too much for just knowing what's going on inside the third party's code.

While wider acceptance of Open-source in the industry is a definitely good thing, but beware it makes you feel less comfortable working with proprietary closed source softwares once you are used to it. I'm still wondering, how long would closed source software exist in platforms like Java or languages like Ruby and likes?

But please, for the hack's sake, please let us, the software developers on the mother Earth, see the SOURCE CODE.

TDD Evangelizing Experiences

I was discussing about usefulness of developer testing and TDD stuff with one of my friend, Kapil. I told him "you should avoid debugging at all costs" - debugging is painfully mundane and costly activity, "one of the way of avoiding it is; to write automated unit tests", he said, "I avoid it by debugging differently..".

I asked how and what different debugging techniques, and he gave an example of manual technique to avoid seeing what's going in code ("If a file is not found, I'll try renaming or something like that"). I said, "this is no debugging, its kind of troubleshooting to avoid debugging". He would not agree to it.., then I told him why not write an intelligent program which will do the same thing programmatically ?(well, its kinda behavioral testing), then? well then he was interested in listening... and was ready for few pills of TDD :).

With few examples, It was easy to explain him how TDD can change the way you think about developing software, he already knows it a bit, tries to minimize it through other manual hacks (this is partially because PHP has smattering good debuggers and he works on PHP based server apps).

I told him "debugging happens to be one of those trivial activities which doesn't differentiate you from others" (no big deal, everyone can do it, right?); You need to consider audience attitude for a successful systematic evangelizing :).

TDD, in its real sense-the test first, is one thing that I still find difficult to explain to people; It requires you to give it a shot without unveiling providential "Zen" behind it, somethings you should just try without resisting and put in practice if you find it effective.

Once, I explained TDD to my trainee team at C-SAM, I explained it by example, told them what happens when you don't have tests, made them feel the pain of debugging; then write tests and show what great peace of mind this automatic testing gives.

To sum the experiences on evangelizing TDD:

• To a good programmer explain the benefits of writing intellectual tests rather than idiot debugging every time.
• To a novice, give a real example; mock a bad situation and exhibit how to tackle it with TDD.
• To an amateur, explain how tests are better than manual troubleshooting.
  

Kill Debugging, Write tests.

I hope this experiences will find some use and benefits in development.

Taking care of abused Exceptions using JDT AST

How many times you went nuts while dare debugging a large application? and while debugging, faced desperate failure in getting any clue on what went wrong, no exception, no log?

Well, If you work on right smartly large bad code base, this experience is quite common. One of the main reasons, as I think, for this situation is swallowed and mishandled exceptions.

I ran in similar problems once, where it was becoming very hard to find the problems. I came to know mishandled exceptions in many places. And I decided to write a plug-in to manipulate AST and replace it with logging statements.

Here is the code for interesting parts of the plug-in. Created a catch clause visitor, which will visit all the catch clauses in AST. The AST will be parsed for each Compilation Unit in a loop.

private final class CatchClauseVisitor extends ASTVisitor {
@Override
public boolean visit(CatchClause node) {
allCatchClauses.add(node);
return super.visit(node);
}
}

This visitor can be used to collect clauses from AST:

ASTParser parser = ASTParser.newParser(AST.JLS3);
parser.setKind(ASTParser.K_COMPILATION_UNIT);
parser.setSource(lwUnit); // set source
parser.setResolveBindings(true); // we need bindings later on
CompilationUnit unit =(CompilationUnit) parser.createAST(null /* IProgressMonitor */);
unit.accept(new CatchClauseVisitor());

Now that we have all catch clauses, we can iterate on them to execute transformations and rewrite the AST.

try {
for (CatchClause node : allCatchClauses) {
if (list.size() == 0) // SWALLOWED EXCEPTION
putLoggingStatement(node, null, rewrite, null);
else // code to replace SOP statements with logger call
for (Object object : list)
// ....
// computation of the new source code
TextEdit edits = rewrite.rewriteAST(doc,
lwUnit.getJavaProject().getOptions(true));
edits.apply(doc);
String newSource = doc.get();
// update of the compilation unit
lwUnit.getBuffer().setContents(newSource);
lwUnit.save(null, false);

} catch (...){...}

And here is the putLoggingStatement method..

private void putLoggingStatement(CatchClause node, Object object,
ASTRewrite rewrite, ASTNode expStatement) {
MethodInvocation invocation = node.getAST().newMethodInvocation();
setLoggingMethodName(node, invocation);
SimpleName args = node.getAST().newSimpleName(
node.getException().getName().getIdentifier());
invocation.arguments().add(args);
if (expStatement == null) {
ExpressionStatement statement = node.getAST()
.newExpressionStatement(invocation);
ListRewrite listRewrite = rewrite.getListRewrite(node.getBody(),
Block.STATEMENTS_PROPERTY);
listRewrite.insertFirst(statement, null);
rewrite.replace(node, node, null); // replace the nodes
return;
}
ExpressionStatement statement = node.getAST().newExpressionStatement(
invocation);
setContextInstaceVariable(invocation);
rewrite.replace(expStatement, statement, null);
}
// ....

A number of details are removed for the sake of brevity and I don't claim the code to be professionally written. There are some improvements I can thing of, like, the SOP can be replaced with logging statement, a provision for rethrowing exception statement based on some config (no, no configs!)....

Finally let me tell you that writing the code involving AST can become complex and error prone, but it does a real good job. This small job of carefully writing a custom code manipulation saved me from doing it manually for hundreds of java source files as well as code maintainability got much better, what a relief!

The State of software

There's a thought provoking discussion on pragprog on the current state of software. The discussion was really insightful, and I can't disagree with the group that, quality of softwares has gone bad.

These days, People comfortably accept existence of bugs in most software, which is a fact not acceptable at all but it seems the expectation from software is being relaxed and hence limited existence of better software, is this the failure on our part?

Even after almost fifty years, people are still writing code in historical languages, and are happy with it, this is very questionable compared to other disciplines. If we compare software crafting with other domains, it's quite frustratingly slower in improving, this rant is quite valid. Although few of us have started picking up the lessons from domain, lean software development being one of them.

On optimistic fronts, there are few good news also (note that 73.5% of all stats. are made up ;) ). According to these news there are few outright software project failures and look at the bloody success ratios! they are marginally good in comparison!

Of course, there are few world class development in open source, which have outperformed most traditional stuff (e.g. managed environments), but they are not in sufficiently large number.

IMO, all this deserves evolutionary development, and yes Agile helps, not a silver bullet but helps; quite a lot in controlling four essential aspects of four variables.

And this pragprog topic is worth a read...