What does it mean to "be Agile" today?

Are you using Agile approach in your organisation?
You can often hear:
"Yes, and we are going do make SCRUM certification".
What does it mean in practice?
Read more about other shades of Agile methodologies, effects of PR and buzzword leverage for Agile movement:
http://pragdave.me/blog/2014/03/04/time-to-kill-agile/

Requirements management story

The development of simple specific project management system was set in motion. There were couple of meetings and email "conversations" about requirements and use cases scenarios. There was one administration like system role valid for period of time when system "project" was running. I asked client few times about replacing user of that role in case of illness days off etc.

It is not important now .....

The answers were: it is not important part of story, we can wait for finishing project business if there is enough time or substitute someone in place using the same system user, and finally there is only one administrative user for single "system project". The message was clear: it is not important, keep going, we can handle situation. The application was simple so it was looking like cutting the corners on unused features.

... but I cannot accept that ...

The story was finished till final acceptance tests. One of the users from client side playing such system "administrative role", get few days off. The other administrative user asks now for guidance because "there was something in manual about replacing users of other role in system projects". After reviewing quick "user doable" solutions he had serious objectives about completeness of the system. So either detailed "walk through" procedure will be enough or new feature have to be added to application functionality as a part of service agreement.

The role of Final user adoption

It's story about little detail but it shows common scenario that very often takes place during software development.
During requirements gathering the project team was planning common usage and sanely limited set of functionalities.
Every user thinks in a different way. Some will understand used functionality set, other will need detailed procedures for every specific situation.
Final user adoption is important thing. That is one of the reasons why so many classic waterfall project models fail. There are also stories about complete systems, agreed on paper but unused or misused by users that just didn't like it.

Software development project size and methodology

Here is a short description of software development characteristics for projects of different size and its impact for methodology.

Very simple application (single programmer, tester - one man for all)

Requirements are simple and all known. If you are writing very simple application, using good architecture plan and code writing practices, as a result you should have clean ready to change application. Adding automated tests makes changes easier without software quality drop. You know all details without looking into documentation, and testing new changes and features during implementation. All changes go to the code repository with comments and TODO file.

Simple/medium, application (team of 1+ programmers and other stakeholders)

The requirements should be gathered and agreed before development. You are using some kind of Issue Management System, tracing all change requests. Application is divided into main modules referred in task details. Programmers know what to change and what are dependencies between modules and side effects of implemented changes. Testers know main modules, and are going through test procedures of modules and functionalities. Procedures are short and clean and there is a natural place for agile methodologies.

Large application, (team of 10+ programmers and stakeholders)

Application is quite huge. Preparing requirements is often separate project. Every change request is thoroughly reviewed then approved. Application have many interdependencies caused by re-usability and connection points. Some changes have strictly local effects for given feature, those are less harmful. Other change some lower level service, probably unit tested, but still having impact on bigger process. Automated tests are not covering every possible use of code, manual top level functional tests are last line of defense in quality assurance. In real life every module has many connections with other modules. Testers have very limited knowledge about those dependencies. Everyone needs more plans, documents, procedures and artifacts that is slowing down the whole process. Practical solution is to divide the large project into smaller, less complex sub-projects.

At this level of complexity there are many "process complete" management methodologies. The winning one is the best suitable for given project, reducing risks and allowing to finish project at planned costs and time.

Database related general application performance tips

Most of the serious applications nowadays use some kind of relational database engine. Database becomes main data source and data processing module in application architecture. That's the reason why it's common application performance bottleneck and main scalability factor.

Having design and data flow requirements its possible to prepare database system at the system planning and implementation phase. Don't get to much into performance details at that stage because "premature optimization is the root of the evil". Sometimes system requirements and design meets reality and final expectations, but changes in requirements that could change application data flow are more common. The most accurate results are coming from real data and users usage patterns, so getting system responsiveness metrics is crucial even at the early prototype stages.

Having such data about performance "weak points" is a start point to optimize and improve overall system scalability. It's good to begin from top more abstract layers in application architecture before rushing to lower level database storage tweaking.

There are some tips divided by subsystem scopes and ordered by top to bottom abstraction level:

Application scope

• discuss with project stakeholders responsiveness requirements for various application functionalities
• analyze data usage patterns like writes vs reads, common data usage vs casual reporting, written once or changed often etc.- it gives image what could be improved, and what kind of underlying database mechanism you will need
• remove worst bottleneck (having biggest performance impact) first to get best results
• use cache for most used data (reads and writes if its possible)
• design transactions smart- long transactions cause performance problems
• at first you should use normalized data schema, but there are situations where little denormalization is crucial for good performance (f.e. copying some data to most read table to eliminate need for joining other big tables)

Database system scope

• use indexes where it works best (every index adds performance penalty for data writes)
• use vertical and horizontal data partitioning - move less used data into other tables or use database engine specific features
• configure database and use its features like special table types, special indexes for your best

Operating system scope:

• use database dedicated host or performance cluster - for large scale systems
• check network latency and throughput for large data transmissions
• tune underlying disks structure and file system- separate partitions or disks for database files, use low overhead file system or custom database "raw" partitions (like in Oracle DB)

How to eat spaghetti (code)

I'm thinking about dinner so the subject has a culinary style.

I have spent some time fighting with spaghetti code that grown from adding more and more conditional strands to initial prototype of application framework.

Before major version release of middle size production system I had to add a new feature to the biggest important process. Not so big change - it looked simple. It just involved doubling of some model objects and making two separate subprocesses. After alternating main flow, many side effects occurred, caused by intertwined logic and persistence. I had to do some clean up, cut off tangled strands, standardize conditional flows and reduce code. The results still weren't valid - at least I have found "single point of failure". Because of time constraints I had to fix it with block of ugly code. The project needs change in the process structure to adapt new features in more seamless way.

When you meet that kind of "spaghetti code" there are two known ways of dealing with it:
- pull one strand at a time - clear one flow path - repeat until structure is simple enough
- eat it all at once - analyze requirements and used implementation and rewrite it from start

Reduce structure, simplify, refactor and do not let the code grow into spaghetti again. It will ruin maintainability and extendability of software project in future.

Choosing scatter chart library for web application

So I definitely know that I have to make web based UI for medical diagnostic application. It's not so complicated as I initially though. It is a kind of scatter chart that has to display delivered XY data and use as reference some kind of meta data like description, database id etc. Basic functionality of that module is adding and editing points. It should be somehow interactive - like has clickable points that can fire various events. Graph zooming is also needed.

There are articles with list of interesting solutions: 13 useful graphing solutions, 10 Free Chart Scripts. I reviewed few charting libraries that can be divided into three categories:

• HTML canvas charts
• Java applets
• Flash based charts

There where also other interesting ideas like SVG based interactive chart, or image chart with imagemap and added DHTML tricks, but still without enough support and needed writing more code.

Needed charting solutions should be lightweight so I thought about html canvas drawing, but unfortunately there are serious IE issues - It does not implement such feature, so excanvas ActiveX is used not always successfully (new IE changes are breaking excanvas functionality).

Another good choice from point of customization and extending functionality is Java applet based on JFreeChart, or lighter jchart2d library. Using Java applet was tempting. I made some serious applet work some time ago so I had to make some user experience tests of deploying UI as applet. The large group of users has no technical computer background, and can be confused about accepting prompts for Java installation (Java is not preinstalled with system in here), waiting for runtime download and finally for applet initializing. That may cause serious problems with acceptance tests.

So I tried Open Flash Chart 2- the nice flash based chart library. It has fixed but all basic features I need. Flash player is king of the browsers (large deployments ratio) today. That and smaller than Java's runtime should cause less problems with runtime installation. It has wrapper libraries in most popular programming languages including Java. I have choose third party jofc2 library for quite easy integration with code sample from Open Flash Chart and Wicket example.

Open Flash Chart has enough javascript interface, to dynamically change data, graph style, add tooltips and script onclick events on data points. It has no direct zooming support but can be implemented as changing chart properties by adding external scripted controls. Redrawing after data delivery is fast also.

Near future brings even more choices like Silverlight, improved Applet/JavaFX runtime or adoption of canvas like solutions.

Further reading:
Scatter chart library - moving to Flot
Looking for simple python chart library

Rich GUI dilemma - web or desktop application?

I'm planning additional visualization tools for diagnostic system. The have "additional" role - because will be scarcely used only by expert users. It's a kind of fancy interactive scatter plot with plenty of displaying options including simple data transformations.

Primarily I thought about it as a desktop application. But I when I get initial requirements it seemed that there is no much functionalities and it could be written as module of already existing web application for data gathering.
I have to gather more details , because now I have to guess future boundaries of that part of the system.

The basic pros and cons array:

	Pros	Cons
Web application	- its web application so you need only web browser (maybe Java or Flash runtime automatically downloaded by browser)- easy deploying - better (seamless) integration with the rest of the (web) system	- possibly low client performance depending on used technology
Desktop application	- established for rich GUI - more computational power to use on client side (really?) - targeted for more functionality "richness"	- target client system dependence (but I know that 99% of users uses Windows platform) - loose integration with the rest of web based system (deployment architecture allows it)

Reviewed technologies and solutions:

• Java desktop app
• Java applet
• GWT application
• YUI components
• flash components

Final choice depends on the range of features, and final users preferences. Web version seems more performance constrained - but there are many technological solutions that make that choice as good as its desktop counterpart. At that point there are very little differences between web or desktop deployment outcomes. Nowadays buzz is "web is the future". I have to make my choice based on economic assumptions.

How to get (rid of) Anemic Domain Antipattern coding multitieried enterprisey application

Programmers want to make "good designed", "enterprise like" application. So keeping in mind boxy schema of multitiered application they start to organize code into separate layers like most common DAO,Service and Domain. Very often that way leads to Anemic Domain Antipattern. It's characterized by "dumb" C struct like domain objects without logic in methods.

The mechanism is simple: programmer wants to keep layers separate and dependent only on common interfaces. To reduce wiring and simplify initial development DAO object are connected to Service layer which uses also Domain objects. Service layer is filled by Script like methods, written in procedural style. That way of development is easy at the beginning. So main logic goes to Services "layer" and data operations go to DAO layer. After some time Services become really fat, but domain objects gain only more attributes and accessor methods. As long as application is a kind of a prototype, or just is small everything works quite good.

But to get more benefits of object oriented programming in future, we need net of more "intelligent" domain objects "talking" to each other, not just logic closed in services layer scripts. At this moment that could only be done by total refactoring, or just by designing from scratch using experience from previous implementation. Just keep in mind mantra encapsulation, reusability, simplicity.

The simplest cure to that antipattern is to design "good wiring" at the begging, to give domain object direct access to DAO layer, and other low level services needed by domain logic. Then, domain can evolve easy into more complex structures, needed by bigger "enterprise" applications.

Another point of view on Object Oriented Programming problems

It's not another article about OOP pros and cons.
I want to present you a non orthodox example of dealing with common programming problems without using standard OOP patterns.

The site is for some time on the web already, and presents controversial articles about OOP and SQL criticism.

I don't agree with author's overall point of view, but many ideas are very usable in specific simple applications development, and some of patterns presented there could be found in current RAD frameworks.

Just read and brainstorm.

The art of exceptions catching

Exceptions in programming languages are very useful tool for propagation information about program errors and helping handling critical situations properly. But it is only tool and how it can be useful depends only on implementing programmer.

I can put it in few points what I have learned in theory and verified during practice:

• If you don't know what to do with exceptions don't catch'em - exceptions reveal errors so fix errors - don't hide them under carpet

• Catch exception if you can handle it, you know how to clean up used resources and how to inform developers/users about that event

• Don't use exceptions mechanism (hacky way) in the correct flow of the program

• In Java use unchecked exceptions (RuntimeExceptions descendants) instead of checked Exceptions - it saves "throws SomeException" litter in code forcefully reminding about catching them (it makes sense only in some kind of low level services with inevitable critical situations that can be handled by direct clients)

I think it's only basic set of rules.

Do you have another ideas for exception handling?

User interface form state and interaction with model advice

Advice of the week:

Separate form bean and core model state in more complicated (not naive) user interface related use cases. Propagate changes using events and sensible messaging.

Little real life story:

I have used two different frameworks in two production systems with naive auto binding controls with model objects. In both cases controls were changing almost directly state of underlying model entities. It's good for fast prototypes, but not for production demands and interaction with real users. Creating simple workarounds was good enough, but caused strange quirks in displaying actual state. Some fraction of users were loosing their confidence in system reliability, and called for support. The problem was fixed, by redesigning some UI controls and model entities interaction.

Fighting with duplication - attack of the clones

Last week I proceeded with clean up tasks on one project I'm working now. It's big number of jsp generated reports. Some time ago I setup initial environment for such reports, so even analytics without special JSP or JAVA knowledge could do some reports. It was temporary solution, before jumping to more enterprise tool. Results were not so bad - new reports were added very quick. Everything worked and was usable so enterprise tool is just forgotten mention.

Over two years it overgrowth overall code base and many problem issues and problems were related with those reports. In harder cases I had to dig into that and what I have seen is a massive duplication. I any possible form. So when we found bug in one section of code there was huge probability that the same problem was duplicated more times elsewhere. It was even worse. Instead of new features added to base report there where created new versions of the same report with new parameters or features. And divergence between those reports arises over time. So you can imagine how much time maintenance started cost at some point.

I like to stick a specially with one coding rule: do not create new and remove existing duplication.

Duplication starts when some functional block of code appears at least in two places in code. Sometimes it is just simple one liner - but even then you should consider pros and cons for wrapping that piece of code in some procedural statement.

Let's back to vicious mechanism of avalanche of duplicated code in non programmers environment - when people don't know good "coding" practices. One of reports (let's call it A) was copied with some functional difference (as B). We have got almost two identical files. Then comes another feature that seems to interfering with previous one. Because new feature ticket was assigned only to one of those reports (after some time everybody sees two reports and thinks - those are two separate reports) and new feature seems so "new" and practice before was to create separate functional report (for "clear solution") so there comes new report C derived from A. After some time somebody realized that there is no such feature for B report. Continuing previous process we have now 4 reports with D version. Then comes major change - added new module that shows new values in similar manner so base reports A,B,C,D are copied as 2A,2B,2C,2D and then changed some values and layout. Almost pure copy'n'paste coding.

Having one simple report at the beginning that would have parametrized additional features it's now eight versions of duplicated code with less than 10% differences. Cost of removing bug or implementing new feature is to about eight times bigger than for single but more complicated file. Not including effects of further "extension" mechanism - avalanche just gains new mass in exponential rate.

You could think what kind of procedures you have there to allow that kind of practices. Ok - team of report analytics was told to eliminate visible duplications, but it wasn't enough. When somebody concentrates on complex analytical problem doesn't think much about removing duplication. Even when report is complex and amount of code and sql is enormous. It's just additional burden that seems not helpful for analytical problem solving.

What could help? Some quick course of coding practices and techniques in context of that environment - basics how to write reusable pieces of code, avoid common pitfalls and reduce codebase to ease development and maintenance. And more practical explanatory examples. Sometimes duplication problem is visible but there is no simple solution - then pairing with more experienced programmer would help. It's just organizational issue.

So now the team is fighting with reports bugs and feature requests. And the biggest impact we are gaining now by not touching those new features or bugs but by merging and removing duplicates. It's ironic but number of bugs is decreasing now in reverse exponential rate.

Agility - ready for change

Life is full of surprises and random events affecting every planned action.

Last week my wife and kid get some health problems. I had to change schedule to take more care of my family.

Another example is my Mutual Funds Portofolio project. I have finished first prototype and was ready for "intra" testing. Unfortunately main quote data source has been shut down. It was free service, that I used for testing purposes, so that kind of things are quite probable.

Minor or major change of plan will occur anyway. So, is planning worth anything? Yes, but we have to plan with possible changes in mind.
Most solutions tend to fall between two strategies:
- using enormous resources to analyze every probable option
- analyze core issues and then proactive adapt do changes

Good example of ready for change strategy are agile methodologies. If you are interested in the subject please read introduction at agilemanifesto.org/.