Real life stories behind bug reports

I've found some gem, while browsing through Android reported issues:

My Nexus 10 was on silent mode this yesterday at church. I was using the YouVersion Bible app, taking notes. All of a sudden my Nexus 10 let off a shreaking siren type noise that couldn't be silenced. My church (of about 1500 people) all heard it, it stopped the pastor mid-sentence - as I ran out of the auditorium. Has anyone else experienced something like this? I don't think I can ever trust it to be quiet - in a quiet place again.

source: Aaron.Ky report at https://code.google.com/p/android/issues/detail?id=42447

It shows also why some environments have distance to newest technology. 

99 $items on the $container ...

Do you know the song that goes like:

99 little bugs in the code
99 little bugs in the code
Take one down, patch it around
~
117 little bugs in the code
...

That was about some funny facts we are experiencing some days.

Now something more didactic:
http://www.99-bottles-of-beer.net/

Software development project size and methodology

Here is a short description of software development characteristics for projects of different size and its impact for methodology.

Very simple application (single programmer, tester - one man for all)

Requirements are simple and all known. If you are writing very simple application, using good architecture plan and code writing practices, as a result you should have clean ready to change application. Adding automated tests makes changes easier without software quality drop. You know all details without looking into documentation, and testing new changes and features during implementation. All changes go to the code repository with comments and TODO file.

Simple/medium, application (team of 1+ programmers and other stakeholders)

The requirements should be gathered and agreed before development. You are using some kind of Issue Management System, tracing all change requests. Application is divided into main modules referred in task details. Programmers know what to change and what are dependencies between modules and side effects of implemented changes. Testers know main modules, and are going through test procedures of modules and functionalities. Procedures are short and clean and there is a natural place for agile methodologies.

Large application, (team of 10+ programmers and stakeholders)

Application is quite huge. Preparing requirements is often separate project. Every change request is thoroughly reviewed then approved. Application have many interdependencies caused by re-usability and connection points. Some changes have strictly local effects for given feature, those are less harmful. Other change some lower level service, probably unit tested, but still having impact on bigger process. Automated tests are not covering every possible use of code, manual top level functional tests are last line of defense in quality assurance. In real life every module has many connections with other modules. Testers have very limited knowledge about those dependencies. Everyone needs more plans, documents, procedures and artifacts that is slowing down the whole process. Practical solution is to divide the large project into smaller, less complex sub-projects.

At this level of complexity there are many "process complete" management methodologies. The winning one is the best suitable for given project, reducing risks and allowing to finish project at planned costs and time.

The best programming language 2 - Simple case tale

The University needed temporary solution for generating list of quiz questions, chosen randomly from data prepared in structured text file. The scope of application was nothing comparing to "Enterprise Systems", so I treated it as a kind of entertaining enterprise.

The core characteristics were:

• simple logic
• UI forms for getting parameters from user
• printable or saveable formatted questions list
• low burden installation and use (get it on pendrive or download ready to use).
• main constraints: short time and possibly low cost.

I started looking for similar open source products, but not found needed functionality. So when I checked myself for Not Invented Here syndrome, I started thinking about custom solution.

There were many ways of implementation. I thought about preparing windows exe file with included required dependencies (no installing additional Java, .Net, Python or other runtime).
I started playing in "prototyping with Python"- it was fun and things were going smoothly. But I stopped before making UI. Plain old console was "too lame" for final users, so I planned GUI with TkInter included by default in python installation to avoid porting to windows executable problems.

I get core set of functionalities, but user interface still was barely specified. Then I get many additional requests about "last minute" changes in UI forms (and probably not the last). The second "economic" option was to implement the whole application in Java Script and HTML running under web browser. Every computer has web browser suitable to run that kind of application so there are almost no installation and portability issues. Implementation time should be short too. To speed up things I have changed data format from flat file of records into JSON structure. Porting logic was fast too. The most time took me making forms working.

First version was not so fancy and needed some GUI tweaking like formatting, changing labels and headers. In a case of any further UI changes other people knowing HTML could make fixes instantly. The extra feature was possibility of "deploying" the little application on a web server as HTML page. Saving result to file wasn't working directly. Making it really automatic needed interacting with browser security policies for access to local file system. We decided that it will be still usable if user use copy-paste technique, and place formatted result in external text editor. The default usage was printing questions on paper so nobody saw problem there.

Everything is working as designed, and shows that web browser should be treaded as good platform for small applications. Sometimes little prototyping gives better insight on a whole system.

Pragmatic printable document format

What open file format comes to your mind when you have to produce mass of personalized electronic documents in enterprise system?

I have reviewing pragmatic solutions for given requirements:

• ready for batch print out and producing quite good prints on customer's printers
• keeping documents in electronic version for evidence (content cannot be changed)- the bussiness process is paper based
• have customizable templates (text, formatting, tables)
• total cost of process should be optimized (including printing by customer issues)

PDF seems good choice, it is well distributed format and produces nice print outs. I've tried it and it worked quite well. The cons of that solution popped out during batch processing and were: slow process of generating final PDF and bigger files included containing additional font data (national character issues - no problems with standard fonts).

Another approach was using XHTML as final format with ability to print from web browser. That needs more attention on formatting and automatic flow, but it makes good prints also.

Users can print documents using both formats without problems. XHTML was faster to produce and worked with standard system fonts, so files were in most cases smaller than their PDF counterparts.

That was scenario for paper dominated document flow process. Electronic document processing requirements need also easy and automatic access to important data in document content. The best standard for that seems XML containing structured data. That plus XSLT transformation for description of form view, may be used to produce printable XHTML or PDF (with help of FOP (Formatting Objects Processor)).

New document formats like ODF are based on similar ideas so future lays somewhere here, just new solutions need more adoption.

Another point of view on Object Oriented Programming problems

It's not another article about OOP pros and cons.
I want to present you a non orthodox example of dealing with common programming problems without using standard OOP patterns.

The site is for some time on the web already, and presents controversial articles about OOP and SQL criticism.

I don't agree with author's overall point of view, but many ideas are very usable in specific simple applications development, and some of patterns presented there could be found in current RAD frameworks.

Just read and brainstorm.

How to export data to one big XML file

My company had to add extra export format for data in a system I'm working on. I get specification and XML schema and started analyzing it for best fast solution. Data was ready to export so there left decision about choosing right way to make XML. Deadline was imminent, and everybody was really busy at work. Making export file - was good task for our newest programmer. He never parsed/created XML using SAX/DOM or other libraries, but was envy to use and learn one of the standard.

The main technological problem was the size of output XML. For biggest data case it would generate over 500MB XML with about 50000 nodes at highest level. It would be efficiently generated using only writing partially to stream technique like SAX. I have chose two options- use SAX or own similar simple API - realization time was crucial. Because XML structure was relatively easy - all tags without attributes, so we decided to use plain string buffer technique with addition of couple helper functions.

At the beginning I had objections about not using standard libraries that should help. But the implementation of XML format moved forwards, output file was easy to validate with XML schema. The most time was committed to handle non existing or incomplete data cases. The export function was ready before time. That is one of examples where using simplest tools, meets desired goal.

What are your choices in similar situation?

I'm busy - new Java project

I'm quite busy now, while setting up new project. It will be Medical Diagnostic Expert System using machine learning algorithms. I've got legacy applications preceding that project, that could be used for their core algorithms implementation. Besides that, the whole application must be build from scratch, to meet new requirements. I'm working now on data gathering services and user interface for experts modules. It is designed as web application, implemented in Java and using Spring framework.

Real world update deployment scenario

I'm thinking about automating updates deployment for one of the commercial applications I work on. It's hard to find good ready "out of the box" solution and below are the reasons why:

• maximum simplifying deployment
• it has to manage applications major updates and minor patches
• deployment automatically should stop production server, make backups for easy revert, update application code and database, start server and make some test to check if it's started (that part is already working)
• multiple deployment servers- independent installations
• find out new updates on central server using pull strategy
• it has to manage in almost unique way two platforms- windows and linux
• it has to be scheduled to out of office work hours
• some target servers have outstanding deploy time window- so deployment time could be set for local site
  
• some target servers need patches earlier- like some quick fixes that shouldn't be spread anywhere else
• some target servers are more "conservative"- we need additional agreements with customer to make updates- so we are making updates less frequently
• automatically apply all needed updates and patches up to date- it could be really nice feature

The most tedious work is already automated, but rest still needs manual operation. This is at least 1.5 hour per each full deployment- it looks I can cut it down. So after looking at some commercial solutions, I decided to make custom tool using java ANT and some helpers.

Do you know any utilities that can manage most of those requirements?

Mutual funds portfolio monitoring application - stage one finished

I have finished first stage of development of mutual funds portfolio monitoring application. Major bugs are fixed and all needed stats are calculated. I have been checking it on my investment account. Benchmarks don't lie- second half of 2007 was harsh for mutual fund investors :).

I'm working now on adding more visual appealing features like graphs etc. It needs also some refactoring and implementation of storing precalculated values - almost all features are data processing intensive. But more important are interesting portfolio stats, then I will work on scalability. After that I can think about first public alpha release.

I still don't have external python hosting service for development purposes, so if you know cheap one let me know.

Weakly typed languages need more testing

I have to written and managed some projects using scripting languages. Those languages often have no strong typing or have duck typing like in python. This is their strength on one way, and weakness on another because it's easier to break the code giving mistaken parameter. Most of this errors are visible after running a buggy code.

Some of older programming languages have built coarse in error messages like in PHP or old ASP family. It doesn't help during debugging and can be really frustrating indeed.

Automatic testing is good, everybody can say even if they don't use it. So it's best to use unit testing specially for that kind of code. In that way simple errors caused by non checked and mistaken parameters can be easy found and unit functionality is checked also.

So once again I say- automatic tests are good. Scripting languages like python, perl or php need automatic tests as additional "safety net". Comparing code written in weakly and strong typed language both covered with tests both may have equal quality, so there is no real typed language quality advance over non typed one.Good tests are what gives you quality and confidence.

So next time if you are going to code something think about- how to test it.

Additional insights about Web.py (update 2) - sessions and slowly pages

I have used default branch of web.py framework for prototyping purposes. As my web app needs sessions and there is no bult in session support, I tried to use flup and beaker modules. Meanwhile I realized that there is another branch of web.py with simple sessions.
It is handling session storage on database or files, and has typical session api (although simpler than beaker). It is all what I needed.
If you are using bzr already you can get web.py with sessions directly from repository:

bzr branch http://bazaar.launchpad.net/~karol.tarcak/webpy/webpy.sessions.branch webpy

Another issue is slowly page generation. I think it's matter of using uncached templates, but this is unevitable during development process. Maybe database connection also adds its workload, I have to check it. Now every page loads in about 10 seconds- it's slower than compiling jsp page on the same machine. I'm going to investigate the subject and return to process of happy-snappy coding.

Maybe you found anything what can speed up page generation- please share your experiences. I'm waiting for comments.

Prototyping with Web.py - the light web framework

Welcome again

Last week I started in cooperation with my friend small personal investment tracking application project. It's still in designing stage, maybe I will write something about that later.

I decided to write basic functional prototype using python, having in mind that it can be production environment. Because it is going to be a web application I reviewed current python web frameworks.

There are interesting solutions like Django or Pylons, but I would prefer to use simple but flexible frameworks for that scale of project. So I decided to use web.py - simple framework made by creators of reddit.com.


Web.py has really simple skeleton connecting together http server, controller engine and built in but loosely coupled database module and template engine. It needs minimum configuration to run. In default mode it uses WSGI http server, with option to switch to fastcgi on apache or lighttpd. The same with other modules. You can use other full fledged db framework or better known templating engine. There are no fancy helpers and component support, but it's easy to include existing solutions.

It reminds me another web framework having similar attributes - java Maverick. It have more features, but the same spirit of simplicity.

If you are ready to use light web framework try one of these, or find something like that. Bigger and having more features not always means better.

The best programming language - choose the right tool for the job.

Is there the only one ultimate programming language, what is the best match for every application?

Some of folks may say- "Yes, It's L**. I've used it for X years in production environments and there is nothing in the world better than that". But it's not yet another thread about java vs c++ vs python vs ruby vs php [vs pick_your_language]...

Look at the current situation of existing programming languages. There are probably more programming languages than ever before and they are still evolving, what is IMHO good thing. An evolutionary criterion here is mainly market demand. It regulates need for that kind of tools and the most obsolete ones are vanishing.

Most of the programming languages start existence as a project of small group of enthusiasts or scientists. One person driven projects are quite usual. When it gains wider audience as good candidate for programming environment there is a moment when market influence kicks in.
From business point of view software production process must be optimized to get needed set of features, using possibly smallest resources and time to get final product of desired quality.
So there is economical pressure to keep (man time cost)/(product quality) ratio low.

What is programming environment? The main element is of course programming language concept with its constructs, and a set of basic framework libraries. It must be implemented, so it goes with tools like compilers, interpreters, virtual machines, debuggers and other preparing software to run and monitor software in target system. On the other end there are Integrated Development Environments boosting programmers’ productivity. All of these elements are used during software production process which has strategic impact on software cost and quality. The total cost and quality depends on the combination of lower level facilities of programming environments (I am excluding people skills).

All programming environments have some sweet points there and sometimes hard to reach or lack of other features somewhere else. So there is no one simple and effective tool for creating software.

So how to choose best one for your needs? Productivity depends on domain and size of application, architecture, speed of elementary analyze-write-compile-test cycle, easiness to create small and clean code, good debug information, matching tools power to programmer skills and other minor factors. These factors should be evaluated during short "evaluation runs" to get the complete picture.

How to collect such valuable metrics? We need productivity measuring features seamlessly hooked up into existing tools used during software development and maintenance. Having recorded software feature production time metrics and process structure can give us very valuable information. It gives possibility to look back, find weak points of process and try to improve it. Or, you would try to compare metrics in context of features and application character for different programming languages and choose the best for your needs. But we need tools to acquire those metrics. Some of commercial tools are going into that direction, and I think it's time to incorporate that into platforms supporting creation of open source software and sharing information with wider audience.

There is a very long way (if any exists) to one leading programming environment with simple but powerful programming language and well designed set of libraries etc. We have different competing programming environments find their market niches today. Evaluating and selection of best solution isn't trivial task. But if you can - think about it and choose the right tool for the job.

Maybe one day there will be one ideal small and complete programming language with good framework library, with scripting for small programs and support for big enterprise application features, with changeable features and many sets of well designed libraries and frameworks. Sounds like utopia for me, but who knows?