Chrome OS - the dawn of net(only)books

Google is making good strategic move to get control and share of netbooks, and maybe nettops market. The incoming era of ubiquitous mobile internet connection for everyone, makes concept of thin clients and network systems plausible. The idea probably will be mainly accepted by mainstream population (look at software usage patterns and emerging fast booting systems from BIOS on desktops for basic web/email operations). If the big G stays behind it so we can expect hardware producers answer. Looking at the Android history it can be massive adoption of "new" OS concept.

There were similiar succesful concepts like Eye OS, but I tented to use separate specific web services best suited to one task at once like email, documents editor and sharing etc. Google already has those applications, so its also reason why it could be available as Chrome OS asset.

Some experts are calling that future is only in weba pplications. But some more advanced or geeky users it could be wrong way. It's ok to have most of tools as online service, but there is still place for lower level native (and offline) platform applications. Either Chrome OS "native applications" will be sufficient then, or Chrome OS will be installed as second system on more "fat" devices.

One thing is for sure - current web applications technology is still evolving, so this is only beginning.

Scatter chart library - moving to Flot

After playing with Open Flash Chart 2 trying to make interactive scatter plot, I've found severe glitches that popped up during prototyping. Library worked quite well for standard line and bar charts. But for my specific scatter chart I realized that:

• I have to use hacks to provide 0,0 origin based XY axes (OK I could just use standard left/bottom axes)
• there are problems with larger scale values (I didn't checked exactly but values in range to 10000 rendered OK, somewhere above it is some kind of limitation)
• problem with grid rendering for values in less than zero range
• OFC can't parse scientific notation values (data number formatting is a must)
• some minor problems that could be solved by chart tuning and hacking ranges etc.
• after playing with OFC stable version I've checked last trunk version that worked much better

I summed all problems and thought about potential extendability problems and tried to move to library based on HTML Canvas object. One of the simplest and most mature projects is Flot. It has all required features, including interaction. The only problem I found was rendering under IE using Excanvas emulation object. Standard stable Flot version doesn't work well under IE8. Charts were drawn although without data points. After checking project issues list, I've found that some problems where resolved by using newer JQuery and Excanvas library. I've tried to run Flot from trunk version, and my IE8 problems were solved. As a extra newer Flot version contains more interesting plugins like navigation. BTW maybe it's time to publish next Flot stable version in downloads section. The only disadvantage of that solution is forcing IE users to accept running Excanvas security policy.

Choosing scatter chart library for web application

So I definitely know that I have to make web based UI for medical diagnostic application. It's not so complicated as I initially though. It is a kind of scatter chart that has to display delivered XY data and use as reference some kind of meta data like description, database id etc. Basic functionality of that module is adding and editing points. It should be somehow interactive - like has clickable points that can fire various events. Graph zooming is also needed.

There are articles with list of interesting solutions: 13 useful graphing solutions, 10 Free Chart Scripts. I reviewed few charting libraries that can be divided into three categories:

• HTML canvas charts
• Java applets
• Flash based charts

There where also other interesting ideas like SVG based interactive chart, or image chart with imagemap and added DHTML tricks, but still without enough support and needed writing more code.

Needed charting solutions should be lightweight so I thought about html canvas drawing, but unfortunately there are serious IE issues - It does not implement such feature, so excanvas ActiveX is used not always successfully (new IE changes are breaking excanvas functionality).

Another good choice from point of customization and extending functionality is Java applet based on JFreeChart, or lighter jchart2d library. Using Java applet was tempting. I made some serious applet work some time ago so I had to make some user experience tests of deploying UI as applet. The large group of users has no technical computer background, and can be confused about accepting prompts for Java installation (Java is not preinstalled with system in here), waiting for runtime download and finally for applet initializing. That may cause serious problems with acceptance tests.

So I tried Open Flash Chart 2- the nice flash based chart library. It has fixed but all basic features I need. Flash player is king of the browsers (large deployments ratio) today. That and smaller than Java's runtime should cause less problems with runtime installation. It has wrapper libraries in most popular programming languages including Java. I have choose third party jofc2 library for quite easy integration with code sample from Open Flash Chart and Wicket example.

Open Flash Chart has enough javascript interface, to dynamically change data, graph style, add tooltips and script onclick events on data points. It has no direct zooming support but can be implemented as changing chart properties by adding external scripted controls. Redrawing after data delivery is fast also.

Near future brings even more choices like Silverlight, improved Applet/JavaFX runtime or adoption of canvas like solutions.

Further reading:
Scatter chart library - moving to Flot
Looking for simple python chart library

Rich GUI dilemma - web or desktop application?

I'm planning additional visualization tools for diagnostic system. The have "additional" role - because will be scarcely used only by expert users. It's a kind of fancy interactive scatter plot with plenty of displaying options including simple data transformations.

Primarily I thought about it as a desktop application. But I when I get initial requirements it seemed that there is no much functionalities and it could be written as module of already existing web application for data gathering.
I have to gather more details , because now I have to guess future boundaries of that part of the system.

The basic pros and cons array:

	Pros	Cons
Web application	- its web application so you need only web browser (maybe Java or Flash runtime automatically downloaded by browser)- easy deploying - better (seamless) integration with the rest of the (web) system	- possibly low client performance depending on used technology
Desktop application	- established for rich GUI - more computational power to use on client side (really?) - targeted for more functionality "richness"	- target client system dependence (but I know that 99% of users uses Windows platform) - loose integration with the rest of web based system (deployment architecture allows it)

Reviewed technologies and solutions:

• Java desktop app
• Java applet
• GWT application
• YUI components
• flash components

Final choice depends on the range of features, and final users preferences. Web version seems more performance constrained - but there are many technological solutions that make that choice as good as its desktop counterpart. At that point there are very little differences between web or desktop deployment outcomes. Nowadays buzz is "web is the future". I have to make my choice based on economic assumptions.

The best programming language 2 - Simple case tale

The University needed temporary solution for generating list of quiz questions, chosen randomly from data prepared in structured text file. The scope of application was nothing comparing to "Enterprise Systems", so I treated it as a kind of entertaining enterprise.

The core characteristics were:

• simple logic
• UI forms for getting parameters from user
• printable or saveable formatted questions list
• low burden installation and use (get it on pendrive or download ready to use).
• main constraints: short time and possibly low cost.

I started looking for similar open source products, but not found needed functionality. So when I checked myself for Not Invented Here syndrome, I started thinking about custom solution.

There were many ways of implementation. I thought about preparing windows exe file with included required dependencies (no installing additional Java, .Net, Python or other runtime).
I started playing in "prototyping with Python"- it was fun and things were going smoothly. But I stopped before making UI. Plain old console was "too lame" for final users, so I planned GUI with TkInter included by default in python installation to avoid porting to windows executable problems.

I get core set of functionalities, but user interface still was barely specified. Then I get many additional requests about "last minute" changes in UI forms (and probably not the last). The second "economic" option was to implement the whole application in Java Script and HTML running under web browser. Every computer has web browser suitable to run that kind of application so there are almost no installation and portability issues. Implementation time should be short too. To speed up things I have changed data format from flat file of records into JSON structure. Porting logic was fast too. The most time took me making forms working.

First version was not so fancy and needed some GUI tweaking like formatting, changing labels and headers. In a case of any further UI changes other people knowing HTML could make fixes instantly. The extra feature was possibility of "deploying" the little application on a web server as HTML page. Saving result to file wasn't working directly. Making it really automatic needed interacting with browser security policies for access to local file system. We decided that it will be still usable if user use copy-paste technique, and place formatted result in external text editor. The default usage was printing questions on paper so nobody saw problem there.

Everything is working as designed, and shows that web browser should be treaded as good platform for small applications. Sometimes little prototyping gives better insight on a whole system.

Successful simple web based MMORPG

Massively Multiplayer Online Role Playing Games are at the peak of popularity. People like to play with other live players and compete with them. Game producers are gaining more and more profits from that kind of games, giving better and better visual and audio experience in pseudo 3d worlds. Is there a place for a technically simpler web based games?

One of my friends told me to look at the one classic game of that type: Fallen Sword produced by Hunted Cow Studios Ltd.. Game started, as I know in 2006 and today has about 2 700 000 registered accounts. There is probably massive amount of inactive players, but the number is still impressive. As I checked during day there were always above 2000 players online. So people like not only WoW 3D style games.

The main game concepts remind some of popular MUD games, with added graphics, auction house and trade between players suport. You can buy special game money units, for real money using for example PayPal account. Those units called Fallen Sword Points maybe used to boost game character skills or other attributes. Although there is no way out for money as company states in terms of use, in game economics is quite strong. People are playing game for fun, but sometimes are trying to trade game money for little services, like giving temporary skills boosts called buffs, or "selling" fan-art avatars. Those gray areas are acceptable, in other cases the real winner is producer.

The game lacks some responsibility that can be gained through more AJAX like components, but overall speed is acceptable. I'm trying to figure out other interesting features - and ideas for game people may have. I like those kind of the non highest tech but very playable games. I had one web based game started and unfinished prototype in past, so I'm interested in subject from a business point of view too.

Good luck of Wicket beginner with grain of problems

It was worky and little disorganized week. Functional prototype of application I wrote in Wicket is working and waiting for client acceptation. When I decided to use Wicket for that project I have almost no bad feelings about using new framework. Framework looked reasonable and was written using best software patterns and practices. In practice It works indeed.

Because nothing is ideal, I want to share with some problems I have met while charging at Wicket:

• WicketTester can't load my page resource files - maybe it's a problem with Spring classloader.I have skipped more complicated testing, and made only the simplest TestUnits. It's bad practice, but almost all application features was and will be heavily manual used and checked.


• Automatic "hot" class deployment - it's problem with Spring classloader and proxying. I've tried to set class reloading only for non Spring managed page classes, but with no luck.


• Editor in ListView component - problem is finally solved but it took me all day to get it work with validation and all editing features. I have read two books and some tutorials. Where was the answer? I should more carefully read API docs, where the problem of reusing list items is addressed - so remember that editor with validation based on Wicket ListView needs setReuseItems(true) option !

Going back to work.

I have used Wicket for small web application

More than week ago I have started implementing first module for Medical Diagnostic Expert System. It's quite simple application for manual data entry and editing. There is one catch - forms have to be flexible in some way and dynamically configurable. The whole project will work as web application, so data entry editor will work in that way too.

As this is the new fresh project I have possibility to use the newest Java features and frameworks. The multi tier architecture application will use Spring for easy separation of concerns between layers. That plus JPA on top of Hibernate should be sufficient for future development of modules and expansions (the final project boundary is blurry).

I thought about using Spring MVC framework, but after little research I decided to use Wicket, to set up user interface. Wicket is using view component model, with main page component containing other components and controls with event handlers. Binding controls is made by marking special wicket attributes in html template. Additional markup is not intrusive, so even raw html template viewed in web browser is looking good (sounds like Tapestry). The basic ideas are similar to that used in ASP.NET. Developing in Wicket is more Java code centric than in other popular frameworks, with clean separation of html template design.

For quick start up I have used QWicket generated application template and tailored it to work with JPA and my core module. Application should work under Postresql database, but for easy of deployment of prototype for testing purposes I stayed with preconfigured HSQLDB. Switching back to Postgres is just as easy, as changing few lines in properties file.

I'm unexperienced Wicket user but development goes quite smooth. Using documentation, tutorials and books is really helpful at that stage. As far so good. I'm finishing UI prototype and can concentrate now on implementing core module functionalities.

I'm busy - new Java project

I'm quite busy now, while setting up new project. It will be Medical Diagnostic Expert System using machine learning algorithms. I've got legacy applications preceding that project, that could be used for their core algorithms implementation. Besides that, the whole application must be build from scratch, to meet new requirements. I'm working now on data gathering services and user interface for experts modules. It is designed as web application, implemented in Java and using Spring framework.

Good method of web frameworks comparison for programmers

If you are programmer and want to choose the web frameworks suitable for your project, one of the best ways of framework evaluation (not fastest) is writing simple prototype that uses concepts similar to your project requirements. Opinions of other experienced frameworks users also counts, but sometimes framework selection is based on personal preferences. Thanks to prototype code all framework sweet spots and "need to be done" areas are easily exposed. Prototyping gives also practical measure of framework impact on overall development productivity.

Last week I have done little research about new trends in Java web frameworks, and found very good and detailed frameworks comparison based on prototyping on a Peter Thomas blog.
Here are articles comparing Wicket framework to Spring MVC, JSF and GWT.
Good job Peter Thomas!

By the way, I have found many Wicket positives in his articles, that I give that framework a try in my next project requiring component based UI.

Google Chrome - one more step on the way for perfect client for web applications

Google Chrome browser gets a lot of attention from the start. It is browser build from scratch for better performance, usability and security. Isn't it good solution for web applications, even for enterprise level? As Forrester Research analyst Sheri McLeish says in Information Week article Google's Chrome browser "is not top of mind for IT organizations". It's still in beta, and Google is the only main supporter with its unclear reasons.

There can't be "perfect" browser for everyday surfing, and in the same time using enterprise web applications. But it's good step on the way to "perfection". In worst case the best features could be adopted by other web browser players. On the other hand, if chrome goes out from Beta stage, it can take big share of users by its usability and performance.

I will be watching progress.

Firefox2 and css caching issue - part 2

I have tried out to change settings for expire headers, and figured it that application adds its own headers to dynamic generated css files. Then apache2 mod-expire tries to add own settings.
After tweaking application settings there are resulting headers:

Date: Mon, 1 Mar 2008 06:46:02 GMT
Server: Apache/2
Cache-Control: max-age=600, max-age=604800
Expires: Mon, 6 Mar 2008 06:46:02 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 18128
Content-Type: text/css


Expire times:

• max-age=600 - given by apache2 global settings


• max-age=604800 - given by applications itself

It's ugly - but surprisingly caching works under Firefox with these headers. I haven't checked yet which expiration time is taken by Firefox. I need to clean it and remove one of extra headers anyway.

Below are old headers for the same resource fort comparison - when Firefox caching was not working:


Date: Sat, 01 Mar 2008 07:28:31 GMT
Last-Modified: Sat, 01 Mar 2008 07:26:56 GMT
Expires: Sat, 01 Mar 2008 07:36:56 GMT
Cache-Control: max-age=600
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 18128
Keep-Alive: timeout=10, max=98
Connection: Keep-Alive
Content-Type: text/css


There are removed last-modified and keep-alive headers, but it's hard to understand how it can change Firefox caching.
I have to check difference between 600 and 604800 expire age, that may be solution.

Probably all headers should look like those added by apache mod-expire for static files:


Date: Mon, 1 Mar 2008 06:46:04 GMT
Server: Apache/2
Last-Modified: Thu, 6 May 2007 02:52:18 GMT
Accept-Ranges: bytes
Cache-Control: max-age=604800
Expires: Mon, 6 Mar 2008 06:46:04 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 3588
Content-Type: application/x-javascript


Firefox 2 and css caching issue

Last week I had strange issue with Firefox cache. One of web application needed tweaking to be more responsive. After playing with apache configuration I tested page load speed using YSlow.

All files have expiry headers and all text files are deflated. Results are still mediocre. Some of dynamically generated css files aren't cached by Firefox 2. IE works as it supposed to do - any subsequent request is handled blazingly fast. I tried it on different installations of Firefox to exclude my browser optimizations side effects. Still the same.

Below are HTTP response headers of css that isn't going to FF cache:

Date: Sat, 01 Mar 2008 07:28:31 GMT
Last-Modified: Sat, 01 Mar 2008 07:26:56 GMT
Expires: Sat, 01 Mar 2008 07:36:56 GMT
Cache-Control: max-age=600
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 18128
Keep-Alive: timeout=10, max=98
Connection: Keep-Alive
Content-Type: text/css


I can't see anything suspicious. Content type is proper, all expire headers are set. Requested file name is named as css.php - does FF makes some assumptions on file extension? I tried turning off Cache-Control header and leave only Expires, but it doesn't work too.

It must be simple detail. Anybody has any clues?

Mutual funds portfolio monitoring application - stage one finished

I have finished first stage of development of mutual funds portfolio monitoring application. Major bugs are fixed and all needed stats are calculated. I have been checking it on my investment account. Benchmarks don't lie- second half of 2007 was harsh for mutual fund investors :).

I'm working now on adding more visual appealing features like graphs etc. It needs also some refactoring and implementation of storing precalculated values - almost all features are data processing intensive. But more important are interesting portfolio stats, then I will work on scalability. After that I can think about first public alpha release.

I still don't have external python hosting service for development purposes, so if you know cheap one let me know.

web.py update 5 - DbUtils pooling and apache MPM module

My application using web.py is still growing. One day happened something suprising. After testing on internal WSGI server, I deployed application at testing machine with Apache, and started checking. Login screen appeared and everything was looking ok. But after login I waited very long time and nothing happened. After couple of minutes (long timeout?) application finished request with exception inside DBUtils code. I've checked logs, another browser- still the same.

I googled exception code and found that it's something related with exhausted connection pool. Application grows and communication with database complicates too. I've add additional code closing connections. Still nothing.
Then I have found some interesting posts about DBUtils connection pooling method and problem related to apache MPM module.

Apache MPM modules are directly handling HTTP requests. DBUtils creates pool for application process. My default Apache configuration was MPM prefork. That means every http request is handled by one single threaded MPM process. DBUtils created pool for every apache handler process- that exhausted database connections.

There is simple solution if you use dedicated apache server- using MPM worker that works better with DBUtils multi threaded pooling.
Once again, the day was saved.

web.py update 4 - apache cgi configuration

Finally I decided to put my web.py application on test server. I updated python to 2.5, and installed needed packages.

Then I decided to configure application as fast-CGI under existing Apache2 installation with bunch of other running applications. Short and good installation manual can be found at webpy page.

I tried to configure fast-CGI and then CGI setup, but something went wrong. All configuration examples are using url rewriting, and my Apache does not have proper module. I could of course install mod-rewrite for apache, but I realized that I don't need it yet. Simple web.py application should work even without mod-rewrite. It could be nice feature on constrained hosted systems.

After needed modifications my apache config file for CGI looks like that:

<VirtualHost *:80>
ServerName mywebpyapp.domain.com

Alias /static /var/www/mywebpyapp/static
ScriptAliasMatch ^/(.*) /var/www/mywebpyapp/run-cgi.py/$1

<Directory "/var/www/mywebpyapp">
Options ExecCGI
AllowOverride None
Order allow,deny
Allow from all
</Directory>
</VirtualHost>


and my separate CGI script run-cgi.py:

#!/usr/local/bin/python2.5
import cgitb; cgitb.enable()
import sys
import web, app.controller

def cgidebugerror():
_wrappedstdout = sys.stdout
sys.stdout = web._oldstdout
cgitb.handler()
sys.stdout = _wrappedstdout

web.internalerror = cgidebugerror

#web.webapi.internalerror = web.debugerror
web.config.db_parameters = dict(dbn='postgres', user='user', pw='password', db='myappdb',host='localhost',port='5432')
web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)

if __name__ == "__main__":
urls=app.controller.urls
web.run(urls, globals(), web.reloader)


So my application is working now behind apache and it is ready for production deployment.

web.py update 3 (and tough bugs)

My new application on web.py framework works great. I eliminated slowness issue during installation of new web.py environment.
It was caused by opening db connection without pooling- and it took really long time (about 10 s). So next time I properly installed DBUtils. Run tests and found exception raised by web.py database. I had to make quick fix in my web.py installation - changed name of one of parameters in DBUtils function call.

Few days ago I tested web.py session support and authentication in my application. It worked good, when something broke next day. I had feeling that session isn't properly stored or recalled- and my authorization module redirected me to login page. Access logs showed me that actually I requested only login page, not any other. Strange. I was looking for bug in web.py, but there had to be another reason I thought at end.
I used another web browser- in my case IE. Login and session worked well, better than under Firefox. The reason causing problem was performance enhancing plugin for firefox I installed few days ago to tune in new Firefox. It used strange caching algorithm, and every time I hit some application page login page was loaded. Probably it was first page cached by plugin. I uninstalled it and reviewed other browser modifications.

Once again experience shows that the most time consuming bugs are simple and just hidden in area assumed for sure as free of bugs (in my case I was looking in server app, but bug was caused by browser client modification).

Additional insights about Web.py (update 2) - sessions and slowly pages

I have used default branch of web.py framework for prototyping purposes. As my web app needs sessions and there is no bult in session support, I tried to use flup and beaker modules. Meanwhile I realized that there is another branch of web.py with simple sessions.
It is handling session storage on database or files, and has typical session api (although simpler than beaker). It is all what I needed.
If you are using bzr already you can get web.py with sessions directly from repository:

bzr branch http://bazaar.launchpad.net/~karol.tarcak/webpy/webpy.sessions.branch webpy

Another issue is slowly page generation. I think it's matter of using uncached templates, but this is unevitable during development process. Maybe database connection also adds its workload, I have to check it. Now every page loads in about 10 seconds- it's slower than compiling jsp page on the same machine. I'm going to investigate the subject and return to process of happy-snappy coding.

Maybe you found anything what can speed up page generation- please share your experiences. I'm waiting for comments.

Prototyping with Web.py - the light web framework

Welcome again

Last week I started in cooperation with my friend small personal investment tracking application project. It's still in designing stage, maybe I will write something about that later.

I decided to write basic functional prototype using python, having in mind that it can be production environment. Because it is going to be a web application I reviewed current python web frameworks.

There are interesting solutions like Django or Pylons, but I would prefer to use simple but flexible frameworks for that scale of project. So I decided to use web.py - simple framework made by creators of reddit.com.


Web.py has really simple skeleton connecting together http server, controller engine and built in but loosely coupled database module and template engine. It needs minimum configuration to run. In default mode it uses WSGI http server, with option to switch to fastcgi on apache or lighttpd. The same with other modules. You can use other full fledged db framework or better known templating engine. There are no fancy helpers and component support, but it's easy to include existing solutions.

It reminds me another web framework having similar attributes - java Maverick. It have more features, but the same spirit of simplicity.

If you are ready to use light web framework try one of these, or find something like that. Bigger and having more features not always means better.