The best programming language 2 - Simple case tale

The University needed temporary solution for generating list of quiz questions, chosen randomly from data prepared in structured text file. The scope of application was nothing comparing to "Enterprise Systems", so I treated it as a kind of entertaining enterprise.

The core characteristics were:

• simple logic
• UI forms for getting parameters from user
• printable or saveable formatted questions list
• low burden installation and use (get it on pendrive or download ready to use).
• main constraints: short time and possibly low cost.

I started looking for similar open source products, but not found needed functionality. So when I checked myself for Not Invented Here syndrome, I started thinking about custom solution.

There were many ways of implementation. I thought about preparing windows exe file with included required dependencies (no installing additional Java, .Net, Python or other runtime).
I started playing in "prototyping with Python"- it was fun and things were going smoothly. But I stopped before making UI. Plain old console was "too lame" for final users, so I planned GUI with TkInter included by default in python installation to avoid porting to windows executable problems.

I get core set of functionalities, but user interface still was barely specified. Then I get many additional requests about "last minute" changes in UI forms (and probably not the last). The second "economic" option was to implement the whole application in Java Script and HTML running under web browser. Every computer has web browser suitable to run that kind of application so there are almost no installation and portability issues. Implementation time should be short too. To speed up things I have changed data format from flat file of records into JSON structure. Porting logic was fast too. The most time took me making forms working.

First version was not so fancy and needed some GUI tweaking like formatting, changing labels and headers. In a case of any further UI changes other people knowing HTML could make fixes instantly. The extra feature was possibility of "deploying" the little application on a web server as HTML page. Saving result to file wasn't working directly. Making it really automatic needed interacting with browser security policies for access to local file system. We decided that it will be still usable if user use copy-paste technique, and place formatted result in external text editor. The default usage was printing questions on paper so nobody saw problem there.

Everything is working as designed, and shows that web browser should be treaded as good platform for small applications. Sometimes little prototyping gives better insight on a whole system.

PyCha chart drawing library working under SPING

I've ported PyCha chart drawing library to work under SPING instead of Cairo. I've discussed that idea with PyCha lead developer - Lorenzo Gil Sanchez.

Lorenzo said he will still stick with Cairo, that is good library ported and available for many systems. Anyway I've ported PyCha to SPING and tested with SPING creating SpingCha off-project.


Here is lastly created PyCha news group , where you can find posts about PyCha development.
And - if anyone is interested - working SpingCha release in files section.

Blending scripting languages in modern enterprise environment

Is your company using "enterprisey" technologies like Java or .NET? Do you want to use flexible scripting languages like Python or Ruby?
There are interesting possibilities for popular and "mature" programming languages like those above. Many popular languages has implementations that work under Java or .NET environment, like Jython, JRuby, IronPython, IronRuby and others. It's quite old thing but it still worth mentioning it. The main problem with such solutions is their maturity and completeness, so checking and quick research is needed before doing decision.

You can also find implementations of old veterans like COBOL and REXX for new platforms. There are also single platform languages that have cool features of scripting languages like Groovy or beanshell for JVM.

Additional languages are primarily used in configuration, defining custom and complex business rules, adding Domain Specific Language features etc. Word of warn - do not create local Babel Tower by multiplying number of existing languages in single project. But it's nice to have to have one scripting language blended into main project platform.

Other case is porting applications from scripting to target platform. Constraints are narrow - all used third party libraries have to work on both platforms, so additional work is needed often. Practical example - here is interesting tutorial how to deploy web.py python application under Jython and JVM.

Rolling my own simple python chart library

Two weeks ago I started looking for simple python chart library. There are interesting solutions based on cairo, GD and alike libraries. I was surprised there is no simple chart drawing library using "core" python installation environment. I found of course nice working PyCha using cairo, but there were problems on one testing servers with cairo library installation.

Portability is one of my priorities so I've been trying to find more flexible solution. So I've found SPING (Simple Platform INdependent Graphics former now as PIDDLE). It supports numerous backends like PDF, PIL, QuickDraw, PostScript, Illustrator, piddleVCR, OpenGL, Tk, wxWindows and support for others is in development. Looks promising. Ok so it gives me basic drawing capabilities. What about beatiful charts? After online research, I decided that something like PyCha would be enough. But PyCha uses cairo ... . Then idea has come- rewrite PyCha to use SPING. Idea is not so stupid- PyCha library is quite small and porting should be doable.

So I started rewriting PyCha fo SPING (codename SpingCha). The biggest part of work is porting calls typical for cairo interface to more generic SPING. I used Interface pattern with simple cairo like API. After two days I have almost working implementation, tested on original PyCha examples. By saying almost - I mean it needs better color, fonts and antialiasing handling to produce more beautiful charts.

After finishing and cleaning up code I'm planning to publish it under derived GNU Lesser General Public License. Maybe I will talk with author of PyCha to just include my ideas in main project. Meanwhile if you are interested to get SpingCha just add your comment, I will know if I need to speed up things, and release it.

UPDATE: About further story you can read new post

Looking for simple python chart library

I'm looking for simple chart library. I need it to draw stock quote and portfolio benchmarking charts.

I'm trying now pycha (python charts). It's simple but ready for most common applications. One drawback- I'm unable to setup needed cairo library on testing system, yet.

I will try to made it, but maybe there is another solution.

Do you know any python chart drawing library with less dependencies?

Update:
It is still one of the simplest and best chart utilities. I'm also using JavaScript generated charts in web browser.

web.py update 5 - DbUtils pooling and apache MPM module

My application using web.py is still growing. One day happened something suprising. After testing on internal WSGI server, I deployed application at testing machine with Apache, and started checking. Login screen appeared and everything was looking ok. But after login I waited very long time and nothing happened. After couple of minutes (long timeout?) application finished request with exception inside DBUtils code. I've checked logs, another browser- still the same.

I googled exception code and found that it's something related with exhausted connection pool. Application grows and communication with database complicates too. I've add additional code closing connections. Still nothing.
Then I have found some interesting posts about DBUtils connection pooling method and problem related to apache MPM module.

Apache MPM modules are directly handling HTTP requests. DBUtils creates pool for application process. My default Apache configuration was MPM prefork. That means every http request is handled by one single threaded MPM process. DBUtils created pool for every apache handler process- that exhausted database connections.

There is simple solution if you use dedicated apache server- using MPM worker that works better with DBUtils multi threaded pooling.
Once again, the day was saved.

web.py update 4 - apache cgi configuration

Finally I decided to put my web.py application on test server. I updated python to 2.5, and installed needed packages.

Then I decided to configure application as fast-CGI under existing Apache2 installation with bunch of other running applications. Short and good installation manual can be found at webpy page.

I tried to configure fast-CGI and then CGI setup, but something went wrong. All configuration examples are using url rewriting, and my Apache does not have proper module. I could of course install mod-rewrite for apache, but I realized that I don't need it yet. Simple web.py application should work even without mod-rewrite. It could be nice feature on constrained hosted systems.

After needed modifications my apache config file for CGI looks like that:

<VirtualHost *:80>
ServerName mywebpyapp.domain.com

Alias /static /var/www/mywebpyapp/static
ScriptAliasMatch ^/(.*) /var/www/mywebpyapp/run-cgi.py/$1

<Directory "/var/www/mywebpyapp">
Options ExecCGI
AllowOverride None
Order allow,deny
Allow from all
</Directory>
</VirtualHost>


and my separate CGI script run-cgi.py:

#!/usr/local/bin/python2.5
import cgitb; cgitb.enable()
import sys
import web, app.controller

def cgidebugerror():
_wrappedstdout = sys.stdout
sys.stdout = web._oldstdout
cgitb.handler()
sys.stdout = _wrappedstdout

web.internalerror = cgidebugerror

#web.webapi.internalerror = web.debugerror
web.config.db_parameters = dict(dbn='postgres', user='user', pw='password', db='myappdb',host='localhost',port='5432')
web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)

if __name__ == "__main__":
urls=app.controller.urls
web.run(urls, globals(), web.reloader)


So my application is working now behind apache and it is ready for production deployment.

web.py update 3 (and tough bugs)

My new application on web.py framework works great. I eliminated slowness issue during installation of new web.py environment.
It was caused by opening db connection without pooling- and it took really long time (about 10 s). So next time I properly installed DBUtils. Run tests and found exception raised by web.py database. I had to make quick fix in my web.py installation - changed name of one of parameters in DBUtils function call.

Few days ago I tested web.py session support and authentication in my application. It worked good, when something broke next day. I had feeling that session isn't properly stored or recalled- and my authorization module redirected me to login page. Access logs showed me that actually I requested only login page, not any other. Strange. I was looking for bug in web.py, but there had to be another reason I thought at end.
I used another web browser- in my case IE. Login and session worked well, better than under Firefox. The reason causing problem was performance enhancing plugin for firefox I installed few days ago to tune in new Firefox. It used strange caching algorithm, and every time I hit some application page login page was loaded. Probably it was first page cached by plugin. I uninstalled it and reviewed other browser modifications.

Once again experience shows that the most time consuming bugs are simple and just hidden in area assumed for sure as free of bugs (in my case I was looking in server app, but bug was caused by browser client modification).

Additional insights about Web.py (update 2) - sessions and slowly pages

I have used default branch of web.py framework for prototyping purposes. As my web app needs sessions and there is no bult in session support, I tried to use flup and beaker modules. Meanwhile I realized that there is another branch of web.py with simple sessions.
It is handling session storage on database or files, and has typical session api (although simpler than beaker). It is all what I needed.
If you are using bzr already you can get web.py with sessions directly from repository:

bzr branch http://bazaar.launchpad.net/~karol.tarcak/webpy/webpy.sessions.branch webpy

Another issue is slowly page generation. I think it's matter of using uncached templates, but this is unevitable during development process. Maybe database connection also adds its workload, I have to check it. Now every page loads in about 10 seconds- it's slower than compiling jsp page on the same machine. I'm going to investigate the subject and return to process of happy-snappy coding.

Maybe you found anything what can speed up page generation- please share your experiences. I'm waiting for comments.

Prototyping with Web.py - the light web framework

Welcome again

Last week I started in cooperation with my friend small personal investment tracking application project. It's still in designing stage, maybe I will write something about that later.

I decided to write basic functional prototype using python, having in mind that it can be production environment. Because it is going to be a web application I reviewed current python web frameworks.

There are interesting solutions like Django or Pylons, but I would prefer to use simple but flexible frameworks for that scale of project. So I decided to use web.py - simple framework made by creators of reddit.com.


Web.py has really simple skeleton connecting together http server, controller engine and built in but loosely coupled database module and template engine. It needs minimum configuration to run. In default mode it uses WSGI http server, with option to switch to fastcgi on apache or lighttpd. The same with other modules. You can use other full fledged db framework or better known templating engine. There are no fancy helpers and component support, but it's easy to include existing solutions.

It reminds me another web framework having similar attributes - java Maverick. It have more features, but the same spirit of simplicity.

If you are ready to use light web framework try one of these, or find something like that. Bigger and having more features not always means better.