My own HTPC v0.8

During holidays and weekend I've played more with my HTPC.

At first I reconnected and fitted all cables to make more order behind PC and for "easy" access to keyboard and mouse (still on the cable). Cards in low profile PCI(e) slots are secured too.

I've tried to get better image quality on TV, but either Radeon tv-out don't have mode suitable to work with CRT 100hz Flicker Free TV, or it needs serious tweaking with tools like Powerstrip. I've tried various settings, but Radeon allows to change tv resolution up to 1024x768 and some quality settings. Nothing about vertical refresh rate. So I use best setting I get - 768x576 PAL resolution with visible flickering on high contrast static images like text in web browser. Radeon has deflicker feature but it works by filtering and lowering contrast in flickering areas so image is more fuzzy and misses details. Flickering isn't so bad, but I miss stable 100hz picture. It's basically suited for viewing SDTV programs and for that purpose it looks good.

I thought about purchasing Windows Media Center (my Windows Vista Bussiness licence doesn't include it) , but I will stay with Leadtek PVR2 software. It does it job, I managed to set up remote for work in TV and radio mode. I missed remote configuration for other than PVR2 functions, so I have searched for other software. I have to try commercial solution like Grider, but maybe there is something more cheap and specific. At least I found small utility called Winfast Remote Control Support. It has basic functions allowing controlling other applications like Winamp and sending key codes, so maybe it will be enough. I even tried to control mouse cursor from remote but it's to slow and skippy (max remote "resolution" is about 4 ticks per second). The best option would be controlling separate applications/windows in context aware way. Anyway it's hard to control PC without keyboard, so my next buy will be probably nice wireless keyboard/mouse set.

Because I get analog TV without EPG I have tried to get EPG from internet. It was quite easy because PVR2 can import XMLTV format files. I get wide tv programs coverage in one file. Some additional settings to map channel names to match those from XMLTV and it starts working. I have to set up automatic update of XMLTV file yet, and find way to add more XMLTVs for other programs.

Next I have tried to record some shows from EPG. Few clicks and at the evening I can watch all my favorite shows. Hibernating and waking up due to scheduled recording is working very well.

I think I have device I wanted. The most fun is setting it up and tweaking for my needs. Probably buying low price PVR will save me some money and much time, but I wasn't sure about final usability and match for my needs. Internet connection gives greater possibilities like youtube and other videos, IP-TV is other great option. I don't spend too much time for TV, but now I have a freedom of choice when I can watch my favorite show, and better manage my time.

My own HTPC v0.5

Before weekend I have bought tv tuner and gfx card with tv out connector, get my new small form factor low profile case PC with Windows XP, and initially assembled my HTPC. I called it v0.5 because of unsolved hardware and software problems.

I have low profile PC case, and low profile cards - unfortunately with long brackets only. I have asked local retailer about short brackets, but without luck. To check if everything is working I plugged cards without brackets. (Don't do try it at home - I had damaged two cards in that way in past). I will try to get short ones, or just cut the long I have and remake it into low profile standard.

I planned to connect tv-out to crt tv via scart connector using RGB signal. Unfortunately it seems that Radeon supports only YUV standard on tv-out. YUV to RGB converter is an option, but its extra cost and resulting picture quality may be degraded (it depends on converter). There is also more exotic option for some cards like Radeon 2400, to make simple VGA-SCART connector for RGB and sync signals, and force graphics card to work with TV frequencies. I have decided to use more popular S-Video connection- it should give almost as good quality as RGB. In near future I plan to upgrade my tv to some kind of HDTV LCD/Plasma set - most of current connection problems will be irrelevant.

I have tried to set display native tv resolution mode. I have now PAL 768x576 60hz resolution but it seems like it's interlaced. I have "100hz" TV set with internal picture scan, but it probably works only for standard 50hz PAL signal. I have tweaked TV driver with Powerstrip but still no luck.

TV tuner is Leadtek Winfast 2000 XP Expert - cheap and simple analog tuner cart without hardware encoding with remote control. Many reviews recommended that model. There is a lot of newer hardware, but it's hard to get reliable review for it. I'm still have most of programs in SDTV standard so it's sufficient for my needs. That plus separate tv and radio antenna input gives good integration with lasting analog sources. Picture quality is good enough- the same signal connected directly to tv gives noticeably sharper picture and more crispy details. It may be caused by tv tuner or graphics card tv-out mode.

Leadtek adds to tuner its software Leadtek Winfast PVR2. It has basic (and my priority) PVR functionalities like recording, time shifting and scheduled recording. It has it's own design - not so simple and aesthetic like Windows Media Center, but quite good and usable. Most used functions like channel switching, viewing and recording, can be operated from Leadtek remote. One sad thing I realized that separate application Winfast radio wont work with remote (maybe it's something with configuration). I hope I will also find way to use remote for other applications. I'm going to buy small wireless keyboard also, to get comfortable control over HTPC.

I'm planning to fix hardware problems and add more software features in future. Stay tuned.

Work now, fun later

Next busy week.

Last fixes for next version of product I am working on few years already.

On the second hand another project - business trip with prototype presentation.

Meanwhile at home - planning and research for building custom HTPC. I will publish later more details about hardware configuration.

All important work tasks should be finished before end of this year, then I will play with my HTPC.

I want media PC for christmas

Last weekend I was partying at my friend's place. I have opportunity to play his self assembled PC media center (they are calling ti Home Theater PC, right?). He used Windows Media Center as a basic software player. It was working very nice with Logitech remote for media playing, and wireless keyboard for more specific windows operations. The Windows Media Center interface is very aesthetic. I felt lack of more "power user" features, like changing delay between pictures in slide-show.

Because I have planned to install something like that HTPC, I think I'm now ready to assembly my own PC entertainment center for main reasons:
- cable programs recording for later viewing and other DVR features
- integration of media sources like cable and internet
- ability to play computer games with family

Good luck of Wicket beginner with grain of problems

It was worky and little disorganized week. Functional prototype of application I wrote in Wicket is working and waiting for client acceptation. When I decided to use Wicket for that project I have almost no bad feelings about using new framework. Framework looked reasonable and was written using best software patterns and practices. In practice It works indeed.

Because nothing is ideal, I want to share with some problems I have met while charging at Wicket:

• WicketTester can't load my page resource files - maybe it's a problem with Spring classloader.I have skipped more complicated testing, and made only the simplest TestUnits. It's bad practice, but almost all application features was and will be heavily manual used and checked.


• Automatic "hot" class deployment - it's problem with Spring classloader and proxying. I've tried to set class reloading only for non Spring managed page classes, but with no luck.


• Editor in ListView component - problem is finally solved but it took me all day to get it work with validation and all editing features. I have read two books and some tutorials. Where was the answer? I should more carefully read API docs, where the problem of reusing list items is addressed - so remember that editor with validation based on Wicket ListView needs setReuseItems(true) option !

Going back to work.

User interface form state and interaction with model advice

Advice of the week:

Separate form bean and core model state in more complicated (not naive) user interface related use cases. Propagate changes using events and sensible messaging.

Little real life story:

I have used two different frameworks in two production systems with naive auto binding controls with model objects. In both cases controls were changing almost directly state of underlying model entities. It's good for fast prototypes, but not for production demands and interaction with real users. Creating simple workarounds was good enough, but caused strange quirks in displaying actual state. Some fraction of users were loosing their confidence in system reliability, and called for support. The problem was fixed, by redesigning some UI controls and model entities interaction.

I have used Wicket for small web application

More than week ago I have started implementing first module for Medical Diagnostic Expert System. It's quite simple application for manual data entry and editing. There is one catch - forms have to be flexible in some way and dynamically configurable. The whole project will work as web application, so data entry editor will work in that way too.

As this is the new fresh project I have possibility to use the newest Java features and frameworks. The multi tier architecture application will use Spring for easy separation of concerns between layers. That plus JPA on top of Hibernate should be sufficient for future development of modules and expansions (the final project boundary is blurry).

I thought about using Spring MVC framework, but after little research I decided to use Wicket, to set up user interface. Wicket is using view component model, with main page component containing other components and controls with event handlers. Binding controls is made by marking special wicket attributes in html template. Additional markup is not intrusive, so even raw html template viewed in web browser is looking good (sounds like Tapestry). The basic ideas are similar to that used in ASP.NET. Developing in Wicket is more Java code centric than in other popular frameworks, with clean separation of html template design.

For quick start up I have used QWicket generated application template and tailored it to work with JPA and my core module. Application should work under Postresql database, but for easy of deployment of prototype for testing purposes I stayed with preconfigured HSQLDB. Switching back to Postgres is just as easy, as changing few lines in properties file.

I'm unexperienced Wicket user but development goes quite smooth. Using documentation, tutorials and books is really helpful at that stage. As far so good. I'm finishing UI prototype and can concentrate now on implementing core module functionalities.

Younger programming languages and frameworks are better

I have short talk with young student of Computer Science. He said he prefers .NET framework and C# more than Java, because Java seems outdated. I've seen those kind of preferences for some time in new generation of young programmers. I've asked what makes Java so outdated. He said about first release year (sic!), and mess in Java libraries, legacy API and so on. I've no time to ask for more details, but many of those pros and cons come probably from rumors and hypes. Good point for .NET marketing.

Younger programming languages and frameworks are potentially better. It's true for most popular (chosen by ,majority so in some way best) languages supported by community and commerce. Standing on the shoulders of existing giants - in that case language frameworks, gives better opportunity to make better API and add more practical features. New languages and frameworks are addressing most popular trends for its time (like WEB 2.0 and AJAX now). Other possible positive outcome would be cutting of all legacy code (good and bad one), and start everything from scratch, that would tend to include new ideas. All that needs many people, hours of work, and money.

So taking that assumptions and having big support - about every 10 (5?) years it would allow to produce new "cool" language that young generation of programmers will choose.

What is the important thing for developers? It's better to be open for new ideas and be ready for changes. What is good tool for work today, would be old in next 10 years.

Java XML object binding libraries

I have to implement mapping between some domain model object and xml document representation. It's actually working production application. Xml schema could be suited to needs of current application. I'm thinking about two free/open source libraries.

Sun promotes Java add-in support for JAXB standard. It looks heavier, and has harder initial learning curve. Last implementation of JAXB 2.0+ gives tools (lek automatic class generation) and aids to simplify development (annotation based mapping definition). So JAXB as officially supported standard should gain more user base.

The good and tested third party solution is JXM (Java XML Mapping). It's default configuration allows fast implementation and mapping of desired object structures. You only have to keep standard node and attribute names, that is right if you are designing XML from scratch using "java code centric" approach. Other possibilities are also enabled by adding some marshaling/unmarshaling code.

As Blogger.com added polling feature, I've posted poll, so readers could give opinion about their favorite XML/object mapping Java library.

Polishing PHP application - Role of tests

Real life scenario:
Slightly changed requirements from category "eventually" - gives as result quite lot code changes.
An extra problem was that in one production environment it have to work on PHP 4, so application needs downgrading from PHP 5 mainly in XML processing area.

Solution:
Writing more tests for new features.
Gradually code fixing and refactoring.
As a result I have more clean, better working code with new features and potentially new undetected bugs :) .

Savings from using unit and regression testing are almost priceless.

Trying to do some changes like that in PHP production code without automated tests will be really painfull.

I'm busy - new Java project

I'm quite busy now, while setting up new project. It will be Medical Diagnostic Expert System using machine learning algorithms. I've got legacy applications preceding that project, that could be used for their core algorithms implementation. Besides that, the whole application must be build from scratch, to meet new requirements. I'm working now on data gathering services and user interface for experts modules. It is designed as web application, implemented in Java and using Spring framework.

Automated time tracking with Rescue Time

Sometimes I have mixed thoughts about spending all day with computer. I know what I have done that day, but I know I could do that in a more productive way.

Time is passing in a tricky way while you are sitting in a front of a screen. Sometimes tasks categorized as "easy" took only 15 minutes, other time its 3 hours, and you feel only "small" difference between both cases. Other issue is actual context of task that includes communication, looking for references, research etc. Many times work ends on reading "more interesting" site or blog.

Here is an automated time tracking software from Y Combinator startup named RescueTime. It makes possible "Effortless Time Tracking" under Mac, Windows and thanks to "an enterprising Linux fan" for Linux. It automatically tracks time for every application, web address and gives reports by applications, categories, and user defined tags.

Cons: I see only one- development team still works on recognizing java applications. Now all Eclipse and java applications are seen as Java. Something about privacy and tracking web addresses: you can define white/black list for web addresses, and daily hours of tracking (for example since 8:00 till 16:00). Urls are also truncated to not track irrelevant data. So software can be properly configured and used in "a safe" way.

The graphical reports give very informative strategical overview of time usage. You can also set your time management goals and track your success. They also have business plan that enables team activity tracking and comparing results.

Now I'm going to check share of my time wasters.

Good method of web frameworks comparison for programmers

If you are programmer and want to choose the web frameworks suitable for your project, one of the best ways of framework evaluation (not fastest) is writing simple prototype that uses concepts similar to your project requirements. Opinions of other experienced frameworks users also counts, but sometimes framework selection is based on personal preferences. Thanks to prototype code all framework sweet spots and "need to be done" areas are easily exposed. Prototyping gives also practical measure of framework impact on overall development productivity.

Last week I have done little research about new trends in Java web frameworks, and found very good and detailed frameworks comparison based on prototyping on a Peter Thomas blog.
Here are articles comparing Wicket framework to Spring MVC, JSF and GWT.
Good job Peter Thomas!

By the way, I have found many Wicket positives in his articles, that I give that framework a try in my next project requiring component based UI.

Mature programming languages and internationalization

Have you ever tried to implement application using character encoding other than Latin 1? What is deal with that? If you are using modern development platforms like Java or .Net - it just works. Let's omit other problems caused by bad implementation of internationalization issues in framework/library. Multilingual application scenario is simple - use internal UTF character encoding, allow transcoding from external systems at the API level (I/O operations with text files, databases etc.) - and your application will be able to write messages in most standardized character sets including Russian and Chinese.

What about other popular programming languages? Take for example Python 2.5 - recent version that could be used in production environment. It supports UTF strings processing, and character transcoding- but be aware - there are glitches and unimplemented UTF features, causing bad UTF processing. To get it work you have to use (sometimes ugly) workarounds. Here is nice presentation about these issues. There is hope that developers from open source community will fix that till let's say version 3.0.
Ruby is in better situation- as a younger language it has more properly adressed internationalization issues at the design stage. I suspect there is still more work to do.

What about old work horse of web applications development- PHP in newest stable version 5.5.2. It came a long road, but still have problems caused by legacy of non integrated separated libraries. It has long list of options to set encoding like internal encoding, http input output encoding, and many more affecting single libraries.

So what about using mbstring module (multibyte string representation) , setting UTF as internal encoding and sticking to it. Situation is worse than in Python- many even simple string functions can't process multibyte strings in proper way. The Solution is to write own workarounds or use third party libraries like PHP utf8. Here is good list of PHP internationalization issues. Last week I have checked xml import support for old PHP application- it caused problems with Latin2 encoding. The reason was simple: The XML SAX parser supports only ISO-8859-1, UTF-8 and US-ASCII encodings.

Such low detail problems could potentially take a lot of precious project time. So if your next application will use character set other than Latin 1, you should consider good internationalization support by chosen development and production platform.

Summer is ending - three online games to get more fun of life

The summer is going to end - it gets cold and rainy. The weather takes out fun from outdoor activities. So I'm presenting three online games for autumn (winter) gloomy days.

The first one is the Intrusion - platform shoot and action game with bosses etc. reminding old console classics. It works very smooth, and custom physics makes play pretty enjoyable.

The second is Fantastic Contraption - it is more thinking/creativity kind of. It's in an early stage of development, but already gives the fun of play now. The beauty of the game is in that there are so many ways to solve given levels.

The third is Globulos that needs use of tactic and quick thinking. Globulos is in the business for some time, and every new feature added before time made game only more cool.

I wish more successes for those good products, and I'm waiting for more news and features.

Google Chrome - one more step on the way for perfect client for web applications

Google Chrome browser gets a lot of attention from the start. It is browser build from scratch for better performance, usability and security. Isn't it good solution for web applications, even for enterprise level? As Forrester Research analyst Sheri McLeish says in Information Week article Google's Chrome browser "is not top of mind for IT organizations". It's still in beta, and Google is the only main supporter with its unclear reasons.

There can't be "perfect" browser for everyday surfing, and in the same time using enterprise web applications. But it's good step on the way to "perfection". In worst case the best features could be adopted by other web browser players. On the other hand, if chrome goes out from Beta stage, it can take big share of users by its usability and performance.

I will be watching progress.

DVR or custom built PC media center?

Under pressure of family I extended subscription of cable network. That means more potentially interesting programs to view. I still have the same amount of time for all my activities. I agree with the term somebody coined, that is "TV - the weapon of mass distraction". I want to take my time on my way, so I thought about some kind of DVR (Digital Video Recorder). On the other side I have seen cool media center box placed in my friend's living room.

There are many DVD recorders with hard disks and DVR functionalities for good price, but I'm not sure about its usability. I tempted to build my own media center/set top box connected to Internet, but it will probably take more time and money. But it also gives more fun. I have to rethink my needs and choose something useful.

IT pro life anniversary

It passed one year since I started writing this blog. It serves me good as outlet of me geek activities. I'm not a "pro" blogger but I like writing. I think it started when I have been writing posts to closed email groups, sharing ideas and experiences with friends. Now my writings are more public and I can get to more readers interested in that subjects.

I maintained weekly posting that gives 55 posts for now. I get small feedback but almost all positive.

I have made mistakes, some of it I know I could fix now. There is so much things to improve.

I would be grateful for your opinions.

Best Canon CHDK features for everyday photo shooting

I really used to take pictures with my Canon digicam equipped in CHDK software. During summer vacation I took many pictures and some short movie clips. I had to switch full SD card for second empty one - without CHDK. When I started shooting again, I realized how I miss those additional features.

The top most features I use even during casual shooting is zebra and live histogram. Zebra simply highlights under/overexposed areas, that fall beyond camera light range. Histogram helps balance lights level in shadows, middle tones and bright areas. Those two save me many experimental, "light probing" shoots until I get expected result. It's very easy for use feature, even for non experienced photographer.
I don't mention other, less used features that can really expand photographing experience, if you really get into it.

Next time I am installing CHDK on all of my SD cards to not miss opportunities when I get my camera.

Finished summer (and computer) break

I am back. At the computer and broadband internet.

Some break was needed for mind rest. I feel slightly changed my point of view.

It's time to get back to work. I have to check email and all online services I use.

Back on track - but I feel urge for some improvement.

Vacation - planning day with weather service

I am spending vacation at the polish Baltic Sea shore. The place, where the summer weather is a mix of wind, sun and rain. In that circumstances, good weather forecast service is really helpful.
I am checking weather forecasts at ICM institute site. It uses numerical prediction model for 2 day short horizon forecasts. The forecasts are detailed and precise for selected area. Unfortunately forecasts are available only for Europe. That forecast service is really good. Many times it saved me biking in the rain.
So now I am checking when it stops raining so I could walk on the beach.

Driving with GPS navigation makes me stupid

For long distance journeys through unknown roads I use GPS navigation software running on old Pocket PC. Directions given by automate navigator are so precise that you can drive all way even without looking at road signs. And this is the worst part.
At the beginning I felt very confident. When navigation was turned on my natural navigation skills gone away. It felt like some kind of brain damage effect. Part of my thinking was turned off in exchange for blindly listening to navigator instructions. Imagine what is going on when map is not actual or has some glitches. "where is that road it is talking about?". The result is often big surprise and embarrassment.
Its scarry when machine can make you fool and put into potentially dangerous situation on the road. After that experiences I always check my route on map before journey, observe road signs, and listen to navigator as the last resort to ensure that I am right.
I feel much better that way.

Taking 2 week summer vacation

I'm going today at summer vacation. I will be back in two weeks. Meanwhile, I will try to post something using smartphone (I don't take any laptops - it's rest time). The 8 hour car driving in the night is still before me, so I have to finish packing and catch some naps through day.

It's going to be fun.

Microsoft position in open source community

Probably you know the Apache Software Foundation- one of the greatest source of production ready open source solutions, like famous Apache web server. And the Microsoft - the world wide corporation that earns money for their OS, Office suite and server software and seems to stay at opposite side. At 25th of July I've read CNET article about announcement that Microsoft is becoming a sponsor of the Apache Software Foundation.

Are Microsoft trying to change business model? Some kind of mix where open sourced subprojects will be used as parts of commercial products? Or just it's strategic move to have own people in open source community, to influence some future software directions. Officially, it seems that they are trying playing nice with open source. The Redmond giant doesn't want to be an enemy for everyone and sends white flag to open source crowd.

Undoubtedly, behind that are serious political and business reasons.

Diving into Sakai project

I'm diving into Sakai project with intention to evaluate it as Learning Management System for my University.

Description from original web site:

"Sakai is a set of software tools designed to help instructors, researchers and students create websites on the web. For coursework, Sakai provides features to supplement and enhance teaching and learning. For collaboration, Sakai has tools to help organize communication and collaborative work on campus and around the world."

It's mature and well supported platform, targeted for large institutions. It uses some parts of J2EE technology with light approach. It's built on Spring framework and can work on Tomcat server.

Probably there will be need to customize some parts of platform to conform local requirements, so I have to check architecture and enchantment possibilities.

The Java performance upgrades

Suns JVM (not to mention other JVM vendors) gains more and more in performance benchmarks. According to SPECjbb2005 benchmark data presented at Java One 2008, Java 6 get more than 100% speed prior to Java 5 at x64 platform. Such big gains were possible due to fact that applications and JVM were not so optimized to work at 64 bits. Many tests show that applications on x64 undeperformed to that on 32 bit platform.

For 32 bits JVM get smaller but still significant improvements. I felt that on my skin when switched from JVM 5.0 to newest JVM 6.0. Interactive applications work snappier now.

The direction of Java performance upgrades is in concurrency. To get good multi core processor utilization, new applications have to work in multi threaded way. If programmers won't write applications in the "new way", then maybe JVM and OS could better use multi-core as "back end". There is idea of implementing transparent internal concurrency for basic algorithms and data structures like collections or maps from Java libraries.

Latest JVM can raise desktop applications performance by two digits, but there is a lot of custom work for scalable server applications. Scalability still heavily depends on utilization of resources, memory in particular. It's still open field and solutions depend on application.

Tools for static java code checking

Java language with its strong typing is well suitable for static code analysis.
There is variety of tools for early bugs and flaws detection through automated code checking. I mention three open source tools that any java programmer should give a try:
- PMD
- CheckStyle
- FindBugs

All tools have set of flaw detection rules. Every one has some strengths and weaknesses - all three are complementary to each other. I suggest you to run every of it and look at the different kind of flaws they explore. with default rules. It's possible of course to define own set of rules suitable for your needs. I've tried to install eclipse plugins for those tools - only PMD runs without problems. Maybe I should refresh my Eclipse 3.1 installation. PMD has a "copy and paste" duplication detection tool- but somehow it is not working with some characters in UTF-8 encoding. I hope it would be fixed - it's really useful tool.

So install code checkers - and view reports of your code. You can be surprised. It gives the opportunity to chop off the heads of some simple but nasty bugs.

Illness and days without (almost) computer

I had no prepared post for that week. I get ill and I have to stay in bed. For past few days the best thing I was being able to do was sleeping. I'm trying to reduce time while lying in the bed with laptop on my belly, so now is moment I can write anyghing.

Such a pity that during good weather I have to stay at home.

Day off and mobile internet

Disclaimer: That post should be published one day before, but it is today, due to unusual mobile posting problems.

I took one day off to enjoy summer in beautiful nature surroundings.
Being in possession of cell phone with some smartphone possibilities, I have decided to try out mobile internet.

Few days ago, I have turned on gprs service at my cell provider. Everyday I have been surrounded by computers, almost all connected online.

What for do i need internet on my cell? One reason are convenient online services kille weather or dining searching. The other explanation is that I am computer/internet addict.
I have tried some useful software on my nokia s40 phone.
Here is my list:
Opera mini - good for search and news browsing
Gmail mobile - explanation is not needed
Midpssh - nice ssh client - for use in critical situations
Some IM ,SIP VOIP client - still searching
The real pain is text input. It would be nice to use better method than T9

Enough. I am going for a walk

Analysing SleepTracker data for R.E.M. phases

I'm collecting sleep data from my Sleeptracker watch for above 3 months. It's a good time to check what is interesting in there. My goal is to check pattern of my sleep R.E.M. phases.

First, I get my sleep data in csv format stored on www.sleeptracker.net site. Then using simple python script I've converted all night/time data into relative values. Now I have all SleepTracker movement occurrence time saved in "hours after going sleep".

After looking at initial data I was a little bit disappointed. I have not seen clean gaps for non R.E.M phases. Picture was so cluttered. There were many possible reasons. I've choosen one - sometimes I go sleep at different time, but sleep patterns may be anchored to usual hours.

I decided to make more preprocessing to move out "outstanding" data. I removed data instances with non standard "going to bed" times, movements it the first hour of sleep and some made some minor tweakings. When I viewed data again the picture was much clearer.



I rerun data clustering software using standard kmeans method and get some results:

Cluster 1
Mean/Mode: 0.8458
Std Devs: 0.2783

Cluster 2
Mean/Mode: 1.5433
Std Devs: 0.2069

Cluster 3
Mean/Mode: 2.6117
Std Devs: 0.2857

Cluster 4
Mean/Mode: 3.6685
Std Devs: 0.236

Cluster 5
Mean/Mode: 4.8086
Std Devs: 0.2848

Cluster 6
Mean/Mode: 5.9405
Std Devs: 0.4276


That means that probably REM phases occurs about 0.85, 1.5, 2.6, 3.6, 4.8 and 5.9 hours after falling asleep. Common value found in literature is about 1.5 hour between R.E.M. phases.
Bigger standard deviation in later vs earlier phases reflects longer R.E.M. phases at each cycle, what is also found in sleep research. First cycle must be treated in special way - the sleep pattern is different sudden after falling asleep.
The last one can be also less reliable because of standard waking time after about 6 hours.

I decided to pick R.E.M. time values by my self using "smoothed" histogram of sleep data. Local modalities are clearly visible on picture.


Now middle of phase values seems like: 0.8, 1.6 , 2.3 , 2.8, 3.4 , 4 , 4.9. Comparing these results with those from automatic clustering method it seems that "k means" method joined 2 close groups together.
I assume that manually picked results are better.

Time for conclusion.

To get more reliable results I need more data and regular sleep :) . SleepTracker is quite effective by using simple accelerometer movement detection method. Unfortunately many movement events aren't recorded (for example the subtle ones or when the arm is blocked under the pillow). I will try to catch more good data and check results again.

Summer is coming

It's really hot for last few days - real summer weather. I'm often going out for outdoor activities.

I'm finishing production stage of one of bigger projects I'm working now. I'm thinking about summer time and resting in some nice place.

Price of gas is arising, but I'm planning only one long distance trip. For other destinations I'm going to visit some places near by, maybe do some biking.

Anyway- my home computer is more turned off, and my new projects are going on the shelf for some time.

PyCha chart drawing library working under SPING

I've ported PyCha chart drawing library to work under SPING instead of Cairo. I've discussed that idea with PyCha lead developer - Lorenzo Gil Sanchez.

Lorenzo said he will still stick with Cairo, that is good library ported and available for many systems. Anyway I've ported PyCha to SPING and tested with SPING creating SpingCha off-project.


Here is lastly created PyCha news group , where you can find posts about PyCha development.
And - if anyone is interested - working SpingCha release in files section.

Fighting with duplication - attack of the clones

Last week I proceeded with clean up tasks on one project I'm working now. It's big number of jsp generated reports. Some time ago I setup initial environment for such reports, so even analytics without special JSP or JAVA knowledge could do some reports. It was temporary solution, before jumping to more enterprise tool. Results were not so bad - new reports were added very quick. Everything worked and was usable so enterprise tool is just forgotten mention.

Over two years it overgrowth overall code base and many problem issues and problems were related with those reports. In harder cases I had to dig into that and what I have seen is a massive duplication. I any possible form. So when we found bug in one section of code there was huge probability that the same problem was duplicated more times elsewhere. It was even worse. Instead of new features added to base report there where created new versions of the same report with new parameters or features. And divergence between those reports arises over time. So you can imagine how much time maintenance started cost at some point.

I like to stick a specially with one coding rule: do not create new and remove existing duplication.

Duplication starts when some functional block of code appears at least in two places in code. Sometimes it is just simple one liner - but even then you should consider pros and cons for wrapping that piece of code in some procedural statement.

Let's back to vicious mechanism of avalanche of duplicated code in non programmers environment - when people don't know good "coding" practices. One of reports (let's call it A) was copied with some functional difference (as B). We have got almost two identical files. Then comes another feature that seems to interfering with previous one. Because new feature ticket was assigned only to one of those reports (after some time everybody sees two reports and thinks - those are two separate reports) and new feature seems so "new" and practice before was to create separate functional report (for "clear solution") so there comes new report C derived from A. After some time somebody realized that there is no such feature for B report. Continuing previous process we have now 4 reports with D version. Then comes major change - added new module that shows new values in similar manner so base reports A,B,C,D are copied as 2A,2B,2C,2D and then changed some values and layout. Almost pure copy'n'paste coding.

Having one simple report at the beginning that would have parametrized additional features it's now eight versions of duplicated code with less than 10% differences. Cost of removing bug or implementing new feature is to about eight times bigger than for single but more complicated file. Not including effects of further "extension" mechanism - avalanche just gains new mass in exponential rate.

You could think what kind of procedures you have there to allow that kind of practices. Ok - team of report analytics was told to eliminate visible duplications, but it wasn't enough. When somebody concentrates on complex analytical problem doesn't think much about removing duplication. Even when report is complex and amount of code and sql is enormous. It's just additional burden that seems not helpful for analytical problem solving.

What could help? Some quick course of coding practices and techniques in context of that environment - basics how to write reusable pieces of code, avoid common pitfalls and reduce codebase to ease development and maintenance. And more practical explanatory examples. Sometimes duplication problem is visible but there is no simple solution - then pairing with more experienced programmer would help. It's just organizational issue.

So now the team is fighting with reports bugs and feature requests. And the biggest impact we are gaining now by not touching those new features or bugs but by merging and removing duplicates. It's ironic but number of bugs is decreasing now in reverse exponential rate.

CHDK - second life for Canon point and shoot digital cameras

When I've bought my Canon digicam, I've heard rumors about additional software uploaded into Canon cameras. But last year there was still no support for my A570 model. Last week I've found article on Lifehacker mentioning hacking Canon firmware.

I've uploaded CHDK software using SD card reader, and I'm playing now with new features. There is lot of interesting things like making RAW pictures, overriding default range of exposure settings, bracketing and lots of cool of tools like live histogram and under/overexpose region marking.

It feels like I've got new camera - it's indeed second life for that piece of hardware.

I'm trying to get some HDR pictures now using RAW format and auto bracketing features.

So, happy shooting!

JavaOne 2008 - Java+You

Last week Sun organized JavaOne 2008 conference in San Francisco. I weren't on place, but there is lot of interesting information on conferrence web site.
There were tracks for every of Java technologies, applications in web, SOA and some new ideas introduced like JavaFX - new (?) rich interface and scripting platform.

I invite you to reading, and see what are proposals for Java future.

Not too much sports

I'm after weekend bike trip. I had such muscle aches I could barely sleep. Definitely I have to to do something with my fitness.

It's hard for me to schedule such activities as these trips, but I'm thinking about another kind of sport activities. Maybe jogging- it takes less time and is more exhausting.

Maybe I should think about some in home gear like walking treadmill or fitness bike.

Do you have any good ideas how to make work together keyboard time, some kind of sport and dense deadlines?

Urban Terror game - FPS fun with grain of tactics

Once upon a time I started digging in search for good FPS game for Linux. I liked FPS games with more "reality" and tactics demands like Counter Strike. I used to play some dynamic FPS for 15 minutes instead of cup of coffee- in most cases it worked perfectly. I reviewed some game candidates and found something that matches perfectly. My choice is Urban Terror.

The game was made as a mod for Quake 3, and nowadays it has standalone version with installer and q3 engine within it. All game documentation and downloads could be found at official site www.urbanterror.net. Game is free, but has special license for Q3 engine.

You can play on many public servers as well as in local net if you run game on one host in server mode. There is also possibility to play with Q3 bots, but documentation says it's not supported. The game crashed quite often on my Windows XP laptop- but probably it is issue with bots and/or graphic chipset. Under Linux it works like a charm :).

Game is very dynamic, although sometimes it's better to use tactic and think before rushing. Wide choice of weaponry balanced with simple but quite "realistic" health, energy and wounding system makes game very playable. The game is rather popular so you can find interesting information on community sites.

I don't have much time to play games but sometimes it is real fun. Enjoy the game!

Alternative Human Computer Interface for cheap (Wiiiii)

More efficient or just possible with computer interaction in hard cirtcumstances is one of my dreams (so tech geeky). I specially like input methods without standard keyboard. It's good to have touch screen and/or some kind of wearable pointing device.

I've been trying to get some results using web cam and motion analyzing software- to move mouse cursor or activate functions by waving hands. I haven't enough time to get it succesfull.

Then I saw some of Johny Lee Wiimote projects results- it's really impressive. All thanks to internal processing inside wiimote- all you get on output are coordinates of tracked points and values from accelerometers.

I've never been into wii- just another game console. One positive- it was introducing new kind of games. I'm reading about wiimote possibilities and it looks it gives a lot of possibilities for affordable price.

Probably I give up my web cam attempts as Johny Lee wrote on his blog:

The wii remote contains a 1024x768 camera with built-in hardware for IR blob tracking of upto 4 points at 100Hz. This significantly outperforms any $40 webcam I'm aware of. It'll work with a webcam, just not as well and not as easily.


I'm thinking to get one of these to hack it into something usable.

Real world update deployment scenario

I'm thinking about automating updates deployment for one of the commercial applications I work on. It's hard to find good ready "out of the box" solution and below are the reasons why:

• maximum simplifying deployment
• it has to manage applications major updates and minor patches
• deployment automatically should stop production server, make backups for easy revert, update application code and database, start server and make some test to check if it's started (that part is already working)
• multiple deployment servers- independent installations
• find out new updates on central server using pull strategy
• it has to manage in almost unique way two platforms- windows and linux
• it has to be scheduled to out of office work hours
• some target servers have outstanding deploy time window- so deployment time could be set for local site
  
• some target servers need patches earlier- like some quick fixes that shouldn't be spread anywhere else
• some target servers are more "conservative"- we need additional agreements with customer to make updates- so we are making updates less frequently
• automatically apply all needed updates and patches up to date- it could be really nice feature

The most tedious work is already automated, but rest still needs manual operation. This is at least 1.5 hour per each full deployment- it looks I can cut it down. So after looking at some commercial solutions, I decided to make custom tool using java ANT and some helpers.

Do you know any utilities that can manage most of those requirements?

Sleeptracker - gadget for sleep optimizers

In last month I get new gadget- Sleeptracker Pro alarm watch.
Short description how it works from original website:


SLEEPTRACKER® continuously monitors signals from your body that indicate whether you are asleep or awake. Because you wear SLEEPTRACKER® on your wrist like a watch, its internal sensors can detect even the most subtle physical signals from your body. SLEEPTRACKER® finds your best waking moments, so that waking up has never been easier.


Manufacturer created positive buzz amongst community. I have read so many positive reviews that even price tag (about 180$ in US for Pro model) doesn't stopped me out of buying it.

Some remarks about alarm clock features:
- clock with date on quite big display
- one alarm time
- sound and/or vibration alarm (sound and vibration may be too silent for very hard sleepers!)
- back light is nice
- can be worn as casual watch, but can be too bulky for some people

Because it's not normal alarm clock there are additional outcomes:
1) It wakes you in the moment (within set time window) you are better prepared to wake up, so you feel better
It generally "feels" true or it's hard to say without comparing statistics with and without Sleeptracker

2) It is shortening your sleeping time by about 15-30 minutes (it depends on sleep schedule) without negative consequences.
It's strong dependable on sleeping schedule and works best for unoptimized sleep with waking time in the middle of sleep cycle. On the other way every day sleep patterns may vary a little so monitoring device it's catching actual cycles. For me alarm clock sets up between 0 and 40 minutes before usual waking time.

3) It records R.E.M. activity during night, so data can be collected and analyzed to improve sleep habits.
Sleep data can be reviewed on watch or Sleeptracker Pro has USB port to transfer data to computer.




There are important rules I figured out that should be considered before buying that stuff:
- Sleeptracker will not work for you if you don't have enough sleep or have heavy sleep disorders
- alarm sound is silent so get enough of sleep to be ready top wake up in the morning
- it did not worked for my polyphasic sleep schedule (look at get enough sleep point)
- watch is worn on wrist, so if you place your arm under pillow or in other position that could block arm movement, Sleeptracker won't record R.E.M. phase due to lack of movements
- it may won't work for people who don't move their arms at during R.E.M. cycle (it's hard to check but it's possible)
- have I told that you should get enough sleep?

Sleeptracker Pro package contains USB cable, printed manual and little CD with software for windows. Interesting solution is used for connecting USB cable to watch. Cable ends with clip that have 3 springing pins matching connector on the backside of watch. I didn't mentioned the watch is waterproof!.

Sleeptracker software is used for transfer sleep data from watch and tracking that data with a day events like drinking coffee or alcohol, sleeping light and temperature conditions etc. I tried to run that software under WINE on Linux, but USB data transfer did not work.

So I tried to find another solution. I found the PERL script to get data from USB cable on sleeptracker forum. I found two version of script with bugs or unimplemented features so I integrated both to get one better. Instead of desktop software it can store data as text or xml files. The next nice feature of script is export to web version of Sleeptracker software at http://www.sleeptracker.net/. I'm still collecting data so later I report my findings.

Below is mentioned PERL script:

#!/usr/bin/perl -w

# This program downloads records from your Sleeptracker Pro
# and displays them as text, xml, or csv
#
# Sleeptracker and Sleeptracker Pro are trademarks of
#
# Innovative Sleep Solutions
# 2870 Peachtree Road #140
# Atlanta, GA 30305
#
# This program is NOT developed or supported by Innovative Sleep Solutions
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# You can purchase your Sleeptracker or Sleeptracker Pro from:
#
# http://www.sleeptracker.com
# http://www.sleeptracker.de
# http://www.sleeptracker.fr
# ... and some other stores
#

# Chagelog:
# --------------
# 09.12.2007: fixed bug in xml output tag
# 10.12.2007: fixed wrong cecksum calculation
# 11.12.2007: added browser output
# 13.12.2007: fixed seconds from DataA
# 14.12.2007: fixex wrong eff.alarm time, when no alert in window
#
#
#


use Device::SerialPort;
use Time::Local;
use strict;


my $browser="firefox"; # which browser to call
my $uploadURL="http://www.sleeptracker.net/import.php"; # which upload URL to call

my $numArgs = $#ARGV + 1;
my $ttyPath="";
my $format="browser"; #set default

if($numArgs > 1){
$format=$ARGV[0];
$ttyPath=$ARGV[1];
} elsif ($numArgs == 1){
$ttyPath=$ARGV[0];
} else {
help();
exit 1;
}

if (not -w $ttyPath) {
print STDERR "Can't access Sleeptracker at device '$ttyPath'\n";
exit 1;
}

print STDERR "Reading Sleeptracker data from $ttyPath\n\n";

my $serial = Device::SerialPort->new("$ttyPath");
$serial->baudrate(2400);
$serial->databits(8);
$serial->purge_all();
$serial->rts_active(0);
$serial->dtr_active(1);
# Send request
$serial->write("V");
# Wait one second
select(undef, undef, undef, 1);
# Read response
my($count, $data) = $serial->read(250); # goog for more than 70 awake moments

if($count<15){
print STDERR "Error while reading from Sleeptracker!\nWatch showing DATA screen?\n";
exit 1;
}

my @stBytes=();
my $cnt=0;
my $crc=0;
foreach my $byu (split (//, $data)) {
$cnt++;
# for debug
#printf "%03d: %02x = %d \n",$cnt,ord($byu),ord($byu);
push @stBytes,ord($byu);
if($cnt>1 && $cnt<$count-1){
$crc+=ord($byu);
}
}
#printf "%x=%d\n\n",$crc,$crc;
$crc=$crc%256;

# compare checksum
if($crc!=$stBytes[$count-2]){
print STDERR "Checksum Error while reading. $crc!=$stBytes[$count-2]";
exit 1;
}


# calculations and data formating

my $pos=1;

my (undef,undef,undef,$mday,$mon,$year,undef,undef,undef)=localtime();
my $jahr=$year+1900;
my $monat=$mon+1;
my $tag=$mday;

my $stTag=$stBytes[$pos+1];
my $stMonat=$stBytes[$pos+0];
my $stJahr=$jahr;

if($stMonat>$monat || ($stMonat==$monat && $stTag>$tag) ){
$stJahr=$jahr-1;
}

my $datum = sprintf "%02d.%02d.%4d",$stTag,$stMonat,$stJahr;
$pos+=2;

#$unknown=$stBytes[$pos];
$pos+=1;

my $window = sprintf "%2d",$stBytes[$pos];
$pos+=1;

my $toBed= sprintf "%02d:%02d",$stBytes[$pos+0],$stBytes[$pos+1];
my $toBedTS = timelocal(0,$stBytes[$pos+1],$stBytes[$pos],$tag,$monat-1,$jahr);
$pos+=2;

my $alarm =sprintf "%02d:%02d",$stBytes[$pos],$stBytes[$pos+1];
my $alarmTS=timelocal(0,$stBytes[$pos+1],$stBytes[$pos],$tag,$monat-1,$jahr);
$pos+=2;

my $cntData=$stBytes[$pos];
$pos+=1;

my @almostAwake=();
my @almostAwakePst=();

my $lastAwake=$toBedTS;
my $thisAwake=$lastAwake;
my $alltimes="";
for(my $i=0;$i<$cntData;$i++){
my $aaw=sprintf "%02d:%02d:%02d", $stBytes[$pos+($i*3)+0],$stBytes[$pos+($i*3)+1],$stBytes[$pos+($i*3)+2];
my $aawShort=sprintf "%02d:%02d", $stBytes[$pos+($i*3)+0],$stBytes[$pos+($i*3)+1];
$alltimes.=$aawShort.",";
push @almostAwake,$aaw;
$thisAwake = timelocal($stBytes[$pos+($i*3)+2],$stBytes[$pos+($i*3)+1],$stBytes[$pos+($i*3)+0],$tag,$monat-1,$jahr);
if($thisAwake<$lastAwake){
$thisAwake+=(24*60*60); # day has changed
}
my $slept=sprintf "%d",($thisAwake-$lastAwake)/60;
$lastAwake=$thisAwake;
push @almostAwakePst, $slept; #period sleeping time in sec
}
$pos+=($cntData*3);

if($thisAwake<$toBedTS){
$thisAwake+=(24*60*60); # day has changed
}

my $getUp=$almostAwake[@almostAwake-1];

# when the last awake moment was outside the window, the alert goes off at alarm time
if($lastAwake<$alarmTS-$window){
$getUp="$alarm:00";
}

my $dataAVal=($stBytes[$pos+1]*0xff)+$stBytes[$pos];
my $dataASec=$dataAVal%60;
#my $dataA=sprintf "%d:%2d",(($dataAVal-$dataASec)/60),$dataASec ;
#my $dataACalc=sprintf "%d:%2d",(($thisAwake-$toBedTS)/$cntData)/60,(($thisAwake-$toBedTS)/$cntData)%60;
my $dataA=sprintf "%d:%02d",(($dataAVal-$dataASec)/60),$dataASec ;
my $dataACalc=sprintf "%d:%02d",(($thisAwake-$toBedTS)/$cntData)/60,(($thisAwake-$toBedTS)/$cntData)%60;


# output Browser
if($format eq "browser"){
my $URLParam=sprintf "a=%s&w=%s&t=%s&dt=%s&da=%s",$alarm,$window,$toBed,$alltimes,$dataA;
my $callFirefox=sprintf "%s \"%s?%s\" &",$browser,$uploadURL,$URLParam;
print "\n";
print $callFirefox;
print "\n";
my $ret=system($callFirefox);
}

# output Text
if($format eq "text"){
print "Date: $datum\n";
print "To Bed: $toBed\n";
print "Alarm Time: $alarm\n";
printf "Effective Alarm Time: %s\n",$getUp;
print "Window: $window min\n";
print "Data A (Clock): $dataA min\n";
print "Data A (Calculated): $dataACalc min\n\n";

print "Awake moments ($cntData): \n";

my $aawCnt=0;
foreach my $awm (@almostAwake) {
printf " Data %2d: %s (slept: %3d min)\n",$aawCnt+1,$awm,$almostAwakePst[$aawCnt];
$aawCnt++;
}
print "\n";
}

# output CSV
if($format eq "csv"){
printf "%s;%s;%s;%s;%s;%s;%s;%s",$datum,$toBed,$alarm,$getUp,$window,$dataA,$dataACalc,$cntData;
my $aawCnt=0;
foreach my $awm (@almostAwake) {
printf ";%s",$awm;
$aawCnt++;
}
print ";\n";
}

# output XML
if($format eq "xml"){
print "\n";
print " $toBed\n";
print " $alarm\n";
printf " %s\n",$getUp;
print " $window\n";
print " $dataA\n";
print " \n";

my $aawCnt=0;
foreach my $awm (@almostAwake) {
printf " %s\n",$aawCnt+1,$awm,$almostAwakePst[$aawCnt];
$aawCnt++;
}
print " \n";
print "\n";
}



sub help {
print STDERR "Usage: sleeptracker [format] device\n";
print STDERR "format: browser(default), text, csv, xml\n";
}


Polyphasic sleep - sleeping less and better? - Not really

I finished my polyphasic sleep experiment. I just couldn't catch up enough sleep in proper period. Kid wakes me up twice per night - so it's hard to maintain core 4.5 hour sleep block. It was very hard to get up and you can imagine the rest of the story. I guess that my overall "time and productivity" ratio decreased. I lost my dream memory too- it's an early sign of sleep deprivation for me.

So I decided to break it and gain more rest from my classic 6 hours sleep schedule, plus occasional power nap at late afternoon. After about two weeks of adaptation I feel like I had never changed my sleep habits. I feel more rested now, so probably it was good move.

The most important lesson I learned during that experiment:
Shortened sleep time needs maxed out rest results during sleep. Taking naps with regular schedule and doing it in comfortable way (comfy place, silence and darkness) is a must.

Maybe next time when I could organize my bed time without interruptions, I will try polyphasic schedule again. Besides, I'm still working on improving my sleep habits.

Thought of a day- unplanted easter eggs

The funniest easter eggs you can find sometimes browsing source code with distance. There are unintentional pearls of some kind of evil genius.

Digitize your life

Everyday I try to organize my life using mix of paper and electronic means. Sometimes workflow between different areas is seamless, but in most cases I finish up with nothing more that little scrap of papers and information bits. All data is scattered across dispersed information islands. I have to be very fast to visit them all.

Have you heard about MyLifeBits project? It's based on an MEMEX idea of ubiquitous personal information storage. It may include pictures of what you are seeing, sound you are hearing, coordinates of your current position, trail for your electronic activity like emails, phone logs etc. All data is interconnected, tagged and ready for text searching. It's compelling idea but more important are practical applications. The whole stored information may be used as a kind of human memory extension. For successful usage I think it's needed to invent more powerful tools helping seamlessly storing and automatically retrieving important for as bits of information.

Drawing of Bush's theoretical Memex machine (Life Magazine, November 19, 1945),
picture posted by p373
Some rights reserved.

I need now only small part of that information, and probably make simple home brew personal information workflow system using pack of separate gadgets like cellphone with camera with GPS and some software. What is it for? I'm not sure yet, but maybe it's worth to try something like that to use it in future. And again another question - what today information will be worth in future after next 10 years?

I'm going back to figure out how to connect online calendar software with one on my cellphone.

Firefox2 and css caching issue - part 2

I have tried out to change settings for expire headers, and figured it that application adds its own headers to dynamic generated css files. Then apache2 mod-expire tries to add own settings.
After tweaking application settings there are resulting headers:

Date: Mon, 1 Mar 2008 06:46:02 GMT
Server: Apache/2
Cache-Control: max-age=600, max-age=604800
Expires: Mon, 6 Mar 2008 06:46:02 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 18128
Content-Type: text/css


Expire times:

• max-age=600 - given by apache2 global settings


• max-age=604800 - given by applications itself

It's ugly - but surprisingly caching works under Firefox with these headers. I haven't checked yet which expiration time is taken by Firefox. I need to clean it and remove one of extra headers anyway.

Below are old headers for the same resource fort comparison - when Firefox caching was not working:


Date: Sat, 01 Mar 2008 07:28:31 GMT
Last-Modified: Sat, 01 Mar 2008 07:26:56 GMT
Expires: Sat, 01 Mar 2008 07:36:56 GMT
Cache-Control: max-age=600
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 18128
Keep-Alive: timeout=10, max=98
Connection: Keep-Alive
Content-Type: text/css


There are removed last-modified and keep-alive headers, but it's hard to understand how it can change Firefox caching.
I have to check difference between 600 and 604800 expire age, that may be solution.

Probably all headers should look like those added by apache mod-expire for static files:


Date: Mon, 1 Mar 2008 06:46:04 GMT
Server: Apache/2
Last-Modified: Thu, 6 May 2007 02:52:18 GMT
Accept-Ranges: bytes
Cache-Control: max-age=604800
Expires: Mon, 6 Mar 2008 06:46:04 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 3588
Content-Type: application/x-javascript


Firefox 2 and css caching issue

Last week I had strange issue with Firefox cache. One of web application needed tweaking to be more responsive. After playing with apache configuration I tested page load speed using YSlow.

All files have expiry headers and all text files are deflated. Results are still mediocre. Some of dynamically generated css files aren't cached by Firefox 2. IE works as it supposed to do - any subsequent request is handled blazingly fast. I tried it on different installations of Firefox to exclude my browser optimizations side effects. Still the same.

Below are HTTP response headers of css that isn't going to FF cache:

Date: Sat, 01 Mar 2008 07:28:31 GMT
Last-Modified: Sat, 01 Mar 2008 07:26:56 GMT
Expires: Sat, 01 Mar 2008 07:36:56 GMT
Cache-Control: max-age=600
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 18128
Keep-Alive: timeout=10, max=98
Connection: Keep-Alive
Content-Type: text/css


I can't see anything suspicious. Content type is proper, all expire headers are set. Requested file name is named as css.php - does FF makes some assumptions on file extension? I tried turning off Cache-Control header and leave only Expires, but it doesn't work too.

It must be simple detail. Anybody has any clues?

Blending scripting languages in modern enterprise environment

Is your company using "enterprisey" technologies like Java or .NET? Do you want to use flexible scripting languages like Python or Ruby?
There are interesting possibilities for popular and "mature" programming languages like those above. Many popular languages has implementations that work under Java or .NET environment, like Jython, JRuby, IronPython, IronRuby and others. It's quite old thing but it still worth mentioning it. The main problem with such solutions is their maturity and completeness, so checking and quick research is needed before doing decision.

You can also find implementations of old veterans like COBOL and REXX for new platforms. There are also single platform languages that have cool features of scripting languages like Groovy or beanshell for JVM.

Additional languages are primarily used in configuration, defining custom and complex business rules, adding Domain Specific Language features etc. Word of warn - do not create local Babel Tower by multiplying number of existing languages in single project. But it's nice to have to have one scripting language blended into main project platform.

Other case is porting applications from scripting to target platform. Constraints are narrow - all used third party libraries have to work on both platforms, so additional work is needed often. Practical example - here is interesting tutorial how to deploy web.py python application under Jython and JVM.

Polyphasic sleep - sleeping less and better - update 1

Photo author: agoode

I mentioned before that I started polyphasic sleep schedule experiment After few weeks I can say it's quite difficult, but somehow working.
I had one week for trying it without family, but now they are back and are very important part of that experiment.

After transition phase I experimented with two different sleep schedules versions: 3 hours and 4.5 of core sleep version. I must say that best schedule that works for me is 4.5h of core sleep and 3 naps: one before work, one after dinner and additional one at evening. It works probably because of quite long core sleep time only 1.5h shorter than my usual 6h sleep. I maintained that sleep cycle for most all days with some exceptions:
- sometimes I was too tired after whole day or not slept good last night- in that circumstances its possible to miss even an alarm clock
- weekends when my lazy part demands full night sleep
- nightly interruptions caused by my toddler needing nursing

The first two obstacles probably will be gone after better adaptation. Kid is the bigger problem. I managed that by tweaking my sleeping hours and synchronizing with my kid's sleeping cycle. It's still not ideal but seems working. And know I'm logging sleep times for two.

My core sleep isn't equal for every night. Sometimes it fits better my natural rhythm the other time worse. I use precalculated cycle length of about 1.5h. So 4.5 core sleep takes about 3 cycles. If my waking time differs by more than half an hour earlier or later, I feel after that really bad. Grogginess, jet lag effect cumulates together into zombie like state - I need slow warm up to function normally. After first nap things go better. Every nap is crucial for reducing short sleep time effects and I'm usually more energized at the morning. After 2 pm I start to feel worse and I'm slowing till I get next nap after dinner.

But If my sleeping times are in schedule I feel normal, sometimes after naps even more energized than usual . So the key to success is to tune in to natural body rhythm and holding schedule times. As I mentioned before, naps are crucial- every nap missed makes me feel worse till next sleep.

Somebody may ask- is it worth to take such sacrifices? My total sleeping time is shortened from 6.5 to 5.5 hours- so 1 hour gained. It's a small reward if any. My main goal was to redefine my daily sleep cycle to get my works done and have time for my family after work. My family helps me a little so I can get two 20 minutes naps at late afternoon and evening. Everybody gets its goals and is happy. I think its worth to try it making better. Maybe it's weird method, but that's the way I like to try.

Estimated car expenses - verified

Last week I had some serious car servicing and insurance expenses. I decided to check my total cost of car ownership estimates.

I'm 99% city commuter so before I bought a car, besides well visible costs like car price, insurance etc., I had checked couple of factors like fuel consumption and estimated car servicing cost.After spreading all cost for next 5 years my calculated total was about 2650 euro per year (about 3850 $).

Two years later I have more precise data, like servicing invoices, fuel consumption history etc., excluding occasional fees like parking and other. Simple recalculation in spreadsheet and it looks like my average cost of car ownership will be about 2500 euro/year during last 2 and for next 3 years. It looks like my estimate was quite good. I should include predicted car selling price for example after 5 years, but it's not the point for now.

Here are biggest and most important chunks to consider in calculating car TCO:
- initial car value 47%
- insurance 24%
- fuel 20%

Maybe I should think about rolling car cost calculator based on that experiences. I wish all my budgeting estimates will work with that accuracy.

Rolling my own simple python chart library

Two weeks ago I started looking for simple python chart library. There are interesting solutions based on cairo, GD and alike libraries. I was surprised there is no simple chart drawing library using "core" python installation environment. I found of course nice working PyCha using cairo, but there were problems on one testing servers with cairo library installation.

Portability is one of my priorities so I've been trying to find more flexible solution. So I've found SPING (Simple Platform INdependent Graphics former now as PIDDLE). It supports numerous backends like PDF, PIL, QuickDraw, PostScript, Illustrator, piddleVCR, OpenGL, Tk, wxWindows and support for others is in development. Looks promising. Ok so it gives me basic drawing capabilities. What about beatiful charts? After online research, I decided that something like PyCha would be enough. But PyCha uses cairo ... . Then idea has come- rewrite PyCha to use SPING. Idea is not so stupid- PyCha library is quite small and porting should be doable.

So I started rewriting PyCha fo SPING (codename SpingCha). The biggest part of work is porting calls typical for cairo interface to more generic SPING. I used Interface pattern with simple cairo like API. After two days I have almost working implementation, tested on original PyCha examples. By saying almost - I mean it needs better color, fonts and antialiasing handling to produce more beautiful charts.

After finishing and cleaning up code I'm planning to publish it under derived GNU Lesser General Public License. Maybe I will talk with author of PyCha to just include my ideas in main project. Meanwhile if you are interested to get SpingCha just add your comment, I will know if I need to speed up things, and release it.

UPDATE: About further story you can read new post

Polyphasic sleep - sleeping less and better?

I've been planning to change my sleep schedule. The new year had come and it was good time to incorporate new habits. The first goal is to change sleep pattern to boost my creativity and make better time management through all day.

First step was to become more "early bird", it means getting up at least at 5 am. That part was easy.

So it's time for next step- polyphasic sleep. What is this all about? Instead of one big chunk of time dedicated for sleeping (usually 6-8 hours), polyphasic sleep is to take 4-6 20-30 minutes naps thorough all day.
The outcome is: summarized nap time is much shorter than usual sleeping time. And what is more controversial- you can be still rested and refreshed without health side effects. The biggest sleep time cut has Uberman's sleep schedule. One 20-30 minutes nap replaces one sleep cycle that average takes about 1.5 hour. The simplest form of polyphasic sleep is classic sleeping block plus siesta time.

I'm planning to use Everyman's sleep schedule. I have time to take 4.5 hour sleep, then one nap before leaving to work. Now it will be nice if I can afford short power nap during lunch time. And then one nap after work and dinner.

Sounds simple. I've tried to manage that schedule for few days. My first remarks:
- I didn't know that I can be so lazy to get up from bed at unusual time.
- It's hard so I have to keep more discipline.
- It's important to make most of used "sleep cycles". Keeping schedule and proper timing really makes difference.
- During adaptation I you feel sometimes like zombie at certain time of day.

Last week I caught a cold- so I held this experiment. I'm going to continue that as soon as I get better. So stay in touch and wish me luck.

Looking for simple python chart library

I'm looking for simple chart library. I need it to draw stock quote and portfolio benchmarking charts.

I'm trying now pycha (python charts). It's simple but ready for most common applications. One drawback- I'm unable to setup needed cairo library on testing system, yet.

I will try to made it, but maybe there is another solution.

Do you know any python chart drawing library with less dependencies?

Update:
It is still one of the simplest and best chart utilities. I'm also using JavaScript generated charts in web browser.

Mutual funds portfolio monitoring application - stage one finished

I have finished first stage of development of mutual funds portfolio monitoring application. Major bugs are fixed and all needed stats are calculated. I have been checking it on my investment account. Benchmarks don't lie- second half of 2007 was harsh for mutual fund investors :).

I'm working now on adding more visual appealing features like graphs etc. It needs also some refactoring and implementation of storing precalculated values - almost all features are data processing intensive. But more important are interesting portfolio stats, then I will work on scalability. After that I can think about first public alpha release.

I still don't have external python hosting service for development purposes, so if you know cheap one let me know.

web.py update 5 - DbUtils pooling and apache MPM module

My application using web.py is still growing. One day happened something suprising. After testing on internal WSGI server, I deployed application at testing machine with Apache, and started checking. Login screen appeared and everything was looking ok. But after login I waited very long time and nothing happened. After couple of minutes (long timeout?) application finished request with exception inside DBUtils code. I've checked logs, another browser- still the same.

I googled exception code and found that it's something related with exhausted connection pool. Application grows and communication with database complicates too. I've add additional code closing connections. Still nothing.
Then I have found some interesting posts about DBUtils connection pooling method and problem related to apache MPM module.

Apache MPM modules are directly handling HTTP requests. DBUtils creates pool for application process. My default Apache configuration was MPM prefork. That means every http request is handled by one single threaded MPM process. DBUtils created pool for every apache handler process- that exhausted database connections.

There is simple solution if you use dedicated apache server- using MPM worker that works better with DBUtils multi threaded pooling.
Once again, the day was saved.