Jeffrey’s Blog

I wrote, many moons ago (A brief adventure in “On-board Java Development” for Pocket PC 2002), about the prospect of being able to do the entire edit-compile-debug-run-distribute-enjoy programming cycle on the Pocket PC 2002 platform. The conclusion was that Java programming was possible via low-cost or free personal-profile JVMs and Pat Niemeyer’s BeanShell, a Java scripting language, and that the possibility of more joy lurked. I was using a borrowed PDA though, and the exploration ended without fruit. I also concluded that programming in the flagrantly verbose Java language using a pickboard or virtual keyboard truly sucked wind.

Here I report a new effort, in its preliminary stage, aimed at getting an equivalent or better environment together for the Windows Mobile 5/6 Smartphone platform. The same set of tools should be available and workable on the Pocket PC versions of WinMo (for devices with touchscreens), but what testing of that I care to do will only be through the WM6 emulator.

The target hardware is my new but already abused T-mobile Dash with WM6, and incidentally having an on-board MIDP implementation (Esmertec Jbed circa 2007), and text-messaging capable keyboard. This geek gadget, being without a pointing device (no touch screen), is inherently less Java capable, though the MIDP model of Java, designed for phones with T9 keyboards and little else, fits well.

To start with the good news: the Jbed JVM works. You can download MIDP apps as JAD/JAR file pairs onto the Dash, and with an appropriate text editor (see below), edit out the URL part of the MIDlet-Jar-URL entry (leaving only the JAR file name), and then select the JAD or JAR file in an appropriate file explorer, whence the Esmertec gizmo will attempt to install it. The jad/jar files are copied off to some repository, and the silly game you loaded will probably run unless it needs some Nokia or Siemens or JSR extension feature that Jbed hasn’t got (in which case it will probably not install). Performance is really very good considering the Dash’s 206 mHz clock speed. (The spec on JBed is MIDP 2.0, CLCD 1.1 with no extensions.)

You can even do some on-board programming within certain MIDP applications such as Cellular Basic, a Basic language interpreter, or Nikolay Klimchuk’s PCalc, a RPN-style programable calculator, or some other existent MIDP onboard-programming envronment. (I have not yet had much luck with David N. Welton and Wolfgang Kechel’s Hecl, a Tcl implementation, or MID4th, a Forth language interpreter, but still trying.)

On-board programming targeting Jbed would be great, particularly in that the canvas/game screen interface is the best lightweight and Java standard sprite environment available, and the JVM is on the device as delivered. And I am now going to describe how one can do all of the essential development steps — write, compile, preverify, run — for MIDP or regular java VM, using freeware, onboard the smartphone.

That’s the good news. The bad news is that Sun, IBM (J9), NSI (Creme), and Esmertec (Jeode) have all but stopped any effort to continue supporting their earlier “personal profile” capable JVMs for the WinCE environment, and there has been no commercial company that has stepped in to fill the gaping void. It is JVMs of this ilk, rather than simply a MIDlet player, that we really need for compiling (using java based compilers), debugging, testing, etc. Fortunately, there are some free and open-source options, and so it is to the job of getting them working for us that we now turn.

If you are still interested, let’s review the tools we need. Firstly we need a way to create/edit files and specify their type. Of course, WinMo comes with Mobile Word, but all the supplied utilities of the OS are caught in the philosophy of every file has exactly one associated application, and therefore the user has no need to know or be able to change a file’s type. We need something better and the premier freeware choice I found was Total Commander/CE (I used version 2.52ß), a file manager with built-in editor (cut and paste between files, open multiple files simultaneously), zip file manipulation, FTP, and other critical capabilities. The editor is limited to file sizes under 64K, so I suggest also installing Vieka Wordpad, a most excellent editor with a fully file-type-explicit file chooser. (With Wordpad alone, which appears to be copyright-protected freeware, you can do all of the text-file manipulation and renaming that we will need.) Wordpad has a primitive macro facility that is initially configured to aid in creating HTML/Javascript files.

I needed a scripting tool to make the many experiments needed to move the exploration process forward, and since the scripting tool I used, Mortscript, is so generally useful, I recommend installing this valuable freeware as well. In fact, for many onboard-programming noodlers, this basic scripting environment will be the end of the search, since this is a very capable language, lacking only the graphics and window controls that we seek from J2ME. With Mortscript, I was able to conduct the innumerable test launches of the java tools with differing syntax and options necessary to find the magic incantations that do indeed work. (This effort would have been greatly eased if any of the three free “console” programs of yesteryear worked on the smartphone. I spent a ton of time trying to get them to function.)

Because Windows Mobile 5/6 design has led many application developers to forgo an explicit way to cleanly exit their applications, and because both the operating system and some of the applications used in development appear to have memory leaks, a task manager is a requirement in the environment we are building, and memory recovery tools are very useful. Fortunately, the Dash comes with the manufacturer’s own, excellent, task manager, but if your device is not from HTC or otherwise lacks this facility, there are a number of freeware choices. I often use the multifaceted shell (i. e., “Start” menu replacement) Smart Toolkit. While no longer actively developed, this is a surprisingly resource-friendly keystroke-saving front-end for your device, and it has a task manager.

Oxios distributes two tiny programs — Hibernate and CloseApps — useful for recovering lost memory here.

So with a full-coverage subset of these tools installed (plus all available documentation about the tools), we can now talk about Java compilers. The open-source compilers available are divided by technology into C++ based compilers with the IBM developed Jikes being the prime example, and Java-based compilers such as Sun’s Javac, gj (generic java), and Pizza compilers, and DMS’ Kopi compiler suite. Of course, in the case of a java-based compiler, we will also need a compatible Java virtual machine (JVM). While I know from my PDA experience that a working Jikes would be a fast, resource efficient, and accurate solution, I could not find a workable Jikes solution at present. And this already interminable blog entry would never end if I detailed the innumerable, but still not exhaustive, testing of script syntax, JVMs (mainly MySaifu and Kaffe), and compilers needed to find an acceptable working environment. The result is that the compiler Kopisusu (derived from Kopi) and the recently developed MySaifu JVM (http://www2s.biglobe.ne.jp/~dat/java/project/jvm/index_en.html), together with a set of carefully constructed Mortscript scripts make compiling Java programs on the smartphone possible and relatively efficient, and the addition of the Proguard preverification tool allows one to prepare such programs for direct execution on the Jbed midlet manager, and can also be used to prepare jar files for midlet distribution.

So again we must do some installation. Download the MySaifu JVM (I used version 0.4.5) and transfer the 12M cab to your device; install on your card storage. Get the documentation on the command line options too.

Download the Kopisusu (I used version 1.9 Beta 5) jar file; it’s in tar.gz format so you will need a tool like 7zip on Windows to unpack it. Move the ksusu.jar file to your device.

This is about all that we would need to compile and run java programs with the MySaifu JVM, and since that might be where some of my dear readers are really headed, lets take a time out from the environment building and run a few tests to see what we have got.

With one of the editors (Wordpad or Total Commander’s built-in editor) make a java source file (\Hello.java) containing (a zip containing all sources and scripts is here):

// A console Hello app.

public class Hello {
   public static void main(String[] s) {
          System.out.println("Hello, Dogface!!!");
   }
}

Next, make a compiler script file named “\ksHello.mscr” containing (in one line):

Run("\Storage Card\Program Files\Mysaifu JVM\jre\bin\jvm.exe", " -Xmx10M -Xconsole
-Xlogfile:\jout.txt -jar ""\Storage Card\java\ksusu.jar"" --classpath
""\Storage Card\Program Files\Mysaifu JVM\jre\lib\rt.jar"" \Hello.java")

Here adjust the path to jvm.exe, ksusu.jar, and rt.jar to match your environment. This script says “run kopisusu on the jvm with a minimum of 10 megs of ram, specifying rt.jar as the place to reference java libraries, and \Hello.java as the source file.” A successful compile will result in \Hello.class and in any case command line output will appear in the file \jout.txt and in a console window. Execute this script by selecting it in Total comander or the WinMo File Manager, and then be patient because it may be 15-20 seconds before the all clear signal “JVM exit” appears in the console window. You will need your task manager to close down the running JVM after it has finished the compile or you will not have enough memory left to start the JVM again, nor will you be able to open the output file “\jout.txt” in order to view any compile errors. (In fact, this version of MySaifu will require closing after every use.)

If the compile succeeded, then we can run the resulting class file “\Hello.class” with the aid of another script, “\Hello.mscr,” containing the single line:

Run("\Storage Card\Program Files\Mysaifu JVM\jre\bin\jvm.exe", "-Xmx8M -Xconsole
-Xlogfile:\jout.txt  Hello")

Execution of this script should produce a window with:

Hello, Dogface!!!
JVM exit

And at this point, we have demonstrated edit, compile and running of a java program on the smartphone. If MySaifu were more than a slow, pudgy, and buggy work-in-progress, we would be in WinCE programmer’s nirvana at this juncture. In a future post I will address the performance issue — MySaifu is an order of magnitude slower than the commercial JVMs of old, and could be improved by a factor of four without too much trouble, in my view. It’s memory requirements and behavour when starved are also only barely acceptable, though they are quite understandable given its GNU Classpath components.

Creating applications for distribution using MySaifu as the VM is so obviously problematical, that I now describe how to surmount the remaining hurdles to developing for Jbed and other “MIDLet manager” JVMs. Compiling can be done as described, using Kopisusu and MySaifu, with the target MIDP libraries (midp-2.0.jar and cldc-1.1.jar) replacing the rt.jar in the “–classpath” option in the compile script.

There is an additional complication involved in the development of MIDlet applications compared to “plain old” java applications. In order to simplify the verification process that checks java byte code before execution, MIDlets must be “preverified” as part of the compilation process. Recently an open-source java-based preverification tool — Proguard — has become available, and it is fortunately compatible with the MySaifu JVM.

Download Proguard (http://proguard/sourceforg.net) and move proguard.jar to your device. (I used version 4.3; 4.4 is available.)

To compile midlets, you will also need appropriate MIDP and CLDC jar files (for their metadata only). I took the (apparently stubbed — i. e. only containing the metadata, not the object code) files midp_2.0.jar (35k) and cldc_1.1.jar (33k) files from the lib directory of a Windows install of Sun’s Java ME Platform SDK 3.0. This is the appropriate set of files for compiling to run with Jbed.

To test, here is the source of a “hello” midlet (put in \HelloMIDP.java):

import javax.microedition.midlet.*;
import javax.microedition.lcdui.*;

/* An example MIDlet with simple "Hello" text and an exit command. */

public class HelloMIDP extends MIDlet implements CommandListener {
    private Command exitCommand;
    private TextBox textBox;
    private Display display;
    public void commandAction(Command command, Displayable screen) {
        if (command == exitCommand) {
            destroyApp(false);
            notifyDestroyed();
        }
    }
    public HelloMIDP() {
        exitCommand = new Command("Exit", Command.EXIT, 1);
        textBox = new TextBox("Hello MIDlet", "dogface_was_here", 512, TextField.ANY);
        textBox.addCommand(exitCommand);
        textBox.setTicker(new Ticker("Dogface -- you can edit the text in the box.  Try it!!!"));
        textBox.setCommandListener(this);
        display = Display.getDisplay(this);
        display.setCurrent(textBox);
     }

    //Starts the MIDlet
    public void startApp() {  }

    //Pauses the MIDlet
    public void pauseApp() {  }

    //Destroys the MIDlet
    public void destroyApp(boolean unconditional) {  }

}

To compile \HelloMIDP.java to \HelloMIDP.class, use a script like this (\ksHelloMIDP.mscr):

Run("\Storage Card\Program Files\Mysaifu JVM\jre\bin\jvm.exe", " -Xmx10M -Xconsole
-Xlogfile:\jout.txt -jar ""\Storage Card\java\ksusu.jar"" --classpath
\cldc_1.1.jar;\midp_2.0.jar;\ \HelloMIDP.java")

If compilation succeeded, we are ready for the (necessary) preverification step. I put the finished class files in a “\preverified” directory to keep from mixing them with non-preverified class files. Here is a script to preverify \HelloMIDP.class with Proguard and stash the result in the (previously created) “\preverified” directory (\pvHelloMIDP.mscr):

Run("\Storage Card\Program Files\Mysaifu JVM\jre\bin\jvm.exe", " -Xmx10M -Xconsole
-Xlogfile:\jout.txt -jar ""\Storage Card\java\proguard\proguard.jar"" -dontshrink -dontoptimize
-dontobfuscate -microedition -libraryjars \cldc_1.1.jar;\midp_2.0.jar -injars \HelloMIDP.class
-outjars \preverified")

We can now run this MIDP application with Jbed — here is the script for that (\HelloMIDP.mscr):

Run("\Windows\jeodek.exe", "-classpath  \preverified HelloMIDP")

You should have a moment of pure joy when this thing finally runs. On another day I will tell you about:

1) Using Sun’s FREE JavaFX Mobile MIDP (2.1!) manager to test the MIDP apps you develop with this system.

2) The distribution step: making jad/jar file pairs on-board

3) The plan for a faster MySaifu JVM.

4) The reference set you need for on-board Java development.

5) Getting Jikes running.

6) What is still not working.

Notes:

a) To compile mutually interdependent java files, include them all in the compiler script.

b) I expect that MIDlets launched this way can probably not create or write to RMS record stores. this will need investigation.

c) If I have turned off comments on this blog (due to the inevitable spam storm), you can still write to me: {JeffreyRFox} at {msn.com}; make sure to use an appropriate subject line.

d) The scripts and source files for this blog can be downloaded here.

For reasons that I care to only peripherally discuss, I came to possess a T-mobile Dash (aka HTC Excaliber) smartphone with the upgraded Windows Mobile Standard 6 operating system for the last 12 days. For reasons that do not have anything to do with the device — hardware or software — tomorrow the device will probably be returned to the T-mobile store for a refund. Here follows a brief review of the device and its on-board programmability and hardware characteristics.

You can get the raw specs online, and it looks a lot like a Blackberry with a QVGA landscape screen and keyboard. It is only about a third the volume of the Zaurus 5500 and is about at the lower limit of comfortable size for the PDA with keyboard type of device. The keyboard is designed for one-handed use (right-handed), and the layout is obviously Blackberry inspired. There are a number of keys needed in programming that are very awkward to generate (i. e., backslash; square or angle brackets) or missing altogether (control keys). And then you have wasted keys (t-zone key, email key) and unused sequences (fn + shift). Not very promising, but perhaps fixable in software.

The device was apparently first released with Windows Mobile 5, and now with Windows Mobile Standard (meaning Smartphone, no touch screen) 6. The older models can be upgraded for free, and the large number of annoyances reported with WM5 would appear to make that desirable. The upgrade has been criticized as slow, but I did not notice performance to be an issue.

I did not test the Dash as a music player, but in that regard the use of a proprietary connector to headset/speakers would be a deal-breaker for some. Video in some forms played from the Portable Internet Explorer (PIE) browser, but the Adobe Flash plugin I tried to install refused to go.

Aboard were the mobile versions of Word and Excel, and the viewer version of Power Point. I did some editing of text files with Word and saw a powerpoint file rendered (nicely) by the viewer. Trying to use the Excel app did not go well — you have to get an Excel file from your computer to even start playing. Since Word will only save and load *.txt files for plain text, using it for script or html/javascript files would require another application to change file extensions back and forth.

I don’t really care much for phones, so other than to say that it does phone calls, I will refrain from further comment on that. There is a connection manager application that makes turning the phone link and the WiFi on and off easily, and I like to preserve power by turning them off when not in use.

The screen is bright but not of the transflective type I have come to love on the Zaurus. This one looks great, with high contrast and wonderful fonts — but is not really visible in bright sun. Unfortunately, I live in sunny Colorado and like to be perpetually outdoors.

The prospects for doing “on-board programming,” particularly when compared with the Zaurus Linux-based system I have been using for the last many (seven?) years, still appears a little grim. I didn’t think the keyboard would be a problem, but it is; it is too small, and significant key-mapping work will be required. The other major problem is that there is no working console application available, though the search continues. An although Windows CE in its many incarnations has been around for a decade, this device runs a variant that is compatible with only about 10% of the existing software base, judging from my testing of a wide variety of freeware and shareware. I did find many essentials: editor, file explorer, scripting language, registry editor, etc. But without an xterm-like console to glue things together, even the exploration of what is possible is very frustrating, and the design of each tool is effected negatively by the design of the operating system itself.

The freeware programming community is, however, alive. It is not bound by the open-source spirit, and there is little source code to see even in the case of essential utilities. The remains of some long ago porting efforts for GNU tools are scattered about, but they are not presently in good working order, and are clearly under-appreciated.

In my next post, I will detail my effort to reach a state where on-board Java programming is possible, and will also discuss the available scripting languages.

Well there I said it. Can open-source projects sue?

I’ve decided to try a CMS called LightNEasy for the Chess Club (springschess.org) and it is now on-line.

It had been my intention to let Wordpress have the job, but I have become enamored of the idea that “dynamic sites” should be as static as possible. Wordpress, with it’s complete recreation of all web content on every page read, is incredibly slow. This is perhaps not a problem for commercial web servers, but it is for their clients (YOU, dear reader). For me it means even locally testing on my equipment is a tedious prospect. So while I appreciate that there is a very professional finish to the Wordpress system, and that there is a plugin to solve every problem, even this one, I need to move on. This is a difference of computing philosophy. (I recall a Sun guy saying “memory is cheap” back when memory was not cheap. I used to run Windows 3.1 with 1 mb of memory and a 10 meg hard drive. Those were the days.)

LightNEasy is a simple, no-database style, CMS system. It is about 500 kb of php and templates with a javascript based html editor (2.5 mb) attached for content creation duty. It seems to work ok with my mobile browsers, even for creation, so I’m overlooking it’s weaknesses for the moment.

Speaking of weaknesses, Wordpress 2.0 managed to get corrupted on my palmtop because it recorded my local net address as the root of my web blog. I can therefore not get to parts of the blog unless I’m online — i. e., I can’t use a PDA browser to automatically access the content. The forum explains that I have to scrub the database manually — weeee!

I was going to do that, when my system, running wp, suffered a catastrophic failure, and caused a system auto-rebuild, wiping my configuration and tools out. Now I will have to blog about how to rebuild a Zaurus. (Had to do it twice.)

LightNEasy regenerates the static content when you you change it — yaah! That makes the content movable, so generation in one place, hosting elsewhere is possible, even without PHP. It has a sqlite version, and a NO-DATABASE flat file version. It is flat for me. (I tried the sqlite; now I have a “locked” database, whatever that means. I know it means I can’t use it.)

Maybe I can even hack it. I fact I did have to hack something already to get it to work. More later.

But Wordpress still has this blog ’till I get something better.

Update: Spam comment is killing both Light’n'Easy and my Wordpress. Comments being turned off for the time being for Wordpress.

content moved from PDA

It turns out that running the combination of Apache + Mysqld + Qtopia + browser + Wordpress is too much for my stock Zaurus 5500 with 16M swap file on the ramdisk. I am writing this into Lynx using the bare console with Qtopia turned off.

1. Note to self — How do I turn Qte back on? I know there is a shell script somewhere but I have failed to memorize the name.

2. Note to self — must get Apache to automatically serve index.php files. Now being dumped into directories.

3. Note to self — research PHP memory usage in wordpress.
And the performance is miserable too. And what about spell checking? I may have to get that from my browser.

And look now, it got posted (published) and I can go back for an edit! It is really a surprise that this works since the version of lynx I’m using is not even javascript capable.

Answer to 2: To get back to Qtopia use the script /sbin/qt while root. Update: you can just exit the shell and you will drop back into the countdown to Qtopia.

Answer to 1: I fixed the problem with Apache not automatically serving up an index.php file by adding index.php after index.html in the appropriate place in the httpd.conf file, then stopping and restarting Apache (which I do with apachectl.sh stop and apachecrl.sh start).

Update: content moved from PDA

After starting Apache (from root), and mounting and configuring ZGCC to have the C++ shared library available, start mysql as root with

./mysqld –skip-innodb –user=zaurus &

Ok, maybe the shared library should have been set up properly somewhere else long ago. Presently it is part of the ZGCC cramfs image (see my Zaurus web site).

Update: content moved from PDA

Welcome to “Jeff’s Blog.” If you want to know what it’s about, the “about” button may help. More content will come. Some definitions follow:

Zaurus: I own two (or more) Sharp Zaurus SL5500 PDAs. These are small handheld computers with a keyboard. It looks like this:

These palmtop computers are long out of production and becoming rare on Ebay. I use one every day and carry it around with me like a 21st century codpiece. It runs Linux and is stuffed full of the open source software that I am presently interested in, and is well designed to be used in the bright Colorado sunlight for about as long as I can stand to play on its tiny keyboard. The start of this blog coincides with the technical achivement of getting an Apache server, Mysql database server, Wordpress blogging software in PHP configured, and appropriate browser all working simultaneously on the Zaurus.