Uncertaintly Principal and Software Engineering

Uncertainty Principal is a principle of quantum mechanics postulated by Nobel laureate Werner Heisenberg in the 1920s which states that it is not possible to determine both the position and the momentum of a particle at the same instant. The reason he argued, was that the act of “observing” the particle changes its behavior with respect to what is being measured. He was one of the first scientists to bring in probability into the predictable science of physics.

Prior to Heisenberg’s theory most physicists (including Albert Einstein) believed that one day physics and computing would be powerful enough that given an input we would be able to predict the future exactly because we would know how each and every element in the universe would react and therefore be able to model time and space with infinite precision. But with the advent of Heisenberg’s principal and modern quantum physics (which is almost always a function of probability) physics is no where near predictable.

But what does all this have to do with software engineering? Professor Hadar Ziv and Debra Richardson from UCI published a paper arguing that software engineering is inherently uncertain. The reason they argued was the three sources of uncertainty within all softwares:

1. Uncertainty in the problem domain: The first source of uncertainty is the problem domain. The authors argue that since most softwares models “real world” and real world is full of uncertainties. The software that models the real world therefore will inevitably contain those uncertainties.
2. The second source of uncertainty was that in addition to problem domain uncertainties, softwares themselves introduce uncertainties. The authors use concurrent softwares as an example to argue their point.
3. Their third argument is that softwares are tools that depend greatly on human participation; and human behavior is another area where great amount of uncertainties can be introduced (read my entry on Can UIs produce bugs)

Uncertainty Principle in Software Engineering is a great publication for anyone interested in the topic of testing, validation or software engineering in general. You can download a copy from here or visit their site where the download is also available.

Form posts and CacheControl

Let’s say you have an HTML form which posts to another page, and that page in turn posts to other page(s) (maybe even to itself). Consider what happens in a very common scenario such as this:

1. User visists page 1
2. Types in username and password
3. Posts to page 2 (valid user)
4. User sorts a column in page 2 which posts the column name and the order to page 2.
5. Page 2 reloads.

Now if the user pushes the back button when he/she is step 5 he/she will see one of two things: (1) Either the previous page will load (which is the behavior we would like to have) OR (2) the infamous “Warning: Page has Expired” message will appear in the screen.

Most developers seem to assume that the latter is how forms work and ignore this irritation, specially since there is an alternative, hit refresh and press retry — everything works (well not quite! strange things happen when you hit refresh, more on that some other time)!

The proper way to get the desired behavior is to use CacheControl. In an ASP page you can control how the page is cached. You get the warning message when there is no caching. That means you are saying: “always get a new file from the server. Do not cache anything”. When ASP returns to the server it realizes that it was a post and does not automatically re-post the data for you (until you hit retry). You can mark a page as no-cahe using the following statement:

Response.CacheControl = "no-cache"

However, if caching is allowed then a cached-version of the page is displayed when the user hits the back button, you can mark a page for caching using the following statement:

Response.CacheControl = "public"

You can turn your caching method on/off at different times of your pages. However make sure the line appears before any output goes to the browser. For instance, you might switch to caching for almost-static pages but want to turn off caching when dealing with forms and posts.

Shutter speed and your flash

In this document I am going to try to describe the relationships between your camera’s shutter speed setting and your flash. Quite simply put there is no difference (unless you are in FP mode, more on this soon).

The flash is sometimes described as a non-continuous lighting, meaning it lights up only for a split second. The split seconds happens to be so fast that most shutter speed settings will not affect the result. What will affect the result however is the aperture setting on your lens (the wider you open the more light goes through the better exposure you get). The other important factor in flash exposure is the subject to lighting distance. Meaning how far you stand away from the subject when using your flash. The more expensive your flash the higher the distance (since it can produce more brighter outputs).

One important thing to remember is that as you increase your distance from the subject the level of light does not came down linearlly but it comes down inversely. The inverse-square law it states that:

…light output is proportional to the inverse square of the distance. (ie: divide 1 by the distance, then square the result) So if you double the distance you get 1/2 2, or one quarter as much light. If you quadruple the distance you get 1/4 2, or only one sixteenth as much light.

However there is one instance when shutter speed does make a difference. If you are using a flash that’s FP capable (or focal plane/high speed) capable then you can push your shutter speed beyond the normal 1/200s limit to sometimes as high as 1/40000s of a sec. In such scenarios you actually freeze high speed actions even when using a flash. My Canon 580EX for instance, is FP capable and it can literally freeze bullet actions.

You can read far more detailed concepts of flash photography on Canon EOS system from PhotoNotes.org.

Constant/Variable aperture

Ever since I started out in the field of photography, there were few things things that I never got a hang for. For instance, on my Canon 28-135mm IS lens the aperture is not fixed. I didn’t know about this “variable/fixed aperture” concept until few days ago, so let me try to explain what that means.

Variable aperture simply means the higher you go up in the focal length the maximum aperture (the widest point) increases proportionally. For instance my widest aperture is f/3.5 but this is only true when the focal length is set to 28mm. If I zoom in to say about 60mm the maximum aperture changes to f/5.6. Since I mostly do portrait photography I am never in the 28mm range, instead I am always around 70mm – 125mm range, this forces me to stay at a higher aperture which means I am forced to take shots in brighter lighting conditions and also increases the DOF.

However there are constant-aperture lens’ For instance the Canon 70-200 2.8/L IS is a contant-aperture lens. You can be at: 200mm and still be @ f/2.8 if you like. Not that you would want to, all lens’ when wide-open tend to be a little soft. But this is Canon’s one of the most expensive lens and some reviews suggest that it’s equally sharp wide-open!

Option Groups

Not many people know about this. In a HTML dropdown (select) there is a option group tag that groups your options together. For instance:

If you run the code above you will notice users can’t even select the option group (which is perfectly valid). And the automatic indendation within the sub-groups is very useful also. This feature is supported by Opera and IE browsers. The actual version details you will have to find out.

Why run as doesn’t work on explorer.exe?

You will notice in windows no matter how you want to “Run as” explorer.exe it will not work. Almost all programs you can right-click or hold CTRL and right click to get the run as option but with explorer this doesn’t seem work. You cannot even use the command line tool runas /u:administrator to run explorer.

Explorer.exe is a special task. It actually runs all the time and is responsible for the taskbar and other features on the Windows desktop, press CTRL+ALT+DEL right now and you will see explorer.exe is running, you can also kill the process (don’t do it if you have important things going, but for the sake of experiment you can try) and you will notice your taskbar and your desktop will clear away. You can once again run the process by clicking File->New Task->explorer.exe

The reason why run as does not work with explorer is because when a new explorer instance wants to start it looks to the current desktop for an already running instance of explorer.exe to find out what needs to be done (for instance explore c:\) once the new explorer knows what to do it takes over and the previous instance automatically unloads itself (they both share the same PID, we will talk about this more in detail some other time).

Since the current explorer is running as “you” the user on the system the run as option cannot force a new user on an already running instance. And that’s why the run as feature does not work.

A workaround to this problem is to run iexplore to explore the drives (almost all things explorer can do iexplore can). You can do something like:

runas /u:administrator "c:\program files\internet explorer\iexplore.exe"

You can run the above command from the cmd prompt and it will prompt you for the administrator password, once in you can then explore the system (simply type: C:\) with administrative rights! Aaron Margosis at Microsoft also has some interesting thoughts on this if you want to read on.

Can UIs produce bugs?

All developers are aware how programming errors such as logic error, exception error or incorrect input data can produce bugs. But can UIs produce bugs or at least attract the user towards a bug? The answer yes! And these bugs tend to be quite difficult to catch because each user react to the UI in a different way. And the fact that programmers are so used to the behaviors of the default controls in a GUI environment it’s hard for them to test the application for UI confusions.

Unfortunately this is more true in a web platform than in it is in a let’s say windows application. In a windows application the developer has more control and therefore can enforce a UI path whereas in a web-application the user is partially in control. Consider what happens in Gmail for instance, if you write an e-mail and half way through you close the browser by mistake what happens? Gmail will not give you any alerts instead the browser will close and you will have lost everything you typed (assuming you didn’t save a draft).

Now let’s follow through the same steps but this time on a Windows application. Open up outlook and create a new message, now no matter how you want to close the window weather it’s closing the “New message” window or Outlook itself completely you will have to confirm that you don’t want to save the message and still quit.

Now of course the example I am presenting above is obvious enough that no one makes this mistake. Everyone knows that if you are using Hotmail, Yahoo Mail or Gmail you should save your message first before quitting.

But in a complex application (specially when the user is not used to it) this UI problem can get quite scary. This issue came into my attention when I released a fairly large application into production and some of the users were having trouble submiting information to us through the app. After some investigation I realized that it was the UI that was confusing some users.

To make long stories short, in a web environment we as programmers must be cautious as to how to design our UIs because we have several boundaries that we have to work against, the lack of control when it comes to UI in a web environment, the statelessness of web-applications and the constant client-server connection all play a role into introducing strange bugs.

Movable Type 3.15 Bug

It appears that there is a bug in MovableType 3.15, at least for me running on Perl version: 5.8.3. After upgrading from 3.1 to 3.15 I noticed that a user clicks to add a new comment, the comment textarea is not empty by default. It contains a strange “1″? So I digged into this problem a little deeper and realized it had to do with this line in: lib/MT/Template/Context.pm

$comment_text = encode_html($ctx->_hdlr_comment_body({convert_breaks=>0}), 1) || "";

To fix this bug simple replace that line above with the following:

$comment_text = encode_html($ctx->_hdlr_comment_body({convert_breaks=>0}), "") || "";

Notice the only difference being “” instead of 1. Hope that helps.

Nullable Types

A concept called Nullable types will be introduced into the new C# 2.0 speicifcation. When dealing with databases it’s essential that datatypes can be nullable. Currently there is no way to make an integer, for instance nullable. Consider what happens when an integer reading from a database:

int SomeValue=-1;
if(!RecordSet.Eof) {
	SomeValue = RecordSet["SomeValue"];
}
// .. other code here
// ...
// ...
// ...
RecordSet.Close();

if(SomeValue >= 0) Response.Write("SomeValue = " + SomeValue);
else Response.Write("No records found");

In this case, since integers were not nullable we got around this situation by setting SomeValue to -1 in case of an EOF. But what would happen if the database itself would contain -1 for SomeValue? Another solution would be to use a boolean variable to true/false depending on weather or not the recordset is empty. Yes that would work, but that’s an extra variable to keep track of, and often produces poor readibilty.

Nullable types solves by allowing any types to be nullable. The syntax is simple:

int? SomeValue = null;

// .. later in the code we can do

if(SomeValue.HasValue) // This is equivalent to SomeValue == null ** {
	Response.write("SomeValue = " + SomeValue);
}
else {
	Response.write ("No records found");
}

** The concept here is tricky but simple. SomeValue.HasValue is the equivalent to SomeValue==null is AND ONLY IF the “==” operator is not defined for the datatype in question. If the datatype (in this case integer) has already defined a == operator then SomeValue.HasValue may or may not be the same as SomeValue==null (although it could be if operator == (…) { return this.HasValue();} )

More on this later, with a discussion on null propagation.

Optimizing Regular Expressions

Optimizing regular expressions is an exceptionally complex field. A vast majority of the problems within this field is NP or NP-COMPLETE. Meaning that their solutions cannot be bound (time/space) under any definite polynomial.

All APIs (such as: .NET, PHP, ASP (classic VB really), JavaScript) that support regular expressions performs some form of equivalence reduction meaning it tries to reduce the given regular expression into a reduced YET equivalent regular expression. Of course there is no full-proof way of doing this for couple of reasons: (1) There is no way to guarantee that a reduced regular expression is the most efficient regular expresion possible. At this moment there is no GENERAL way of proving that.

I am writing about this topic because I am in the process of learning ASP.NET and I was just testing some of it’s regular expression capabilities so I tried (as you can see I wasn’t thinking):

([a-z]+)*!

And it took longer than I thought. As soon as I changed the regular expression to the equivalent regular expression:

([a-z])*!

It ran much much faster. This is a special case of nesting the Kleene star (*) which always takes precedence over the (+) operator. Most text-books actually don’t even define the (+) operator because it can be simplied to the following:

a+ = aa*

or in our case

[a-z]+ = [a-z][a-z]*

If we take it a little futher we get:

  ([a-z]+)*!
= ([a-z][a-z]*)*!
= ([a-z]*[a-z]*)!
= [a-z]*! (COMPLEX STEP)

The last step I would assume is where .NET couldn’t reduce any further. The last step although easy to see why they are equivalent it’s hard to develop a general algorithm to detect that.