Note that two icons will be placed in your app menu - one for portrait and one for landscape. You'll have to use an alternative launcher (I prefer Home++, but LauncherPro is just as good).
Whenever you turn on or wake the device, it will return to landscape orientation, so you'll have to run the portrait app again. To make this easier, I have both icons on one of my home screens - this way I don't have to fiddle with the menu at all.
Does anyone know how to make a particular orientation the default at start time?