Experimentation with Zenphoto

I’ve been doing some experimentation with Zenphoto. Trying to create a good php feed for Facebook so that I can be a bit more lazy about posting travel updates. Right now I can transcode a bunch of videos and images on my machine, upload them to Zenphoto, visit the gallery and it will generate the rss feeds and thumbs and everything on the fly.

The problem, or a problem, because there have been many, is that Facebook seems to mess up the RSS feeds. I sent them a message to find out if they could do anything about it. A URL like:

http://www.kallies.ca/photos/zp-core/i.php?a=080705-12%20Sudbury%20to%20Port%20Perry&i=DSCN1110.JPG

Gets converted to

http://www.kallies.ca/photos/zp-core/i.php?a=080705-12%20Sudbury%20to%20Port%20Perry&%3Bi=DSCN1110.JPG

So I did some searching about mod_rewrite and stuff, but instead found a nice idea to just replace some of the variables right off the bat. This is much more precise and should not introduce gaping security holes… unless they’re already there. At the top of i.php in zencore I added:

<?php
// Hack around a facebook bug where & gets changed to &amp%3B
if(isset($_GET['amp;i']))
{
  $_GET['i']=$_GET['amp;i'];
}
if(isset($_GET['amp;s']))
{
  $_GET['s']=$_GET['amp;s'];
}
if(isset($_GET['amp;ch']))
{
  $_GET['ch']=$_GET['amp;ch'];
}
if(isset($_GET['amp;t']))
{
  $_GET['t']=$_GET['amp;t'];
}
if(isset($_GET['amp;cw']))
{
  $_GET['cw']=$_GET['amp;cw'];
}
?>

It seems to work. I’m posting it here, because well, I need to have this somehwere so that when I eventually upgrade, I remember what I did.

What I’m not sure about is why the %3B is converted to a semicolon in the variable names.  I know %3B is a semicolon, but why is it *converted*?  I’m surprised that a semicolon is a valid array index in PHP.

Cinelerra is Working

Cinelerra is a bit annoying under the VMWare desktop and on top of that Gnome and junk wastes a lot of resources. The ideal would be to strip Ubuntu down to text-only as a special purpose VMWare VM and run Cinelerra on the Windows desktop through X11 on the make-believe VMWare-Windows network.

Another thing which is very important is to ensure that mounting of filesystems is not a pain in the butt. Having everything in the same place at the same time so that video (huge files) can be edited in-place without moving around all the time. Mounting an SMB share is probably the best way to do this in this kind of config. A bit wasteful on CPU resources, but I can’t think of any other method.

So here’s the attempt, some notes and a resultant screenshot.

(Sorry if this seems terse, it’s a dump of my own notes with some junk in here.)


root@Ubuntu-VMWare:~# apt-get install sysv-rc-conf
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libcurses-perl libcurses-ui-perl
The following NEW packages will be installed:
libcurses-perl libcurses-ui-perl sysv-rc-conf
0 upgraded, 3 newly installed, 0 to remove and 246 not upgraded.
Need to get 383kB of archives.
After unpacking 1421kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://de.archive.ubuntu.com gutsy/universe libcurses-perl 1.13-1 [116kB]
Get:2 http://de.archive.ubuntu.com gutsy/universe libcurses-ui-perl 0.95-6 [242kB]
Get:3 http://de.archive.ubuntu.com gutsy/universe sysv-rc-conf 0.99-6 [24.2kB]
Fetched 383kB in 8s (46.4kB/s)
Selecting previously deselected package libcurses-perl.
(Reading database ... 89401 files and directories currently installed.)
Unpacking libcurses-perl (from .../libcurses-perl_1.13-1_i386.deb) ...
Selecting previously deselected package libcurses-ui-perl.
Unpacking libcurses-ui-perl (from .../libcurses-ui-perl_0.95-6_all.deb) ...
Selecting previously deselected package sysv-rc-conf.
Unpacking sysv-rc-conf (from .../sysv-rc-conf_0.99-6_all.deb) ...
Setting up libcurses-perl (1.13-1) ...
Setting up libcurses-ui-perl (0.95-6) ...
Setting up sysv-rc-conf (0.99-6) ...
root@Ubuntu-VMWare:~#


Don’t need Bluetooth, don’t need X11, don’t need gdm, nvidea, usplash, acpi, acpid, apmd,
rsync?  sshd isn’t default but rsync is?
apport
atd
avahi?
cinestart?
consolekit
cron/anacron?  Why both?
dbus
dhcdbd
HAL
hotkey-something
laptop-thing
powernowd

I don’t have time to read all those docs.  I’ll just disable everything I don’t recognise.  It’s probably all junk anyway.

I don’t need cron on the system either.

So why is sshd not on there?

Oh.  I need to install it.


root@Ubuntu-VMWare:~# apt-get install sshd
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package sshd

OH yeah, forgot.


root@Ubuntu-VMWare:~# apt-get install openssh-server
Reading package lists... Done
Building dependency tree
Reading state information... Done


How are people supposed to know this?

oh right, the gui.  But I don’t need the GUI.


root@Ubuntu-VMWare:~# man vbesave
No manual entry for vbesave
root@Ubuntu-VMWare:~# less /etc/init.d/vbesave


what is vbesave?

Technically don’t need vmware-tools, but I’ll leave them in there.

sysv-rc-conf is nice.  It’d be nicer if I had a doc describing the Distro-blessed method of doing the stuff manually.  No innittab, just a note in init.d to check the Debian policy manual.  The Debian policy manual says nothing about changing runlevels in the absence of /etc/inittab.  So blegh.  Short of manually editing /etc/event.d/rc-default, I can’t see a way to do it.

I can’t fathom why this script exists:


# rc - runlevel compatibility
#
# This task guesses what the "default runlevel" should be and starts the
# appropriate script.

start on stopped rcS

script
runlevel --reboot || true

if grep -q -w -- "-s\|single\|S" /proc/cmdline; then
telinit S
elif [ -r /etc/inittab ]; then
RL="$(sed -n -e "/^id:[0-9]*:initdefault:/{s/^id://;s/:.*//;p}" /etc/inittab || true)"
if [ -n "$RL" ]; then
telinit $RL
else
telinit 2
fi
else
telinit 2
fi
end script

Okay. Let’s set up a line in the fstab to automagically mount the SMB share from Windows on boot.


//192.168.74.1/linux-shared     /home/mike/video        smb     uid=mike,gid=mike,credentials=/home/mike/.video-smbmountcred

mike@Ubuntu-VMWare:~$ ls -l .video-smbmountcred
-r-------- 1 root root 33 2008-12-22 19:40 .video-smbmountcred


Errors dont’ seem to mean antyhing important, probably just warnings.


mike@Ubuntu-VMWare:~$ sudo mount /home/mike/video
17571: session request to 192.168.74.1 failed (Called name not present)
17571: session request to 192 failed (Called name not present)
mike@Ubuntu-VMWare:~$


Added an account to the Windows side of things.  The password is actually some random-generated string.  Note that on my wireless adapters file sharing is not attached to the network stack.  I only have to be careful when I plug in to a LAN, and even then, with cryptic passwords and junk like that, it’s okay.  I can’t get too fancy on the account though because this is XP home and they disabled a lot of the fancier account management

The MS Windows XP firewall blocks non-local subnet connections for file and printer sharing.  Seems to be the default.  Ditto for not allowing file and printer sharing.

Now we reboot to test it all out.  Hopefully, I won’t need to use VMWare console for anything.

….

Surprising.  The mounts come up before the network.  Hmm.

https://help.ubuntu.com/community/Fstab

Looks pretty, even includes examples, pretty similar to what I already did.  Not helpful though.  _network is “nfs-only”.

Okay, setting the /etc/fstab to noauto for the smb filesystem.  I’m putting a command in /etc/rc.local :


/bin/mount /home/mike/video


dhcdthing is required.  That’s back in there.

/boot/grub/menu.list, remove the splash option.  Reboot.

Hmph.  Still no net.  This is stupid.

Ripped out the DHCP daemon.  threw the following in /etc/rc.local before the smb mount.


/sbin/ifconfig eth0 192.168.74.128 up


Reboot.  Works perfectly.

Not that I’d ever deploy a system like that in the field.  Not that I’d deploy a system in the field with Ubuntu.

Now I can launch Cygwin/X, ssh from Windows into my Linux VM, export the display, and run Cinelerra:

Cinelerra running next to WMM on Windows. Kinda.

It’s Working!

Scratch install.  Update the repositories (no actual system update or upgrade).  Add the repositories for Cinelerra.  Apt-get blah blah, execute the install instructions for VMWare tools (so as to get some reasonable performance)

Seems that Cinelerra is still not in the default repositories. Same was true for VMWare tools.

Damned if I know why a year old distro wouldn’t have Cinelerra and requires you to compile VMware tools.  Strange too that VMWare-player is available in the repositories, but VMWare tools is not.

Cinelerra and VMWare Tools Successfully Installed

Nearly picture perfect.   Of course apt-get install vmware-tools and the like didn’t work.  See the following for more info about installing VMWare Tools on Ubuntu 7.10

http://www.linux.com/feature/119570/

Not too painful.  Quirky.  But not too painful.

VMWare tools is working!

My only complaint is that my MP3 player skipped in Windows while rebooting Linux.  I guess that happens when you’re on a 6 year old machine with under a GB of RAM.

Success! Cinelerra on my VMWare environment

I wonder if it actually works?

The next problem is that VMWare Server does not allow for shared folders.  It’s a marketing thing, they want you to buy VMWare Workstation.  The more secure way to do this would be to export an SMB Share on Linux and let it have raw access to the partition.  The Linux machine, with no bridged networking would be protected by NAT.  That and I’d trust Linux to do an SMB export a thousand times before I’d trust Windows.  Sometimes you just don’t have a lot of choices though and I’d rather maintain my screwed up, buggy BSOD-prone EXT2 megapartition native than to require 128 or so MB of RAM to access it through a Linux system under VMWare.  This system just doesn’t have the resources and short of soldering, it is at the hardware limit on RAM.

A good day, but I really need to get away from the computer.

Okay. I’m Cheating

I’m kind of lacking in video editing software on Windows.  I thought about buying a license for a cheapo commercial package, but I want to give Cinelerra one more shot.

The trouble is that I’m travelling.  My Ubuntu distro can no longer reach anything using the external wireless adapter (pppbfft.  Who knows, who cares).  The distro was handy for Cinelerra, but dual-booting is not an option.

I’ve reformatted one of my ext2 partitions to NTFS because, FOSS being so superior, the FOSS IFS driver for EXT2 causes BSODs under what seems to be certain system calls.  It’s pretty reproduceable under WoW, or under… you’ll never guess.  Oh no…

Truecrypt.

So the reason I wasn’t able to encrypt my Windows volume was because I had EXT2 drivers loaded.  Now, it just seems to BSOD when it indexes stuff, so until I totally blow the EXT2 stuff, I’m not going to be able to use Truecrypt at all (!).

I also think this was preventing me from using VMWare.

Now that VMWare is working, I can let Windows handle all the fancy stuff like my suspend/hibernate, network, video, audio, power management bla bla.  I can dedicate a VM specifically to Cinelerra, and because it is behind a NAT system (ironically Windows), it will not be so high a priority to keep the thing up on security updates.

Video performance under Linux was crud anyway, so throwing the CPU intensive work over to VMware, letting it chug out some video, then playing it under Windows is just fine.  Preferable even.  Just kinda wasteful on memory, but this is for something which is an overnight process anyway.

It may also solve the problem that Cinelerra would overheat my machine when running overnight on Ubuntu.  (aggghhh!)

For those who’ll tell me that running Ubuntu 7.10 is hopelessly out of date, I have to only say that I’m already stealing wireless.  Downloading gigabytes of new distros is not an option.  I have a 7.10 iso.  I shouldn’t need another.  My Windows distro is from 2002, and I’m quite sure the service packs don’t exceed a gig.  This should work just fine.

Hopefully the Linux mencode will let me ditch my HORRIFYING Windows mplayer script. Yes, I look forward to having a nice shell environment.  Yes, I still love Linux, but I have a hate-hate relationship with it on the desktop.

Loading Ubuntu in VMware

You should see the video I created in Windows Movie Maker.  The thing is quite simple to use.  Not very capable, I mean, it gets the job done, and deters you from fiddling too much.  It’s got a serious weakness with input file types though.  It can only do AVI.  I don’t like converting everything just to use such a silly thing, but then again, my output medium (Youtube) is so low resolution that it doesn’t really matter I guess.

I do *want* to use Cinelerra though.  Despite being forced to use Linux on the Desktop to get it.

Galleries, Dreamhost, mencoder and ImageMagick

Over the past couple months, I’ve been trying to create regular blog entries about travel. The trouble is that I rarely have time to sit down and write, and writing does take time.

So one thought I had was that if I could post all my photos and videos in nearly-real time, they can supplement the writing very well and very easily. They’d also be a better way to blog… in a way.

A huge impediment to regularly posting photos and video is that the video files, or even just the photographs, are huge. It’s rare that I come upon a high speed uplink, and it’s rude if I visit a cafe and upload a few hundred MB while sipping a coffee.

The trick here, is to compress the files on my machine, and not only that, because there are so many, it needs to be automated. Manual processes are error-prone, tedious and difficult to tweak after the fact.

For automatic conversion, the graphic files are easy. In the Windows world, Irfanview or even Cygwin with ImageMagick can do this without any trouble. The video files are more complex.

During the LBW, a friend of mine, Andi, he tipped me off that mencoder from the Mplayer suite works fine to transcode video into smaller sizes from the command line and to convert video files into flv format.

Fortunately, mencoder, along with mplayer and ImageMagick are indeed all available on Windows. Unfortunately, mencoder was compiled for the windows CMD shell, which means that the cygwin paths don’t work as parameters. This makes life a little more complex, but not as complex as dual booting my Ubuntu distro just to convert photos and videos.

So here’s the 0.1 version of the conversion script. Note the hardcoded paths and other awful things. It works, but it’s not beautiful. I need to create a posting script too, that might be hard because Zenphoto on Dreamhost does not seem to be all that inviting for automated posts.

#!/bin/bash

sourcedir=/cygdrive/g/Photos
targetdir=/cygdrive/g/webgallery
dostempdir=g:\\webgallerytemp
mplayerconvertbat=/cygdrive/g/webgallerytemp/mplayerconvert.bat
mencoderoptions="-of lavf -oac mp3lame -lameopts abr:br=56 -srate 22050 -ovc lavc -lavcopts vcodec=flv:vbitrate=500:mbd=2:mv0:trell:v4mv:cbp:last_pred=3"

OFS=$IFS
IFS='
'

for i in `find $sourcedir -iname *.JPG`
do
if [ ! -e "$targetdir/$i" ]
then
printf "$i"...
mkdir -p "$targetdir/`dirname $i`"
convert -resize 1024 -quality 80 "$i" "$targetdir/$i"
if [ "$?" -eq "0" ]; then echo ... done; fi
fi
done

for i in `find $sourcedir -iname *.MOV`
do
if [ ! -e "$targetdir/$i" ]
then
# create a batch file for each command
cat <$mplayerconvertbat
@ECHO OFF
EOF
filepath=$i
dirname=`dirname $filepath`
basename=`basename $filepath`

IFS="/"
set -- $dirname
# get rid of cygdrive and assign drive letter
shift; shift
dosdir="$1:"

# replace slashes with backslashes
while [ "$1" != "" ]
do
shift
dosdir="$dosdir\\$1"
done

echo "$i"...

echo c:\\utils\\mplayer\\mencoder \"$dosdir\\$basename\" -o \"$dostempdir\\out.flv\" $mencoderoptions >> /cygdrive/g/webgallerytemp/mplayerconvert.bat
u2d /cygdrive/g/webgallerytemp/mplayerconvert.bat
cmd /c g:\\webgallerytemp\\mplayerconvert.bat

mv /cygdrive/g/webgallerytemp/out.flv "$targetdir/$i"
fi

IFS='
'

done
IFS=$OFS

Does anyone have any hints about mass-posts to Zenphoto on Dreamhost? Particularly of files >7MB?

Traveling, Wifi and USB

I set up my Thinkpad X24 as a dual-boot configuration. Aside from some video experiments in Cinelerra, I haven’t done much with it.

While traveling, I noticed that nobody is using 802.11a/b. They’re only using 802.11g. So I picked up a little USB wireless adapter, to try to get wireless working. I didn’t have a CDROM drive, so I figured that ensuring that it worked on Linux was important.

So I went into BestBuy and did a search on the chipset…. Bingo. The Hawking HWUG1 adapter seems to be supported. It even does promiscuous sniffing and stuff.

Anxiously, I take it back to my car, ready to try to get on to some free access points. I plug the thing in to the back of my machine…


[ 391.688000] usb 1-1: configuration #1 chosen from 1 choice
[ 392.172000] ieee80211_init: failed to initialize WME (err=-17)
[ 392.184000] iwlwifi_rc80211_simple: Unknown symbol iwlwifi_ieee80211_rate_control_unregister
[ 392.184000] iwlwifi_rc80211_simple: Unknown symbol iwlwifi_sta_info_put
[ 392.184000] iwlwifi_rc80211_simple: Unknown symbol iwlwifi_sta_info_get
[ 392.184000] iwlwifi_rc80211_simple: Unknown symbol iwlwifi_ieee80211_rate_control_register
[ 392.192000] wmaster0: Selected rate control algorithm 'simple'
[ 392.300000] usbcore: registered new interface driver rt2500usb
[ 392.304000] usbcore: registered new interface driver rt73usb
[ 392.600000] phy0 -> rt2500usb_enable_radio: Error - Register initialization failed.


Ugh. Just try to figure that one out without an Internet connection.

So I boot Windows. It seems to want that driver CD for which I have no CDROM drive.

Blegh.

A few days later I found a student who was able to transfer the contents of my CD to a USB keyfob where I was able to pull the drivers.

Ubuntu: 0
Windows: 0

Windows is working fine with wireless now. I have no time to figure it out in Linux.

Linux is a Tool and a Philosophy

I completely give up on the notion that Linux is superior on the desktop.

It’s not.

No advocate will be able to convince me otherwise. Especially if they “haven’t used Windows in years”. I’m not going to spend time trying to convey the vastly superior performance of my machine under Windows XP than under Ubuntu. I’ve encountered this problem for years under multiple versions of Windows and I’m not willing to compare it any further. It’s not helpful to anyone.

Advocates do a huge disservice to Linux when they make broad claims of ease of use and performance.

That said. Linux is useful to me because I administer Linux servers professionally. It’s a tool. It has particular characteristics and features which can help me. It has many characteristics and features which hinder me.

Now on to getting Ubuntu to use LUKS on my X24.

https://help.ubuntu.com/community/EncryptedFilesystemHowto7

I Add/Removed gparted, created a /boot, did a cp -a to get stuff into the new /boot, did a little dance to get rid of the old /boot fired up grub-install.

I made some mistakes along the way, but I was able to fix them by:

• booting with the LiveCD,
• mounting on /mnt/root and /mnt/root/boot, then
• editing my /mnt/root/boot/grub/menu.lst
• running grub –root-directory=/mnt/root /dev/sda

My old menu options looked like this:


title Ubuntu 7.10, kernel 2.6.22-14-generic
root (hd0,1)
kernel /boot/vmlinuz-2.6.22-14-generic root=UUID=5f0be2ea-329e-476b-91b5-2320b9f20f23 ro quiet splash
initrd /boot/initrd.img-2.6.22-14-generic
quiet


My new ones look like this:


title Ubuntu 7.10, kernel 2.6.22-14-generic
root (hd0,3)
kernel /vmlinuz-2.6.22-14-generic root=UUID=5f0be2ea-329e-476b-91b5-2320b9f20f23 ro quiet splash
initrd /initrd.img-2.6.22-14-generic
quiet


My partition table now looks like this:


root@pigeon:~# fdisk -l /dev/sda

Disk /dev/sda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00043c78

Device Boot Start End Blocks Id System
/dev/sda1 * 1 2433 19543041 7 HPFS/NTFS
/dev/sda2 2434 4866 19543072+ 83 Linux
/dev/sda3 4880 30401 205005465 5 Extended
/dev/sda4 4867 4879 104422+ 83 Linux

Partition table entries are not in disk order


Video Stablizing Filters, Advocacy and Feature Lists

I was just experimenting with Premiere’s capabilities and I’m quite disappointed with the stabilizer. It’s crap.

• Top Left: Adobe Premiere Elements (filling using background), 45 Minutes to process

• Top Right: Deshaker through Virtualdub and AVI Synth: about 50 minutes to process (2×25 minute passes)
• Bottom Left: Power Director (hey, some website with a probably fake review suggested it), 3 minutes (!)
• Bottom Right: Unfiltered (as I say in the video, I think I was dehydrated)

(The 4-way image was produced using AVI Synth… awesome program. )

It seems my expectations are set way too high by Deshaker. As an example of when FOSS beats out the commercial apps, VirtualDub and AVISynth are GPL, although the filter appears to be closed but free for distribution and use. For the Linux zelots, VirtualDub and AviSynth (2.x) are good examples of FOSS packages not available on Linux.

However… it seems that Cinelerra may have the Depan filter which can be used to track a fixed object and alter the borders of the image in order to remove motion from a clip.

http://avisynth.org.ru/depan/depan.html

From the website, Depan is a half implementation of Deshake. DePanEstimate is required to figure out how DePan will move the panning. It’s unclear to me that Cinelerra has this aspect of the filter, but there’s an awesome demo out there of something slightly related:

Now that’s good advocacy. Good documentation would be more useful, but good advocacy gives me some hope.. better than “do it yourself freeloader” or “try a 5-figure commercial package”

Unfortunately, this means means trying to install Cinelerra again. I have to install Ubuntu on my notebook anyway, I may have to have a peek at this silly Ubuntu Studio distribution.

The Promise of Ubuntu 8.04, Data Recovery and Video Editing

There are some promising improvements in Ubuntu 8.04. I’ll have to try an upgrade, maybe even to fix the boot sequence of my Linux drive (after my repairs, it is reporting Error 17 on boot). Simplified X11 configuration, support for multiple monitors of differing sizes, and that new audio system are all promising.

I might just need to fire it up today just so that I can try taking a binary dump of a flash card to try to recover all the photos and videos I lost. The data recovery worked partially. I lost all my MP3s. Most were truncated, all the file names were lost and the organization of them. The same unfortunately applies for a detailed set of films and photos I took of some archaeological sites in Crete, and a hiking trip through the Samarian Gorge.

I’m still very bitter about that.

In the meantime, I’m downloading Adobe Premiere elements. Can somebody explain to me why I might want to consider comparing it to Broadcast2000/Cinelerra/Open Movie Editor

You can watch the history of Linux video editing unfold with these three apps:

http://www.oreillynet.com/pub/a/network/2000/08/11/magazine/broadcast2000.html

http://en.wikipedia.org/wiki/Cinelerra

http://www.openmovieeditor.org/index.html

Of course I can’t actually get it to work, and nothing in Linux seems to have a deshake filter, so while it is almost there, it took 8 years to get there, and there are free tools in Windows that get most of the job done, but still leave me quite willing to fork out $100 for a reasonable tool. Wine doesn’t seem to cut it for the free Windows tools.

Film Gimp/Cinepaint and Open Movie Editor might help me in 2012.  In 2008, I just don’t have time to think about it.

Time to take the Richard Stallman Philosophy. I think it was Richard Stallman, the idea that if there is no free software available to do what you need to do, then it’s reasonable to use commercial software.

…48 hours later, the horror of GPM

From past experience, I unplugged the keyboard so that I wouldn’t accidentally hit enter.  Photorec stops the process without asking for confirmation.

When I got home, I decided to jiggle the mouse so that the screen blanking would clear.  I accidentally touched the right-mouse-button.

Right-click to paste is demented.  Ditto for middle-click to paste.  It seems I highlighted a CR at some point.  BAM.  Quit.  No prompt.  Process over.

I guess before I restart the process, I’ll manually sift through the smoldering heap of 15,783 unnamed jpg files, 2,396 mp3s, 248 oggs and 95 movs to find out if my vacation photos and MP3 collection were recoverable or not.