Too clever by half

It's always frustrating to come across a build error in allegedly-portable code, then to discover that the root of it is someone complicating things in a misguided attempt to improve portability -- usually by introducing complexity instead of removing it.

The macports project in particular is rife with the problems, undoubtedly because most of the "cross-platform" open source projects it ports never considered OS X as a build target.

KDE4 has refused to build on a particular OS X box (Quad G5, 10.5.8, XCode 3.1.4) via macports for quite some time. Currently, the build fails in kdebase4-runtime due to an error in the fishProtocol ctor in fish.cpp:


/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp: In constructor 'fishProtocol::fishProtocol(const QByteArray&, const QByteArray&)':
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp:275: error: cannot convert 'const char* (*)()' to 'const char*' for argument '1' to 'size_t strlen(const char*)'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp: In member function 'void fishProtocol::manageConnection(const QString&)':
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp:1079: error: no matching function for call to 'fishProtocol::writeChild(const char* (&)(), int&)'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp:561: note: candidates are: void fishProtocol::writeChild(const char*, KIO::fileoffset_t)
make[2]: *** [kioslave/fish/CMakeFiles/kio_fish.dir/fish.o] Error 1
make[1]: *** [kioslave/fish/CMakeFiles/kio_fish.dir/all] Error 2
make: *** [all] Error 2

Little did I know at the time that the actual cause for the error was further up:

[ 57%] Generating fishcode.h
cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish && /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/generate_fishcode.sh /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl /opt/local/bin/md5 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/build/kioslave/fish/fishcode.h -f\ 4
sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": unescaped newline inside substitute pattern


The line in question (fish.cpp:275) does a strlen on fishCode, which is not defined anywhere, though there is an #include for fishcode.h .

I searched around, found fishcode.h, and it was close to empty:

#define CHECKSUM "'fb18e850532f42b49f1034b4e17a4cdc /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'"
static const char
*fishCode('
');


Something had gone horribly wrong, somewhere -- fishCode is empty, though that shouldn't cause the error being displayed. Still, builds get wacky, and it's a lead.

Googling around for a copy of fishcode.h, turns up nothing, which is not a surprise as it is obviously autogenerated by the build process. What does turn up, however, are the commands used to generate it:

SUM=`/usr/bin/md5sum
/usr/src/packages/BUILD/kdebase-683581/kioslave/fish/fish.pl | cut -d ' ' -f 1`;
echo '#define CHECKSUM "'$SUM'"' > fishcode.h; echo 'static const char
*fishCode(' >> fishcode.h; sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[
]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'
/usr/src/packages/BUILD/kdebase-683581/kioslave/fish/fish.pl >> fishcode.h; echo
');' >> fishcode.h;

Time to set SUM in the shell and started playing with the sed command, which is obviously what's failing:

bash-3.2$ SUM='fb18e850532f42b49f1034b4e17a4cdc /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'

bash-3.2$ sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[
> ]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'

sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": unbalanced brackets ([])

Something is going wrong, set SUM to a more sane value:

bash-3.2$ SUM='fb18e850532f42b49f1034b4e17a4cdc'

bash-3.2$ sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[> ]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'

sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": unbalanced brackets ([])

bash-3.2$ SUM='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'

bash-3.2$ sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[
> ]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'

sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": unbalanced brackets ([])


Try replacing the embedded newline with a good old '\n':

bash-3.2$ SUM='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'

bash-3.2$ sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[\n]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'

sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": bad flag in substitute command: 'o'

Sed doesn't like the embedded newline; not a surprise. Now it seems like SUM is being interpreted as part of the expression. Let's see what's really going on:

bash-3.2$ SUM='fb18e850532f42b49f1034b4e17a4cdc /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'

bash-3.2$ echo sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[> ]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;' sed -e s/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[
]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/fb18e850532f42b49f1034b4e17a4cdc /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl/;

Argh! Someone's single quotes aren't properly escaped! Right there at '$SUM'!

Simple enough to fix, though. Fortunately, using sed to generate fishcode.h manually allows the kdebase4-runtime build to continue, which saves the time of tracking down where in the build process that sed command is.

It does raise the question, though: was any of this autogeneration really necessary? And if so, wouldn't a perl script (or even a one-liner) be more reliable?

One can argue that perl does not ship on as many OSes as sed (debatable), but the fact that this build worked on a maintainer's machine and not on an end-user's machine (running an OS highly regulated by the manufacturer, no less) pretty much blows the portability argument out of the water: if that's what the maintainer is truly after, well, their methods aren't working either.

OK, end angry-at-overly-convoluted-open-source-code-again rant. The moral: verify your goddamn shell commands!