The Sorry Scheme of Things Entire

Ruby: Make any program a Quine

2012-02-14T14:16:00.000-05:00

One of the amusing diversions available to writers of compiled-language programs was the challenged of writing a quine (after W.V.O., of course): a program which, when executed, prints its own source code.

Attempts in C range with the ugly but obvious (repeating the source code as a string, and essentially printing the string twice) to the downright obfuscated.

Things are different in dynamic languages, of course, and there's a reason why this problem was never a RubyQuiz (UPDATE: it has! with a necessary extra 'piping' condition): any ruby program can be turned into a quine with the following code:

if __FILE__ == $0
puts File.open(__FILE__, 'r') { |f| f.read }
end

But perhaps something more can be done? Perhaps the source code and the bytecode can be displayed.

It turns out that the RubyVM object in YARV-based Ruby interpreters makes this possible:

#!/usr/bin/env ruby

def no_op

$some_variable ||= 1024

end

if __FILE__ == $0

buf = File.open(__FILE__, 'r') { |f| f.read }

code = RubyVM::InstructionSequence.compile(buf)

puts buf

puts code.disasm

end

Running this under ruby-1.9.2-head produces the following output:

bash$ rvm ruby-1.9.2-head do ruby quine.rb

#!/usr/bin/env ruby

def no_op

$some_variable ||= 1024

end

if __FILE__ == $0

buf = File.open(__FILE__, 'r') { |f| f.read }

code = RubyVM::InstructionSequence.compile(buf)

puts buf

puts code.disasm

end

== disasm: <RubyVM::InstructionSequence:<compiled>@<compiled>>==========
== catch table
| catch type: break st: 0029 ed: 0046 sp: 0000 cont: 0046
|------------------------------------------------------------------------
local table (size: 3, argc: 0 [opts: 0, rest: -1, post: 0, block: -1] s1)
[ 3] buf [ 2] code
0000 trace 1 ( 3)
0002 putspecialobject 1
0004 putspecialobject 2
0006 putobject :no_op
0008 putiseq no_op
0010 send :"core#define_method", 3, nil, 0, <ic:0>
0016 pop
0017 trace 1 ( 7)
0019 putstring "<compiled>"
0021 getglobal $0
0023 opt_eq <ic:9>
0025 branchunless 100
0027 trace 1 ( 8)
0029 getinlinecache 36, <ic:2>
0032 getconstant :File
0034 setinlinecache <ic:2>
0036 putstring "<compiled>"
0038 putstring "r"
0040 send :open, 2, block in <compiled>, 0, <ic:3>
0046 setlocal buf
0048 trace 1 ( 9)
0050 getinlinecache 59, <ic:4>
0053 getconstant :RubyVM
0055 getconstant :InstructionSequence
0057 setinlinecache <ic:4>
0059 getlocal buf
0061 send :compile, 1, nil, 0, <ic:5>
0067 setlocal code
0069 trace 1 ( 10)
0071 putnil
0072 getlocal buf
0074 send :puts, 1, nil, 8, <ic:6>
0080 pop
0081 trace 1 ( 11)
0083 putnil
0084 getlocal code
0086 send :disasm, 0, nil, 0, <ic:7>
0092 send :puts, 1, nil, 8, <ic:8>
0098 leave ( 7)
0099 pop
0100 putnil ( 11)
0101 leave
== disasm: <RubyVM::InstructionSequence:no_op@<compiled>>===============
0000 trace 8 ( 3)
0002 trace 1 ( 4)
0004 putnil
0005 defined 7, :$some_variable, false
0009 branchunless 17
0011 getglobal $some_variable
0013 dup
0014 branchif 22
0016 pop
0017 putobject 1024
0019 dup
0020 setglobal $some_variable
0022 trace 16 ( 5)
0024 leave ( 4)
== disasm: <RubyVM::InstructionSequence:block in <compiled>@<compiled>>=
== catch table
| catch type: redo st: 0000 ed: 0011 sp: 0000 cont: 0000
| catch type: next st: 0000 ed: 0011 sp: 0000 cont: 0011
|------------------------------------------------------------------------
local table (size: 2, argc: 1 [opts: 0, rest: -1, post: 0, block: -1] s3)
[ 2] f<Arg>
0000 trace 1 ( 8)
0002 getdynamic f, 0
0005 send :read, 0, nil, 0, <ic:0>
0011 leave

Web page mirroring (wget) in Ruby

2012-02-04T22:25:00.001-05:00

Doing a basic search for 'wget in ruby' or 'web page mirroring in ruby' generally turns up a number of unimaginative (not to mention lazy) examples of invoking wget from a sub-shell. While these solutions often meet the immediate needs of their authors, they provide no guidance for those who need to implement the mirroring of remote documents as part of a larger data processing or data management application.

The following are the results of implementing web page mirroring (as performed by non-recursive wget, or by Firefox's Save As...[Web Page, Complete] feature) in such an application. The provided code will download an HTML document, parse it for JS, CSS, and image files (referred to as resources), download those files to the destination directory, rewrite the references to those resources so they refer to the local (downloaded) files, then write the updated HTML document to the destination directory.

The bulk of the work is done by Nokogiri, which parses the HTML document and supports XPath queries on its elements. Fetching remote documents is handled by Ruby's built-in net/http module.

#!/usr/bin/env ruby
# faux-wget.rb : example of performing wget-style mirroring
require 'nokogiri'
require 'net/http'
require 'fileutils'
require 'uri'

The implementation uses a single RemoteDocument class which takes a URI in its constructor and provides the high-level method mirror which downloads and localizes the document at the URI.

The mirror method invokes four helper methods:

html_get will download the remote document
Nokogiri::HTML parses the downloaded document
process_contents extracts resource tags from the document
save_locally localizes all resources

The document parsing is straighforward: Nokogiri does all the work, and process_contents simply performs XPath queries to build lists of CSS, JS and IMG tags. The document fetching is also pretty straightforward; support for response codes in the 3xx (redirect) and 4xx-5xx series (not found,
unauthorized, internal error, etc) adds only a couple of lines of code

The save_locally method begins by creating the destination directory if it does not exist. If the document contains a BASE tag, it is removed - which makes the mirrored page behave better in most web browsers. Next, the localize method is invoked on all of the resource tags. Finally, the document is written to the destination directory.

The localize method performs all of the interesting work.

It begins with a random delay of tens of milliseconds, which prevents most web servers from blocking the barrage of requests when downloading the resource files. The resource URL is obtained from the HREF attribute of CSS tags, or from the SRC attribute of JS and IMG tags.

This URL can be absolute or relative (e.g. the file "http://foo.com/images/bar.jpg" may be referenced as "images/bar.jpg"), so an absolute URL is generated from the document URI if needed. The URL is then converted to a local path, meaning that the downloaded resource can end up in the directory "images" inside the destination directory.

The remote resource file is then downloaded to this path, and the URL in the tag attribute (which is part of the original document's source) is replaced with the local path to the downloaded resource.

The local path generation logic in localize_url is a bit simplistic (resulting in ugly directory trees), and the mirroring process itself is non-recursive (i.e. HREF attributes in A tags are detected but ignored). The caller can easily use the RemoteDocument object to provide additional services: the parsed contents of the HTML document are available in the contents property, the metadata tags (e.g. keywords) are available in the meta property, and all links found in the document are available in the links property.

=begin rdoc
Wrap a URI and provide methods for download, parsing, and mirroring of remote HTML document.
=end
class RemoteDocument
attr_reader :uri
attr_reader :contents
attr_reader :css_tags, :js_tags, :img_tags, :meta, :links

def initialize(uri)
@uri = uri
end

=begin rdoc
Download, parse, and save the RemoteDocument and all resources (JS, CSS,
images) in the specified directory.
=end
def mirror(dir)
source = html_get(uri)
@contents = Nokogiri::HTML( source )
process_contents
save_locally(dir)
end

=begin rdoc
Extract resources (CSS, JS, Image files) from the parsed html document.
=end
def process_contents
@css_tags = @contents.xpath( '//link[@rel="stylesheet"]' )
@js_tags = @contents.xpath('//script[@src]')
@img_tags = @contents.xpath( '//img[@src]' )
# Note: meta tags and links are unused in this example
find_meta_tags
find_links
end

=begin rdoc
Extract contents of META tags to @meta Hash.
=end
def find_meta_tags
@meta = {}
@contents.xpath('//meta').each do |tag|
last_name = name = value = nil
tag.attributes.each do |key, attr|
if attr.name == 'content'
value = attr.value
elsif attr.name == 'name'
name = attr.value
else
last_name = attr.value
end
end
name = last_name if not name
@meta[name] = value if name && value
end
end

=begin rdoc
Generate a Hash URL -> Title of all (unique) links in document.
=end
def find_links
@links = {}
@contents.xpath('//a[@href]').each do |tag|
@links[tag[:href]] = (tag[:title] || '') if (! @links.include? tag[:href])
end
end

=begin rdoc
Generate a local, legal filename for url in dir.
=end
def localize_url(url, dir)
path = url.gsub(/^[|[:alpha]]+:\/\//, '')
path.gsub!(/^[.\/]+/, '')
path.gsub!(/[^-_.\/[:alnum:]]/, '_')
File.join(dir, path)
end

=begin rdoc
Construct a valid URL for an HREF or SRC parameter. This uses the document URI
to convert a relative URL ('/doc') to an absolute one ('http://foo.com/doc').
=end
def url_for(str)
return str if str =~ /^[|[:alpha:]]+:\/\//
File.join((uri.path.empty?) ? uri.to_s : File.dirname(uri.to_s), str)
end

=begin rdoc
Send GET to url, following redirects if required.
=end
def html_get(url)
resp = Net::HTTP.get_response(url)
if ['301', '302', '307'].include? resp.code
url = URI.parse resp['location']
elsif resp.code.to_i >= 400
$stderr.puts "[#{resp.code}] #{url}"
return
end
Net::HTTP.get url
end

=begin rdoc
Download a remote file and save it to the specified path
=end
def download_resource(url, path)
FileUtils.mkdir_p File.dirname(path)
the_uri = URI.parse(url)
if the_uri
data = html_get the_uri
File.open(path, 'wb') { |f| f.write(data) } if data
end
end

=begin rdoc
Download resource for attribute 'sym' in 'tag' (e.g. :src in IMG), saving it to
'dir' and modifying the tag attribute to reflect the new, local location.
=end
def localize(tag, sym, dir)
delay
url = tag[sym]
resource_url = url_for(url)
dest = localize_url(url, dir)
download_resource(resource_url, dest)
tag[sym.to_s] = dest.partition(File.dirname(dir) + File::SEPARATOR).last
end

=begin rdoc
Attempt to "play nice" with web servers by sleeping for a few ms.
=end
def delay
sleep(rand / 100)
end

=begin rdoc
Download all resources to destination directory, rewriting in-document tags
to reflect the new resource location, then save the localized document.
Creates destination directory if it does not exist.
=end
def save_locally(dir)
Dir.mkdir(dir) if (! File.exist? dir)

# remove HTML BASE tag if it exists
@contents.xpath('//base').each { |t| t.remove }

# save resources
@img_tags.each { |tag| localize(tag, :src, File.join(dir, 'images')) }
@js_tags.each { |tag| localize(tag, :src, File.join(dir, 'js')) }
@css_tags.each { |tag| localize(tag, :href, File.join(dir, 'css')) }

save_path = File.join(dir, File.basename(uri.to_s))
save_path += '.html' if save_path !~ /\.((html?)|(txt))$/
File.open(save_path, 'w') { |f| f.write(@contents.to_html) }
end
end

Finally. the source code file needs a driver method to make it useful from the command line. This one takes a URL and a destination directory as its parameters:

# ----------------------------------------------------------------------
if __FILE__ == $0
if ARGV.count < 2
$stderr.puts "Usage: #{$0} URL DIR"
exit 1
end

url = ARGV.shift
dir = ARGV.shift
doc = RemoteDocument.new(URI.parse(url))
doc.mirror(dir)
end

JRuby and RubyGems

2011-11-09T16:36:00.001-05:00

For reasons too tangential to cover here, it was decided to port a Ruby 1.8.7 project to JRuby (>= 1.6.0). This worked well enough, except for the fatal flaw in JRuby: using gems.

To begin with, getting *any* gem with native extensions to build in JRuby requires some manual intervention.

The reason? The RVM JRuby has its Ruby include files under cext/src/include/, instead of lib/native/include like the other RVM Rubies. This appears to be a known issue.

A symlink handily solves the problem:

bash$ cd /usr/local/rvm/rubies/jruby-head
bash$ ln -s cext/src/include lib/native/include

Making this change will allow most gems with native extensions (e.q. pg, sqlite
3, etc) to be built.

The qtbindings gem, however, still fails with an error like the following:

bash$ sudo rvm jruby-head do gem install qtbindings

CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:91 (MESSAGE):

Could NOT find Ruby (missing: RUBY_INCLUDE_DIR)

Call Stack (most recent call first):

/usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:252 (_FPHSA_FAILURE_MESSAGE)

cmake/modules/FindRuby.cmake:249 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)

CMakeLists.txt:18 (FIND_PACKAGE)

-- Configuring incomplete, errors occurred!

make: [build] Error 1 (ignored)

cd ext/build; make

make[1]: Entering directory `/usr/local/rvm/gems/jruby-head/gems/qtbindings-4.6.3.4/ext/build'

make[1]: *** No targets specified and no makefile found. Stop.

make[1]: Leaving directory `/usr/local/rvm/gems/jruby-head/gems/qtbindings-4.6.3.4/ext/build'

make: *** [build] Error 2

Unpacking the gem, fixing the extconf.rb to properly find JRuby, and repacking the gem gets past these problems, but ultimately fails when compiling the actual C extension code for Qt4.

The reason? Qt4 uses heavy inspection of the stack frame in order to implement its signal/slot mechanisms, and JRuby does not provide access to this information. Until equivalent functions for rb_frame_callee() and the like are provided by JRuby, Qt4 will remain incompatible with it.

Postscript: For the curious, the following lines (or variants thereof) can be added to extconf.rb to fix the Cmake build issues:

file.puts "-DRUBY_EXECUTABLE=/usr/local/rvm/rubies/jruby-head/bin/jruby \\"

file.puts "-DRUBY_LIBRARY=/usr/local/rvm/rubies/jruby-head/lib/native/x86_64-Linux/libjruby-cext.so \\"

file.puts "-DRUBY_LIB_PATH=/usr/local/rvm/rubies/jruby-head/lib/native/x86_64-Linux \\"

file.puts "-DRUBY_INCLUDE_DIR=/usr/local/rvm/rubies/jruby-head/cext/src/include/ruby \\"

file.puts "-DRUBY_CONFIG_INCLUDE_DIR=/usr/local/rvm/rubies/jruby-head/cext/src/include/ruby \\"

file.puts "-DCMAKE_INSTALL_PREFIX=/usr/local/rvm/rubies/jruby-head \\"

The lines go in the 'else' block of the 'if windows' condition, where the other cmake options are defined (around line 216, or search for '-DENABLE_SMOKE').

Qt4 and Ruby1.9.x

2011-09-26T01:01:00.003-04:00

After a long period with no Qt4 support for Ruby 1.9, the qtbindings gem has stepped in to fill the void. This project packages the qtruby4-qtruby library from the korundum project, so the API is unchanged.

Installation is straightforward:

sudo gem install qtbindings

or for rvm users:

sudo rvm gem install qtbindings

Compatible with 1.8 as well.

Determining installed version of Ubuntu

2011-09-25T23:03:00.002-04:00

Neither uname nor motd are of much use in determining the version of Ubuntu installed on a third-party system, and searching launchpad.net for version details of key packages in dpkg is annoying.

Fortunately there is a utility called lsb_release that provides this information:

bash$ lsb_release -si
Ubuntu
bash$ lsb_release -sr
11.04
bash$ lsb_release -sc
natty

Complete all the way down to the silly nickname for each release.

Since this is an LSB utility, it's probably been in place for some time, silently waiting to provide more detail than uname. Seriously, would a mention in the SEE ALSO section of the uname manpage be *so* hard to add?

First impressions, redux

2011-09-21T14:30:00.000-04:00

The R500 bit the dust over a year and a half ago. Most PC manufacturers were busy trying to crank out netbooks or a Macbook Air ripoff at the time, so there was no suitable replacement -- had to limp along with a Macbook Pro (ugh!) running Ubuntu until spring of this year.

What replaced the R500? A Thinkpad X1. It weighs twice as much, has no transreflective screen, and is a bit bulkier, but it is paradise after using that Macbook Pro for a year. Dedicated pageup/down keys! Home and End keys! A Delete key! THREE mouse buttons! A hardware kill switch for wifi and bluetooth! A freaken eSATA connector!

But why no grumpy install post detailing how to get Linux speaking to all the hardware? Because everything just works! Even the volume control buttons and the fingerprint reader. No joke. You can thank IBM's interest in Linux for that.

On top of the compatibility, it's a nice looking machine, powerful (64-bit debian runs great on a dual-core i5 with 8 GB of RAM), and tough -- GorillaGlass over the display, MILSPEC (assuming 810) shock and spill resistance (including a keyboard that drains *through* to the bottom of the laptop), the works.

The one caveat: it's best to use non-standard video drivers, as the standard debian/ubuntu Intel video driver became a bit crashy. Easily fixed, though, by adding the xorg-edgers PPA.

.vimrc and UTF-8

2011-09-18T20:59:00.000-04:00

Quick .vimrc config to display UTF-8 characters correctly:

" support UTF-8 automatically when not on console
if has('gui_running') && has('multi_byte')
set encoding=utf-8
set fileencoding=utf-8
set fileencodings=utf-8
endif

Note that the gui_running requirement ensures that this will be used by gvim and by vim run from a terminal emulator (such as urxvt or mlterm), not by vim running from a virtual terminal or console.

Fixing strings(1)

2011-09-07T20:42:00.002-04:00

The first step in any examination of a binary is to run strings(1) on it.

This usually results in output like the following:
bash$ strings /bin/ls
FXL9GX

|"H9

GXL9FX

|"H9

F@L9G@

...which, needless to say, sucks.

The problem here is that a 'string' is any sequence of printable ASCII characters of a minimum length (usually 4).

The output of strings(1) can be made a bit more usable by discarding any string that does not contain a word in the system dictionary.

An off-the cuff implementation of such a filter in Ruby would be:

#!/usr/bin/env ruby

DICT='/usr/share/dict/american-english-insane'

ARGF.lines.each do |line|

is_str = false

line.gsub(/[^[:alnum:]]/, ' ').split.each do |word|

next if word.length < 4

is_str = true if (`grep -Fxc '#{word}' #{DICT}`.chomp.to_i > 0)

end

puts line if is_str

end

This would then be passed the output of strings(1):

bash$ strings /bin/ls | ./strings.rb

__gmon_start__

_fini

clock_gettime

acl_get_entry

acl_get_tag_type

...

This is super-slow for large executables, and will miss sprintf-style formatting strings that contain only control characters (e.g. "%p\n"), but for the general case it produces useful output.

Directions for improvement: load one or more dictionary files and perform lookups on them in Ruby. Search for english-like words by taking 4-letter prefixes and suffixes of each 'word' in the string and searching for dictionary words that start or end with that prefix/suffix. Provide early-exit from the inner field look when a match is found. Allow matches of sprintf formatting strings, URIs ('http://'), etc.

Ruby END and DATA

2011-08-11T02:05:00.004-04:00

The __END__ keyword in Ruby causes the parser to stop executing the source file; it is often used for appending documentation such as a LICENSE file to the end of a source code file.

More interesting is the fact that the contents of the file following the __END__ keyword are available via the global IO object named DATA.

This means that it is possible to include test data -- even binary data -- at the end of a Ruby source file:

bash$ cat od.rb
#!/usr/bin/env ruby
if __FILE__ == $0
offset = 0
while (buf = DATA.read(16)) do
bytes = buf.unpack 'C*'
puts "%08X: %s\n" % [ offset, bytes.map { |b| " %03o" % b } ]
offset += 16
end
end
__END__
bash$ cat /bin/true >> od.rb
bash$ ./od.rb
00000000: 177 105 114 106 002 001 001 000 000 000 000 000 000 000 000 000
00000010: 002 000 076 000 001 000 000 000 220 044 100 000 000 000 000 000
00000020: 100 000 000 000 000 000 000 000 060 226 001 000 000 000 000 000
...

knotify4 uses 100% CPU

2011-08-08T14:56:00.000-04:00

This is true, and has been for awhile. It tends to occur when a laptop is suspended while online, then resumed while offline. A nice side effect is that battery life gets reduced by about 75% while offline, which is generally when one wants the longest battery life.

There are plenty of bug reports open on this, but it's pretty clear that the KDE/Kubuntu guys either have no clue how to fix this, or cannot be bothered.

Since knotify4 isn't really all that useful (especially when using E17 for a WM), there is a brutal hack that will effectively silence it:

sudo mv /usr/bin/knotify4 /usr/bin/knotify4.orig

sudo cp /bin/true /usr/bin/knotify4

Needless to say, the original file should be restored before doing an upgrade.

Disabling startup (init.d) services in Ubuntu

2011-08-08T11:43:00.000-04:00

Ubuntu never has made obvious what the "Ubuntu way" of removing services from System-V run levels is. The GUI tools in GNOME and KDE are incomplete, and a quick investigation of the run level directories shows that they are filled automatically -- so that symlinks added and removed manually might, in the future, get ignored.

Fortunately, the README in /etc/init.d ends with the following advice:

Use the update-rc.d command to create symbolic links in the /etc/rc?.d
as appropriate. See that man page for more details.

The man page lists the following forms for invoking upgrade-rc.d:

update-rc.d [-n] [-f] B name remove

update-rc.d [-n] B name defaults [NN | SS KK]

update-rc.d [-n] name start|stop R NN runlevel [ runlevel ]...

update-rc.d [-n] B name disable|enable [ S|2|3|4|5 ]

The following command will remove the service collectd from all run levels:

bash$ sudo update-rc.d collectd disable
update-rc.d: warning: collectd start runlevel arguments (none) do not match LSB Default-Start values (2 3 4 5)
update-rc.d: warning: collectd stop runlevel arguments (none) do not match LSB Default-Stop values (0 1 6)
Disabling system startup links for /etc/init.d/collectd ...
Removing any system startup links for /etc/init.d/collectd ...
/etc/rc0.d/K95collectd
/etc/rc1.d/K95collectd
/etc/rc2.d/S95collectd
/etc/rc3.d/S95collectd
/etc/rc4.d/S95collectd
/etc/rc5.d/S95collectd
/etc/rc6.d/K95collectd
Adding system startup for /etc/init.d/collectd ...
/etc/rc0.d/K95collectd -> ../init.d/collectd
/etc/rc1.d/K95collectd -> ../init.d/collectd
/etc/rc6.d/K95collectd -> ../init.d/collectd
/etc/rc2.d/K05collectd -> ../init.d/collectd
/etc/rc3.d/K05collectd -> ../init.d/collectd
/etc/rc4.d/K05collectd -> ../init.d/collectd
/etc/rc5.d/K05collectd -> ../init.d/collectd

The command to remove a service from a specific runlevel should be the following:

sudo update-rc.d BASENAME disable `runlevel | cut -d ' ' -f 2`

...however a quick experiment shows that the runlevel argument is ignored:

bash$ sudo update-rc.d collectd disable 2

sudo update-rc.d collectd disable 2
update-rc.d: warning: collectd start runlevel arguments (none) do not match LSB Default-Start values (2 3 4 5)
update-rc.d: warning: collectd stop runlevel arguments (none) do not match LSB Default-Stop values (0 1 6)
Disabling system startup links for /etc/init.d/collectd ...
Removing any system startup links for /etc/init.d/collectd ...
/etc/rc0.d/K95collectd
/etc/rc1.d/K95collectd
/etc/rc2.d/S95collectd
/etc/rc3.d/S95collectd
/etc/rc4.d/S95collectd
/etc/rc5.d/S95collectd
/etc/rc6.d/K95collectd
Adding system startup for /etc/init.d/collectd ...
/etc/rc0.d/K95collectd -> ../init.d/collectd
/etc/rc1.d/K95collectd -> ../init.d/collectd
/etc/rc6.d/K95collectd -> ../init.d/collectd
/etc/rc2.d/K05collectd -> ../init.d/collectd
/etc/rc3.d/K05collectd -> ../init.d/collectd
/etc/rc4.d/K05collectd -> ../init.d/collectd

perf-backed disassembly

2011-08-07T22:12:00.001-04:00

Since 2.6.31 or thereabouts, the Linux kernel has come with a built-in performance counter known as perf.

The common form of perf is well-known to be useful in gathering performance statistics on a running program:

bash$ perf stat -cv ./a.out

cache-misses: 11313 2020574449 2020574449
cache-references: 62031796 2020574449 2020574449
branch-misses: 17909 2020574449 2020574449
branches: 606684832 2020574449 2020574449
instructions: 6324531571 2020574449 2020574449
cycles: 6408533747 2020574449 2020574449
page-faults: 304 2019963367 2019963367
CPU-migrations: 7 2019963367 2019963367
context-switches: 205 2019963367 2019963367
task-clock-msecs: 2019963367 2019963367 2019963367

Performance counter stats for './a.out':

11313 cache-misses # 0.006 M/sec
62031796 cache-references # 30.709 M/sec
17909 branch-misses # 0.003 %
606684832 branches # 300.344 M/sec
6324531571 instructions # 0.987 IPC
6408533747 cycles # 3172.599 M/sec
304 page-faults # 0.000 M/sec
7 CPU-migrations # 0.000 M/sec
205 context-switches # 0.000 M/sec
2019.963367 task-clock-msecs # 0.996 CPUs

2.027948307 seconds time elapsed

The events to be recorded can be specified with the -e option in order to refine the output:

bash$ perf stat -e cpu-clock -e instructi

ons

Performance counter stats for './a.out':

2026.748812 cpu-clock-msecs

6324293589 instructions # 0.000 IPC

2.032519896 seconds time elapsed

A list of available events can be obtained via perf list:

bash$ perf list | head

List of pre-defined events (to be used in -e):

cpu-cycles OR cycles [Hardware event]

instructions [Hardware event]

cache-references [Hardware event]

cache-misses [Hardware event]

branch-instructions OR branches [Hardware event]

branch-misses [Hardware event]

bus-cycles [Hardware event]

The perf toolchain also includes the utility perf top, which can be used to monitor a single process, or which can be used to monitor the kernel:

bash$ sudo perf top 2>/dev/null

-------------------------------------------------------------------------------

PerfTop: 0 irqs/sec kernel:-nan% exact: -nan% [1000Hz cycles], (all, 4 CPUs)

-------------------------------------------------------------------------------

samples pcnt function DSO

_______ _____ ______________________ __________________

77.00 39.3% intel_idle [kernel.kallsyms]

13.00 6.6% __pthread_mutex_unlock libpthread-2.13.so

13.00 6.6% pthread_mutex_lock libpthread-2.13.so

12.00 6.1% __ticket_spin_lock [kernel.kallsyms]

7.00 3.6% schedule [kernel.kallsyms]

6.00 3.1% menu_select [kernel.kallsyms]

6.00 3.1% fget_light [kernel.kallsyms]

6.00 3.1% clear_page_c [kernel.kallsyms]

Where things start to get interesting, however, is with perf record. This utility is generally used along with perf report to record the performance counters of a process, and review them later.

This can be used, for example, to generate a call graph:

bash$ perf record -g -o /tmp/a.out.perf ./a.out

[ perf record: Woken up 1 times to write data ]

[ perf record: Captured and wrote 0.148 MB /tmp/a.out.perf (~6461 samples) ]

bash$ perf report -g -i /tmp/a.out.perf

# Events: 1K cycles

# Overhead Command Shared Object Symbol

# ........ ............. ............. ......

99.90% a.out a.out [.] main

--- main

__libc_start_main

0.10% a.out [l2cap] [k] 0xffffffff8103804a

--- 0xffffffff8105f438

0xffffffff8105f675

...

Once perf data has been recorded, the perf annotate utility can be used to display a disassembly of the instructions that were executed:

bash$ perf annotate -i /tmp/a.out.perf |more

------------------------------------------------

Percent | Source code & Disassembly of a.out

------------------------------------------------

: Disassembly of section .text:

: 0000000000400554

0.00 : 400554: 55 push %rbp
0.00 : 400555: 48 89 e5 mov %rsp,%rbp
0.00 : 400558: 48 81 ec 30 00 0c 00 sub $0xc0030,%rsp
0.00 : 40055f: 48 8d 85 d0 ff fb ff lea -0x40030(%rbp),%rax
0.00 : 400566: ba 00 00 04 00 mov $0x40000,%edx
0.00 : 40056b: be 00 00 00 00 mov $0x0,%esi
0.00 : 400570: 48 89 c7 mov %rax,%rdi
0.00 : 400573: e8 b0 fe ff ff callq 400428 <memset@plt>
0.00 : 400578: c7 45 fc 00 00 00 04 movl $0x4000000,-0x4(%rbp)

...

4.21 : 4006a5: 8b 45 d0 mov -0x30(%rbp),%eax
15.54 : 4006a8: 83 c0 01 add $0x1,%eax
4.97 : 4006ab: 89 45 d0 mov %eax,-0x30(%rbp)
4.87 : 4006ae: 8b 45 d0 mov -0x30(%rbp),%eax
17.79 : 4006b1: 83 c0 01 add $0x1,%eax
4.36 : 4006b4: 89 45 d0 mov %eax,-0x30(%rbp)
4.72 : 4006b7: 48 83 45 f0 01 addq $0x1,-0x10(%rbp)
0.00 : 4006bc: 48 8b 45 f0 mov -0x10(%rbp),%rax

...

As to be expected from Torvalds and company, the utilities include a number of options for generating parser-friendly output, limiting reporting to specified events and symbols, and so forth. Check the man pages for details.

Easy E(17)

2011-07-31T13:46:00.009-04:00

Ubuntu finally ships with a working E17, but there is a lingering problem: most of the modules are gone!

To get a full-featured install of E17, replete with modules and UI enhancements (finally! the settings panel entries re available from the main menu!), it is best to install from SVN.

Building E17 has always been an ordeal. Fortunately, the script easy_e17.sh makes downloading, building, and installing E17 from SVN to be quite a simple affair, once all of the dependencies are met.

To begin with, download the easy_e17 script from http://omicron.homeip.net.

There is a bug in the script which must be fixed if packages are going to be built. According to this bug report, the line (around 36 or 37)

packages_full="$efl_basic $bin_basic $e_modules_bin $e_modules_extra $efl_extra
$bin_extra"

should be

packages_full="$efl_basic $bin_basic $e_modules_bin $efl_extra $bin_extra $e_modules_extra"

... in order to get all the internal dependencies straight. Failure to do this results in the following error:
e_mod_main.c:1527:32: error: ‘ethumb_client’ undeclared (first use in this function)

Next, install all necessary dependencies to build E17 and the modules:

bash$ sudo apt-get install autopoint libudev-dev libgcrypt11 libgcrypt11-dev
bash$ sudo apt-get install libasound2-dev libasound2 libxine-dev
bash$ sudo apt-get install paman padevchooser paprefs pavucontrol pavumeter
bash$ sudo apt-get install libiptcdata-dev libmpd-dev cython libxcb-shape0-dev

According to this guide, the full list of dependencies is as follows:

bash$ sudo apt-get install xterm make gcc bison flex subversion cvs automake1.10 autoconf autotools-dev autoconf-archive libtool gettext libpam0g-dev libfreetype6-dev libpng12-dev zlib1g-dev libjpeg62-dev libtiff4-dev libungif4-dev librsvg2-dev libx11-dev libxcursor-dev libxrender-dev libxrandr-dev libxfixes-dev libxdamage-dev libxcomposite-dev libxss-dev libxp-dev libxext-dev libxinerama-dev libxft-dev libxfont-dev libxi-dev libxv-dev libxkbfile-dev libxres-dev libxtst-dev libltdl7-dev libglu1-xorg-dev libglut3-dev xserver-xephyr libdbus-1-dev liblua5.1-0-dev libasound2-dev libudev-dev autopoint libxml2-dev

Chances are, however, that most of these are already installed.

The E17 build makes some assumptions about the location of the gcrypt libraries, so they must be symlinked:

bash$ sudo ln -s /usr/lib/x86_64-linux-gnu/libgcrypt.a /lib/x86_64-linux-gnu
bash$ sudo ln -s /usr/lib/x86_64-linux-gnu/libgcrypt.la /lib/x86_64-linux-gnu
bash$ sudo ln -s /usr/lib/x86_64-linux-gnu/libgcrypt.so /lib/x86_64-linux-gnu

...other wise linker errors such as the following will appear:
../../src/lib/.libs/libeet.so: undefined reference to `gcry_cipher_setiv'
../../src/lib/.libs/libeet.so: undefined reference to `gcry_cipher_setkey'

The Xine development headers (libxine-dev) must be installed in order for emotion to compile; otherwise an error like this will appear:
configure: error: Xine, Gstreamer or VLC backends must be selected to build Emotion
Naturally, Gstreamer or VLC headers could be installed instead.

The asound2-dev and the pulseaudio tools are required to get the mixer module working. Without them, the error
No ALSA mixer found!

will appear in the mixer module settings. Note that pulseaudio requires that the current user be added to the audio group; this can be done with the following command:

sudo adduser $USERNAME audio

Once all this is done, it is a simple matter to install E17 with the following command:

./easy_e17.sh --instpath=/usr/local/e17 --srcpath=/home/$USER/src/e17 -i

This downloads the E17 source code to ~/src/e17, and sets the install directory to /usr/local/e17. There should be a nice success message if everything goes well.

Next, build all of the modules:

./easy_e17.sh --instpath=/usr/local/e17 --srcpath=/home/$USER/src/e17 --packagelist=full -i

Upgrading is just as straightforward:

./easy_e17.sh --instpath=/usr/local/e17 --srcpath=/home/$USER/src/e17 --packagelist=full -u

Final notes:

* The composite module causes problems: when using the Software engine, it causes typing delays that make terminals near unusable; when using the OpenGL engine, windows do not update until there is a focus change (BIG problem). It should be disabled.

* The shelf containing the systray module must be given the stacking order Above Everything; otherwise, the applications in the E17 systray will not receive mouse clicks (VERY BAD).

* Determining the fonts used by various modules is easier when the source is available. The font classes used in the tclock module, for example, are in E-MODULES-EXTRA/tclock/tclock.edc. The time is displayed in text_class: "module_large", and the date is displayed in text_class: "module_small". The Settings->Look->Fonts dialog can be used to determine the font used for these classes by enabling (via the checkbox at bottom-right) the font class Modules::Small or Modules::Large, and setting the font (e.g. Sans/Regular/12 pixels) to be used for each.

UPDATE : Recent builds now fail when using --packagelist=full. Best stick with --packagelist=half if the full build fails.

Static DNS servers in /etc/resolv.conf

2011-07-31T00:27:00.000-04:00

It's an old problem: NetworkManager overwrites /etc/resolv.conf with each connection, forcing one to use the local ISP's crappy DNS servers instead of one's tried-and-true public DNS servers.

The fixes for this are legion: chattr resolv.conf, disable dhclient, replace NetworkManager.

It turns out that dhclient can be configured to include specified DNS servers (up to twp before a warning is displayed) in the resolv.conf.

Simply add the following line to /etc/dhcp/dhclient.conf :

prepend domain-name-servers 4.2.2.1, 4.2.2.4;

Obviously, replace 4.2.2.1 and 4.2.2.4 with whatever name servers are desired. The semicolon, by the way, is extremely important -- don't leave it out.

Ubuntu, you're fired.

2011-07-29T14:49:00.005-04:00

Attempted to upgrade a workstation (dual-xeon, sata3 ssd for /; sata 3 hdd for /usr, /var, /opt, /usr/local; sata2 mirrored raid (via dmraid) for /home) from 10.04 to 11.04 via do-release-upgrade (in update-manager-core) last night. Why do that? Because Ubuntu only updates software packages for its newest release, so if you need, say, Python 2.7 in order to work on a project, you're SOL unless you upgrade.

Everything went well until the final reboot, which left the machine at:

error: the symbol 'grub_xputs' not found.

Verified that the grub software is all present:

grub rescue> ls (hd0,1)/boot
... lots of modules and such...

Everything seemed to be there, so attempted to boot manually, under the assumption it was all due to misconfiguration:

grub rescue> set root=(hd0,1)
grub rescue> set kernel=/boot/vmlinuz-2.6.32-33-generic
grub rescue> set initrd=/boot/initrd.img-2.6.32-33-generic
grub rescue> boot
error: no loaded kernel

Second attempt, loading the grub modules manually (per Ubuntu Grub2 Docs and this thread):

grub rescue> set prefix=(hd0,1)/boot/grub
grub rescue> insmod normal
error: the symbol 'grub_xputs' not found.
grub rescue> insmod linuxgrub
error: the symbol 'grub_xputs' not found.

For some reason, the symbols in the grub2 bootstrap code do not match the ones expected by the modules. One of them is the wrong version. Looks to be a reported bug.

The solution, per Ubuntu Grub2 Docs and this fix, is to boot a LiveCD (which must be the same version as the newly-installed-but-broken Ubuntu) and run grub-install. This turns out to be a huge delay, as the upgrade took place without a LiveCD, so an ISO must be downloaded. It is also a huge pain when one is out of blank CDs, and has to press a usbstick into service (accomplished per Ubuntu Install Docs).

It also doesn't work. The new error on reboot is

error: file not found

grub rescue> insmod linux

ELF header smaller than expected

It turns out that 10.04 upgrades to 10.10, NOT 11.04, so the LiveCD contained the wrong version of grub!

Fortunately, there is a guide to purging and reinstalling grub once booted from any LiveCD.

The steps are (details such as hdd devices and mount points are specific to this workstation):

bash$ sudo bash

bash# mount /dev/sdk1 /mnt

bash# mount /dev/sdf3 /usr

bash# mount /dev/sdf4 /var

bash# mount --bind /dev /mnt/dev

bash# mount --bind /dev/pts /mnt/dev/pts

bash# mount --bind /proc /mnt/proc

bash# mount --bind /sys /mnt/sys

bash# chroot /mnt

bash-chroot# apt-get update

bash-chroot# apt-get purge grub grub-common grub-pc

bash-chroot# apt-get install grub-common grub-pc

bash-chroot# #NOTE: Be sure to select /dev/sdk for Grub2 install location

bash-chroot# exit

bash# umount /mnt/dev/pts

bash# umount /mnt/dev

bash# umount /dev/sys

bash# umount /mnt/proc

bash# umount /mnt/var

bash# umount /mnt/usr

bash# umount /mnt

bash# reboot

That does the trick.

But you know what would really be nice? A way to upgrade the OS, including the kernel and all packages, WITHOUT TOUCHING THE GODDAMN BOOT-LOADER. Seriously, it's an upgrade, the bootloader already works -- don't mess with it!

Pre-defined preprocessor definitions (GCC)

2011-07-18T15:31:00.001-04:00

The predef project is extremely convenient for looking up architecture-dependent, os-dependent, compiler-dependent, or standard C/C++ compiler macros, but it is not available when working offline.

To get around this, the preprocessor definitions defined automatically by GCC can be viewed with this command line:

bash$ gcc -dM < /dev/null

#define __DBL_MIN_EXP__ (-1021)
#define __UINT_LEAST16_MAX__ 65535
#define __FLT_MIN__ 1.17549435082228750797
...

This will output the #define commands as they would be executed by the preprocessor. To get a list of the non-expanded macros, use -dN :

bash$ gcc -dN < /dev/null
# 1 ""
# 1 ""
#define __STDC__
#define __STDC_HOSTED__
#define __GNUC__
#define __GNUC_MINOR__
...

This is a bit more noisy, as undefined macros (e.g. "# 1") are included, but may be more suitable for parsing.

/usr/bin/say linux

2011-07-17T18:16:00.000-04:00

Anyone who has spent any time fiddling with OS X on the command line will have discovered /usr/bin/say, and incorporated it into a few shell scripts to provide background notifications ("build complete", etc).

While Linux doe snot provide this command out of the box, there is a package called festival which provides text-to-speech capability:

apt-get install festival festival-freebsoft-utils

The usage is quite straightforward:

echo "this is only a test" | festival --tts

A simple alias can be used to replace /usr/bin/say on the command line:

alias say='festival -tts'

A more script-friendly solution would be to create a small wrapper script that passes /usr/bin/say arguments (filenames or STDIN) to festival:

#!/usr/bin/env ruby

def say(str)
`echo '#{str}' | festival --tts`
end

if __FILE__ == $0
say(ARGF.read)
end

RattleR

2011-07-08T01:05:00.000-04:00

Rattle is a useful data analysis UI for R.

Unfortunately, it's a bit hard to get started from a shortcut such as an XDG .desktop file. The usual method (ala Rkward and RCommander), `R_DEFAULT_PACKAGES="$R_DEFAULT_PACKAGES rattle" R`, doesn't work. Instead, one is left doing the `library(rattle); rattle()` commands in an R terminal.

A first stab with Rscript fails:

#!/usr/bin/env Rscript
library(rattle)
rattle()

This starts Rattle, then exits both Rattle and R. Obviously, rattle() is spawning a thread and returning immediately, causing RScript to reach EOF and exit. The same happens when the commands are sent to R, r, or Rscript via STDIN.

Adding a line to wait for an enter keypress seems a likely workaround:

#!/usr/bin/env Rscript
library(rattle)
rattle()

system('bash -c read')

This starts Rattle, and pauses R waiting for an enter key as expected, but Rattle does not receive input. Whatever it's doing for its UI, it is not properly threaded as locking up the parent also locks up the child --- probably some sort of "greenthreads" ala Ruby or Python.

It turns out, however, that this is not an unknown problem. A query of "R_DEFAULT_PACKAGES rattle" turns up pages that provide a fix for the original method: specifying Rattle's dependencies.

The correct command is therefore

R_DEFAULT_PACKAGES="datasets,utils,grDevices,graphics,stats,rattle" R -q --save

Note the -q, to suppress the startup text, and --save (or, alternately, --no-save) to suppress the confirm-quit message. This should be launched in a .desktop file with Terminal=true. R will still wait around for a q() or Ctrl-D, but that is a small price to pay.

Installing Google Go system-wide on Ubuntu

2011-05-14T17:35:00.000-04:00

Just some quick notes. This assumes all necessary packages for C dev are installed (sudo apt-get install bison ed gawk gcc libc6-dev make).

As root (sudo bash):

bash# apt-get install mercurial mercurial-common mercurial-git

bash# cd /usr/local

bash# hg clone -u release https://go.googlecode.com/hg/ go
bash# cd go/src

bash# export GOBIN=/usr/local/bin

bash# ./all.bash

After this, the Go toolchain will be installed in /usr/local/bin, with all supporting files in /usr/local/go. Ubuntu should already have /usr/local/bin in the path for all users.

To test (as a normal user):

bash$ cd /tmp
bash$ cat >hello.go <
> package main
> 
> import "fmt"
> 
> func main() {
> fmt.Printf("hello, world\n")
> }
> EOF
bash$ 6g hello.go
bash$ 6l hello.6
bash$ ./6.out
hello, world
bash$ rm hello.go hello.6 6.out

More on OSS vs Commercial users

2011-05-14T16:36:00.000-04:00

The striking difference in attitude between commercial users and open source users (e.g. more professionalism, gratitude and patience in the former than the latter) is most likely due to one of Cialdini's Influences: Commitment (in the book, "commitment and consistency").

It seems that the initial decision to spend money on software causes an emotional investment in the user. One would therefore expect that when a user has had the choice of software forced upon them (e.g. the use of Windows in a corporate environment, the use of an unfamiliar version control utility in a dev department, etc), they would be as impatient, rude, and generally unprofessional as a typical open source user.

Informal observation bears this out, but it would be nice to see some studies demonstrating the effects -- a simple review of ticketing systems for similar projects (Apache/IIS, Eclipse/VisualStudio, etc) should bear some fruit.

Not as constant as one might think

2011-03-24T00:18:00.001-04:00

One of those Ruby gotchas that has to lead to a bug report before it finally burns itself to memory:

irb(main):001:0> TMP="1234"
=> "1234"
irb(main):002:0> t = TMP
=> "1234"
irb(main):003:0> t.succ!
=> "1235"
irb(main):004:0> TMP
=> "1235"

Eh? Isn't TMP a constant, after all?

Apparently this has to do with t being assigned a reference to a (shared) String:

irb(main):005:0> a="456"
=> "456"
irb(main):006:0> b=a
=> "456"
irb(main):007:0> b.succ!
=> "457"
irb(main):008:0> a
=> "457"

Needless to say, this will do the trick:

t = TMP.dup

This will raise an error on t.succ!, for obvious reasons :

TMP='123'

TMP.freeze

Moral: "Constants" are NOT frozen and their values are not copied on assignment!

Using git to synchronize and backup home directories

2011-03-23T01:09:00.003-04:00

With git-home-history gone, and nothing suitable to take its place, one must handroll a decent solution to version controlling a home directory. Since git can synchronize repositories, it should also be possible to use it to synchronize the contents of a home directory on multiple machines.

This example will assume that a desktop computer (hostname 'desktop')contains the master ('remote') repository, and that a laptop (hostname 'laptop') contains the slave (local) repository.

NOTE: It is important to be aware of which files should be left out of the repo. Because a laptop and a desktop will have different graphics hardware, the settings for GUI applications such as web browsers and window managers should not be in the repo. Also, files which change a lot (cache files, .bash_history, etc) should be kept out of the repo as they will always cause merge conflicts. Finally, private keys should be left out of the repo.

Finally, need it be said that the home directories of both machines should be backed up before trying this?

Desktop: Create and fill the repository

To begin with, create an empty git repo:

bash$ cd ~

bash$ git init .

bash$ touch .gitignore

bash$ git add .gitignore

Next, modify the .gitignore file to select which files or directories to leave out of the repo:

bash$ vi .gitignore

.*

Desktop

Downloads

Templates

tmp

mnt

*.log

*core

*.swp

*.swo

*.bak

This example ignores backup, swap, and core files, as well as directories that probably shouldn't be shared between the two machines (Desktop, Downloads, Templates, tmp, mnt). Note that all hidden files are left out of the repo by default (.*).

Now add all allowed files into the repo:

bash$ git add .

Now add exceptions to the .gitignore file. These will be config files that are shared between the two machines:

bash$ git add -f .bashrc .xsessionrc .vimrc .gvimrc .gdbinitrc  .ssh/config .local/share/applications

If Firefox was configured correctly (i.e. by making a tarball of ~/.mozille/firefox on the desktop machine and extracting it to ~ on the laptop machine, instead of letting Firefox generate its own config), then the bookmarks file can be added as well:

bash$ git add -f .mozilla/firefox/*.default/bookmarks.html

This of course holds true for non-config data in the Firefox dir, such as .mozilla/firefox/*.default/ReadItLater (UPDATE: but not zotero, as it updates itself even while it is not being modified) .

Finally, commit all of the contents to the repo:

bash$ git commit -m 'Initial home dir checkin'

The git directory now has a starting version of the home directory checked in. It can be reviewed with a tool such as QGit to ensure nothing is missing or unwanted:

bash$ qgit &

NOTE: To make the following operations go smoothly, the following line must be added to .git/config :

[receive]

    denyCurrentBranch = false

Desktop : Create a script to auto-commit

At this point. it is useful to create a shell script that performs a commit in the background.

bash$ mkdir -p bin

bash$ vi bin/git_commit_homedir.sh

#!/bin/sh                                                                       
cd ~

git add .
git commit -m 'automated backup' . 

bash$chmod +x bin/git_commit_homedir.sh 

Laptop : Clone the repository

On the laptop, clone the repository from the desktop:

bash$ cd ~

bash$ mkdir -p tmp/git-repo

bash$ cd tmp/git-repo

bash$ git clone desktop:/home/$USER

Note that the repo was cloned to a temporary directory so that it will not overwrite any local files. This is important!

Move the git metadata directory to the home directory:

bash$ cd $USER

bash$ mv .git ~

Retrieve any missing files (i.e. that exist on the desktop but not on the laptop, such as .gitignore) from the repository:

bash$ cd ~

bash$ git checkout \*

Laptop: Add local changes

Create a branch for the changes that will be made next:

bash$ git checkout -b laptop

Add any local exclusions to the .gitignore file:

bash$ echo .pr0n >> .gitignore

Add any additional files to git:

bash$ git add TODO NOTES

Commit the branch:

git commit -m 'laptop additions'

Now merge the branch into master:

bash$ git checkout master

bash$ git merge laptop

Verify that the changes are suitable:

bash$ qgit &

Finally, push the changes to the desktop:

bash$ git push 

Desktop: Generate canonical file versions

The desktop will now have all its files set to the laptop versions.

At this point, files that have been modified should be reviewed and editted, so that a canonical version will be stored in the repo and used by both the desktop and the laptop. QGit makes the review process fairly simple.

Note that some config files will have to source local config files that lie outside the repository (i.e they are excluded in /gitignore). For example, .bashrc might have a line like

[ -f ~/.bash_local.rc ] && . ~/.bash_local.rc

...and .vimrc might have a line like

if filereadable(expand("$HOME/.vim_local.rc"))

    source ~/.vim_local.rc

endif

The files .bashrc_local.rc and .vim_local.rc will be listed in .gitignore, and will have machine-specific configuration such as custom prompts, font size (e.g. in .gvimrc), etc.

Once the canonical versions of the files have been created, they are committed :

bash$ git commit -, 'canonical version' .

bash$ git tag 'canonical'

Laptop: Pull canonical versions

The canonical versions can now be pulled down to the laptop. Note that any supporting files (e.g. .bash_local.rc) will have to be created on the laptop.

bash$ git pull

Laptop & Desktop : Add cron job

In order for git to automatically track changes to the home directory, both the laptop and the desktop will need to add a cron job for running git_commit_homedir.sh .

The following crontab will run git every two hours:

bash$ crontab -e

0 */2 * * * /home/$USER/bin/git_commit_homedir.sh 2>&1 > /dev/null

...of course $USER must be replaced with the actual username.

Note: Some provision must be made for pushing the laptop repo to the desktop. This can be done in a cron job, but is probably better suited to an if-up (on network interface up) script.

UPDATE: Be careful when pushing; the desktop must be forced to update its working tree, or its next commit will delete files on the laptop. The following script will do the trick:

 #!/bin/sh                                                                       
cd $HOME

git push && ssh desktop 'git reset --merge `git rev-list --max-count=1 master`'

Of course passwordless ssh should be set up for this to work. A similar problem exists when pulling from the server: a "git checkout \*" must be performed to create any missing files.

Desktop: Add backup script and cron job

At this point, a backup script and cron job can be added to the desktop server. The directory ~/.git is all that needs to be backed up; a shell script can rsync it to a server.

Running .desktop files from the command line

2011-03-10T02:48:00.000-05:00

After making a few .desktop files with custom commands in them (e.g.RXVT settings), it is useful to be able to run those same commands from the occasional terminal.

When KDE is installed, the kioclient can be used for this purpose:

kioclient exec file:/PATH_TO_DESKTOP_FILE

Custom desktop files are stored in ~/.local/share/applications, so this is easy to wrap in a shell function for inclusion in ~/.bashrc :

xdg-exec () {
    kioclient exec file:${HOME}/.local/share/applications/${1}.desktop
}

Example:

bash$ xdg-exec rxvt-unicode

Ruby, Qt4, and AbstractItemModel

2011-03-03T00:13:00.000-05:00

There are no good examples of using QAbstractItemModel in Ruby. For most purposes, QStandardItemModel will suffice. In this particular case, which calls for a lazily-loaded tree, QStandardItemModel will not cut it.

What follows is a simple implementation. The number of columns is limited to 1, items are read-only, there is no DnD support, and the model only supports the Display role for item data. Needless to say, these features are easy enough to implement, and would only distract from the example of subclassing Qt::AbstractItemModel.

The Model

=begin rdoc

A basic tree model. The contents of all tree nodes are determined by the ModelItems, not the Model, so they may be loaded lazily.

=end

class Model < Qt::AbstractItemModel

  signals 'dataChanged(const QModelIndex &, const QModelIndex &)'

=begin rdoc
The invisible root item in the tree.
=end
  attr_reader :root 

  def initialize(parent=nil)

    super

    @root = nil

end

=begin rdoc

Load data into Model. This just creates a few fake items as an example. A full implementation would create and fill the top-level items after creating.
Note: @root is created here in order to make clearing easy. A clear() method just needs to set root to ModelItem.new('').

=end

  def load()

    @root = ModelItem.new('',nil)

    ModelItem.new('First', @root)

    ModelItem.new('Second', @root)

end

=begin rdoc

This treats an invalid index (returned by Qt::ModelIndex.new) as the index of @root.

All other indexes have the item itself stored in the 'internalPointer' field.

See AbstractItemModel#createIndex.

=end

  def itemFromIndex(index)

    return @root if not index.valid?

    index.internalPointer

end

=begin rdoc

Return the index of the parent item for 'index'.

The  key here is to treat the invalid index (returned by Qt::ModelIndex.new)  as the index of @root. All other (valid) indexes are generated by  AbstractItemModel#createIndex. Note that the item itself is passed as the third parameter (internalPointer) to createIndex.

See ModelItem#parent and ModelItem#childRow. 

=end

  def index(row, column=0, parent=Qt::ModelIndex.new)

    item = itemFromIndex(parent)

    if item

      child = item.child(row)

      return createIndex(row, column, child) if child

end

    Qt::ModelIndex.new

end

=begin rdoc
Return the index of the parent item for 'index'.
This is made a bit complicated by the fact that the ModelIndex must be created by AbstractItemModel. 

The parent of the parent is used to obtain the 'row' of the parent. If the parent is root, the invalid Modelndex is used as usual.
=end

  def parent(index)

    return Qt::ModelIndex.new if not index.valid?

    item = itemFromIndex(index)

    parent = item.parent

    return Qt::ModelIndex.new if parent == @root

    pparent = parent.parent

    return Qt::ModelIndex.new if not pparent

    createIndex(pparent.childRow(parent), 0, parent)

end

=begin rdoc

Return data for ModelItem. This only handles the case where Display Data (the text in the Tree) is requested.

=end

  def data(index, role)

    return Qt::Variant.new if (not index.valid?) or role != Qt::DisplayRole

    item = itemFromIndex(index)

    item ? item.data : Qt::Variant.new

end

=begin rdoc

Set data in a ModelItem. This is just an example to show how the signal is emitted.

=end

  def data=(index, value, role)

    return false if (not index.valid?) or role != Qt::DisplayRole

    item = itemFromIndex(index)

    return false if not item 

    item.data = value.to_s

    emit dataChanged(index, index)

    true

end

  alias :setData :data=

=begin rdoc

Delegate rowCount to item.

See ModelItem#rowCount.

=end

  def rowCount(index)

    item = itemFromIndex(index)

    item ? item.rowCount : 0

end

=begin rdoc

Only support 1 column

=end

  def columnCount(index)

1

end

=begin rdoc

All items can be enabled only.

=end

  def flags(index)

    Qt::ItemIsEnabled

end

=begin rdoc

Don't supply any header data.

=end

  def headerData(section, orientation, role)

    Qt::Variant.new

end

end

The ModelItem

=begin rdoc
An example of a ModelItem for use in the above Model. Note that it does not need to descend from QObject.
 
The ModelItem consists of a data member (the text displayed in the tree), a parent ModelItem, and an array of child ModelItems. This array corresponds directly to the Model 'rows' owned by this item.

=end

class ModelItem

  attr_accessor :data

  attr_accessor :parent

  attr_reader :children

  def initialize(data, parent=nil)

    @data = data

    @parent = parent

    @children = []

    parent.addChild(self) if parent

end

=begin rdoc
Return the ModelItem at index 'row' in @children. This can be made lazy by using a data source (e.g. database, filesystem) instead of an array for @children.
=end

  def child(row)

    @children[row]

end

=begin rdoc
Return row of child that matches 'item'. This can be made lazy by using a data source (e.g. database, filesystem) instead of an array for @children.
=end

  def childRow(item)

    @children.index(item)

end

=begin rdoc
Return number of children. This can be made lazy by using a data source (e.g. database, filesystem) instead of an array for @children.
=end

  def rowCount

    @children.size

end

=begin rdoc
Used to determine if the item is expandible.
=end

  def hasChildren

    childCount > 0

end

=begin rdoc
Add a child to this ModelItem. This puts the item into @children.
=end

  def addChild(item)

    item.parent=self

    @children << item

end

end

This should serve as a basic implementation of an AbstractItemModel.

Realistically, ModelItem would be subclassed to represent different types of items in the data source, each of which would also (likely) be subclassed from ModelItem. This allows a browsable tree to be created for navigating data hierarchies.

The way it goes

2011-02-08T20:34:00.000-05:00

Toy Code : Working source code which is extensively peer-reviewed, obsessively tested, well-documented, and released once. Often used in articles, examples, mailing lists, and blog posts/comments.

Production Code: Working source code which is rarely reviewed, infrequently tested, poorly documented, and regularly released. Responsible for the majority of the world's electronic infrastructure.

Ruby: Array of Hashes to 2-D Array

2011-01-16T15:48:00.002-05:00

Quick Ruby trick.

It is often useful to convert an Array of Hashes (representing a table of objects) to an Array of column names (table header) and an Array of rows (table data). The canonical example would be taking the result of a DB query (ala Sequel) and displaying it in an HTML table (ala DataTables).

Without further ado, here is a proper map/inject one-liner:

array.inject( [data.first.keys, []] ) { |memo, row| memo[1] << memo[0].map { |key| row[key] } ; memo }

A 2-dimensional Array is returned. The first dimension contains the column names, and the second contains the row data (in the same order as the column names, which is the tricky part as Hash#keys can return the keys in any order).

The columns can be sorted (or otherwise ordered specifically) by manipulating data.first.keys when it is initially stored in memo[0].

Example of usage:

irb > data = [ { a: 1, b: 2, c: 3 }, { c: 9, b: 8, a: 7 } ]
irb >  cols, rows = data.inject( [data.first.keys, []] ) { |arr, row| arr[1] << arr[0].map { |key| row[key] } ; arr }

irb > cols
=> [:a, :b, :c]
irb > rows
=> [[1, 2, 3], [7, 8, 9]]

QScintilla and getSelection()

2011-01-06T03:11:00.001-05:00

The recent Ubuntu upgrade and subsequent Ruby woes were caused, of course, by the desire to install libqscintilla-ruby, a package only available on 10.10 (and even then, only for 1.8).

For the most part, QScintilla works fine in Ruby... until one encounters methods like this:

void getCursorPosition(int *line, int *index) 

void getSelection(int *lineFrom, int *indexFrom, int *lineTo, int *indexTo) 

The docs provide some hint as to the problem:

If there is a selection, *lineFrom is set to the line number in which the selection begins and *lineTo is set to the line number in which the selection ends. (They could be the same.) *indexFrom is set to the index at which the selection begins within *lineFrom, and *indexTo is set to the index at which the selection ends within *lineTo. If there is no selection, *lineFrom, *indexFrom, *lineTo and *indexTo are all set to -1.

How does one pass an integer by reference in Ruby?

The answer: one doesn't. These functions take integer arguments and return nil, making them entirely useless in Ruby.

The Python guys did it right:

line_fro, idx_fro, line_to, idx_to = getSelection

The Ruby guys, of course, were lazy, and routed all calls directly to libqscintilla.so regardless of the sanity of their argument lists.

There is a way to make things work, however, thanks to ScintillaBase.

This, the base class of the Scintilla widget, provides the following method:

long SendScintilla(unsigned int msg, unsigned long wParam=0, long lParam=0)

At the top of the base class documentation are a bunch of enums that look promising:

...

 SCI_SETSELECTIONSTART =  2142,   

 SCI_GETSELECTIONSTART =  2143,  
 SCI_SETSELECTIONEND =  2144,   

 SCI_GETSELECTIONEND =  2145, 

...

Sure enough, these turn out to be the values for the msg parameter.

It makes for short work to add the following methods to the Scintilla object:

def get_sel_start

    # SCI_GETSELECTIONSTART 

    self.SendScintilla(2143), 0, 0)

end

def get_sel_end

    # SCI_GETSELECTIONEND
    self.SendScintilla(2145), 0, 0)
end

def get_current_pos
    # SCI_GETCURRENTPOS
    self.SendScintilla(2008), 0, 0)
end

Yes, the messages have to be passed by number, as the symbols for the bulk of the messages have not been defined as constants in Ruby (lazy! bad! lazy!), as can be determined by examining Qsci::ScintilaBase.constants.

Ubuntu getting crappier and crappier

2011-01-03T03:40:00.003-05:00

Stupidly made the decision to upgrade to 10.10 (for a single package!) on the main laptop (mbpro 5,5) and poof! No wireless!

Seriously, it's been three releases since an ethernet cable was required to do an upgrade.

Piece.
Of.
Shit.

Only question now is how many hours of productivity are going to be lost fixing what *was* a perfectly working system before the upgrade.

The sad fact is that os x and windows are just as unreliable for upgrades (while being less usable), and FreeBSD refuses to even glance at this hardware.

A decade ago, this shit used to *work*!

Running log of the fixes:

* apt-get install gnome-icon-theme to get NetworkManager running again. There is no fix for Wicd; it has apparently "stopped working".

* apt-get install gnome-alsa-mixer to un-mute the sound. Kmixer has been reduced from 7 or so channels to 1 (without a mute option).

* uninstall ruby 1.9 via apt-get. Download and install rvm (system-wide as this is a workstation), then do 'sudo rvm --default use 1.9.2' to make 1.9 the system-wide default (might roll that back later).

* reinstall all 1.9 gems.

* rebuild the passenger apache module ('sudo passenger-install-apache2-module') and update the config files to point to the new location.

* fix all rack-based webapps to include the line
$: << File.dirname(__FILE__)
in config.ru and in each Sinatra::Base application file, as something got screwed in the ruby-passenger-rack environment, and PassengerRoot is no longer in the Ruby module path.

See? A smooth, seamless upgrade! You almost don't even notice that it happened!

Apache + Passenger + Sinatra

2010-12-16T00:51:00.001-05:00

OK, this seems like a pretty normal situation, and like most normal situations it apparently never occurs in nature (judging by the available docs).

An existing Apache webserver is to have a new webapp added to it, in a subdirectory of the DocumentRoot. Sinatra is the framework to be used, and Passenger is going to route requests from Apache to Sinatra without going through any mod_proxy or mod_rewrite business.

Assume the following: Apache2, an Ubuntu system, and all of the relevant gems have been apt-get installed. The web server root directory is /var/www, and the webapp will be in /var/www/timon.

First, update the Apache config file (/etc/apache2/apache2/conf):

<VirtualHost *:80>
   ServerName Apemantus
   DocumentRoot /var/www
 
   <Directory />
     Options -Indexes
     AllowOverride None
   </Directory>
 
   <Directory /var/www/>
    AllowOverride AuthConfig
    Order allow,deny
   </Directory>
 
  SetEnv RUBYLIB '/var/www/timon'
  PassengerEnabled on
  PassengerAppRoot /var/www/timon
  RackBaseURI /timon
 </VirtualHost>

The bold lines indicate what must be added.

SetEnv : This just allows the code in subdirectories of the webapp to be easily required.
PassengerEnabled : Seems fairly obvious.
PassengerAppRoot : The full filesystem path to the webapp directory.
RackBaseURI : The relative (to DocumentRoot) path to the webapp directory.

Next, verify that the Passenger options (/etc/apache2/mods-enabled/passenger.conf) are correct:

<IfModule mod_passenger.c>

PassengerRoot /usr
PassengerRuby /usr/bin/ruby
PassengerMaxPoolSize 10
PassengerDefaultUser www-data

</IfModule mod_passenger.c>

These options are pretty straightforward. The only thing to note is that PassengerRuby can be set to a specific version of Ruby, e.g. jruby or ruby1.9.

Now, create an empty Rack-friendly directory structure for the webapp:

bash# cd /var/www

bash# mkdir timon

bash# mkdir timon/public

bash# mkdir timon/tmp 

The use of public for static pages and tmp for restart.txt is well-documented.

Next, a Rack config file must be provided. This will be named config.ru (/var/www/timon/config.ru) and will have the following contents:

#!/usr/bin/env ruby

require 'rubygems'

require 'sinatra'

# Disable Sinatra's default Webrick instance

set :run, false

# Include timon.rb, the main webapp script

require 'timon'

run Sinatra::Application

Finally, the app itself must be provided. This will be named timon.rb (/var/www/timon/timon.rb) and will have the following contents:

#!/usr/bin/env ruby

require 'rubygems'

require 'sinatra'

get '/' do

  "<b>TIMON!</b>"

end

# Local 404 handler

not_found do

  "Timon NotFound exception"

end

# Local error handler

error do

  "Timon Error: " + env['sinatra_error'].name

end

Debugging can be made a bit more straightforward by adding some basic logging code to config.ru:

#!/usr/bin/env ruby

require 'rubygems'

require 'sinatra'

# Disable Sinatra's default Webrick instance 

set :run, false

# Local logging

FileUtils.mkdir_p 'log' unless File.exists?('log')

log = File.new('log/sinatra.log', 'a')

$stdout.reopen(log)

$stderr.reopen(log)

# Include timon.rb, the main webapp script 

require 'timon'

run Sinatra::Application

That's it.

Nothing much to it, really, but the lack of RackBaseURI in the relevant examples really makes debugging this kind of thing difficult.

Ruby CGI and Javascript

2010-11-30T22:41:00.001-05:00

After a brief perusal of the Ruby CGI module documentation, it doesn't seem like there is a good way to generate Javascript from within it.

Due to the method_missing way that the CGI module handles HTML tags, however, it turns out to be quite simple: invoke cgi.script, passing all tag parameters as a hash, and pass the Javascript code as a string in the block:

  cgi.out {
    cgi.html {
      cgi.head {
        cgi.title {
          "Test"
        } +

        # Javascript library to include
        cgi.script( 'src' => 'dygraph-combined.js', 
                    'type' => 'text/javascript') {
        }
      } +

      cgi.body {
        cgi.br +
        cgi.div( 'id' => 'graphdiv' ) +
        cgi.br +
        cgi.script( 'type' => 'text/javascript') {

          # Javascript to execute
          'g = new Dygraph(
                  document.getElementById("graphdiv"),
                  "Date,Temperature\n" +
                  "2008-05-07,75\n" +
                  "2008-05-08,70\n" +
                  "2008-05-09,80\n"
              );
          '
        }

}

}

internationalization regex

2010-10-18T17:38:00.000-04:00

Just a quick vim regex that's useful when adding internationalization to a file:

:%s/\("\([^\\]\\"\|[^"]\)\+"\|'\([^\\]\\'\|[^']\)\+'\)/gettext(\1)/

Wraps all quoted strings in a gettext() call, preserving inner (and escaped) quotes. Doing this in vim instead of sed allows a cursory review of the changes in case any of them * shouldn't* be changed (e.g. log strings).

Indeed.

2010-08-16T02:59:00.000-04:00

I was astonished to learn how accomplished both of my children are in programming. It is a skill that an entire generation of adolescents is learning underground, much the way we used to pick up dirty words.

-- S. Milgram, 1982

Controlling a command-line interpreter in Ruby

2010-08-15T02:30:00.001-04:00

There seem to be a lot of posts to mailing lists, forums, etc in regards to controlling a child process in Ruby via STDIN and STDOUT. Most of the discussion (with this notable exception) ends with "use open3!" or "use open4!".

Anyone who has ever tried to control (i.e., tried to send more than one command to) an interpreter using either of these recognise their shortcomings immediately: the child process STDOUT cannot be read until STDIN has been closed.

Fortunately, the pty module (helpfully mentioned in the notable exception) allows the interpreter to be controlled properly as long as the OS supports psuedo-terminals -- that is, as long as it is a UNIX-like OS (i.e. Linux, OS X, *BSD... basically every desktop/server OS but Windows).

It is a bit tricky to get working well, due to the problem of not knowing for sure whether the child process is preparing more data to write to STDOUT, or is waiting for another command on STDIN.

The following implementation wraps the praat speech analysis software. It uses the praat prompt ('Praat > ') to determine when the child process is ready for more input (i.e., when it can stop reading from STDOUT).

#!/usr/bin/env ruby

require 'pty'

module Praat

  class Interpreter
    attr_reader :stdout, :stdin, :pid

    PROMPT='Praat > '

    def initialize(program='praat')
      @stdout, @stdin, @pid = PTY.spawn( 'praat', '-' )

      # Read initial prompt from pipe
      read_until_prompt

      if block_given?
        yield self
        @stdin.close
        @stdout.close
        @stdin = @stdout = @pid = nil
      end

    end

    def read_until_prompt
      outbuf = buf = ''

      # Read from child STDOUT until > 0 bytes have been read
      # (i.e. wait for child process to finish reading input)
      while buf.length == 0
        begin
          IO.select([@stdout])  # block until child process is ready
          @stdout.read_nonblock( 1024, buf )
        rescue Exception => e
          buf = ''              # READ failure! Try again.
        end
      end

      # Read from child STDOUT until 0 bytes are read or a line ending in a
      # prompt (i.e. next input prompt) was encountered.
      while buf.length > 0
        outbuf << buf

        # complete read if next input prompt is encountered
        break if outbuf =~ /#{PROMPT}$/

        begin
          buf = ''
          @stdout.read_nonblock( 1024, buf )
        rescue Errno::EAGAIN => e
          IO.select([@stdout])  # block until child process is ready
          retry
        rescue Exception => e
          buf=''                # READ failure. Exit loop.
        end

      end

      # Return output of interpreter as an array of lines
      return outbuf.split("\n").each { |x| x.chomp! }
    end

    # Send a command to the interpreter. Returns an array of the output.
    # If include_prompts is true, lines beginning with a prompt will NOT be
    # stripped from the output.
    def send( command, include_prompts=false )
      @stdin.write(command + "\n")

      outbuf = read_until_prompt

      # Ignore all ECHOed lines before the first (input) prompt
      first_prompt = outbuf.find_index { |x| x =~ /^#{PROMPT}/ }
      result = outbuf.slice(first_prompt, outbuf.length-first_prompt)

      # Return full results, or the results with prompt lines removed
      result = result.select {|x| x !~ /^#{PROMPT}/} if not include_prompts

      yield result if block_given?

      return result
    end

  end

end

if __FILE__ == $0

  puts 'Testing block implementation'
  Praat::Interpreter.new() { |p| puts p.send('echo BLOCK TEST') }

  Praat::Interpreter.new() do |p| 
    p.send('echo Full Output', true).each { |line| puts "\t" + line }
  end

  puts 'Testing object implementation'
  praat = Praat::Interpreter.new()

  lines = []
  lines.concat praat.send('echo OBJ TEST 1' )
  lines.concat praat.send('echo OBJ TEST 2' )
  lines.concat praat.send('echo OBJ TEST 3' )

  puts lines.inspect
end

Examining .gem files

2010-08-15T02:11:00.000-04:00

The gem(1) utility has a lot of useful commands for managing gem repositories. It only provides two commands for manipulating .gem files, however: build and install.

When building your own gems, or examining a gem before install, it is useful to be able to take a look at the contents without extracting the gem to a temporary directory.

The structure of the gem is simple: a tar file containing a tarball of the gem contents, and a compressed file containing the metadata:

bash#  tar -tf /tmp/test.gem 

tar: Record size = 19 blocks

data.tar.gz

metadata.gz

Both tar and gzip support input and output on STDOUT, so it is easy enough to write shell functions to access the contents and metadata:

gem_contents () {

  tar -xOf $1 data.tar.gz | tar -ztf -

  return $?

}

gem_metadata () {

  tar -xOf $1 metadata.gz | gzip -d

  return $?

}

This provides quick access to the contents and metadata of a gem file:

gem_contents /tmp/test.gem

gem_metatdata /tmp/test.gem

Unit Testing in C

2010-08-15T02:10:00.003-04:00

There are a number of unit test frameworks for C, but they all seem to lack something.

Sure, each can be used to perform automated testing, and some will even generate test stubs for you. But all of them are limited to API testing; that is, the only functions that can be unit-tested are the public API exported in the header files.

C differs from object-oriented languages in that most of the work -- the units to be tested, as it were -- is done in static functions that are not exported. This makes unit-testing, as it stands, all but useless from a C programmer's perspective; there is no compelling reason to use unit tests over the usual "test programs running on test data and returning 0 or 1 to the makefile" approach.

With some small work, however, it is possible to apply a unit test framework to a C project, and to perform actual unit tests (i.e. on static functions). It is a bit ugly, it is a bit invasive (naturally each .c file must contain its own unit tests), but it is scalable and maintainable.

Unit Testing with CHECK

check is a unit testing framework for C (manual). It is primarily designed to be used with the GNU autotools (autoconf, automake, etc), and the documentation is not clear on integrating check with a standard Makefile-based project. The unit tests developed here will serve as an example.

To begin, assume a project with the following directory structure:

Makefile

stuff.c

stuff.h

tests/
tests/Makefile 

The main program, the shared library libstuff.so, has its API in stuff.h, its code in stuff.c, and is built by the Makefile. The unit tests run by check will be in the tests director.y

stuff.h

#ifndef STUFF_H

#define STUFF_H

double do_stuff( double i  );

#endif

stuff.c

#include <math.h>
#include "stuff.h"

static double step1( double i ) { return i * i; }
static double step2( double i ) { return i + i; }
static double step3( double i ) { return pow( i, i ); }

double do_stuff( double i ) { return step3( step2( step1( i ) ) ); }

Makefile

# Library Makefile

#  ------------------------------------------------------------------- 

NAME        =    stuff

LIBNAME     =    lib$(NAME).so

ARCHIVE     =    lib$(NAME).a

DEBUG       =     -DDEBUG -ggdb

OPT         =    -O2 

ERR         =    -Wall

INC_PATH     =    -I. 

LIB_PATH    =    

#----------------------------------------------------------------

CC          =     gcc

LD           =    ld

AR          =    ar rc

RANLIB      =    ranlib

RM          =    rm -f

LIBS        =    -lm 

CC_FLAGS    =     $(INC_PATH) $(DEBUG) $(OPT) $(ERR) -fPIC

LD_FLAGS    =    $(LIB_PATH) $(LIBS) -shared  -soname=$(LIBNAME)

SRC         =     stuff.c

OBJ         =     stuff.o

#----------------------------------------------------------  Targets

all: $(LIBNAME)

.PHONY: all clean check

$(ARCHIVE): $(OBJ)

    $(AR) $(ARCHIVE) $^

    $(RANLIB) $(ARCHIVE)

$(LIBNAME): $(ARCHIVE)

    $(LD) $(LD_FLAGS) --whole-archive $<  --no-whole-archive -o  $@ 

.c.o: $(SRC)

    $(CC) $(CC_FLAGS) -o $@ -c $<

clean:

    [ -f $(LIBNAME) ] && $(RM) $(LIBNAME)|| [ 1  ]

    [ -f $(ARCHIVE) ] && $(RM) $(ARCHIVE)|| [ 1 ]

    [ -f $(OBJ) ] && $(RM) $(OBJ) || [ 1 ]

    cd tests && make clean

check: $(LIBNAME)

    cd tests && make && make  check

So far, this is all pretty straightforward stuff. The Makefile in the tests directory will contain all of the check-specific settings.

tests/Makefile

# Unit-test Makefile

#--------------------------------------------------------- Definitions

TGT_NAME    =    stuff

TGT_SRC     =    ../stuff.c

OPT         =    -O2  -fprofile-arcs -ftest-coverage

ERR          =    -Wall

INC_PATH    =    -I.  -I..

LIB_PATH    =    -L..

LD_PATH     =     ..

#--------------------------------------------------------- 

CC          =     gcc

RM           =    rm -f

CHECK_LIBS  =     -lcheck

LIBS        =     -l$(TGT_NAME)

# NOTE: UNIT_TEST  enables the static-function test case in stuff.c

CC_FLAGS    =     $(INC_PATH) $(OPT) $(ERR) -DUNIT_TEST

# NOTE: check libs must be enclosed by --whole-archive  directives

LD_FLAGS    =     $(LIB_PATH) -Wl,--whole-archive \

                 $(LIBS) $(CHECK_LIBS) \

                 -Wl,--no-whole-archive

# Test Definitions (to be added later)

TESTS       =  

#----------------------------------------------------------  Targets

all: $(TESTS)

.PHONY: all clean check

clean:

    $(RM) $(TESTS) *.gcno *.gcda

check:

        @for t in $(TESTS); do                          \

                LD_LIBRARY_PATH='$(LD_PATH)' ./$$t;     \

        done

There are a few things to take note of here.

First, the compiler options profile-arcs and test-coverage cause check to perform test coverage profiling. See the the manual for details.

Next, libcheck.a (there is no .so) is added to the linker options, enclosed in --whole-archive directives so that the linker will not discard what it thinks are unused object files. This last point is important; when linking a static library into a dynamic library, --whole-archive must be used.

Finally, the preprocessor definition UNIT_TEST is added to the compiler options. This will be used to ensure that unit tests do not get built into a distribution version of libstuff.so.

Simple Unit Test

The first unit test will be a simple one that ensures the library links with no errors. This is a common problem when developing a shared library; unresolved symbols will not be reported until an executable is linked to the library.

tests/test_link.c

#include <stuff.h>

int main(void) { return (int) do_stuff( 32.0 ); }

Add a make target for this first test.

tests/Makefile

# Test 1 : Simple test to ensure that linking against the  library  succeeds

TEST1        =     test_link

TEST1_SRC    =      test_link.c 

TEST1_FLAGS  =     -l$(TGT_NAME)

TESTS        =     $(TEST1 

# ...

$(TEST1): $(TEST1_SRC)

    $(CC) $(CC_FLAGS) $(LD_FLAGS) $(TEST1_FLAGS) -o $@ $^

Note that the lines before # ... go in the definitions (top) part of the Makefile, while the lines after it go in the targets (bottom) part of it.

This first test can now be run:

bash# make check
gcc -I. -DDEBUG -ggdb -O2 -Wall -fPIC -o stuff.o -c stuff.c
ar rc libstuff.a stuff.o
ranlib libstuff.a
ld -lm -shared -soname=libstuff.so --whole-archive libstuff.a --no-whole-archive -o libstuff.so
cd tests && make && make check
make[1]: Entering directory `stuff/tests'
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_link test_link.c
make[1]: Leaving directory `stuff/tests'
make[1]: Entering directory `stuff/tests'
make[1]: Leaving directory `stuff/tests'

Success! Note that the link test does not involve check; make will error out if the linking fails. Pedantic unit testers may want to take the route of making the test fail first (by invoking, say, do_nothing() in test_link.c) in order to convince themselves that it works.

Testing Static Functions

Testing a static function requires embedding the test code in the file that contains the function. The UNIT_TEST preprocessor directive will prevent the unit test from being compiled in non-unit-test binaries.

First, append the test code to the end of the library code.

stuff.c

/*  =================================================================  */

/* UNIT TESTS */

#ifdef UNIT_TEST

#include <check.h>
#include <stdlib.h>    /*  for rand() */

START_TEST (test_step1)
{
    double d =  (double) rand();
    fail_unless( step1(d) == (d * d),  "Step 1 does not square" );
}
END_TEST

START_TEST (test_step2)
{
    double d =  (double) rand();
    fail_unless( step2(d) == (d + d),  "Step 2 does not double" );
}
END_TEST

START_TEST (test_step3)
{
    double d =  (double) rand();
    fail_unless( step3(d) == pow(d, d),  "Step 3 does not exponentiate" );
}
END_TEST

TCase * create_static_testcase(void) {
    TCase * tc =  tcase_create("Static Functions");
    tcase_add_test(tc, test_step1);
    tcase_add_test(tc,  test_step2);
    tcase_add_test(tc, test_step3);
    return tc;
}

#endif      

The START_TEST and END_TEST macros are provided by check, as are the tcase_ routines. The strategy here is to define a single exported function, create_static_testcase, which the unit-test binaries will link to.

The next step is to add a unit test suite to the test directory.

tests/test_stuff.c

#include <check.h>
#include <math.h>
#include <stdlib.h>

#include <stuff.h>

#define SUITE_NAME "Stuff"

/*  -----------------------------------------------------------------  */
/* TESTS */

START_TEST (test_stuff_diff)
{
    double d = (double) rand();
    fail_unless ( do_stuff(d) != d, "do_stuff doesn't!" );
}
END_TEST

START_TEST (test_stuff_rand)
{
    double d = (double) rand();
    double dd = d * d;
    double sumdd = dd + dd;
    fail_unless ( do_stuff(d) == pow(sumdd, sumdd), "Incorrect result"  );
}
END_TEST

/*  -----------------------------------------------------------------  */
/* SUITE */

extern TCase * create_static_testcase(void);

Suite * create_suite(void) {
    Suite *s = suite_create( SUITE_NAME );

    /* Create test cases against library API */
    TCase *tc_core = tcase_create ("Core");
    tcase_add_test(tc_core, test_stuff_diff);
    tcase_add_test(tc_core, test_stuff_rand);
    suite_add_tcase(s, tc_core);

    /* Create test cases against static functions */
    suite_add_tcase( s, create_static_testcase() );

    return s;
}

int main( void ) {
    int num_fail;
    Suite *s = create_suite();
    SRunner *sr = srunner_create(s);
    srunner_run_all (sr, CK_NORMAL);
    num_fail = srunner_ntests_failed (sr);
    srunner_free (sr);
    return (num_fail == 0) ? EXIT_SUCCESS : EXIT_FAILURE;
}

This file creates a test suite that contains two tests of the library API (defined in test_stuff.c) and three tests of the static functions (defined in stuff.c). The entire suite will be run when the unit-test binary is executed.

Finally, the new test must be added to tests/Makefile, as discussed earlier.

tests/Makefile

# Test 2 : Actual unit tests against source code

# NOTE: this cannot link against the library as it  incorporates the  source code

TEST2         =    test_$(TGT_NAME)

TEST2_SRC     =     $(TEST2).c \

             $(TGT_SRC)

TEST2_FLAGS  =

TESTS        =     $(TEST1) $(TEST2)

# ...

$(TEST2): $(TEST2_SRC)

    $(CC) $(CC_FLAGS) $(LD_FLAGS) $(TEST2_FLAGS) -o $@ $^

Now all of the tests will be run when the make target is invoked:

gcc -I. -DDEBUG -ggdb -O2 -Wall -fPIC -o stuff.o -c stuff.c
ar rc libstuff.a stuff.o
ranlib libstuff.a
ld -lm -shared -soname=libstuff.so --whole-archive libstuff.a --no-whole-archive -o libstuff.so
cd tests && make && make check
make[1]: Entering directory `stuff/tests'
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_link test_link.c
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_stuff test_stuff.c ../stuff.c
make[1]: Leaving directory `stuff/tests'
make[1]: Entering directory `stuff/tests'
Running suite(s): Stuff
100%: Checks: 5, Failures: 0, Errors: 0
make[1]: Leaving directory `stuff/tests'

Up and running with Ruport

2010-03-17T04:04:00.001-04:00

The need to revamp an aging report generator at work led to a bit of research and thus to Ruport, an impressive Ruby framework for report generation.

Ruport has suffered the same fate as many a young open source project: fast and frenzied development ("release early and often" and all that cal) has resulted in enough API changes that many of the examples and documentation useless. The best info to date is in the Ruport Book, but even this book does not get one up-and-running with anything more than the most trivial projects (e.g. printing reports for a CSV table).

Let's define "up and running" as "generating reports from data in an existing database" -- the problem that most people are likely trying to solve, but which somehow doesn't appear as a ready example in most of the Ruport documentation. What follows is a quick demonstration of getting Ruport "up and running".

First, install the requisite gems. Note: PostgreSQL is being used as the database in this example instead of MySQL, both because it is a better RDBMS and because the world has enough MySQL examples already.

gem install ruport       

# DBI support                                                       
gem install ruport-util

gem install pg
gem install dbi
gem install dbd-pg
# ActiveRecord and AAR support 

gem install activerecord
gem install acts_as_reportable
# Extras - not covered in this example

gem install gruff
gem install documatic

The database is assumed to have the following configuration:

* host : db

* port : 15434

* database: stuff

* user: dba

* password : secret

The 'stuff' database has the table 'complaint' which will the reports will pull data from. Note that the name is not 'complaints', as ActiveRecord would expect (and create) -- the point here is to connect to an existing database, most likely managed by someone else (who, even more likely, is neither a Rails developer nor a fan of the 'database is subservient to the code' approach).

Generating a Report Using the Ruby DBI

Connecting to a database table via the Ruby DBI requires the Ruport::Query class, which has been moved to the ruport-util gem. Ruport::Query maintains a list of named database sources, with the default source being named, surprisingly, :default.

Generating a report therefore becomes a matter of adding a database connection via Ruport::Query#add_source, then instantiating a Ruport::Query and printing its result:

require 'rubygems'
require 'ruport'
require 'ruport/util'

Ruport::Query.add_source( :default, 
                          :user => 'dba', 
                          :password => 'secret',
                          :dsn => 'dbi:Pg:database=stuff;host=db;port=15434' 
                        )
puts Ruport::Query.new( [['select * from complaint']] ).result

All in all, nice and straightforward, except for that argument to Ruport::Query#new. According to the docs, this should be as simple as

Ruport::Query.new( 'select * from complaint' )

...but this fails, and a look at the source shows that Ruport::Query invokes the each() method of the sql argument (which, being a String, as no each() method). To make matters worse, if sql is an array, then the each() method of sql.first() is invoked, so the nested array is needed. This is very likely a bug, and may be fixed in future versions, rendering this example and caveat obsolete.

Generating a Report Using ActiveRecord

The ActiveRecord support in Ruport is a bit more complicated. In particular, the acts_as_reportable (AAR) mixin must be added to every ActiveRecord table class.

Note that with ActiveRecord, a table must be wrapped by a class derived from ActiveRecord::Base. For existing databases, this class can be empty -- although it may be necessary to override the table_name method if the database schema does not conform to ActiveRecord's expectations.

require 'rubygems'
require 'ruport'
require 'ruport/acts_as_reportable'

ActiveRecord::Base::establish_connection( :adapter='postgresql', 
                                          :host=>'db', 
                                          :port='15434',
                                          :database=>'stuff'
                                          :username => 'dba', 
                                          :password => 'secret' )

class Complaint < ActiveRecord::Base
   acts_as_reportable
   def table_name()
       return 'complaint'
  end
end

puts Complaint.report_table

The report_table method is what AAR exposes to generate a Ruport report for the table.

These quick examples should suffice to get Ruport connected to an existing database without spelunking through the docs and gem source. Once connected, it's simply a matter of playing with Ruport and following the (rest of the) docs.

Using GIT as a hierarchical database

2009-12-22T17:39:00.004-05:00

A quick look into the 'plumbing'' of git (the Pro Git book is good) reveals that it can easily store (versioned) data that is not based on the contents of files checked into the repository. This means that it can be used as a database that is based on trees rather than tables or key:value pairs.

Git stores four kinds of objects: BLOBs, trees, commits, and tags. A BLOB is raw data : it is identified by the SHA1 digest of its contents, so the same data will never be stored twice. A tree is a hierarchy: it can contain BLOBs or other tree objects. A commit object is basically a snapshot of a hierarchy (tree) at a point in time, and a tag assigns a name to a BLOB, tree, or commit.

In terms of using git as a database, these components can be considered as follows:

BLOB: raw data (values).
Tree: Parent-child relations between data, and a mapping of a database entries with a raw value.
Commit: A version of the data.
Tag: A label, e.g. the name of a table, or a name for a particular version of the data.

Consider the expression 'x * y + 2'. It can be expressed as a tree:

+(* ( x, y), 2)

Another way to express this is in a filesystem-like tree:

/root/
/root/type # 'operator'
/root/value # '+'
/root/left/type # 'operator'
/root/left/value # '*'
/root/left/left/
/root/left/left/type # 'variable'
/root/left/left/value # 'x'
/root/left/right/
/root/left/right/type # 'variable'
/root/left/right/value # 'y'
/root/right/
/root/right/type # 'integer'
/root/right/value # '2'

In this schema, entries named 'left' and 'right' are trees, while entries named 'type' and 'value' are BLOBS. Each tree contains at least a type and value entry, with left and right entries existing only if children are present.

To begin, create a Git repo:

# mkdir /tmp/expr
# cd /tmp/expr
# git init

Next, enter the raw data into git using git-hash-object:

# echo 'operator' | git-hash-object -w --stdin
e196e4b5d49f41231b184a124b3ff52bb00ef9f6
# echo 'variable' | git-hash-object -w --stdin
6437a1374bfa97cfdac6611fc5d2d650abaf782b
# echo 'integer' | git-hash-object -w --stdin
82539ed1cd5cbb2c26a500a228a865d8fbbbcd02
# echo '*' | git-hash-object -w --stdin
72e8ffc0db8aad71a934dd11e5968bd5109e54b4
# echo '+' | git-hash-object -w --stdin
fd38861632cb2360cb5acb6253e7e281647f034a
# echo 'x' | git-hash-object -w --stdin
587be6b4c3f93f93c489c0111bba5596147a26cb
# echo 'y' | git-hash-object -w --stdin
975fbec8256d3e8a3797e7a3611380f27c49f4ac
# echo '2' | git-hash-object -w --stdin
0cfbf08886fca9a91cb753ec8734c84fcbe52c9f

Note that git-hash-object can be run without the -w (write) option in order to generate the SHA1 digest of the data without adding it to git.

The tree hierarchy can be created with git-update-index:

# # Root: Operator '+'
#git-update-index --add --cacheinfo 100644 e196e4b5d49f41231b184a124b3ff52bb00ef9f6 'root/type'
# git-update-index --add --cacheinfo 100644 fd38861632cb2360cb5acb6253e7e281647f034a 'root/value'

# # Root Left Child: Operator '*'
# git-update-index --add --cacheinfo 100644 e196e4b5d49f41231b184a124b3ff52bb00ef9f6 'root/left/type'
# git-update-index --add --cacheinfo 100644 72e8ffc0db8aad71a934dd11e5968bd5109e54b4 'root/left/value'

# # * Left Child: Variable 'x'
# git-update-index --add --cacheinfo 1006446437a1374bfa97cfdac6611fc5d2d650abaf782b 'root/left/left/type'
# git-update-index --add --cacheinfo 100644 587be6b4c3f93f93c489c0111bba5596147a26cb 'root/left/left/value'

# # * Right Child: Variable 'y'
# git-update-index --add --cacheinfo 1006446437a1374bfa97cfdac6611fc5d2d650abaf782b 'root/left/right/type'
# git-update-index --add --cacheinfo 100644 975fbec8256d3e8a3797e7a3611380f27c49f4ac 'root/left/right/value'

# # Root Right Child: Integer '2'
# git-update-index --add --cacheinfo 100644 82539ed1cd5cbb2c26a500a228a865d8fbbbcd02 'root/right/type'
# git-update-index --add --cacheinfo 100644 0cfbf08886fca9a91cb753ec8734c84fcbe52c9f 'root/right/value'

# git-write-tree
7fe6589700060de813d529c48937b7678c297d42
# git-ls-tree 7fe6589700060de813d529c48937b7678c297d42
040000 tree b430c300c59dd80764892644c637ba6e29785c09 root
# git-ls-tree -r 7fe6589700060de813d529c48937b7678c297d42
100644 blob 587be6b4c3f93f93c489c0111bba5596147a26cb root/left/left/value
100644 blob 975fbec8256d3e8a3797e7a3611380f27c49f4ac root/left/right/value
100644 blob e196e4b5d49f41231b184a124b3ff52bb00ef9f6 root/left/type
100644 blob 72e8ffc0db8aad71a934dd11e5968bd5109e54b4 root/left/value
100644 blob 82539ed1cd5cbb2c26a500a228a865d8fbbbcd02 root/right/type
100644 blob 0cfbf08886fca9a91cb753ec8734c84fcbe52c9f root/right/value
100644 blob e196e4b5d49f41231b184a124b3ff52bb00ef9f6 root/type
100644 blob fd38861632cb2360cb5acb6253e7e281647f034a root/value
# git-ls-files
root/left/left/value
root/left/right/value
root/left/type
root/left/value
root/right/type
root/type
root/value
# git-ls-files --stage \*/value | cut -d ' ' -f 2 | git-cat-file --batch
587be6b4c3f93f93c489c0111bba5596147a26cb blob 2
x

975fbec8256d3e8a3797e7a3611380f27c49f4ac blob 2
y

72e8ffc0db8aad71a934dd11e5968bd5109e54b4 blob 2
*

0cfbf08886fca9a91cb753ec8734c84fcbe52c9f blob 2
2

fd38861632cb2360cb5acb6253e7e281647f034a blob 2
+

# git-ls-files --stage \*/value | cut -d ' ' -f 2 | xargs git-show
x
y
*
2
+

An ls of the repository will show only the .git directory: the data is stored as BLOBs and paths in the GIT database, not as a directory tree on the filesystem.

In the first attempt, git-ls-files returns the database entries lexically ordered by path. A more database-like approach would be to store the nodes of the expression in a single directory (i.e. a table) and linking them via parent and child references. This highlights a small flaw in the plan: it is not possible to get the SHA1 of a tree without writing the index to disk.

# # Remove previous attempt
# rm -r .git
# git init

# # Node 1 Operator '+'
# git-update-index --add --cacheinfo 100644 `echo '+' | git-hash_object -w --stdin` 'nodes/1/value'
# git-update-index --add --cacheinfo 100644 `echo 'operator' | git-hash_object -w --stdin` 'nodes/1/type'

# # Node 2: Operator '*'
# git-update-index --add --cacheinfo 100644 `echo '*' | git-hash-object -w --stdin` 'nodes/2/value'
# git-update-index --add --cacheinfo 100644 `echo 'operator' | git-hash-object -w --stdin` 'nodes/2/type'

# # Node 3: Variable 'x'
# git-update-index --add --cacheinfo 100644 `echo 'x' | git-hash-object -w --stdin` 'nodes/3/value'
# git-update-index --add --cacheinfo 100644 `echo 'variable' | git-hash-object -w --stdin` 'nodes/3/type'

# # Node 4: Variable 'y'
# git-update-index --add --cacheinfo 100644 `echo 'y' | git-hash-object -w --stdin` 'nodes/4/value'
# git-update-index --add --cacheinfo 100644 `echo 'variable' | git-hash-object -w --stdin` 'nodes/4/type'

# # Node 5: Integer '2'
# git-update-index --add --cacheinfo 100644 `echo '2' | git-hash-object -w --stdin` 'nodes/5/value'
# git-update-index --add --cacheinfo 100644 `echo 'integer' | git-hash-object -w --stdin` 'nodes/5/type'

# git write-tree
d986912eb66eb446ddaaa188a5cc1068c8cf951b
# git ls-tree `git ls-tree d986912eb66eb446ddaaa188a5cc1068c8cf951b | grep nodes | awk '{print $3}'`
040000 tree 945999b7c15c9b745333847bf3d341b7f74a784f 1
040000 tree 86a40cb9ccbc91a00ce06c9e9ca3e132f5e22ab6 2
040000 tree 9a4b76bbc0b9a20185333ca44321ba438eb6d71c 3
040000 tree f556930b3cd058453894e0cc386fe6ca10908abd 4
040000 tree 36fde2340dd25814793c1346ebbb0175049665dd 5

# # Set Node 1 children
# git-update-index --add --cacheinfo 100644 945999b7c15c9b745333847bf3d341b7f74a784f 'nodes/1/left'
# git-update-index --add --cacheinfo 100644 36fde2340dd25814793c1346ebbb0175049665dd 'nodes/1/right'

# # Set Node 2 parent and children
# git-update-index --add --cacheinfo 100644 945999b7c15c9b745333847bf3d341b7f74a784f 'nodes/2/parent'
# git-update-index --add --cacheinfo 100644 f556930b3cd058453894e0cc386fe6ca10908abd 'nodes/2/left'
# git-update-index --add --cacheinfo 100644 f556930b3cd058453894e0cc386fe6ca10908abd 'nodes/2/right'

# # Set Node 3 parent
# git-update-index --add --cacheinfo 100644 86a40cb9ccbc91a00ce06c9e9ca3e132f5e22ab6 'nodes/3/parent'

# # Set Node 4 parent
# git-update-index --add --cacheinfo 100644 86a40cb9ccbc91a00ce06c9e9ca3e132f5e22ab6 'nodes/4/parent'

# # Set Node 5 parent
# git-update-index --add --cacheinfo 100644 945999b7c15c9b745333847bf3d341b7f74a784f 'nodes/5/parent'

# git-write-tree
e6b41858f6e7350914dbf1ddbdc2e457ee9bb4d3
# git ls-tree -r e6b41858f6e7350914dbf1ddbdc2e457ee9bb4d3
100644 blob 945999b7c15c9b745333847bf3d341b7f74a784f nodes/1/left
100644 blob 36fde2340dd25814793c1346ebbb0175049665dd nodes/1/right
100644 blob e196e4b5d49f41231b184a124b3ff52bb00ef9f6 nodes/1/type
100644 blob fd38861632cb2360cb5acb6253e7e281647f034a nodes/1/value
100644 blob f556930b3cd058453894e0cc386fe6ca10908abd nodes/2/left
100644 blob 945999b7c15c9b745333847bf3d341b7f74a784f nodes/2/parent
100644 blob f556930b3cd058453894e0cc386fe6ca10908abd nodes/2/right
100644 blob e196e4b5d49f41231b184a124b3ff52bb00ef9f6 nodes/2/type
100644 blob 72e8ffc0db8aad71a934dd11e5968bd5109e54b4 nodes/2/value
100644 blob 86a40cb9ccbc91a00ce06c9e9ca3e132f5e22ab6 nodes/3/parent
100644 blob 6437a1374bfa97cfdac6611fc5d2d650abaf782b nodes/3/type
100644 blob 587be6b4c3f93f93c489c0111bba5596147a26cb nodes/3/value
100644 blob 86a40cb9ccbc91a00ce06c9e9ca3e132f5e22ab6 nodes/4/parent
100644 blob 6437a1374bfa97cfdac6611fc5d2d650abaf782b nodes/4/type
100644 blob 975fbec8256d3e8a3797e7a3611380f27c49f4ac nodes/4/value
100644 blob 945999b7c15c9b745333847bf3d341b7f74a784f nodes/5/parent
100644 blob 82539ed1cd5cbb2c26a500a228a865d8fbbbcd02 nodes/5/type
100644 blob 0cfbf08886fca9a91cb753ec8734c84fcbe52c9f nodes/5/value

Of course, pulling the contents of these files from the DB results in a list of expression tokens ordered by node number:

# git ls-files --stage '*/value' | awk '{print $2}' | xargs git-show
+
*
x
y
2

While correct, this is not useful for building an infix expression. A new tree is needed that acts as an index for the first tree, so that listing the new tree will return the tokens in infix order. This is accomplished by using the names 'left', 'operator', and 'right' for node contents, so that lexical ordering of the index-tree will return tokens in that order.

# # Root: Operator '+'
# git-update-index --add --cacheinfo 100644 `echo '+' | git-hash-object -w --stdin` 'infix/operator'

# # Root Left Child: Operator '*'
# git-update-index --add --cacheinfo 100644 `echo '*' | git-hash-object -w --stdin` 'infix/left/operator'

# # * Left Child: Variable 'x'
# git-update-index --add --cacheinfo 100644 `echo 'x' | git-hash-object -w --stdin` 'infix/left/left/variable'

# # * Right Child: Variable 'y'
# git-update-index --add --cacheinfo 100644 `echo 'y' | git-hash-object -w --stdin` 'infix/left/right/variable'

# # Root Right Child: Integer '2'
# git-update-index --add --cacheinfo 100644 `echo '2' | git-hash-object -w --stdin` 'infix/right/value'

# git-ls-files --stage infix/\* | cut -d ' ' -f 2 | xargs git-show | awk '{printf $0 " "}'
x * y + 2

Here, the 'infix' tree re-creates the hierarchy of the original expression, much as the first attempt did -- only this time, the names of the files and directories are chosen to force an infix ordering when sorted by name. Only the files containing the values to display are included in this tree, and these map to the same SHA1 digest as those in the 'nodes' tree. Note that git only stores one copy of each BLOB, so the infix tree only uses as much space as is required to represent the tree: the value BLOBs are shared with the original table.

Listing of subtrees correctly returns sub-expressions:

# git-ls-files --stage infix/left/\* | cut -d ' ' -f 2 | xargs git-show | awk '{printf $0 " "}'
x * y

Once the data is has been created, the database can be committed, creating a base version:

# echo 'Node database created' | git commit-tree e6b41858f6e7350914dbf1ddbdc2e457ee9bb4d3
f1b038fb0ea444f7a05ef9c48d74aef6ac3d8b7e

Searching the database relies on git-grep, using the --cached option to search the BLOBs instead of the filesystem. The search may be limited to specific paths:

# git-grep --cached 'operator' -- root

root/left/type:operator
root/type:operator
# git-grep --cached '*' -- expr
expr/left/operator:*
# git-grep --cached 'x' -- '*/variable'
expr/left/left/variable:x
# git-grep --cached '*''*/left/*'
expr/left/operator:*
root/left/value:*

Removing entries involves removing the index entries, not the BLOB:

# git-update-index --force-remove 'infix/operator'
# git-ls-files --stage infix\*
100644 587be6b4c3f93f93c489c0111bba5596147a26cb 0 infix/left/left/variable
100644 72e8ffc0db8aad71a934dd11e5968bd5109e54b4 0 infix/left/operator
100644 975fbec8256d3e8a3797e7a3611380f27c49f4ac 0 infix/left/right/variable
100644 0cfbf08886fca9a91cb753ec8734c84fcbe52c9f 0 infix/right/value

This means that the values stay in the database, so that changes can be reversed. The actual 'database' therefore is nothing but a collection of named links to BLOBs. Searching and browsing of the database is done by operating on paths, the 'live' data, while inserting, updating, and deleting database entries requires changing what blobs the paths point to.

It goes without saying that the git commands provided above can be wrapped with shell functions (or Ruby, or Python, or any other scripting language) that create database entries, populate indexes, and commit changes to the data. With a small amount of work, a reasonable version-controlled hierarchical database can be put together, all contained in the .git dir.

More complex projects could mix files managed by git and the above tree-only approach; this is especially useful when storing binary content, which is awkward to deal with when stored in BLOBs (generally requires unzipping). In such cases, finding the top level (root) of the Git repository is useful, and git makes this easy as well:

# mkdir stuff
# cd stuff
# git-rev-parse --git-dir
/tmp/expr/.git

The importance of documentation

2009-12-21T00:49:00.000-05:00

Went back to Python after three or so years of Ruby, and found immediately that loading modules from . is now broken.

Checked the docs, which are in a horrible state (it's in the Language Ref, no wait it's in the Library Ref even though it's a language feature, no wait it's in a PEP, no wait that is out of date and the real documentation has been posted to a mailing list), and found stuff like this:

From Python Docs: The Module Search Path:

"When a module named spam is imported, the interpreter searches for a file named spam.py in the current directory, and then in the list of directories specified by the environment variable PYTHONPATH. This has the same syntax as the shell variable PATH, that is, a list of directory names. When PYTHONPATH is not set, or when the file is not found there, the search continues in an installation-dependent default path; on Unix, this is usually .:/usr/local/lib/python.

"Actually, modules are searched in the list of directories given by the variable sys.path which is initialized from the directory containing the input script (or the current directory), PYTHONPATH and the installation- dependent default."

Well, that's the theory at any rate. Let's test it with actual code:

# mkdir /tmp/test-py
# cd /tmp/test-py
# mkdir -p snark/snob
# touch snark/__init__.py snark/snob/__init__.py snark/snob/stuff.py
# echo -e '#!/usr/bin/env python2.5\nimport os\nprint(os.getcwd())\nimport snark.snob.stuff\n'> a.py
# chmod +x a.py
# mkdir bin
# cp a.py bin/b.py

What would you expect to happen when this is run? Surely having the module in the current directory means that running either a.py or b.py from . will work, right?

# ./a.py
/tmp/py-test
# ./bin/b.py
/tmp/py-test
Traceback (most recent call last):
File "./bin/b.py", line 4, in
import snark.snob.stuff
ImportError: No module named snark.snob.stuff

Nope! And look at that -- according to Python's own system command, the current working directory (as mentioned in their docs) is the same for both scripts!

Explicitly setting PYTHONPATH fixes this:

# PYTHONPATH=. bin/b.py
/tmp/py-test

..but really, shouldn't the docs be a bit less misleading?

And, for that matter, why the hell is the location of the script used instead of the working directory anyways? Either be sane and use the current working directory, or be slightly less sane and check both.

The whole reason for using Python on this project in the first place was to revert to a language and interpreter that is more stable and more professional (in terms of management style) than Ruby, but this kind of crap certainly gives one pause. If thirty minutes with the language turns up something as obviously broken as the module loader, one shudders to think what is going to come up over the course of the project.

Sad days for the Python boys. The interest of Fowler and his crew must have really given Ruby a leg up in terms of quality.

'standalone' rattle

2009-12-06T15:49:00.000-05:00

Rattle is a decent data-mining utility built on top of GNU R, with one small problem: it is impossible to run outside of the R terminal (e.g. from the shell or a desktop icon/menu).

What this means, on Linux, is that to run Rattle one must start R in a terminal and enter the following commands:

> library(rattle)
Loading required package: pmml
Loading required package: XML
Rattle: Graphical interface for data mining using R.
Version 2.5.3 Copyright (C) 2006-2009 Togaware Pty Ltd.
Type 'rattle()' to shake, rattle, and roll your data.
> rattle()

A Ctrl-D to log out of R is also required.

The big problem (ignoring R's inconsistent handling of stdin/out, which can mostly be solved by using littler) is the R Gtk module starts a separate GUI thread, and quitting R does not do a wait on child threads -- it just kills them.

A workaround to launch rattle from outside of R is to provide a simple wrapper script:

#!/usr/bin/env ruby

require 'open3'

r_prog = `which R`.chomp

Open3.popen3( "#{r_prog} --interactive --no-save --no-restore --slave" ) do | st
din, stdout, stderr |
stdin.puts "library(rattle)"
stdin.puts "rattle()"
puts stdout.read
end

This runs R and launches rattle(), but leaves an R process open in the background once rattle() has exited.

Adding a q() call to the script merely terminates the Rattle GUI child thread. In addition, rattle provides no indication that it is running:

> ls()
character(0)
> library(rattle)
Loading required package: pmml
Loading required package: XML
Rattle: Graphical interface for data mining using R.
Version 2.5.3 Copyright (C) 2006-2009 Togaware Pty Ltd.
Type 'rattle()' to shake, rattle, and roll your data.
> ls()
[1] "Global_rattleGUI" "crs"
[3] "crv" "on_aboutdialog_response"
[5] "rattleGUI" "suppressRattleWelcome"
[7] "viewdataGUI"
> rattle()
> ls()
[1] "Global_rattleGUI" "crs"
[3] "crv" "on_aboutdialog_response"
[5] "rattleGUI" "suppressRattleWelcome"
[7] "viewdataGUI"
# Use Ctrl-Q to exit the Rattle GUI
> ls()
[1] "Global_rattleGUI" "crs"
[3] "crv" "on_aboutdialog_response"
[5] "rattleGUI" "suppressRattleWelcome"
[7] "viewdataGUI"

Once the Rattle library has loaded, there is no way to tell from within R if the Rattle GUI is running or not. Modifying Rattle to create a variable when the GUI loads successfully, and to remove it when the GUI quites, would undoubtedly fix this.

Too clever by half

2009-11-25T19:21:00.002-05:00

It's always frustrating to come across a build error in allegedly-portable code, then to discover that the root of it is someone complicating things in a misguided attempt to improve portability -- usually by introducing complexity instead of removing it.

The macports project in particular is rife with the problems, undoubtedly because most of the "cross-platform" open source projects it ports never considered OS X as a build target.

KDE4 has refused to build on a particular OS X box (Quad G5, 10.5.8, XCode 3.1.4) via macports for quite some time. Currently, the build fails in kdebase4-runtime due to an error in the fishProtocol ctor in fish.cpp:

/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp: In constructor 'fishProtocol::fishProtocol(const QByteArray&, const QByteArray&)':
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp:275: error: cannot convert 'const char* (*)()' to 'const char*' for argument '1' to 'size_t strlen(const char*)'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp: In member function 'void fishProtocol::manageConnection(const QString&)':
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp:1079: error: no matching function for call to 'fishProtocol::writeChild(const char* (&)(), int&)'
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.cpp:561: note: candidates are: void fishProtocol::writeChild(const char*, KIO::fileoffset_t)
make[2]: *** [kioslave/fish/CMakeFiles/kio_fish.dir/fish.o] Error 1
make[1]: *** [kioslave/fish/CMakeFiles/kio_fish.dir/all] Error 2
make: *** [all] Error 2

Little did I know at the time that the actual cause for the error was further up:

[ 57%] Generating fishcode.h
cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish && /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/generate_fishcode.sh /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl /opt/local/bin/md5 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/build/kioslave/fish/fishcode.h -f\ 4
sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": unescaped newline inside substitute pattern

The line in question (fish.cpp:275) does a strlen on fishCode, which is not defined anywhere, though there is an #include for fishcode.h .

I searched around, found fishcode.h, and it was close to empty:

#define CHECKSUM "'fb18e850532f42b49f1034b4e17a4cdc /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'"
static const char
*fishCode('
');

Something had gone horribly wrong, somewhere -- fishCode is empty, though that shouldn't cause the error being displayed. Still, builds get wacky, and it's a lead.

Googling around for a copy of fishcode.h, turns up nothing, which is not a surprise as it is obviously autogenerated by the build process. What does turn up, however, are the commands used to generate it:

SUM=`/usr/bin/md5sum
/usr/src/packages/BUILD/kdebase-683581/kioslave/fish/fish.pl | cut -d ' ' -f 1`;
echo '#define CHECKSUM "'$SUM'"' > fishcode.h; echo 'static const char
*fishCode(' >> fishcode.h; sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[
]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'
/usr/src/packages/BUILD/kdebase-683581/kioslave/fish/fish.pl >> fishcode.h; echo
');' >> fishcode.h;

Time to set SUM in the shell and started playing with the sed command, which is obviously what's failing:

bash-3.2$ SUM='fb18e850532f42b49f1034b4e17a4cdc /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'

bash-3.2$ sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[
> ]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'

sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": unbalanced brackets ([])

Something is going wrong, set SUM to a more sane value:

bash-3.2$ SUM='fb18e850532f42b49f1034b4e17a4cdc'

bash-3.2$ sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[> ]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'

sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": unbalanced brackets ([])

bash-3.2$ SUM='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'

bash-3.2$ sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[
> ]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'

sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": unbalanced brackets ([])

Try replacing the embedded newline with a good old '\n':

bash-3.2$ SUM='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'

bash-3.2$ sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[\n]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;'

sed: 1: "s/\\/\\\\/g;s/"/\\"/g;s ...": bad flag in substitute command: 'o'

Sed doesn't like the embedded newline; not a surprise. Now it seems like SUM is being interpreted as part of the expression. Let's see what's really going on:

bash-3.2$ SUM='fb18e850532f42b49f1034b4e17a4cdc /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl'

bash-3.2$ echo sed -e 's/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[> ]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/'$SUM'/;' sed -e s/\\/\\\\/g;s/"/\\"/g;s/^[ ]*/"/;/^"# /d;s/[
]*$/\\n"/;/^"\\n"$/d;s/{CHECKSUM}/fb18e850532f42b49f1034b4e17a4cdc /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdebase4-runtime/work/kdebase-runtime-4.3.3/kioslave/fish/fish.pl/;

Argh! Someone's single quotes aren't properly escaped! Right there at '$SUM'!

Simple enough to fix, though. Fortunately, using sed to generate fishcode.h manually allows the kdebase4-runtime build to continue, which saves the time of tracking down where in the build process that sed command is.

It does raise the question, though: was any of this autogeneration really necessary? And if so, wouldn't a perl script (or even a one-liner) be more reliable?

One can argue that perl does not ship on as many OSes as sed (debatable), but the fact that this build worked on a maintainer's machine and not on an end-user's machine (running an OS highly regulated by the manufacturer, no less) pretty much blows the portability argument out of the water: if that's what the maintainer is truly after, well, their methods aren't working either.

OK, end angry-at-overly-convoluted-open-source-code-again rant. The moral: verify your goddamn shell commands!

Ubuntu, bluetooth, and the E70

2009-09-12T16:36:00.000-04:00

A few quick notes on connecting a phone (in this case the venerable Nokia E70, the last with the excellent gull-wing keyboard design) to ubuntu via bluetooth.

Assuming the kernel modules and all necessary utilities (bluetooth, bluez, obexftp, obexfs, obecpushd, obex-data-server, irda-utils, cobex, gammi, kbluetooth) are installed, the first step is to create an rfcomm device:

mknod /dev/rfcomm0 c 216 0

Next, bind the device to the phone's bluetooth MAC address:

sudo rfcomm bind 0 00:12:D1:AD:FD:5E

The MAC address can be obtained using

hcitool scan

Note that the link and the bind have to be performed at boot; the file /etc/bluetooth/rfcomm.conf can be modified to perform this (it contains a sufficiently clear example).

Once these steps are performed, utilities like cobex and kbluetooth should just work.

With the E70, though, they don't, so it's necessary to use Obex to get data to and from the device.

The standard, longhand way to do this is with obex-ftp:

obexftp -b $MAC_ADDR -B 11 -l

The -b option is the bluetooth MAC address; the -B option is the channel (usually 10 or 11), and the -l option is the command to execute (ls). The man page lists commands.

Better that obexftp is obexfs, which mounts the phone as a filesystem:

mkdir ~/e70
obexfs -b $MAC_ADDR -B 11 ~/e70

This mounts the phone at the mount point ~/e70, where its phone and MMC memory can be accessed directly.

Doxygen gotchas

2009-09-08T23:26:00.001-04:00

Finally took an afternoon to learn doxygen and have been commenting up the latest project for the past 24 or so hours.

Doxygen is easy to get up to speed with, but there are a couple of gotchas that aren't made clear in the documentation. These are listed below in no particular order.

Standalone pages

This is the first thing one notices when mucking about with doxygen: how can the index.html (aka "Main Page") be modified?

The answer is just as quick to find: create a document with a \mainpage tag. But what document? Surely adding a header file just to create doxygen tags is a bit silly?

Indeed. Instead, create a directory (e.g. doc) and use it for standalone doxygen files. A standalone file is basically a text file consisting of only the C++ comment, e.g.:

/*!
 \mainpage The Main Page
 \section sec_1 Section 1
 ...section 1 stuff...
 \section sec_2 Section 2
 ... section 2 stuff...

<HR>
<b>\ref todo%lt;/b>
*/

Name this something like 'main.dox', and add the .dox extension to the FILE_PATTERNS definition in the project Doxyfile:

FILE_PATTERNS += *.dox

These standalone doxygen files are incredibly useful for stuff like installation instructions, HowTos, and FAQs.

They are also useful for defining groups. A file groups.dox can contain group definitions like the following:

/*!                                                                         
\defgroup outer_group "Outer Group"
This module contains outer-group stuff.

\defgroup inner_group "Inner Group"
\ingroup outer_group
These things are in a group in a group.
*/

This provides a single, easy-to-maintain place for group definitions, so that header files and such just have to use "\ingroup". It seems like pretty good practice to put most files and classes in groups -- it makes for more expedient browsing.

Global namespace

Ok, there's a nice 'Namespaces' folder in the tree pane, and what does it contain? The project namespace. What about all those singletons ill-advisably implemented as globals (purely for narrative effect)?

Turns out there is no way to list the global namespace. This seems like a huge oversight -- if there's one thing you want to know about a project, it's how many lame globals the developers used.

Adding a 'globals' page seems like a good way to circumvent it, except for one slight problem -- if you create a standalone doc with "\page globals Global Namespace" in it, doxygen creates a page in the tree called "Global Namespace" ... with its own internal(?) version of the global namespace in it. This means, basically, that globals defined in your project are not there -- it only contains stuff like (for example) qRegisterMetaType invocations. It looks like 'globals' is an undocumented, and not particularly working, doxygen 'special command'.

A workaround is to use xrefitems. Add a line like the following to doxygen (remember, 'globals" is not an allowed name):

ALIASES                +=  "globalfn=\xrefitem global \"Functions\" \"Globals\""

Now, use code like the following to document your global function:

/*! \fn void GlobalSingletonFactoryMess( bool suck_less )
       \globalfn void GlobalSingletonFactoryMess( bool suck_less )
       \param suck_less : Make singletons suck less
*/
void GlobalSingletonFactoryMess( bool );

A page called 'Globals' will appear under 'Related Pages', and will contain the function prototype, with a link to its documentation in the appropriate .h file. Of course, it's called a 'Member', but one can't have everything.

Xrefitems

Speaking of xrefitems, see that second argument in the ALIASES line above? The one that's set to "Functions", and is supposedly the header for the section that contains the xref items?

Yeh, that argument does nothing. Doxygen ignores it. Go on, try it. Set it to "Doxygen, please format my hard drive". Or have it make fun of your boss or users. It makes no difference. That text will never appear.

Multiple namespaces in header files

This one is just plain wacky. Or rather, it's just plain lazy. Of the doxygen parser.

Let's say you have a header file where you declare an interface and an abstract base class implementing that interface (never mind why, it's an example):


namespace Mine {
class MyInterface {
public:
virtual ~MyInterface() {}
virtual void doStuff() = 0;
};
} /* close namespace */

/* interface must be registered in gobal namespace */
Q_DECLARE_INTERFACE( Mine::MyInterface, "com.me.Mine.MyInterface/1.0");

namespace Mine {
class MyClass : public QObject, public MyInterface {
Q_OBJECT
Q_INTERFACES(Mine::MyInterface)
public:
MyClass( QObject *parent=0 );
virtual void doStuff();
};
}

Guess what happens? MyClass never appears in the documentation. The second namespace block leaves the doxygen parsers as befuddled as ... your favorite befuddlement simile.

The solution of course is to put the interface class in its own header file, which is no big deal ... but really, you shouldn't *have* to.

Random \example tag links

OK, there's an example in docs/examples/, that dir is safely added to the Doxyfile EXAMPLE_PATH, and the file shows up where it's supposed to in the doc tree under Examples.

Looking at the class for which it is an example, though, you see nothing -- or maybe it gets linked to a few rather arbitrary methods, or to methods outside of the class altogether.

What's going on?

Well, it turns out that doxygen decides to out-clever you, and only put a link to the example file in elements that appear in the example. Thus, if your class never directly appears as a type in the file (e.g. 'PluginManager::plugin("MyPlugin").status()' appears instead of 'Plugin p(PluginManager::plugin("MyPlugin"); p.status()'), then the documentation for your class will not be linked to the example, no matter how close to the \class tag you put the \example tag. Doxy knows best, eh? Surely you have no idea where you want your examples linked from.

The fix is to rewrite the example so that the class/function/whatever appears clearly and distinctly.

Using qmake without a build target

2009-08-11T18:35:00.000-04:00

One of the greatest weaknesses of Qt's qmake utility has been its over-reliance on templates. The available templates are standard stuff for Qt projects: app, lib, and subdirs (new with Qt4, I believe). These are all well and good when one is just slapping together a bunch of Qt code, but what about when interfacing with other build systems? A command template would be quite useful.

The follow project file is a workaround. The lib template is used (although app could be used as well) with empty HEADERS and SOURCES variables. The qmake variables that specify which build tools to use are set to a NOP (no-operation) tool such as echo or true. An "extra" target is created for the command to be executed, and set to be a pre-dependency of the main target.

Background: There is a config.py in . that has sip generate all C++ code, as well as the Makefile, in the directory 'sip'. The python module (sip_test.so) is output to the same directory. This project filed is invoked, along with many others, from a subdirs project file in the parent directory.



-----------------------------------------------------
# sip_test.pro
TEMPLATE    = lib
TARGET      = sip_test

# Check that sip exists
unix {
      !system(sip -V > /dev/null) {
              error( "SIP does not exist!" )
   }
}

# sip build command and makefile targets
sip.commands          =    python ./config.py && cd sip && make
QMAKE_EXTRA_TARGETS   +=   sip
PRE_TARGETDEPS        =    sip

# compiler toolchain override
unix {
     # 'true' should exist on Unix and OS X. Win32 folk can
     # use TYPE.EXE or something.
      NOP_TOOL    = true
}

QMAKE_CC        =    $$NOP_TOOL
QMAKE_CXX       =    $$NOP_TOOL
QMAKE_LINK      =    $$NOP_TOOL
QMAKE_LN_SHLIB  =    $$NOP_TOOL

# Copy the .so created by sip.commands to the current directory
QMAKE_POST_LINK        =    cp sip/$${TARGET}.so .
-----------------------------------------------------

With this project file, the standard "qmake && make" command works as expected.

Toshiba R500 Backlight Button

2009-05-30T17:50:00.000-04:00

Keep forgetting to u/l this script for toggling the r500 backlight using toshset:

#!/bin/sh

TOSHSET=/usr/local/bin/toshset

STATE=`sudo $TOSHSET -trmode 2>/dev/null | grep -a transreflective | \
cut -f 3 -d ' '`

if [ $STATE = "on" ]
then
NEW_STATE="off"
else
NEW_STATE="on"
fi

sudo $TOSHSET -trmode $NEW_STATE

Xev shows the keycode for the backlight (and the 'i' internet/information/idiot key, so they'll both do the same thing) to be 180, which is already taken by XF86HomePage in Ubuntu. If for some reason this key is undefined (i.e. nothing shows up on `xmodmap -pke | grep '^keycode 180'`), add the following line to ~/.Xmodmap (which might need to be symlinked to ~/.xmodmaprc on non-Ubuntu Linuxes):

keycode 180 = XF86HomePage

In E17, edit the keyboard bindings using Settings-> Settings Panel -> Input -> Key Bindings. For XF86HomePage, select the Launch : Defined Command option, and point it to the above shell script (e.g. in ~/bin).

Note that the script uses sudo to launch toshset, so you will either need to change this to invoke an GUI sudo program like gtksu, or add a line like the following to /etc/sudoers:

$USER ALL=(root) NOPASSWD: /usr/local/bin/toshset

where $USER is your username. Note that this allows any toshset command to be run as root without a password from this user's session, which could be exploited if someone were sufficiently motivated.

Linking static libraries into a shared library

2009-03-26T23:03:00.000-04:00

The ability to link static libraries into a shared library -- e.g.

gcc -shared -o libstuff.so lib1.a lib2.a lib3.a

-- disappeared in one of the 3.x series. This broke my tendency to build applications as static libraries, linked into a shared library, linked to by an executable consisting of little more than a main().

Turns out all that was missing was a GNU ld flag to prevent ld from optimizing the unused object code out of the shared library.

The solution is to wrap the .a files in --whole-archive flags, e.g.

gcc -shared -o libstuff.so -Wl,-whole-archive lib1.a lib2.a lib3.a -Wl,-no-whole-archive

Insipid Ibex

2009-03-06T15:19:00.001-05:00

Finally upgraded (K)Ubuntu to 8.10. Thanks for breaking my xorg config, guys, by assuming that HAL could actually do a better job that my hand-picked settings. This is easily fixed by uncommenting all of the lines commented out by HAL, but it's amazing that it had to be done at all, since the HAL change made X flat-out stop working. Also, for some reason, ~/.xsession no longer works -- one has to use ~/.xsessionrc instead. Those wacky Ubuntites.

Upgraded E17 shortly afterwards, using the packages at

deb http://greenie.sk/ubuntu intrepid e17

The new version of E has compiz support (which has to almost entirely be disabled for it to run at E17's proper speed) and looks nice. The modules all seem to work, for once.

But it crashes!

The first one came immediately, and was fixed by wiping ~/.e, which contained all of my old config settings. Apparently E17 can't reliably be upgraded with an existing config. Thanks for moving to a binary config file, guys, so that this is actually a far larger problem than it should be. Really, how hard is it to use the binary config only at runtime, and compile from text to binary whenever a config change is made?

Some of the other crashes appear to be from the OpenGeu inclusion of compiz. After removing the calls to emerald and ecomorph from enlightenment_startup.sh, and stripping out the -evil command line option (which, along with -good and -psychotic, did not seem to make things stable), it's become pretty stable, if a little buggy. Dialog boxes occasionally do not disappear when closed, requiring a restart of E17.

Also, some E17 idiocies remain -- things that seemed to be oversights before, but now (having been a pretty stable WM for the past three years) appear to be truly asinine design decisions.

Specifically:

* There is no way to modify the properties of an application (short of adding it to iBar, or modifying the desktop file directly). A dialog exists for this (it's used by ibar and when creating a new application), so it can't be much extra work.

* One cannot cut/copy/paste between E17 widgets and X. This is nice to be able to do when, say, setting an icon file path (since the desktop files for KDE applications do not seem to be handled correctly by E17, and therefore have no icon files). Hope you like typing.

* The binary only config, mentioned above. Really, there is no excuse for this -- the rest of the industry learned that lesson in the 90s. Even Windows allows its abhorred registry to be read from and written to text files.

Aside from that, Ibex works well, and the upgrade was generally painless. The only sticky points were the Xorg conf file (their mistake), E17 (my fault for running it), and the patch I needed to add to get toshset supported by the latest kernel (the maintainer's fault, for not getting it merged after 2 years of reliable performance).

more ffox plugins

2009-02-11T18:14:00.000-05:00

These are more for the UI folk:

Colorzilla:Look up a pixel's color value

Unicode lookup/converter : Character lookup utility

MeasureIt : Ruler widget for measuring distance in pixels

Vim links

2009-02-03T22:52:00.000-05:00

People either love vi or hate it; if you hate it, sod off and take your parentheses with you.

It's good to have a printout of the vi reference card handy, and a cheat sheet or two saved locally:

Vi Reference Card (PDF)
Graphical Vim Cheat Sheet (GIF)
Vim Cheat Sheet

In support of the "learn a new vim command every day" effort, both Vim Tips and the Vim Cookbook have proved interesting, as has the piping feature.
Speaking of pipes, pv is quite an interesting utility. So is histring, but it doesn't seem to be available (or maintained) except in rpm.

Vim has a ton of plugins worth checking out, including:
FuzzyFinder
OmniComplete
TagList
RenameWithCScope
C Call Tree Explorer
SourceCodeObedience
VimDebug
MiniBufferExplorer
SnippetsEmu
AllFold
Fly
SessionManager
Alternate
RunView
Scratch*
VcsCommand
MultVals
GitCommit
GitDiff
GitFile
Git Branch Info
There is also NERDTree, but its moderate utility does not justify its horrible name.

Most languages have syntax and snippet plugins, such as:
C
Python
Ruby/Rails
Perl
SQL
x86 and x86-64 asm
XML/HTML

Finally, to forestall the inevitable requests, here is a sample .vimrc .

DNS

2009-02-03T22:30:00.000-05:00

DNS timeouts have been pretty bad lately; decided to take action and make resolve.conf actually be usable. Of course, it gets overwritten every time DHCP is used to configure an interface, so the name servers have to be added to dhclient.conf .

There are quite a lot of reliable servers to choose from:

OpenDNS
TechFAQ public dns server list
dnsserverlist.org
dnsserverlist.org organized by round trip time

The last two are particularly useful: dnsserverlist.org recommends three DNS servers based on your current IP, and provides an option for sorting their list by round-trip-time.

"Permanent" DNS servers can be added by editing /etc/dhcp3/dhclient.conf and adding 'prepend domain-name-servers' lines:

prepend domain-name-servers 216.224.112.14;
prepend domain-name-servers 67.198.198.213;
prepend domain-name-servers 69.111.95.106 ;
prepend domain-name-servers 208.67.222.222;
prepend domain-name-servers 208.67.220.220;
prepend domain-name-servers 4.2.2.1;
prepend domain-name-servers 4.2.2.2;
prepend domain-name-servers 151.197.0.38;
prepend domain-name-servers 151.202.0.84;
prepend domain-name-servers 151.203.0.84;

Remember to list these in reverse-order: the last server prepended will be the first line in resolv.conf.

The ultimate solution, of course, is to set up a local caching DNS server.

Unfortunate words

2009-01-30T03:57:00.000-05:00

leverage: To utilize or exploit in an unspecified, hand-wavy, and generally unplanned manner.
We leverage the core compentency of our acquisitions in this sector to maximise ROI throughout the project lifecycle.

pundit: A cheerleader in the sport of politics.
Pundits are claiming that the candidate's gaffe is an opportunity to appeal to blue-collar voters, in stark contrast to his opponent's overly intellectual demeanor.

punditry: The presentation of half-considered opinion as research- or poll-backed fact. Generally considered the province of pundits and journalists.
(See above)

blogosphere: Amateur hour in the world of publish-or-perish.
Lo, it is easier for a shark to stop swimming, than for a blogger to refrain from banal commentary.

Productive Programmer

2009-01-28T02:15:00.000-05:00

Just tore through Neal Ford's Productive Programmer (978-0-596-51978-0), ended up with mixed feelings about it.

The first part of the book is pretty decent, with pointers to some good tools. This is also a weakness: the book is not going to be a timeless classic, and will likely be of half utility in two to five years.

The second part has good advice, but is mostly a rehash of agile methodologies. Agile has (or rather, has appropriated) a lot of good ideas, but it is ultimately a methodology, and methodologies lead to two things: zealotry and incomplete projects.

The advice given tends towards the obvious ("use the command line", "learn keyboard shortcuts", "make macros and scripts"), and UNIX is presented as something used by Other People.

In fact the book seems to assume that the reader is encumbered by the two most enfeebling technologies out there: Windows and Java. Windows is inescapable as a platform, but one always has the choice not to code like a Windows programmer (excuse me, "developer"). Java seems to be the language of choice in... well, in companies that I have no interest in working for, so nothing lost there. Buying into the Windows Way or the Java Way means you've created a lot of your own problems, and if this book helps you solve some of them, more power to you.

For the non-Windows-and/or-Java Way programmer, stick with the classics by Brooks, Hunt/Thomas, Kernighan, Plauger, Bentley, Fowler. Maybe some Beck, but you have to be careful with evangelicals.

Free as in...

2009-01-27T23:12:00.000-05:00

Finally got sick of building custom Linux kernels just to change CONFIG_HZ from the inane default of 250 to something that makes the 10ms timer in my (proprietary) device driver work reliably (instead of firing 83 times a second). Switched to hi-resolution timers, and immediate encountered this:

FATAL: modpost: GPL-incompatible module my_module.ko uses GPL-only symbol 'hrtimer_cancel'

Turns out, after reading stuff like http://www.linuxdevices.com/articles/AT9161119242.html and http://www.linuxdevices.com/articles/AT5041108431.html, that the lunatics have taken over the asylum: specific kernel subsystems can be restricted so that only GPL modules can call them.

This is ludicrous: why should GPL code care who calls it? This has nothing to do with distribution; this is *usage*.

Not to mention entirely unlike the "free as in speech" ethos that the FSF claims to support. "Free speech" generally doesn't imply "say anything you like, just don't say it in kernel mode!".

Between this GPL nonsense and the changing of the CONFIG_HZ default value in a minor release, the Linux kernel is proving itself quite unreliable. I may have to move this entire product over to FreeBSD.

Smart Search

2008-06-18T17:39:00.000-04:00

By far the most useful plugin I've added to Firefox in the past half-year or so:

https://addons.mozilla.org/en-US/firefox/addon/188

SmartSearch adds a context menu item that allows you to send selected text (or whatever word you right-clicked on) to any one of the links in the Bookmarks 'Quick Searches' folder (Bookmarks->Organize Bookmarks->Bookmarks Menu->Quick Searches). I had to create this folder myself, because at some point I short-sightedly deleted it as being useless.

The SmartSearch page on quicksearches has a number of handy links such as a dictionary, thesaurus, IMDB, amazon, netcraft, whois, wayback archive, and google maps. I added a few more for google images and some source code pages:

Google Images Quicksearch
Google Code Quicksearch
Koders Quicksearch
Krugle Code Quicksearch (requires JS)
Codease SmartQuery
Codease FullText Query

Obviously many more can come in handy, and the code search engines can be trimmed down to just one or two (probably Koders and Codease).

The good news is that new Quicksearches are simple to create: navigate to the search page, right-click on the search form, select 'Add a keyword for this search', invent some useful keyword, and use the Quick Searches folder for the 'Create in' location.

Nightly Tester Tools

2008-05-27T17:43:00.000-04:00

Got sick of the Intel driver crashing X11, and Ubuntu releasing their upgraded packages only on the recent version (Horny Heron or something). Bit the bullet, did the upgrade, and...

...Firefox 3beta got installed! What the hell?!? Poof go all (well, most) of my extensions, you know those handy things I use for, oh, locking down Firefox, anonymous browsing, other small conveniences that make using such a poorly-performing browser worthwhile.

Included in these was Zotero, in which I have quite a number of documents stored for offline viewing, and which *still* has no independent UI for accessing them (even the Open Office plugin requires a running instance of Firefox). Suckfest à la mode, just what I ordered!

A search of the Zotero forums, though, turns up a link to the Nightly Tester Tools plugin, which can be used to run an extension for a previous version of Firefox. Zotero is now up and running on Firefox 3beta.

Version 3 does seem to be a bit faster than 2, and hasn't been too crashy. The theme/extension writers should really get their act together and update/port their stuff before this puppy hits the street.

Pulseaudio, Firefox, Flash 9

2008-05-07T03:27:00.000-04:00

Upgrading to the latest flash plugin (9) kills audio in Firefox. Awesome!

Based on the PulseAudio information on http://www.pulseaudio.org/wiki/PerfectSetup
and the library provided by revolutionlinux, it took only a couple of steps to get things working:

cd /usr/src
git-clone http://git.0pointer.de/repos/libflashsupport.git
cd libflashsupport
sudo ./bootstrap.sh
sudo ./configure --prefix=/usr
sudo make
sudo make install

Then restart firefox.

It probably helps to have /etc/firefox/firefoxrc contain the line

FIREFOX_DSP="padsp"

...and to have /etc/asound.conf contain the lines

pcm.!default {
type pulse
}

ctl.!default {
type pulse
}

pcm.pulse {
type pulse
}

ctl.pulse {
type pulse
}

...but libflashsupport is the magic bit.

VimMate

2008-05-05T18:52:00.000-04:00

Yet another Textmate-style extension to Vim:

http://vimmate.rubyforge.org/

This one integrates a project manager with Vim. Jury is still out on whether it will prove useful, but coupled with the Vim Debugger plugin it could prove quite useful for the dynamic languages.

The Sorry Scheme of Things Entire

Ruby: Make any program a Quine

Web page mirroring (wget) in Ruby

JRuby and RubyGems

Qt4 and Ruby1.9.x

Determining installed version of Ubuntu

First impressions, redux

.vimrc and UTF-8

Fixing strings(1)

Ruby __END__ and DATA

knotify4 uses 100% CPU

Disabling startup (init.d) services in Ubuntu

perf-backed disassembly

Easy E(17)

Static DNS servers in /etc/resolv.conf

Ubuntu, you're fired.

Pre-defined preprocessor definitions (GCC)

/usr/bin/say linux

RattleR

Installing Google Go system-wide on Ubuntu

More on OSS vs Commercial users

Not as constant as one might think

Using git to synchronize and backup home directories

Running .desktop files from the command line

Ruby, Qt4, and AbstractItemModel

The way it goes

Ruby: Array of Hashes to 2-D Array

QScintilla and getSelection()

Ubuntu getting crappier and crappier

Apache + Passenger + Sinatra

Ruby CGI and Javascript

internationalization regex

Indeed.

Controlling a command-line interpreter in Ruby

Examining .gem files

Unit Testing in C

Up and running with Ruport

Using GIT as a hierarchical database

The importance of documentation

'standalone' rattle

Too clever by half

Ubuntu, bluetooth, and the E70

Doxygen gotchas

Using qmake without a build target

Toshiba R500 Backlight Button

Linking static libraries into a shared library

Insipid Ibex

more ffox plugins

Vim links

DNS

Unfortunate words

Productive Programmer

Free as in...

Smart Search

Nightly Tester Tools

Pulseaudio, Firefox, Flash 9

VimMate

Ruby END and DATA