Disasm Wordcloud

In a recent discussion of what the possible applications of R to binary analysis are, the usual visualizations (byte entropy, size of basic blocks, number of times a function is called during a trace, etc) came to mind. Past experiments with tm.plugins.webmining, however, also raised the following question: Why not use the R textmining packages to generate a wordcloud from a disassembled binary?

Why not, indeed.

The objdump disassembler can be used to generate a list of terms from a binary file. The template Ruby code for generating a list of terms is a simple wrapper around objdump:

# generate a space-delimited string of terms occurring in target at 'path'
terms = `objdump -DRTgrstx '#{path}'`.lines.inject([]) { |arr, line|
  # ...extract terms from line and append to arr...
  arr
}.join(" ")  

The R code for generating wordclouds has been covered before. The code for disassembly terms can be more simple, as the terms have already been extracted from the raw text (disassembly):

library('tm')
library('wordcloud')

# term occurrences must be in variable "terms"
corpus <- Corpus(VectorSource(terms))
tdm <- TermDocumentMatrix(corpus)
vec <- sort(rowSums(as.matrix(tdm)), decreasing=TRUE)
 df <- data.frame(word=names(vec), freq=vec)

# output file path must be in variable "img_path"
png(file=img_path)
# minimum frequency should be higher than 1 if there are many terms
wordcloud(df$word, df$freq, min.freq=1) 
dev.off()
                             
The most interesting terms in a binary are the library functions that are invoked. The following regex will extract the symbol name from call instructions:

terms = `objdump -DRTgrstx '#{path}'`.lines.inject([]) { |arr, line|
  arr << $1 if line =~ /<([_[:alnum:]]+)(@[[:alnum:]]+)?>\s*$/
  arr
}  

When run on /usr/bin/xterm, this generates the following wordcloud:

The other obvious terms in a binary are the instruction mnemonics. The following regex will extract the instruction mnemonics from an objdump disassembly:

terms = `objdump -DRTgrstx '#{path}'`.lines.inject([]) { |arr, line|
  arr << $1 if line =~ /^\s*[[:xdigit:]]+:[[:xdigit:]\s]+\s+([[:alnum:]]+)\s*/
  arr
}  

When run on /usr/bin/xterm, this generates the following wordcloud:       

Of course, there is always the possibility of generating a wordcloud from the ASCII strings in a binary. The following Ruby code is a crude attempt at creating a terms string from the output of the strings command:

 terms = `strings '#{path}'`.gsub(/[[:punct:]]/, '').lines.to_a .join(' ')

When run on /usr/bin/xterm, this generates the following wordcloud:     

Not as nice as the others, but some pre-processing of the strings output would clear that up.

There is, of course, a github for the code. Note that the implementation is in Ruby, using the rsruby gem to interface with R.

Fixing strings(1)

The first step in any examination of a binary is to run strings(1) on it.

This usually results in output like the following:
bash$ strings /bin/ls
FXL9GX

|"H9

GXL9FX

|"H9

F@L9G@

...which, needless to say, sucks.

The problem here is that a 'string' is any sequence of printable ASCII characters of a minimum length (usually 4).

The output of strings(1) can be made a bit more usable by discarding any string that does not contain a word in the system dictionary. 

An off-the cuff implementation of such a filter in Ruby would be:

#!/usr/bin/env ruby

DICT='/usr/share/dict/american-english-insane'

ARGF.lines.each do |line|

  is_str = false

  line.gsub(/[^[:alnum:]]/, ' ').split.each do |word|

    next if word.length < 4

    is_str = true if (`grep -Fxc '#{word}' #{DICT}`.chomp.to_i > 0)

  end

  puts line if is_str

end 

This would then be passed the output of strings(1):

bash$ strings /bin/ls | ./strings.rb

__gmon_start__

_fini

clock_gettime

acl_get_entry

acl_get_tag_type

...

This is super-slow for large executables, and will miss sprintf-style formatting strings that contain only control characters (e.g. "%p\n"), but for the general case it produces useful output.

Directions for improvement: load one or more dictionary files and perform lookups on them in Ruby. Search for english-like words by taking 4-letter prefixes and suffixes of each 'word' in the string and searching for dictionary words that start or end with that prefix/suffix. Provide early-exit from the inner field look when a match is found. Allow matches of sprintf formatting strings, URIs ('http://'), etc.