realurl

The inability of browsers to deal with shortened URLs led to the writing of a quick one-liner for verifying the final destination of a URL.

Add this line to ~/.bashrc :

alias realurl='ruby -r net/http -e "ARGV.map{ |s| while(r=Net::HTTP.get_response(URI(s)) and r.code[0] == \"3\"); s=r[\"location\"];end;puts s}"'

Then use the realurl alias to look up URLs:

bash# realurl http://goo.gl/KjdQYT
https://www.blackhat.com/us-16/briefings.html#weaponizing-data-science-for-social-engineering-automated-e2e-spear-phishing-on-twitter?Wolfesp18

The expanded Ruby code is as follows:

#!/usr/bin/env ruby
# expand a shortened URL by following all redirects
require 'net/http'

def expand_url(str)
  while(hdr=Net::HTTP.get_response(URI(str)) and hdr.code[0] == '3')
    str = hdr['location']
  end
  puts str
end

if __FILE__ == $0
  ARGV.each do |arg|
    puts expand_url arg
  end
end 

Sure, it would be nice to print the page Title attribute as well, but that means loading up Nokogiri and ends up being a bit of overkill.

UPDATE: I've been making pretty heavy use of a shell function to get the page title via wget:

pagetitle () 
{ 
    wget --quiet -O - "$1" | sed -n -e 's!.*.*!\1!p'
}

RIP inittab

Your getty configs are all in /lib/systemd/system/getty@.service now.

[Service]
# the VT is cleared by TTYVTDisallocate
ExecStart=-/sbin/agetty --noclear %I $TERM                                      
Type=idle
Restart=always
RestartSec=0
UtmpIdentifier=%I
TTYPath=/dev/%I
TTYReset=yes
TTYVHangup=yes
TTYVTDisallocate=yes
KillMode=process
IgnoreSIGPIPE=no
SendSIGHUP=yes

Control via /etc/systemd/logind.conf -- see man 5 logind.conf for more info.

xdg-open arcanery

In the midst of playing with tycat and terminology on a virtual console*, it became apparent that xdg-open wasn't doing its job in regards to opening PDF documents.

  bash# xdg-open /tmp/dw.pdf 
  Filename "file:///tmp/dw.pdf" does not exist or is not a regular file

A simple check of the default handler shows that everything is fine:
  bash# xdg-mime query default application/pdf
  evince.desktop
..but for fun, let's try to change it and see what happens.
  bash# xdg-mime default apvlv.desktop application/pdf
  bash# xdg-open /tmp/dw.pdf 
  Filename "file:///tmp/dw.pdf" does not exist or is not a regular file
Nothing.

Grepping for the error in /usr/bin/xdg-open comes up empty. It's odd that two unrelated programs (evince and apvlv) would have the incorrect error message (the file does exist and is regular), or that they would both fail to handle file:// URIs. How does xdg-open actually route these requests?

Turns out it depends on your window manager. If you are running Enlightenment, xdg-open performs the following:

open_enlightenment()
{
    enlightenment_open "$1"

    if [ $? -eq 0 ]; then
        exit_success
    else
        exit_failure_operation_failed
    fi
}

Replacing enlightenment_open "$1" with open_generic "$1" works. Can we invoke open_generic when enlightenment_open fails? Turns out we can't, because enlightenment_open returns 0 (success) even when the file isn't opened.

Fixing enlightenment_open might do the trick. The relevant code (enlightenment.git/ src/bin/e_open.c) is here:

        char **itr;

        int ret = EXIT_SUCCESS;

        for (itr = cmds; *itr != NULL; itr++)

          {

             /* Question: should we execute them in parallel? */

             int r = system(*itr);

             if (r < 0)

               fprintf(stderr, "ERROR: %s executing %s\n", strerror(errno),

                       *itr);

             free(*itr);

             if (r > 0) /* Question: should we stop the loop on first faiure? */

               ret = r;

          }

        free(cmds);

        return ret;

It turns out that the status of what enlightenment_open is calling is ignored. But what is it calling? Adding the line fprintf(stderr, "Trying %s\n", *itr); after the system() call uncovers what is actually happening:

bash# make src/bin/enlightenment_open

  CC       src/bin/src_bin_enlightenment_open-e_open.o

  CCLD     src/bin/enlightenment_open

bash# src/bin/enlightenment_open /tmp/dw.pdf 

Filename "file:///tmp/dw.pdf" does not exist or is not a regular file

TRYING evince-previewer 'file:///tmp/dw.pdf

Where the hell did evince-previewer come from? To make a long story short, xdg-open (and enlightenment_open, and all other *_open tools) uses /usr/bin/mimeopen as a backend. 

This utility operates on files, not mime-types:

bash# mimeopen -a /tmp/dw.pdf 

Please choose an application

1) Print Preview  (evince-previewer)

2) Document Viewer  (evince)

3) vprerex  (vprerex)

4) GNU Image Manipulation Program  (gimp)

use application #

Cancelled

OK, so that is where evince-previewer is coming from. A quick test shows that the actual bug is caused by evince-previewer not understanding file URIs:

bash# evince-previewer file:///tmp/dw.pdf

Filename "file:///tmp/dw.pdf" does not exist or is not a regular file

bash# evince file:///tmp/dw.pdf

(evince:21480): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to allocate widget with width -82 and height 15

That Gtk warning accompanies evince loading and displaying the file just fine. 

Changing the handler for PDFs using mimeopen turns out to work just fine:

bash# mimeopen -d /tmp/dw.pdf 

Please choose a default application for files of type application/pdf

1) Print Preview  (evince-previewer)

2) Document Viewer  (evince)

3) vprerex  (vprerex)

4) GNU Image Manipulation Program  (gimp)

5) Other...

use application #2

Opening "/tmp/dw.pdf" with Document Viewer  (application/pdf)

(evince:21510): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to allocate widget with width -82 and height 15

bash#  xdg-open /tmp/dw.pdf 

(evince:21550): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to allocate widget with width -82 and height 15

xdg-open /tmp/a.pdf

(evince:21602): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to allocate widget with width -82 and height 1

Of course, that doesn't mean that evince-previewer is the only bug. Since evince-previewer returns 1 (failure), enlightenment_open should be returning 1 instead of 0:

        int ret = EXIT_FAILURE;

        for (itr = cmds; *itr != NULL; itr++)

          {

             int r = system(*itr);

             if (r < 0)

               fprintf(stderr, "ERROR: %s executing %s\n", strerror(errno),

                       *itr);

             free(*itr);

             ret =  r;

          }

...and xdg-open should check the return value and fallback to open_generic:

    enlightenment_open "$1"

    if [ $? -eq 0 ]; then

        exit_success

    else

        open_generic "$1"

        if [ $? -eq 0 ]; then

            exit_success

        else 

            exit_failure_operation_failed

        fi

    fi

Let's not even get started on why xdg-open doesn't propagate changes back to mimeopen. Wait, why doesn't it?

UPDATE: looks like there's something deeply wrong with with the xdg-* utilities:

bash# xdg-settings get default-web-browser
xdg-settings: unknown desktop environment

The problem here is that xdg-settings performs its own check for desktop environment (modularity, people!) and, of course, implements this poorly. If the desktop environment is detected (as is Enlightenment), but is not KDE|Gnome|XFCE|MATE,  it returns "unknown desktop environment". If the environment is not detected, a generic handler is called. There are some very confused people working on this software.

The fix is to modify xdg-settings, changing the default case statement
*)
    exit_failure_operation_impossible "unknown desktop environment"
    ;;

to

*)
    dispatch_specific generic "$@"
    ;;

* For those interested: install tslib and rebuild EFL from source, then run the following on a virtual terminal:
ELM_ENGINE=fb /usr/local/bin/terminology -f=nexus/20

Yes, the -b option for a backgroud image will work, as will tytls, tyop, and tycat.

Static DNS servers with Connman

For some reason, Connman uses dnsmasq for name resolution, resulting in extremely slow hostname lookups. This can be verified using the cmcc tool:

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc show $i | grep Nameservers; done
Nameservers = [ 127.0.0.1 ]

Nameservers.Configuration = [  ]


The quick fix is to use cmcc to manually specify name servers:

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc edit $i nameservers 8.8.8.8 8.8.4.4; done

unknown property: 8.8.8.8

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc show $i | grep Nameservers; done

Nameservers = [ 8.8.8.8 8.8.4.4 ]

Nameservers.Configuration = [ 8.8.8.8 8.8.4.4 ]

Alternatively, the default DNS server supplied by the router can be used by specifying auto as the nameserver:

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc edit $i nameservers auto; done
unknown property: auto

bash$ for i in `cmcc services | grep -e '^*' | awk '{ print $2 }'`; do cmcc show $i | grep Nameservers; done

Nameservers = [ 192.168.0.1 ]

Nameservers.Configuration = [  ]

Note that the cmcc tool has a bug in its command-line parsing ("argv" is passed to service.SetProperty in cmd_edit_nameservers, but the nameserver arguments are not popped from argv and thus are treated as additional properties to be parsed once cmd_edit_nameservers returns) -- but as cmcc show proves, the changes have been made.

Returning to Pre-Framebuffer console fonts

At some point in time, the various Linuxes out there decided that the large, blocky, easy-to-read console (as in Virtual Console, not Terminal Emulator) fonts should be replaced with tiny, hard-to-read fonts that make it a chore to edit standard 80-column scripts/source/config files in vim.

Experimenting with various grub and kernel options to change the resolution, fix i915.modeset, and so forth make for a lot of rebooting to get things just right.

Enter the Debian console-setup package. This provides a wizard that ultimately calls setfont and stty to configure the Linux Virtual Consoles.

The dialog-based wizard is invoked via sudo dpkg-reconfigure console-setup .  Each screen is simple enough to navigate on its own; however, the following selections will configure the consoles to display an old, 90s-to-early-oughts style of font:

1. UTF-8 [default, just press ENTER]
2. Combined - Latin; Slavic Cyrillic; Greek [default]
3. VGA
4. 28x16

The result should produce a nice 80 column x 25 row vim display that is visible from halfway across the room (and to old geezers a few inches away). Chuck it on a Macbook air to really get some double-takes from passers by.

Determining installed version of Ubuntu

Neither uname nor motd are of much use in determining the version of Ubuntu installed on a third-party system, and searching launchpad.net for version details of key packages in dpkg is annoying.

Fortunately there is a utility called lsb_release that provides this information:


bash$ lsb_release -si
Ubuntu
bash$ lsb_release -sr
11.04
bash$ lsb_release -sc
natty

Complete all the way down to the silly nickname for each release.

Since this is an LSB utility, it's probably been in place for some time, silently waiting to provide more detail than uname. Seriously, would a mention in the SEE ALSO section of the uname manpage be *so* hard to add?

Fixing strings(1)

The first step in any examination of a binary is to run strings(1) on it.

This usually results in output like the following:
bash$ strings /bin/ls
FXL9GX

|"H9

GXL9FX

|"H9

F@L9G@

...which, needless to say, sucks.

The problem here is that a 'string' is any sequence of printable ASCII characters of a minimum length (usually 4).

The output of strings(1) can be made a bit more usable by discarding any string that does not contain a word in the system dictionary. 

An off-the cuff implementation of such a filter in Ruby would be:

#!/usr/bin/env ruby

DICT='/usr/share/dict/american-english-insane'

ARGF.lines.each do |line|

  is_str = false

  line.gsub(/[^[:alnum:]]/, ' ').split.each do |word|

    next if word.length < 4

    is_str = true if (`grep -Fxc '#{word}' #{DICT}`.chomp.to_i > 0)

  end

  puts line if is_str

end 

This would then be passed the output of strings(1):

bash$ strings /bin/ls | ./strings.rb

__gmon_start__

_fini

clock_gettime

acl_get_entry

acl_get_tag_type

...

This is super-slow for large executables, and will miss sprintf-style formatting strings that contain only control characters (e.g. "%p\n"), but for the general case it produces useful output.

Directions for improvement: load one or more dictionary files and perform lookups on them in Ruby. Search for english-like words by taking 4-letter prefixes and suffixes of each 'word' in the string and searching for dictionary words that start or end with that prefix/suffix. Provide early-exit from the inner field look when a match is found. Allow matches of sprintf formatting strings, URIs ('http://'), etc.

/usr/bin/say linux

Anyone who has spent any time fiddling with OS X on the command line will have discovered /usr/bin/say, and incorporated it into a few shell scripts to provide background notifications ("build complete", etc).

While Linux doe snot provide this command out of the box, there is a package called festival which provides text-to-speech capability:

apt-get install festival festival-freebsoft-utils

The usage is quite straightforward:

echo "this is only a test" | festival --tts

A simple alias can be used to replace /usr/bin/say on the command line:

alias say='festival -tts'

A more script-friendly solution would be to create a small wrapper script that passes /usr/bin/say arguments (filenames or STDIN) to festival:


#!/usr/bin/env ruby                                                             


def say(str)
  `echo '#{str}' | festival --tts`
end


if __FILE__ == $0
  say(ARGF.read)
end