URL encoding/decoding with shell script

Quite some time ago, I wrote a simple shell script called "urldecode" which decodes the "escaped", or URL-encoded, string using the "printf" utility from GNU coreutils. However, today when I tried to write a shell script to generate a short URL using tinyurl.com, I face the problem to have a string to be URL-encoded. So, after reading the page "Percent-encoding" on Wikipedia, I finished my "urlencode" script.


Let me talk about the decoding part first. Decoding an URL-encoded string is relatively simple. Since the "printf" utility accepts the "\xHH" format string, where "HH" is 1 to 2 digits of a byte with hexadecimal value, the only necessary pre-processing for the target string would be replacing the '%' characters in the string into '\x' strings. After that, just pass the processed string to "printf" to get the converted string. The following code is my implementation of the above-mentioned process:

#!/bin/bash
#
# urldecode - decoding the URL-encoded string
#
# (C)2010 Shang-Feng Yang <storm_DOT_sfyang_AT_gmail_DOT_com>
#
# License: GPLv3

ENC_STR=$@
[ "${ENC_STR}x" == "x" ] && {
TMP_STR="$(cat - | sed -e 's/%/\\x/g')"
} || {
TMP_STR="$(echo ${ENC_STR} | sed -e 's/%/\\x/g')"
}
PRINTF=/usr/bin/printf
exec ${PRINTF} "${TMP_STR}\n"

The "urldecode" script can read the string from either STDIN or the script calling argument. This script has an obvious shortcoming that, since the whole string is passed as the format string to the "printf" utility, the operation will fail if the length of the encoded string is too long.

For the encoding part, it becomes a little more complicated. At first, I was thinking about finding the reserved characters, escaping them, and then replacing the original characters with the escaped one. For that purpose, I wrote a short script to find the corresponding ASCII byte value of a given character, called "char2hex":

#!/bin/bash
#
# char2hex - returning the hexadecimal value of the given characters
#
# (C)2012 Shang-Feng Yang <storm_DOT_sfyang_AT_gmail_DOT_com>
#
# License: GPLv3

function usage() {
echo -e "Usage:\n"
echo -e "\t$(basename $0) CHARACTER(S)_TO_CONVERT\n"
}

CHAR=$1

[ "x${CHAR}" == "x" ] && { usage; exit 1; }

echo -n "${CHAR}" | od -A n -t x1 | tr -d ' '

This script is quite straight-forward. The only thing that is worth-mentioned is the reason for the '-n' option to the "echo" command. By default, "echo" will append a newline character to what it printed, so you will get an "additional" "0a" from the output. The '-n' option turns off this behavior.

This approach seems to be relatively elegant and simple, but the implementation could potentially be a nightmare. For one thing, it could be because I'm not smart enough, but I can not figure out a simple way to "pick up" and pass to the "char2hex" script the reserved characters from the input string or input stream by using simple shell syntax or simple utilities. It either could take too much effort to just do that, or the efficiency of the script could be quite low due to heavy I/O. It is apparently not an acceptable way to do this kind of thing for such a lazy guy like me.

After reading both the sections "Percent-encoding reserved characters" and "Percent-encoding the percent character" from the Wikipedia page "Percent-encoding", I found that the reserved characters that need to be encoded are not much, so it is practical to implement the "encoding" by using the "lookup table" method. So, the solution is stupid but simple:

#!/bin/bash
#
# urlencode - escaping the reserved characters using URL-encoding
#
# (C)2012 Shang-Feng Yang <storm_DOT_sfyang_AT_gmail_DOT_com>
#
# License: GPLv3

STR=$@
[ "${STR}x" == "x" ] && { STR="$(cat -)"; }

echo ${STR} | sed -e 's| |%20|g' \
-e 's|!|%21|g' \
-e 's|#|%23|g' \
-e 's|\$|%24|g' \
-e 's|%|%25|g' \
-e 's|&|%26|g' \
-e "s|'|%27|g" \
-e 's|(|%28|g' \
-e 's|)|%29|g' \
-e 's|*|%2A|g' \
-e 's|+|%2B|g' \
-e 's|,|%2C|g' \
-e 's|/|%2F|g' \
-e 's|:|%3A|g' \
-e 's|;|%3B|g' \
-e 's|=|%3D|g' \
-e 's|?|%3F|g' \
-e 's|@|%40|g' \
-e 's|\[|%5B|g' \
-e 's|]|%5D|g'

The "urlencode" script is too simple for me to explain it. It also accepts the target string from either STDIN or the command argument. The following demonstrates the usage of the scripts:

$ urlencode http://en.wikipedia.org/wiki/Percent-encoding
http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FPercent-encoding
$ echo 'http://en.wikipedia.org/wiki/Percent-encoding' |urlencode
http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FPercent-encoding
$ urldecode $(urlencode http://en.wikipedia.org/wiki/Percent-encoding)
http://en.wikipedia.org/wiki/Percent-encoding
$ urlencode http://en.wikipedia.org/wiki/Percent-encoding |urldecode
http://en.wikipedia.org/wiki/Percent-encoding

PS. Due to my "upgrading" the old template into new one, there are some formatting error in the code and terminal blocks... I probably will fix them by modifying underlying CSS of the new template in the future if I got enough motivation...

Read more ...

"Speaking Mandarin Chinese" in Hollywood

It's quite often to see some scenes in either TV shows or movies that the characters speak something they claim to be "Mandarin Chinese". Some characters even claim to be very fluent in it. However, for a native Traditional Chinese speaker from Taiwan like me, most of the time, those so called Chinese on screen can hardly be understandable, if it could be understood at all.

It is quite strange, since there should be lots of native Chinese speakers near the production locations of these shows or movie. Is it that hard to find a decent language consultant to make sure the proper pronunciation of a few lines? Or the Hollywood just too proud to admit the fact that, they can't do it right when they are so self-centered and so used to laugh at those whom didn't speak proper English? It's quite painful to hear a character you love to speak something that has nothing resemble to what they claim to be, if that "thing" could be called a language at all.


Read more ...

Grabbing the vanity card of TBBT into an image

The producer of the TV show "The Big Bang Theory", Mr. Chuck Lorre, always shows the vanity card in the end of each episode. He also posts the same cards on his own website along with those for other shows he produced.

Recently, for some reason, I would like to attach as an image in a e-mail the vanity card for a specific episode of the show from the website. I prefer the image to only contain the content of the card rather than the whole page. This, of course, could be done with screen capturing and cropping of the image using something like GIMP or ImageMagick. However, since I'm a lazy guy, and the chance that I will do this more than once is quite high, manually screen capturing and cropping is certainly not an option for me. Fortunately, I have some ideas on how to do this automatically.


To grab the web page into an image on command line, there are lots of possible ways to do this. The weapon of choice is the still-buggy-but-quite-useful wkhtmltoimage from the project wkhtmltopdf. wkhtmltoimage uses WebKit and Qt to render a given page directly into an image. The great thing about this tool is that, it supports CSS and JavaScript from the page, while you can replace the CSS with your own version and can also append some JavaScripts before rendering happens.

At first, I was trying to render the page into an image, and then pass the image into ImageMatick's convert to cut out only the block of the "vanity card" in the page. However, this approach was proven to be problematic, since it is hard to automatically determine the cropping parameters needed for the "-crop" option of convert. After inspecting the HTML and CSS sources of the page, I decided to experiment with the "visibility" attribute in the CSS definition. I downloaded the CSS file, set the "visibility" attribute to "hidden" for the top most selector (the "#container" selector block in this case), turned on the visibility only for the "#content" block, and supplied the customized CSS to wkhtmltoimage. This gave me an rendered image that only shows the "card" block in the center of a white background. The white "border" then can be easily removed using the "-trim" option of convert.

Although the downloading-and-modifying-CSS approach was a success, supplying a whole modified CSS to wkhtmltoimage is not elegant and could have some potential side-effects. Therefore, the better approach is taking advantage of the ability for wkhtmltoimage to run JavaScripts to alter the "visibility" attribute for appropriate selectors after the page is done loading. Here is my final "one-liner" solution to my problem:

$ wkhtmltoimage \
--run-script "document.getElementById('container').style.visibility='hidden';" \
--run-script "document.getElementById('content').style.visibility='visible';" \
http://chucklorre.com/index-bbt.php?p=364 - \
| convert - -trim tbbt.jpg

The generated JPEG image, "tbbt.jpg", only contains the "card" I want.

The principle behind this could also be applied to other pages. I, as usual, wrote a script to save me some typing that can take an optional production number argument to grab the card for an specific episode. However, since it is an very simple script, I won't bother to post the code here...

Read more ...

ann2srt v0.3

Although all the bug fixing, testing, and cleaning up have been done several days ago, I was a little too lazy to write... Anyway, here is the "official release notice" of ann2srt version 0.3.

Thanks to the commenter L who helped me on testing and debugging the script on Cygwin, version 0.3 of ann2srt now can handle the annotations other than Traditional Chinese language that have newlines and commas in them, and also can run correctly under Cygwin environment on Win32 platform.


Due to the fact that version 0.2 script uses CSV (Comma-Separated Values) as an intermediate format, the version 0.2 script will fail if the annotation has newline or comma in it. To fix this, in version 0.3, tr is used to eliminate newlines in the annotation. To address the "comma" problem, the delimiter for the intermediate stream is changed from comma to "|".

The version 0.2 script, technically speaking, should be able to run correctly without any modification under Cygwin environment. However, since Windows uses "DOS style" newline characters that consists CR+LF, if any of the external programs used in the script were Win32 binary, or if the input annotation file was in DOS format, the execution of the script becomes unpredictable. To fix this, tr is used again to convert the annotation and the output of the Win32 XMLStarlet from DOS format into UNIX format.

Let's cut to the chase. Here is the source of the version 0.3 script:

#!/bin/bash
#
# Convert the youtube annotation into SRT subtitle
#
# By Shang-Feng Yang
# Version: 0.3
# License: GPL v3
#
# Changelog:
# * v0.3 (Oct/19/2011):
# - Fix the parsing errors caused by comma and newline characters in
# some English annotations
# - Adding transparent dos2unix conversion for compatibility under Cygwin
# * v0.2 (Jan/19/2011):
# - Sort the annotations using the "begin" time as key
# - Minor bugs fixing
# * v0.1 (Dec/7/2010):
# - Initial release


ANN=$1
SRT=$(basename ${ANN} .xml).srt
IFS=$'\n'
I=0

function usage() {
echo -e "Usage:\n"
echo -e "\t$(basename $0) ANNOTATION_FILE\n"
}

function parseXML() {
cat ${ANN} | tr -d '\r' |tr '\n' ' ' | xmlstarlet sel -t -m 'document/annotations/annotation' -v 'TEXT' -o '|' -m 'segment/movingRegion/rectRegion' -v '@t' -o '|' -b -n | tr -d '\r'
}

function reformatTime() {
local H=$(echo $1 |cut -d ':' -f 1)
local M=$(echo $1 |cut -d ':' -f 2)
local S=$(echo $1 |cut -d ':' -f 3)
printf '%02d:%02d:%06.3f' ${H} ${M} ${S} |tr '.' ','
}

function time2sod() {
# Convert time in HH:MM:SS.SSS format into second-of-the-day value
local SOD=$(echo $1 | awk -F ":" '{printf("%f\n", $1*3600+$2*60+$3);}')

echo ${SOD}
}

[ "x${ANN}" = "x" ] && { usage; exit 1; }
[ -f ${ANN} ] || { usage; exit 1; }
[ -f ${SRT} ] && rm ${SRT}
[ -f ${SRT}.tmp ] && rm ${SRT}.tmp

for LINE in $(parseXML); do
C=$(echo ${LINE} |cut -d '|' -f 1)
B=$(echo ${LINE} |cut -d '|' -f 2)
E=$(echo ${LINE} |cut -d '|' -f 3)
echo "$(time2sod ${B})#${B}#${E}#${C}" >> ${SRT}.tmp
done

grep "###" ${SRT}.tmp && {
echo "\"${ANN}\" has no valid annotation!" >&2
rm ${SRT}.tmp
exit 1
}

for LINE in $(cat ${SRT}.tmp|sort -n -t '#'); do
(( I++ ))
C=$(echo ${LINE} |cut -d '#' -f 4)
B=$(reformatTime $(echo ${LINE} |cut -d '#' -f 2))
E=$(reformatTime $(echo ${LINE} |cut -d '#' -f 3))
echo -e "${I}\n${B} --> ${E}\n${C}\n" >> ${SRT}
done

rm ${SRT}.tmp

The version 0.3 script can also be downloaded from here to avoid typos caused by copy-and-paste:
http://dl.dropbox.com/u/1382119/tmp/ann2srt

In fact, I just found that the customized "code block" loses all indentations after the blogger updates. Please download the correct script from the link above.

Read more ...

ann2srt v0.2

Last time in my post "Converting Youtube's annotation into SRT subtitle, I released a bash script called "ann2srt" v0.1. Version 0.1 was a pretty crude one that did not deal with the sorting of the subtitles in SRT file, and could possibly be problematic for some SRT parser. Yesterday, I spent some time to improve the script with the sorting functionality, and also fixed some minor bugs in v0.1.


The sorting is achieved by using awk/gawk to convert the "beginning" time of the annotation into seconds and then passing the results into sort for sorting. Since sort is part of the GNU coreutils, and awk/gawk should be installed on most of the distributions, this change should not be a big deal for most people.

Here is the code for v0.2:

#!/bin/bash
#
# Convert the youtube annotation into SRT subtitle
#
# By Shang-Feng Yang <storm_DOT_sfyang_AT_gmail_DOT_com>
# Version: 0.2
# License: GPL v3
#
# Changelog:
# * v0.2 (Jan/19/2011):
# - Sort the annotations using the "begin" time as key
# - Minor bugs fixing

function usage() {
echo -e "Usage:\n"
echo -e "\t$(basename $0) ANNOTATION_FILE\n"
}

function parseXML() {
cat ${ANN} |xmlstarlet sel -t -m 'document/annotations/annotation' -v 'TEXT' -o ',' -m 'segment/movingRegion/rectRegion' -v '@t' -o ',' -b -n
}

function reformatTime() {
local H=$(echo $1 |cut -d ':' -f 1)
local M=$(echo $1 |cut -d ':' -f 2)
local S=$(echo $1 |cut -d ':' -f 3)
printf '%02d:%02d:%06.3f' ${H} ${M} ${S} |tr '.' ','
}

function time2sod() {
# Convert time in HH:MM:SS.SSS format into second-of-the-day value
local SOD=$(echo $1 | awk -F ":" '{printf("%f\n", $1*3600+$2*60+$3);}')

echo ${SOD}
}

ANN=$1
SRT=$(basename ${ANN} .xml).srt
IFS=$'\n'
I=0

[ "x${ANN}" = "x" ] && { usage; exit 1; }
[ -f ${ANN} ] || { usage; exit 1; }
[ -f ${SRT} ] && rm ${SRT}
[ -f ${SRT}.tmp ] && rm ${SRT}.tmp

for LINE in $(parseXML); do
C=$(echo ${LINE} |cut -d ',' -f 1)
B=$(echo ${LINE} |cut -d ',' -f 2)
E=$(echo ${LINE} |cut -d ',' -f 3)
echo "$(time2sod ${B})#${B}#${E}#${C}" >> ${SRT}.tmp
done

grep "###" ${SRT}.tmp && {
echo "\"${ANN}\" has no valid annotation!"
rm ${SRT}.tmp
exit 1
}

for LINE in $(cat ${SRT}.tmp|sort -n -t '#'); do
(( I++ ))
C=$(echo ${LINE} |cut -d '#' -f 4)
B=$(reformatTime $(echo ${LINE} |cut -d '#' -f 2))
E=$(reformatTime $(echo ${LINE} |cut -d '#' -f 3))
echo -e "${I}\n${B} --> ${E}\n${C}\n" >> ${SRT}
done

rm ${SRT}.tmp

The usage should be the same with v0.1.

Read more ...

Converting Youtube's annotation into SRT subtitle

It has been a long time since my last blog. Well, I'm a lazy guy, and English is apparently not my native language. Besides, there were lots of things that weren't exciting enough for me to write a long article on the blog, so I usually write short comments on the my Buzz instead.

Any way, let's cut to the chase.

These days, more and more people like to use annotation to add "subtitles" onto Youtube videos rather than to use caption. There already are lots of on-line/off-line "Youtube downloaders" that can download either videos, the corresponding captions, or both of them at once, such as get_flash_videos, clive, youtube-dl, Google2SRT, and Youtube Subtitle Ripper, etc. However, there is not much information available about how to download the annotations and convert them into SRT subtitles. Today, I found the solution.


First of all, I found this comment on the blog post about how to download the annotations in XML format. And yes, I do write a script to download the caption and annotation using wget, but it is a simple script that is not worth to mention. After downloading the annotation in XML, next step would be converting it into some subtitle format.

Although there are many subtitle formats available, and the converting algorithm is possibly existing in the Google2SRT source code, I decide to write my own bash script that converts the XML into the SRT format, which is one of the simplest subtitle format.

The script I wrote, called ann2srt, uses the XMLStarlet as the XML parsing tool. Other than that, the script only uses the bash built-ins and coreutils like cut and tr. For now, the generated SRT could have some compatibility problems with some players. This is because the annotations in the XML are not in chronicle order. Adding the sorting is possible, but since mplayer can handle the out-of-order subs correctly, I'll leave it this way for now. Here is the code of ann2srt:

#!/bin/bash
#
# Convert the youtube annotation into SRT subtitle
#
# By Shang-Feng Yang <storm_dot_sfyang_at_gmail_dot_com>
# Version: 0.1
# License: GPL v3

function usage() {
echo -e "Usage:\n"
echo -e "\t$(basename $0) ANNOTATION_FILE\n"
}

function parseXML() {
cat ${ANN} |xmlstarlet sel -t -m 'document/annotations/annotation' -v 'TEXT' -o ',' -m 'segment/movingRegion/rectRegion' -v '@t' -o ',' -b -n
}

function reformatTime() {
H=$(echo $1 |cut -d ':' -f 1)
M=$(echo $1 |cut -d ':' -f 2)
S=$(echo $1 |cut -d ':' -f 3)
printf '%02d:%02d:%02.3f' ${H} ${M} ${S} |tr '.' ','
}

ANN=$1
SRT=$(basename ${ANN} .xml).srt
IFS=$'\n'
I=0

[ -f ${ANN} ] || { usage; exit 1; }
[ -f ${SRT} ] && rm ${SRT}

for LINE in $(parseXML); do
(( I++ ))
C=$(echo ${LINE} |cut -d ',' -f 1)
B=$(echo ${LINE} |cut -d ',' -f 2)
E=$(echo ${LINE} |cut -d ',' -f 3)
echo -e "${I}\n$(reformatTime ${B}) --> $(reformatTime ${E})\n${C}\n" >> ${SRT}
done

A sidenote for mplayer users: When playing videos with subs generated by this script, remember to turn on the SSA/ASS support by using the "-ass" option. Due to the nature of the annotations, it is possible that several annotations occupy the same time period, and the built-in SRT parser of mplayer will only show one of them, while they will be stacked when -ass is enabled.

SRT is a quite simple format that did not support any special effect, of which the annotations possess such as position and color of the annotations. The next version of the script will be one that converts the annotations into SSA/ASS format -- only if I have the motive to improve it...

Read more ...

Rotate the Cube in Compiz with wmctrl

The Cube plugin in Compiz or Compiz-Fusion could change the desktop into a virtual cube, and each face of the cube is one of the virtual desktop. Normally, the "rotation" of the cube -- rotate the cube to swith to other virtual desktop on the face -- could be done by keyboard or mouse. However, what if I wanted to rotate it with some shell script?

There are many ways to control the rotation of the cube, such as doing so with the DBus objects provided by Compiz. Besides that, is it possible to rotate the cube with wmctrl, a useful command line tool that could interact with EWMH/NetWM compatible Window Manager? Although wmctrl does not directly support cube rotation in Compiz, the answer is still "YES!"

To control the cube rotation with wmctrl, first of all, we should find out how Cube plugin actually does when managing the virtual desktops. Take my laptop for example. I use the Compiz-Fusion came with Ubuntu 7.10, and the display resolution was set to 1024x768 due to the low hardware specification of my laptop. Let's see what we get with "wmctrl -d" when we are on the first face of the cube:

$ wmctrl -d
0 * DG: 4096x768 VP: 0,0 WA: 0,25 1024x718 N/A

The "DG" in the output is Desktop Geometry, "VP" is the coordinates of the Viewport Position, and "WA" is the coordinates and the geometry of the WorkArea. The last "N/A" is the desktop name, and, since I did not give specific name to the virtual desktop, it is "Not Available." From the output of wmctrl, we can assume that, the Cube is actually a very wide virtual desktop that wrap around the cube. In order to verify our assumption, let's rotate to the right hand side, and run "wmctrl -d" in the second face of the cube:

$ wmctrl -d
0 * DG: 4096x768 VP: 1024,0 WA: 0,25 1024x718 N/A

We can find that, the only difference between these two tests is that, the X coordinate of the VP is changed! With this result, we can conlude that, the Cube is actually a very wide desktop, and the virtual desktop we see on each face is actually a viewport to that desktop. This also explains that why the application windows could be shown across the boundary of two faces. Known that each face of the cube is actually a viewport, now we can achieve the rotation with wmctrl by simply changing the current viewport position!

I wrote a simple BASH script for this:

#!/bin/bash
#
# compiz-rotate-wmctrl - Rotate the cube using wmctrl
#
# Author: Shang-Feng Yang
# Released under GPLv3

VER="1.0"


function rotate() {
  # The target face number (begins with 0)
  TVPN=$(( $1 % ${NF} ))

  # The X coordinate of the target viewport
  TVPX=$(( ${TVPN} * ${WW} ))

  # Change to the target viewport
  wmctrl -o ${TVPX},0
}

function usage() {
  echo -e "$(basename $0) v${VER}\n"
  echo -e "Usage:\n"
  echo -e "\t$(basename $0) {left|right|#}\n"
  echo -e "\tWhere:\n"
  echo -e "\t\tleft - rotate the cube to the left"
  echo -e "\t\tright - rotate the cube to the right"
  echo -e "\t\t# - rotate to #th face (begins with 0)\n\n"
  echo -e "Author: Shang-Feng Yang <storm dot sfyang at gmail dot com>"
  echo -e "Released under GPLv3"
}

# The action to be performed. $ACT could be 'left' or 'right' to rotate
# left or right, accordingly. $ACT could also be the number of the face
# to rotate into.
ACT=$(echo $1 |tr '[A-Z]' '[a-z]')

[ "x$ACT" == "x" ] && { usage; exit 1; } || {
  case $ACT in
    left|right|[0-9]|[0-9][0-9])
      ;;
    *)
      usage
      exit 1
      ;;
  esac
}


# The informations about the desktop
INFO=$(wmctrl -d)
# The width of the desktop
DW=$(echo "${INFO}"| awk '{sub(/x[0-9]+/, "", $4); print $4}')
# The width of the workarea
WW=$(echo "${INFO}"| awk '{sub(/x[0-9]+/, "", $9); print $9}')
# The number of faces on the cube
NF=$(($DW/$WW))
# The X coordinate of the viewport
CVPX=$(echo "${INFO}" |awk '{sub(/,[0-9]+/, "", $6); print $6}')
# Current number of the face in all faces (begins with 0)
CVPN=$(( ${CVPX} / ${WW} ))

[ "$ACT" == "right" ] && {
  ACT=$(( ${CVPN} + 1 ))
} || {
  [ "$ACT" == "left" ] && {
    ACT=$(( ${CVPN} - 1 ))
  }
}

rotate ${ACT}

To use the script,

• if you didn't specify the parameters, or gave wrong parameters, a short usage information would be shown:
  

  
  $ compiz-rotate-wmctrl
  compiz-rotate-wmctrl v1.0
  
  Usage:
  
  compiz-rotate-wmctrl {left|right|#}
  
  Where:
  
  left - rotate the cube to the left
  right - rotate the cube to the right
  # - rotate to #th face (begins with 0)
  
  
  Author: Shang-Feng Yang <storm dot sfyang at gmail dot com>
  Released under GPLv3
  

  
  


• or you specify "left" to achive a left hand rotation:
  

  
  $ compiz-rotate-wmctrl left
  

  
  


• or "right" for a right hand rotation:
  

  
  $ compiz-rotate-wmctrl right
  

  
  


• or given a face number, begins with 0, to rotate to the specified face:
  

  
  $ compiz-rotate-wmctrl 3
  

  
  

Where could this script be used? Well, it would be much easier to use such a kind of script to control the rotation when you use touchscreen to control the computer, or when you remotely control the desktop through network or bluetooth PAN.

Read more ...