@h12.io

A Single Command to Download a Website

9 April 2017
wget --recursive --level=2 --no-parent --no-clobber --convert-links --continue
[URL]

How to Remove Old Kernels in Ubuntu

10 September 2016
sudo apt install byobu
sudo purge-old-kernels

Reference:

Making Code Block Wrap in Jira

5 September 2016

It is a hack.

Set Jira Administration -> System -> Announcement Banner -> Announcement:

<style type="text/css">
pre {
    white-space: pre-wrap!important;
}
</style>

One-liner to check MongoDB Connections

2 September 2016
mongo --eval "JSON.stringify(db.currentOp(true))" | \
tail -n +3 | \
jq -r .inprog[].client | \
sed 's/\(.*\):.*/\1/' \
| sort | uniq -c | sort -nr
  • dump connection information in standard JSON format
  • remove MongoDB header
  • extract IP:port as a list
  • trim port
  • sort IPs by freqencies

Counting Word Frequencies in One Line

9 August 2016
cat book.txt | \
tr '!()[]{};:",<.>?“”‘’*/\r' ' ' | \
tr ' ' '\n' | \
grep -a -P "^[\p{L}\p{N}\-']+\$" | \
grep -a -P -v "^[\p{N}\-']+\$" | \
sed "s/'s\$//" | \
sed "s/^'//" | sed "s/'\$//" > words.txt

cat words.txt | \
sort | uniq -c | \
sort -nr | \
cut -c9- > words_desc.txt
  • replace punctuations with space; remove \r from `\r\n’
  • one word per line
  • keep only words composed of unicode letters, numbers, hyphen and apostrophe
  • remove pure numbers
  • remove ’s
  • remove starting and ending apostrophe
  • output words.txt

  • sort and count unique words

  • sort by freqency in descending order

  • trim the frequency column

A Note on Mobi Format

3 August 2016

ua: User-Agent parser in Go

18 July 2016

Design

The UA detection algorithm is a pipeline containing 3 steps:

  1. Scan: scan products and its corresponding comments from a UA string.
  2. Parse: parse device info from products.
  3. Detect: detect device model by looking up in a database and get related info, e.g. screen resolution.

Scan

The scanner scans the user agent string into products. Each product has its own name, version and comments. e.g.

For user agent string:

Mozilla/5.0 (Linux; U; Android ROM v3; en-us; ALCATEL ONE TOUCH 991 Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

Its products are:

  • Mozilla/5.0
    • Linux
    • U
    • Android ROM v3
    • en-us
    • ALCATEL ONE TOUCH 991 Build/GRK39F
  • AppleWebKit/533.1
    • KHTML, like Gecko
  • Version/4.0
  • Mobile
  • Safari/533.1

Parse

The parser then parses the scanned products and fill the information into a Device struct, e.g.

Mozilla:  5.0
Linux:    true
Security: U
Android:  ROM v3
Locale:   en-us
Model:    ALCATEL ONE TOUCH 991
Build:    GRK39F
Webkit:   533.1
Version:  4.0
Mobile:   true
Safari:   533.1

Detect

The detector is then able to use the parsed Device structure to detect the device type & related info: vendor, screen resolution, tablet-or-not, etc, by searching in a device database.

Design Goals

Accuracy

The accuracy of the detection is determined by the completeness of the device database, then we might have to migrate the data from multiple sources. Automatic tests will be adapted from those resources.

References

Detectors

Parsers