Golang

Overview While Go provides goroutines, channels and selects as first-class citizens to support concurrent programming, it is not trivial to combine these elements to address important concerns of goroutine orchestration, e.g. error handling, panic recovery, goroutine leak prevention, goroutine reuse, goroutine throttle and logging. The package provides a mini-framework to address those cross-cutting concerns. Quick start go get -u h12.io/run Here is an example illustrating the usage of the goroutine pool and the group. The task is described in the “Google Search 2.0” page from this slide. ...

Server side pagination is intrinsically not accurate, as long as the data is dynamic. The data items could be inserted, deleted or changed on the server side while the user goes forward and backward among the pages. However, there is an algorithm that can keep the pagination as stable as possible: encode the id and sorting fields of last value in a page as the continue-token return the continue-token along with each page the client must pass the continue-token to fetch the next page the next page starts with value > continue-token || (value == continue-token && value.id > continue-token.id) Reference Pagination with Dynamic Data Loading

A BUID is a 128-bit unique ID composed of two 64-bit parts: shard and key. It is not only a unique ID, but also contains the sharding information, so that the messages with the same BUID could be stored together within the same DB shard. Also, when a message is stored in a shard, the shard part of the BUID can be trimmed off to save the space, and only the key part needs to be stored as the primary key. ...

Design The UA detection algorithm is a pipeline containing 3 steps: Scan: scan products and its corresponding comments from a UA string. Parse: parse device info from products. Detect: detect device model by looking up in a database and get related info, e.g. screen resolution. Scan The scanner scans the user agent string into products. Each product has its own name, version and comments. e.g. For user agent string: Mozilla/5.0 (Linux; U; Android ROM v3; en-us; ALCATEL ONE TOUCH 991 Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Its products are: ...

Turn on “2-Step Verification” so that an “App password” can be generated go get gopkg.in/gomail.v2 Send your Email like the example below: package main import ( "gopkg.in/gomail.v2" ) func main() { m := gomail.NewMessage() m.SetHeader("From", "[email protected]") m.SetAddressHeader("To", "to_adress@xxx", "to_name") m.SetAddressHeader("Cc", "cc_adress@xxx", "cc_name") m.SetHeader("Subject", "Hello! TEST!") m.SetBody("text/html", "Hello! <b>TEST</b>!") d := gomail.NewPlainDialer("smtp.gmail.com", 587, "[email protected]", "the App password") if err := d.DialAndSend(m); err != nil { panic(err) } }

h12.io/sej provides composable components of distributed, persisted message queue and allows trading off between reliablilty, latency and throughput with minimal devops overhead. Package Organization h12.io/sej: writer, scanner and offset shard: sharding hub: copying across machines cmd/sej: command line tool SEJ Directory [root-dir]/ [sej-dir]/ jnl.lck jnl/ 0000000000000000.jnl 000000001f9e521e.jnl ...... ofs/ reader1.ofs reader1.lck reader2.ofs reader2.lck ...... Journal File format segment_file = { message } . message = offset timestamp type key value size . offset = uint64 . timestamp = int64 . type = uint8 . key = key_size { uint8 } . key_size = int8 . value = value_size { uint8 } . value_size = int32 . size = int32 . All integers are written in the big endian format. ...

h12.io/realtest provides real test environment for Go unit testing, based on Docker. Includes: Queue Kafka cluster (with zookeeper) Database: MySQL MongoDB Cache Redis Configuration service ZooKeeper

json-schema generates Go struct from a JSON Schema specification. Decimal types are supported for accurate currency calculation.

Aliyun OSS (Object Storage Service) Go SDK is a client SDK to access Aliyun OSS API, implemented in the Go programming language. Installation go get -u github.com/aliyun/aliyun-oss-go-sdk/oss go test -v -cover github.com/aliyun/aliyun-oss-go-sdk/oss Highlights Complete set of Aliyun OSS API Thouroughly tested 100% test coverage intuitive table driven tests full test suite completes within 2 seconds Lint clean golint go fmt goimports go vet race detector Idiomatic & elegant response is returned as a parsed object error is returned as a Go error named options for setting headers & parameters Great extensibility clean and orthogonal implementation users can easily extend the SDK when a new API method is supported No third party dependencies Documentation Overview API Object Bucket Object Optional Headers and Parameters Multipart Upload Cross-Origin Resource Sharing (CORS) Object Lifecycle Management Extending the SDK Differences with Python SDK HTTP header User-Agent, e.g. aliyun-sdk-go/0.1.1 (Linux/3.16.0-51-generic/x86_64;go1.5.1) Go HTTP client does not support 100-Continue (will be supported after Go 1.6, see https://github.com/golang/go/issues/3665) HTTP header keys are automatically converted into canonical format, e.g. x-oss-acl becomes X-Oss-Acl Go GET request does not have redundant “Content-Length: 0” header Parameters will be omitted if the argument is not set Go always sends URL parameters and headers in canonical order License licensed under the Apache License 2.0 ...

Install go get -u h12.io/kpax Talk I gave a talk about Kpax at Gopher Meetup Shanghai 2016. See the slides. Design The client is built on top of Kafka Wire Protocol (i.e. low-level API). The protocol related types & marshal/unmarshal functions are automatically generated by wipro from the HTML spec). (-) means to be done. Sub packages model is an abstraction model for request, response, broker and cluster broker is a lazy, asynchronous and recoverable round tripper that talks to a single Kafka broker cluster is a metadata manager that talks to a Kafka cluster proto contains both low level API and a “middle” level facade producer: fault tolerant high-level producer (batching and partitioning strategy) consumer: fault tolerant high-level consumer (consumer group and offset commit) log: replaceable global logger cmd kpax: command line tool to help with Kafka programming Compatibility Compatible with Kafka Server 0.8.2. ...