Projects

Overview While Go provides goroutines, channels and selects as first-class citizens to support concurrent programming, it is not trivial to combine these elements to address important concerns of goroutine orchestration, e.g. error handling, panic recovery, goroutine leak prevention, goroutine reuse, goroutine throttle and logging. The package provides a mini-framework to address those cross-cutting concerns. Quick start go get -u h12.io/run Here is an example illustrating the usage of the goroutine pool and the group. The task is described in the “Google Search 2.0” page from this slide. ...

A BUID is a 128-bit unique ID composed of two 64-bit parts: shard and key. It is not only a unique ID, but also contains the sharding information, so that the messages with the same BUID could be stored together within the same DB shard. Also, when a message is stored in a shard, the shard part of the BUID can be trimmed off to save the space, and only the key part needs to be stored as the primary key. ...

Design The UA detection algorithm is a pipeline containing 3 steps: Scan: scan products and its corresponding comments from a UA string. Parse: parse device info from products. Detect: detect device model by looking up in a database and get related info, e.g. screen resolution. Scan The scanner scans the user agent string into products. Each product has its own name, version and comments. e.g. For user agent string: Mozilla/5.0 (Linux; U; Android ROM v3; en-us; ALCATEL ONE TOUCH 991 Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Its products are: ...

h12.io/sej provides composable components of distributed, persisted message queue and allows trading off between reliablilty, latency and throughput with minimal devops overhead. Package Organization h12.io/sej: writer, scanner and offset shard: sharding hub: copying across machines cmd/sej: command line tool SEJ Directory [root-dir]/ [sej-dir]/ jnl.lck jnl/ 0000000000000000.jnl 000000001f9e521e.jnl ...... ofs/ reader1.ofs reader1.lck reader2.ofs reader2.lck ...... Journal File format segment_file = { message } . message = offset timestamp type key value size . offset = uint64 . timestamp = int64 . type = uint8 . key = key_size { uint8 } . key_size = int8 . value = value_size { uint8 } . value_size = int32 . size = int32 . All integers are written in the big endian format. ...

h12.io/realtest provides real test environment for Go unit testing, based on Docker. Includes: Queue Kafka cluster (with zookeeper) Database: MySQL MongoDB Cache Redis Configuration service ZooKeeper

json-schema generates Go struct from a JSON Schema specification. Decimal types are supported for accurate currency calculation.

Aliyun OSS (Object Storage Service) Go SDK is a client SDK to access Aliyun OSS API, implemented in the Go programming language. Installation go get -u github.com/aliyun/aliyun-oss-go-sdk/oss go test -v -cover github.com/aliyun/aliyun-oss-go-sdk/oss Highlights Complete set of Aliyun OSS API Thouroughly tested 100% test coverage intuitive table driven tests full test suite completes within 2 seconds Lint clean golint go fmt goimports go vet race detector Idiomatic & elegant response is returned as a parsed object error is returned as a Go error named options for setting headers & parameters Great extensibility clean and orthogonal implementation users can easily extend the SDK when a new API method is supported No third party dependencies Documentation Overview API Object Bucket Object Optional Headers and Parameters Multipart Upload Cross-Origin Resource Sharing (CORS) Object Lifecycle Management Extending the SDK Differences with Python SDK HTTP header User-Agent, e.g. aliyun-sdk-go/0.1.1 (Linux/3.16.0-51-generic/x86_64;go1.5.1) Go HTTP client does not support 100-Continue (will be supported after Go 1.6, see https://github.com/golang/go/issues/3665) HTTP header keys are automatically converted into canonical format, e.g. x-oss-acl becomes X-Oss-Acl Go GET request does not have redundant “Content-Length: 0” header Parameters will be omitted if the argument is not set Go always sends URL parameters and headers in canonical order License licensed under the Apache License 2.0 ...

Install go get -u h12.io/kpax Talk I gave a talk about Kpax at Gopher Meetup Shanghai 2016. See the slides. Design The client is built on top of Kafka Wire Protocol (i.e. low-level API). The protocol related types & marshal/unmarshal functions are automatically generated by wipro from the HTML spec). (-) means to be done. Sub packages model is an abstraction model for request, response, broker and cluster broker is a lazy, asynchronous and recoverable round tripper that talks to a single Kafka broker cluster is a metadata manager that talks to a Kafka cluster proto contains both low level API and a “middle” level facade producer: fault tolerant high-level producer (batching and partitioning strategy) consumer: fault tolerant high-level consumer (consumer group and offset commit) log: replaceable global logger cmd kpax: command line tool to help with Kafka programming Compatibility Compatible with Kafka Server 0.8.2. ...

Database scheme extractor & code generator in Go. Supported database: MySQL SQLite

Gombi is a combinator-style scanner & parser library written in Go. It is practical, reasonably fast and extremely easy to use. Quick start go get -u h12.io/gombi Design Combinator parsers are straightforward to construct, modular and easily maintainable, compared to parser generators like Lex/Yacc. Internal DSL no additional code generation and compilation. Composable a subset of the syntax tree is also a parser. a language can be easily embedded into another one. Gombi is inspired by but not limited to parser combinators. Unlike a combinator parser, Gombi neither limits its API to functional only, nor limits its implementation to functional combinators. Go is not a pure functional language as Haskell, so cloning a combinator parser like Parsec to Go will only lead to an implementaion much worse than Parsec. Instead, Gombi is free to choose any Go language structures that are suitable for a modular and convenient API, and any algorithms that can be efficiently implemented in Go. ...