Go efficient text segmentation; support english, chinese, japanese and other.
Dictionary with double array trie (Double-Array Trie) to achieve, Sender algorithm is the shortest path
based on word frequency plus dynamic programming.
Support common and search engine two participle mode, support user dictionary, POS tagging, run JSON RPC service.
Text Segmentation speed single thread 9MB/s，goroutines concurrent 42MB/s (8 nuclear Macbook
Pro).
Install / update
go get -u github.com/go-ego/gse
复制代码Build-tools
go get -u github.com/go-ego/re
复制代码re gse
To create a new gse application
$ re gse my-gse
复制代码re run
To run the application we just created, you can navigate to the application folder and execute:
$ cd my-gse && re run
复制代码Use
package main
import (
"fmt"
"github.com/go-ego/gse"
)
func main() {
// Load the dictionary
var segmenter gse.Segmenter
segmenter.LoadDict()
// segmenter.LoadDict("your gopath"+"/src/github.com/go-ego/gse/data/dict/dictionary.txt")
// Text Segmentation
text := []byte("你好世界, Hello world.")
segments := segmenter.Segment(text)
// Handle word segmentation results
// Support for normal mode and search mode two participle, see the comments in the code ToString function.
fmt.Println(gse.ToString(segments, false))
}复制代码License
Gse is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), base on sego
近期评论