datasketch: Big Data Looks Small¶
datasketch gives you probabilistic data structures that can process and search very large amount of data super fast, with little loss of accuracy.
This package contains the following data sketches:
Data Sketch 
Usage 

estimate Jaccard similarity and cardinality 

estimate weighted Jaccard similarity 

estimate cardinality 

estimate cardinality 
The following indexes for data sketches are provided to support sublinear query time:
Index 
For Data Sketch 
Supported Query Type 

MinHash, Weighted MinHash 
Jaccard Threshold 

MinHash, Weighted MinHash 
Jaccard TopK 

MinHash 
Containment Threshold 
datasketch must be used with Python 2.7 or above and NumPy 1.11 or above. Scipy is optional, but with it the LSH initialization can be much faster.
Note that MinHash LSH and MinHash LSH Ensemble also support Redis and Cassandra storage layer (see MinHash LSH at Scale).
Install¶
To install datasketch using pip
:
pip install datasketch
This will also install NumPy as dependency.
To install with Redis dependency:
pip install datasketch[redis]
To install with Cassandra dependency:
pip install datasketch[cassandra]
To install with Scipy for faster MinHashLSH initialization:
pip install datasketch[scipy]