datasketch: Big Data Looks Small

datasketch gives you probabilistic data structures that can process and search very large amount of data super fast, with little loss of accuracy.

This package contains the following data sketches:

Data Sketch	Usage
MinHash	estimate Jaccard similarity and cardinality
Weighted MinHash	estimate weighted Jaccard similarity
HyperLogLog	estimate cardinality
HyperLogLog++	estimate cardinality

The following indexes for data sketches are provided to support sub-linear query time:

Index	For Data Sketch	Supported Query Type
MinHash LSH	MinHash, Weighted MinHash	Jaccard Threshold
LSHBloom	MinHash, Weighted MinHash	Jaccard Threshold
MinHash LSH Forest	MinHash, Weighted MinHash	Jaccard Top-K
MinHash LSH Ensemble	MinHash	Containment Threshold
HNSW	Any	Custom Metric Top-K

datasketch must be used with Python 3.9 or above, NumPy 1.11 or above, and Scipy.

Note that MinHash LSH and MinHash LSH Ensemble also support Redis and Cassandra storage layer (see MinHash LSH at Scale).

Install

To install datasketch using pip:

pip install datasketch

This will also install NumPy as dependency.

To install with Redis dependency:

pip install datasketch[redis]

To install with Cassandra dependency:

pip install datasketch[cassandra]

To install with Bloom filter dependency:

pip install datasketch[bloom]

Contributing

We welcome contributions from everyone. Whether you’re fixing bugs, adding features, improving documentation, or helping with tests, your contributions are valuable.

Development Setup

The project uses uv for fast and reliable Python package management. Follow these steps to set up your development environment:

1. Install uv: Follow the official installation guide at https://docs.astral.sh/uv/getting-started/installation/

2. Clone the repository:

   git clone https://github.com/ekzhu/datasketch.git
   cd datasketch
   

3. Set up the environment:

   # Create a virtual environment
   # (Optional: specify Python version with --python 3.x)
   uv venv
   # Activate the virtual environment (optional, uv run commands work without it)
   source .venv/bin/activate
   
   # Install all dependencies
   uv sync
   

4. Verify installation:

   # Run tests to ensure everything works
   uv run pytest
   

5. Optional dependencies (for specific development needs):

   # For testing
   uv sync --extra test
   
   # For Cassandra support
   uv sync --extra cassandra
   
   # For Redis support
   uv sync --extra redis
   
   # For all extras
   uv sync --all-extras
   

Learn more about uv at https://docs.astral.sh/uv/

Development Workflow

1. Fork the repository on GitHub if you haven’t already.

2. Create a feature branch for your changes:

   git checkout -b feature/your-feature-name
   # Or for bug fixes:
   git checkout -b fix/issue-description
   

3. Make your changes following the project’s coding standards.

4. Run the tests to ensure nothing is broken:

   uv run pytest
   

5. Check code quality with ruff:

   # Check for issues
   uvx ruff check .
   
   # Auto-fix formatting issues
   uvx ruff format .
   

6. Commit your changes with a clear, descriptive commit message:

   git commit -m "Add feature: brief description of what was changed"
   

7. Push to your fork and create a pull request on GitHub:

   git push origin your-branch-name
   

8. Respond to feedback from maintainers and iterate on your changes.

Guidelines

• Follow PEP 8 style guidelines

• Write tests for new features

• Update documentation as needed

• Keep commits focused and atomic

• Be respectful in discussions

For more information, check the GitHub issues for current priorities or areas needing help. You can also join the discussion on project roadmap and priorities.