Faiss - 部署安装与Demo
2019-09-26 本文已影响0人
红薯爱帅
1. 概述
Faiss is a library for efficient similarity search and clustering of dense vectors.
It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.
It also contains supporting code for evaluation and parameter tuning.
Faiss is written in C++ with complete wrappers for Python/numpy. Some of the most useful algorithms are implemented on the GPU.
It is developed by Facebook AI Research.
- github repository
https://github.com/facebookresearch/faiss
2. Setup
2.1. 通过conda安装faiss，简单方便
- dockerfile已经ok，可以尝试下
ARG IMAGE
FROM ${IMAGE}
ARG FAISS_CPU_OR_GPU
ARG FAISS_VERSION
RUN apt-get update && \
apt-get install -y curl bzip2 && \
curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh > /tmp/conda.sh && \
bash /tmp/conda.sh -b -p /opt/conda && \
/opt/conda/bin/conda update -n base conda && \
/opt/conda/bin/conda install -y python=3.6.9 && \
/opt/conda/bin/conda install -y -c pytorch faiss-${FAISS_CPU_OR_GPU}=${FAISS_VERSION} && \
apt-get remove -y --auto-remove curl bzip2 && \
apt-get clean && \
rm -fr /tmp/conda.sh
ENV PATH="/opt/conda/bin:${PATH}"
how to build a faiss-docker image
- references
https://anaconda.org/pytorch/faiss-cpu
https://anaconda.org/pytorch/faiss-gpu
https://github.com/plippe/faiss-docker
2.2. 源码安装
- 测试没过，卡在cuda上，有机会再试试
RUN apt-get update -y && apt-get install -y libopenblas-dev python-numpy python-dev swig python-pip curl
ENV BLASLDFLAGS=/usr/lib/libopenblas.so.0 \
PYTHON=python
RUN cd /opt \
&& git clone https://github.com/facebookresearch/faiss.git \
&& cd faiss && git checkout v1.3.0 \
&& pip3 install matplotlib==2.2.3 python-config numpy -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com \
&& mv example_makefiles/makefile.inc.Linux ./makefile.inc \
&& ./configure --with-python=python3 \
&& make -j $(nproc) \
&& make install \
&& make -C gpu -j $(nproc) \
&& make -C gpu/test \
&& make -C python gpu \
&& make -C python build \
&& make -C python install
3. How to use
3.1. build a index
dimensions = 128
INDEX_KEY = "IDMap,Flat"
index = faiss.index_factory(dimensions, INDEX_KEY)
3.2. is GPU used
if USE_GPU:
res = faiss.StandardGpuResources()
index = faiss.index_cpu_to_gpu(res, 0, index)
3.3. add features to index with ids
index.add_with_ids(features, ids)
3.4. search
scores, neighbors = index.search(siftfeature, k=topN)
3.5. save index and reload
faiss.write_index(index, "large.index")
index1 = faiss.read_index("large.index")
