Je9dZddlZddlZddlmZmZmZddlmZm Z m Z  ddl m Z m Z mZmZdZGd d eZGd d eZed k(rddlZej.yy#e$r edwxYw)a This module implements a bloom filter probabilistic data structure and an a Scalable Bloom Filter that grows in size as your add more items to it without increasing the false positive error_rate. Requires the bitarray library: http://pypi.python.org/pypi/bitarray/ N)range_fn is_string_iorunning_python_3)unpackpackcalcsize) QBitArrayQFile QDataStream QIODevicez&pybloom_pyqt requires QtCore.QBitArrayc dk\rd\}}ndk\rd\}}nd\}}dz|z}|dkDrtj nR|dkDrtj n<|d kDrtj n&|d kDrtj ntj | j |zztt\}}|r|d z }t fd td |D  fd}| fS)Nl)Qi)I)Hrirc 3jK|]*}td|j,yw)rN)rdigest).0ihashfns L/usr/lib/python3/dist-packages/spyder/utils/external/pybloom_pyqt/pybloom.py z!make_hashfuncs..)s23S! -44673s03rc3Ktr=t|tr|jd}nHt|jd}n-t|tr|jd}n t|}d}D]V}|j }|j |t|jD]}|z|dz }|k\syXyw)Nzutf-8rr) r isinstancestrencodeunicodecopyupdaterr) keyrsalthuintfmtnum_bits num_slicessaltss r _hash_makerz#make_hashfuncs.._hash_maker,s #s#jj)#hoog.#w'jj)#h  D A HHSMsAHHJ/ Xo%Q ?   s CC C) hashlibsha512sha384sha256sha1md5 digest_sizedivmodlentupler) r,r+fmt_code chunk_sizetotal_hash_bits num_saltsextrar.r*rr-s `` @@@rmake_hashfuncsr>sG%* g %*%**nz1O 3  3  3  fh**j8 9Cj#c(3Iu Q 3#Ay13 3E*  creZdZdZddZdZdZdZddZdZ dZ d Z d Z d Z d Zed ZdZdZy) BloomFilters 0g?rN) ValueErrorintmathceillogabs_setupr r+bitarray)selfcapacity error_rater,bits_per_slices r__init__zBloomFilter.__init__GsJ""BC C#BC C!|34 4488C*, make_hashesr)rKrMr,rNrLrRs rrIzBloomFilter._setupdsV$$,  "^3  (6t7;7J7J)L%$+r?c|j}|j}|j|}d}|D]}|||zsy||z }y)7Tests a key's membership in this bloom filter. rFT)rNrJrS)rKr&rNrJhashesoffsetks r __contains__zBloomFilter.__contains__nsZ,,==!!#& %AFQJ' n $F %r?c|jS)z6Return the number of keys stored by this bloom filter.rRrKs r__len__zBloomFilter.__len__{s zzr?cv|j}|j}|j|}d}|j|jkDr t dd}|D]3}|s |r |||zsd}|jj ||z||z }5|r|xjdz c_y|s|xjdz c_yy)z Adds a key to this bloom filter. If the key already exists in this filter it will return True. Otherwise False. TzBloomFilter is at capacityrFr)rJrNrSrRrL IndexErrorsetBit) rKr& skip_checkrJrNrVfound_all_bitsrWrXs raddzBloomFilter.adds==,,!!#& :: %9: : %A.&1*9M!& MM ! , n $F  %  JJ!OJ JJ!OJr?czt|j|j}t|j|_|S)z,Return a copy of this bloom filter. )rArLrMr rJ)rK new_filters rr$zBloomFilter.copys/!@ ' 6 r?c|j|jk7s|j|jk7r td|j}|j|jz|_|S)zd Calculates the union of the two underlying bitarrays and returns a new bloom filter object.zTUnioning filters requires both filters to have both the same capacity and error raterLrMrCr$rJrKother new_blooms runionzBloomFilter.unionsc ==ENN *5#3#33EF FIIK &//%..@ r?c$|j|SrQrkrKris r__or__zBloomFilter.__or__zz%  r?c|j|jk7s|j|jk7r td|j}|j|jz|_|S)zk Calculates the intersection of the two underlying bitarrays and returns a new bloom filter object.zPIntersecting filters requires both filters to have equal capacity and error ratergrhs r intersectionzBloomFilter.intersectionsc ==ENN *5#3#33BC CIIK &//%..@ r?c$|j|SrQ)rrrns r__and__zBloomFilter.__and__s  ''r?ct|}|jtjrt |}|j |j |j|j|j|j|j|j|j|j|j|j||jz|j|j!yy)zWrite the bloom filter to file object `f'. Underlying bits are written as machine values. This is much more space efficient than pickling the object.N)r openr WriteOnlyr writeBytesFILE_FMT writeFloatrMwriteIntr,rNrLrRrJflushclose)rKpathfouts rtofilezBloomFilter.tofiles $K 66)%% &a.C NN4== ) NN4?? + LL ) LL,, - LL ' LL $ 4== GGI GGI 'r?c t|}|jtjst d|zt |}|j }||jk7r t d|j}|j}|j}|j}|j} t} |d} | j||||| t| _ || jz | S)zYRead a bloom filter from file-object `f' serialized with ``BloomFilter.tofile''. zunable to open file zunexpected file formatr) r rvr ReadOnlyrCr readBytesry readFloatreadIntr rIrJ) clsr~rdatafile_fmtrMr,rNrLrRrJfilters rfromfilezBloomFilter.fromfiles $Kvvi(()3d:; ;1~>># s|| #56 6^^% \\^ <<> ;Q j*nhN#+  r?c@|jj}|d=|S)NrS)__dict__r$rKds r __getstate__zBloomFilter.__getstate__s! MM    m r?c|jj|t|j|j\|_|_yrQ)rr%r>r,rNrSrrs r __setstate__zBloomFilter.__setstate__s8 Q(6t7;7J7J)L%$+r?N)MbP?)F)__name__ __module__ __qualname__ryrOrIrYr]rcr$rkrorrrtr classmethodrrrr?rrArADs^H1:L 4 ! ((4 Lr?rAceZdZdZdZdZddefdZdZdZd Z d Z d Z d Z e d Ze dZdZedZdZy)ScalableBloomFilterrrz.as4!1::4sumrr\s rrLzScalableBloomFilter.capacity^s4t||444r?ct|SrQ)r7r\s rrRzScalableBloomFilter.countcs 4yr?c |jt|j|j|j|j |j |jtdt|jt|jdkDr|j}ddt|jzz}|jdt|zg}|jD]E}|j}|j||j|j|z G|j||jt|g|yy)zDSerialize this ScalableBloomFilter into the file-object `f'. UC -./ t|| q Its4<<'899I GGD8I.. /L,, 6 a ##AFFHu$45 6 FF9  GGD2\2 3 !r?c |}|jt|j|jt |jtd|jt d\}|dkDrfdd|zz}|jt |}t||}|D]1}|j j tj||3|Sg|_|S)z7Deserialize the ScalableBloomFilter in file object `f'.rrrr) rIrryreadr rrrAr)rrrnfilters header_fmtbytesfilter_lengthsfls rrzScalableBloomFilter.fromfiles vcllAFF8CLL3I,JKL5!&&%"9:  a<x/JFF8J/0E#J6N$ C%%k&:&:1b&AB C   FN r?c:td|jDS)z7Returns the total number of elements stored in this SBFc34K|]}|jywrQr[rs rrz.ScalableBloomFilter.__len__..s1q1771rrr\s rr]zScalableBloomFilter.__len__s1DLL111r?N)rrrSMALL_SET_GROWTHLARGE_SET_GROWTHryrOrIrYrcr$rkropropertyrLrRrrrr]rr?rrrsH(+&.% , 4!5540 2r?r__main__)__doc__rEr/utilsrrrstructrrr qtpy.QtCorer r r r ImportErrorr>objectrArrdoctesttestmodrr?rrs  ;;))@DD /doL&oLd\2&\2~ zGOOO @ > ??@s A!!A/