q&fddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z mZmZmZmZmZmZmZddlmZdZdZd*d Zd Zd Zd+d Zd ZdZd,dZ dZ!dZ"dZ#GddZ$dZ%dZ&dZ'dZ(dZ)dZ*GddZ+dZ,dZ-dZ.dZ/dZ0d Z1d!Z2d"Z3ejhd#Z5d$Z6d%Z7d&Z8d'Z9d(Z:d)Z;y)-N)_) repository) bookmarksbundle2 cacheutilerror narrowspecphasespycompat requirementsscmutilstore transactionutil)nodemapcdt|}|tjz}|j||S)a determine the final set of requirement for a new stream clone this method combine the "default" requirements that a new repository would use with the constaint we get from the stream clone content. We keep local configuration choice when possible. )setrequirementsmodSTREAM_FIXED_REQUIREMENTSupdate)default_requirementsstreamed_requirementsr s 7/usr/lib/python3/dist-packages/mercurial/streamclone.pynew_stream_clone_requirementsr"s4+,LO===L-. c@|jtjz}|S)zthe set of requirement the new clone will have to support This is used for advertising the stream options and to generate the actual stream content.)r rr)reporequiredformatss rrr/s% OEEE rc|j}|j}|j}||jd}|syt |ry|j ryd}|j retj|d}t|jdg}t|jjdg}t||z}|r|sy|r|syt} |jdr#| jtjd | fS|jd} | s/|jj j#t%dyt| j'd } | |j(z } | rx|jj j#t%d d j+t-| z|jj j#t%d y| } d | fS)aWhether it is possible to perform a streaming clone as part of pull. ``bundle2`` will cause the function to consider stream clone through bundle2 and only through bundle2. Returns a tuple of (supported, requirements). ``supported`` is True if streaming clone is supported and False otherwise. ``requirements`` is a set of repo requirements from the remote, or ``None`` if stream clone isn't supported. sstream-preferred)FNFsclient)rolesstreams streamreqss=warning: stream clone requested but server has them disabled ,sGwarning: stream clone requested but client is missing requirements: %s , sQ(see https://www.mercurial-scm.org/wiki/MissingRequirement for more information) T)rremotestreamclonerequestedcapablelenheads canusebundle2 bundle2mod getrepocapsrgetremotebundle2capsbooladdrREVLOGV1_REQUIREMENTuiwarnrsplit supportedjoinsorted) pulloprrr$streamrequestedbundle2supported local_capslocal_supportedremote_supportedr streamreqs missingreqss rcanperformstreamcloner?:s ;;D ]]F11O ..)<=  4y|| ++DyA jnnY;<v77;;IrJK2B BC  )5L ~~i ==>B  ?^^M2  KKNN  "  ))$/0  4>>1  KKNN  ***VK01 2  KKNN  /  !  rcddlm}t|\}}|sy|j}|j}d}|j dr9|j 5}|jdij}ddd|jjtd|j 5}|jdij}dddj} t| } | dk(rtj"td| d k(rtj"td | d k7rtj"td |j} t%t| j'd d\} } |j+5t-||| | t/|j0||_|j3|j|j0|j4|j6_t;j<|t?j@||r|jBjE|||jGdddy#1swYxYw#1swYxYw#t$r tj td| wxYw#tt(f$r tj td| wxYw#1swYyxYw)zPossibly perform a legacy stream clone operation. Legacy stream clones are performed as part of pull but before all other operations. A legacy stream clone will not be performed if a bundle2 stream clone is supported. r localrepoNs branchmapsstreaming all changes s stream_out'unexpected response from remote server:soperation forbidden by servers$locking the remote repository failedrs%the server sent an unknown error code )$rBr?rr$r&commandexecutor callcommandresultr1statusrreadlineint ValueErrorr ResponseErrorAbortmapr3 TypeErrorlock consumev1rr resolvestorevfsoptionsfeaturessvfsoptionsrwritereporequirementsrpost_stream_cleanup _branchcachesreplace invalidate) r7rBr4r rr$ rbranchmapefplresp filecount bytecounts rmaybeperformlegacystreamclonerds3F;I|  ;;D ]]FJ ~~l#  # # % B|R8??AJ B GGNN1/01    !7Q ]]=" - 4 4 67 A 1v  qykk!<=>> kk!CDEE kk!DEFF A "3a(89 9 $Iy19     &<< GGT&&    %%d+##D)     & &tZ 8 E B B 77  !! 8 91     " !! 8 91   s=!I-0!I:* J/#J3"CK%-I7:J)J03/K"%K.ctj|jvry|jj dddsyt j |}|r|jj ddSy)z5Whether streaming clones are allowed from the server.Fsservers uncompressedT) untrustedsuncompressedallowsecret)rREPO_FEATURE_STREAM_CLONErUr1 configboolr hassecret)rsecrets rallowservergenerationrksc++4==@ 77  iD  I  d #F ww!!)-GHH rc>|jj|||S)Nphase obsolescence)rwalk)rmatcherrnros r_walkstreamfilesrrs ::??7%l? KKrchgd}j5jjdtD]a}|j D]L}|j j j}|s+j|j|f||z }Nctdddtjjdt|fzjjjfd}t||fS#1swYzxYw)aEmit content for version 1 of a streaming clone. This returns a 3-tuple of (file count, byte size, data iterator). The data iterator consists of N entries for each file being transferred. Each file entry starts as a line with the file name and integer size delimited by a null byte. The raw file data follows. Following the raw file data is the next file entry, or EOF. When used on the wire protocol, an additional line indicating protocol success will be prepended to the stream. This function is not responsible for adding it. This function will obtain a repository lock to ensure a consistent view of the store is captured. It therefore may raise LockError. r scanning Ns%d files, %d bytes to transfer c3PKD]\}}r jjd||fzdtj||fz|dd5}|dkr|j |n t j ||D]}|dddy#1swYxYww)Nssending %s (%d bytes) s%s%d srbF) auditpathilimit)r1debugr encodedirreadr filechunkiter)namesizer_chunk debugflagentriesrrVs remitrevlogdataz"generatev1..emitrevlogdata!s! $JD$ 8D$<GH!6 == =dEU3 $r5=''$-'!%!3!3Bd!C$# $  $ $ $ $ $sAB&:B B&B# B&)rRr1ryrrfiles file_sizervfsappendunencoded_path_test_sync_point_walk_1_test_sync_point_walk_2r'rVr) r total_bytesentryfrrrrrVs ` @@@r generatev1rs &GK &  m$%d+ -E[[] -KK 7 NNA$4$4i#@A9,K  - - %&D!GGMM+s7|[.II 99D!!I $ w<n&6 66C&&sA$D(>1D((D1c#Kt|sdy t|\}}}dd||fz|D]}|y#tj$rdYywxYww)aEmit content for version 1 of streaming clone suitable for the wire. This is the data output from ``generatev1()`` with 2 header lines. The first line indicates overall success. The 2nd contains the file count and byte size of payload. The success line contains "0" for success, "1" for stream generation not allowed, and "2" for error locking the repository (possibly indicating a permissions error for the server process). s1 Ns2 s0 s%d %d )rkrr LockError)rrbrcitrs rgeneratev1wireprotor3su ! & #-d#3 9b L  9- --  ?? s%A<AAAAAcdk7r tdt}djt|fd}||fS)a@Emit content for version 1 of a stream clone bundle. The first 4 bytes of the output ("HGS1") denote this as stream clone bundle version 1. The next 2 bytes indicate the compression type. Only "UN" is currently supported. The next 16 bytes are two 64-bit big endian unsigned integers indicating file count and byte count, respectively. The next 2 bytes is a 16-bit big endian unsigned short declaring the length of the requirements string, including a trailing . The following N bytes are the requirements string, which is ASCII containing a comma-delimited list of repo requirements that are needed to support the data. The remaining content is the output of ``generatev1()`` (which may be compressed in the future). Returns a tuple of (requirements, data generator). UNs.we do not support the compression argument yetr"c3Kdt\}}}jjtd||fzt j d||t j dt dzdzdk(sJjjtd|td  }|jd |D]!}|jt | |#|jyw) NsHGS1swriting %d bytes for %d files >QQ>Hrrbundlebytestotalunitrstep) rr1rJrstructpackr' makeprogressr incrementcomplete)rbrcrprogressr compressionrrequiress rgenzgeneratebundlev1..genks #-d#3 9b  0 1Y 4J J kk&)Y77kk%X!233e###77'' iL ( (   E   CJ  /K  sDD)rMrr5r6)rrr rrs`` @rgeneratebundlev1rOsJ,eJKK(.Lyy -.H6  rc |j5|jjtd|t j |fz|jj td|td}|jdt j}|jd5|jj|j|5t|D]N}t|dr|j}nIg}|r|dd k(s,|j|j!d |s#|dd k(s,d j#|} |j%d d \} } t'| } |jj0r3|jj3d| t j | fzt5j6| } |j| dd5} t j8|| D].} |j;t=| | j?| 0 dddQ ddd|jAddddt j|z }|dkrd}|jC|jjtdt j ||t j ||z fzdddy#t(t*f$r t-j.td |wxYw#1swY;xYw#1swYxYw#1swYxYw#1swYyxYw)a Apply the contents from version 1 of a streaming clone file handle. This takes the output from "stream_out" and applies it to the specified repository. Like "stream_out," the status line added by the wire protocol is not handled by this function. !%d files to transfer, %s of data clonerrr) expectedcountrK rrrrCsadding %s (%s) wT)backgroundcloserwrNclearfilecacheMbP?(transferred %s in %.1f seconds (%s/sec) )"rRr1rJrrrcrrtimerrrVbackgroundclosingrangehasattrrKrr{r5r3rLrMrQr rNrryr decodedirr|rr'writer\r)rr_rbrcrstartir`l_bufr}r~pathofprelapseds rrSrSs H   3 4$..34 5 77'' hKyq{(      h '" 1,,TWWI,N -y)-Ar:.KKM !##(U2Y%-?!LL4$)U2Y%-?HHUO%&WWUA%6 d"4y ww(( /49M2NN!??40D4tD-%)%7%7$%G-E$..CJ.?IIe,---3- -B OO4O 0E" 1H**,& a<G  : ;y)y723  CH H T' 2#11HI1--5 - -" 1" 13H H s|B#L>4(L2AL&8L&L& K'3A2L&%A L / L&;L2B L>'/L L&L#L&&L/ +L22L; 7L>>Mc|jd}|dk7r!tjtd|zt j d|jd\}}t j d|jdd}|j|}|j dstjtd t|jdjd }|||fS) NrDrs<only uncompressed stream clone bundles are supported; got %srrrrs@malformed stream clone bundle: requirements not properly encodedr") r{r rOrrunpackendswithrrstripr3)r_rrbrc requireslenrr s rreadbundle1headerrs''!*Kekk %     "===Iy--rwwqz215Kww{#H   U #kk 5   xu-33D9:L i --rcNt|rtjtdt |\}}}||j z }|r9tjtddj t|zt||||tj|y)zApply the content from a stream clone bundle version 1. We assume the 4 byte header has been read and validated and the file handle is at the 2 byte compression identifier. s2cannot apply stream clone bundle on non-empty repo4unable to apply stream clone: unsupported format: %sr#N) r'r rOrrr4r5r6rSrrY)rr_rbrcr r>s r applybundlev1rs  4ykk C D  *;2)>&Iy,/Kkk E Fjj ,- .  dB 9- %rceZdZdZdZdZy)streamcloneapplierzClass to manage applying streaming clone bundles. We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle readers to perform bundle type-specific functionality. c||_yN)_fh)selffhs r__init__zstreamcloneapplier.__init__s rc.t||jSr)rr)rrs rapplyzstreamcloneapplier.applysT488,,rN)__name__ __module__ __qualname____doc__rrrrrrs -rrsccLg}|js|jd|S)z!list snapshot file from the stores phaseroots) publishingr)rfnamess r_walkstreamfullstorefilesr's" F ??  m$ Mrcb|\}}}}|tk7r|S||||||j|fS)z actually copy the snapshot files) _filefullr5)rcopyvfsmapsrcr}ftypedatas r _filterfullr/s@"Cud  ud6#;#3#3D#9: ;;rc.eZdZdZdZdZdZdZdZy)TempCopyManageraManage temporary backup of volatile file during stream clone This should be used as a Python context, the copies will be discarded when exiting the context. A copy can be done by calling the object on the real path (encoded full path) The backup path can be retrieved using the __getitem__ protocol, obj[path]. On file without backup, it will return the unmodified path. (equivalent to `dict.get(x, x)`) c d|_d|_yr)_copies_dst_dir)rs rrzTempCopyManager.__init__Es  rc|jd}tj|i|_tjd|_|S)NzCopies context already opens hg-clone-)prefix)rr ProgrammingErrorr mkdtempr)rmsgs r __enter__zTempCopyManager.__enter__Is@ << #/C((- -  (( =  rctjj|}tj||j \}}tj |||j|<tj||d|S)z"create a backup of the file at src)rdirT)hardlink) osrbasenamer mkstemprcloserr copyfiles)rrrfddsts r__call__zTempCopyManager.__call__Qs^!!#&""&dmmDC   S sC$/ rc:|jj||S)zreturn the path to a valid version of `src` If the file has no backup, the path of the file is returned unmodified.)rr,)rrs r __getitem__zTempCopyManager.__getitem__Zs ||S))rc|jjD]}tj|tj|j d|_d|_y)zdiscard all backupsN)rvaluesr tryunlinktryrmdirr)rargskwarstmps r__exit__zTempCopyManager.__exit__asH<<&&( C NN3   dmm$  rN) rrrrrrrrrrrrrr7s  *rrct|jt|ji}|j|j vsJ|S)z$make a (src -> vfs) map for the repo) _srcstorerV _srccachecachevfsrr)rrs r_makemapr js< 4994==F 886==? ** * Mrc #Kt|}|j|jvrtjd|Dcgc] \}}||||f}}}t |}dx}}|D]4\}} }|j D]} |dz }|| j| z }6|jjtd|td} | jdt5} | 5|D]I\}} }|j D]0} | js| | j| j2K||fd} |D]\}} }|j!|| | |d}|D]\}}}|t#j$t |t#j$||d}|D]3}|t |z }| t |z } | j| |5||k7std }tj&||||fz d d d d d d y cc}}w#1swYxYw#1swYy xYww) zactually emit the stream bundle9repo.vfs must not be added to vfsmap for security reasonsrrrrrT)rrcopies max_changesetpreserve_file_counts=clone could only read %d bytes from %s, but expected %d bytesN)r rrr rr'rrr1rrrr is_volatiler5r get_streamsr uvarintencoderO)rrrvfs_keyr^ max_linkrev file_count totalfilesizekrrrrtotalbytecountr entry_streamsr}streamr~rcrrs r_emit2rws d^F  xx6==?"$$ H  BII'1!,IGId)K!""J. 3 .A !OJ Q[[- -M .. ww## ) M( $H OOA  $EdH$E  5IAsAWWY 5==!"2"234 5 5 -''" EKCaMM)$( *M'4 E"fd ((T33((..  # EU+I"c%j0NOON3K  $-C ++cYd,C&CDD% E E$E$E$EJ$E$E$E$EsPAIH(BI,H:/)H.CH.-)H.H:I.H7 3H::I?Ic # Kt|}|j|jvrtjd|Dcgc] \}}||||f}}}t |}t |}|j jtd|td}|jdt5}|5|D]Z\} } }|jD]A} | j| | js!|| j| jC\dt!j"||D]\} } }|j%|| ||} t!j"t | | D]O\}}}| t!j"t |t!j"|||Ed{Q|j' ddddddycc}}w70#1swYxYw#1swYyxYww)z$actually emit the stream bundle (v3)rrsentryrrN)rrrr)r rrr rr'r1rrrrrrrr5rrrrr)rrrrr^total_entry_countrrrrrrrrr}rr~s r_emit3r s d^F  xx6==?"$$ H  BII'1!,IGIG d)Kww## )  x[$H  OOA  !dH!  5IAsAWWY 5 C ==!"2"234 5 5    !233" !KCaMM) *M $$S%78 8&3 ""fd ((T33((.. !!!  "     !!!!JF"3!!!!s\AHG.A$H:H=:G68C G6G4 G6H%H4G66G? ;HH Hcyz+a function for synchronisation during testsNrrs rrrrcyr"rr#s rrrr$rc#K|j|jJd}|s|r!tj|j||}|j }t ||||}|D] }t|ftj|D]?}|jj|stj|d}t|fAyw)zemit a seris of files information useful to clone a repo return (vfs-key, entry) iterator Where `entry` is StoreEntry. (used even for cache entries) NrmT) entry_pathr) _currentlock_lockrefr matchrootrrrr r cachetocopyr existsrSimpleStoreEntryr ) rincludesexcludesincludeobsmarkersrqrnrrr}s r _entries_walkr2s   T]] + 77 7G8""499hA! !E & G !%  !%%d+% ==   %** Ee$ $%s B)C,$Cc|j5|jjdt||||}t ||}t |}|\}}t |dddt|fS#1swYxYw)aEmit content for version 2 of a streaming clone. the data stream consists the following entries: 1) A char representing the file destination (eg: store or cache) 2) A varint containing the length of the filename 3) A varint containing the length of file data 4) N bytes containing the filename (the internal, store-agnostic form) 5) N bytes containing the file data Returns a 3-tuple of (file count, file size, data iterator). rtr/r0r1N)rRr1ryr2rnextrr) rr/r0r1rchunksfirstrtotal_file_sizes r generatev2r9s &  m$ /  g&V &+# O%&D!  ..#&&s AA;;Bc|j5|jjdt||||}t |t |}t |}|Jt|dddt|S#1swYxYw)aEmit content for version 3 of a streaming clone. the data stream consists the following: 1) A varint E containing the number of entries (can be 0), then E entries follow 2) For each entry: 2.1) The number of files in this entry (can be 0, but typically 1 or 2) 2.2) For each file: 2.2.1) A char representing the file destination (eg: store or cache) 2.2.2) A varint N containing the length of the filename 2.2.3) A varint M containing the length of file data 2.2.4) N bytes containing the filename (the internal, store-agnostic form) 2.2.5) M bytes containing the file data Returns the data iterator. XXX This format is experimental and subject to change. Here is a XXX non-exhaustive list of things this format could do or change: - making it easier to write files in parallel - holding the lock for a shorter time - improving progress information - ways to adjust the number of expected entries/files ? rtr4N) rRr1ryr2r listr5rr)rr/r0r1rr6r7s r generatev3r<9s2  &  m$ /   d7m,V }}% &D! M! & &s ABB c'K|d}|dd}|5|rt|5ddddnddddy#1swYxYw#1swYyxYww)Nrr)nested)ctxsthisrests rr>r>esb 7D 8D         s- A  A5 A A > AA A c 0j5jjtd|t j |fzt j }jjtd|td}|jdt}j|jvrtjdjd5fd|jD}t|5t!|D]}t j"|d} || } t j$|} t j$|} t j"|| } jj&r4jj)d | | t j | fz| | d 5}t j*|| D].}|j-t/| |j1|0 d d d  d d d j3dd d d t j |z }|dkrd}jjtdt j |j4|t j |j4|z fz|j7d d d y #1swYxYw#1swYxYw#1swYxYw#1swYy xYw)zApply the contents from a version 2 streaming clone. Data is read from an object that only needs to provide a ``read(size)`` method. rrrrrrc3TK|]}|jj!ywrrr1.0rrs r zconsumev2.. NsC))$''2N%(radding [%s] %s (%s) rrwrNTrrr)rRr1rJrrrcrrrr rrr rrr>r readexactlyuvarintdecodestreamrryr|rr'rr\posr)rr_rbfilesizerrrr?rrrnamelendatalenr}rrrs` r consumev2rQqs <  3 4$..23 4  77'' hKxak(  $ 88v}} &((L   h ' 1NfmmoND -y)-A**2q1C +C"66r:G"66r:G++B8Dww(( 4"D$..*ABC T4-C%)%7%7'%J-E$..CJ.?IIe,---- -, OO4O 01 14**,& a<G  : ;x||,x||g56   y<<P-- - - 1 1/<<sWCL 1"LB>K4A K'  K4'LBL 'K1,K44K= 9LL L  Lc j5tj}tj|}jj t d|zjjt d|t d}|jdd}t}j|jvrtjdjd5fd|jD}t|5t!|D]a}tj|} | dk(r4jj"rjj%d|zt!| D]}tj&|d } || } tj|} tj|} tj&|| }jj"r8d }|| |tj(| fz}jj%||| z }| |d 5}tj*|| D]}|j-| d d d |j/d d d d d j1dd d d tj|z }|dkrd}t d}tj(|}tj(||z }||||fz}jj ||j3d d d y #1swYxYw#1swYxYw#1swYxYw#1swYy xYw)zApply the contents from a version 3 streaming clone. Data is read from an object that only needs to provide a ``read(size)`` method. s%d entries to transfer rsentriesrrrc3TK|]}|jj!ywrrDrEs rrGzconsumev3..rHrIsentry with no files [%d] rrJrrwNrTrrr)rRrrrLr1rJrrrr rrr rrr>rrryrKrcr|rrr\r)rr_r entrycountrbytes_transferredrr?rrbrrrOrPr}rrrr byte_count bytes_secs` r consumev3rXs A --b1  q45DE77'' hK:(  $ 88v}} &((L   h ' 1NfmmoND /z*/A $ 8 8 "&":":2">#//G<77,,":CCt~~g/F#GGC GGMM#.)W4) t_1)-););Bg)N1 # % 01111"&&A&.-/ /8 OO4O 0= 1@**,& a<G<=^^$56 NN#4w#>?  GY// sCAA`11+ / / 1 13AAsVC!M 3"MD#M8.L; 'MM!BM ;MMM MM M  M)cddlm}|Dcgc]}||jvs|}}|r9tjt ddj t|zt||||t|j||_ |j|j|j|j|j_t!j"|t%j&|ycc}wNrrArr#)rFrBr4r rOrr5r6rQrr rTr1rUrVrWrrXrrY)rr_rbrNr rBrr>s r applybundlev2r\s*Fat~~.E1FKFkk E Fjj ,- .  dB 8,5 D"88 ""DMMDII !!$' %#Gs C4C4cddlm}|Dcgc]}||jvs|}}|r=td}|dj t |z}t j|t||t|j||_ |j|j|j|j|j_t!j"|t%j&|ycc}wrZ)rFrBr4rr5r6r rOrXrr rTr1rUrVrWrrXrrY)rr_r rBr[r>rs r applybundlev3r^s*Fat~~.E1FKFGH uzz&-..kk# dB5 D"88 ""DMMDII !!$' %!Gs C6C6c dg fd}|D]\}}||}||}|j|} |j|} tjj| } tjj | st j | |j|t j| | d|dj dS)NTc0dd<td_y)NFrscopying)rtopic)rrsr copy_usedz_copy_files..copy_used0s :rrF)rno_hardlink_cbcheck_fs_hardlink) r5rrdirnamer-rmakedirs register_filecopyfiler) src_vfs_map dst_vfs_maprrrbrrsrc_vfsdst_vfssrc_pathdst_pathrers ` @r _copy_filesro-svH'4a.a.<<%<<%''//(+ww~~g& MM' "d#  a[$#   '( A;rc J|jDchc]}|tjvr|}}|jDchc]}|tjvr|}}||k(sJ|j5|j5t j |}|j d}d}tjj|rd}t|ddd}t|}t|} t|} td|D|z} |jjt!d| t!d  } |D cgc](\} }|j#D]}| |j$f*}}} }t'| | || }|r.psAAc!A$**,/As&(slinkingsfiles)rarrslinked %d files scopied %d files s localclone)r rWORKING_DIR_REQUIREMENTSrRr bookmarksvfsr5rrr-r2r;r sumr1rrrrrorrhrryrrrcleanup_undo_filesr2vfs_map)src_repo dest_repor[src_store_requirementsdest_store_requirements src_book_vfs srcbookmarksbm_countrri dest_vfs_map total_filesrrr^rrr dst_book_vfs dstbookmarksrtrs r local_copyrKs && O<< < '' O<< <  "%< << <  9M ]]_, : %11(;L',,\:LHww~~l+#"& G 7mG"8,K#I.LAAAHLK{{// m!x[0H#*GQ!**++E #; eXNH(55i@ +00>  lL9Y, :Z  &C&C # +,  " "= 1 &R OO ! !" % & &&y||'8'8):K:KLs9M9M RC, :, :h & &k9M9Ms[I0I5.J?CJ-I: 1AJAJJ +rQrXr\r^rorrrrrs "     [|HV$L77t87tQ h.8&. - -    <00f @EF2!j66%B/B)X BJGT&.&,