}e'dZdZddlmZmZmZmZmZmZm Z ddlZddl Z ddl m Z m Z dZGddeZGd d eZy) a This module contains utility function and classes to inject simple ast transformations based on code strings into IPython. While it is already possible with ast-transformers it is not easy to directly manipulate ast. IPython has pre-code and post-code hooks, but are ran from within the IPython machinery so may be inappropriate, for example for performance mesurement. This module give you tools to simplify this, and expose 2 classes: - `ReplaceCodeTransformer` which is a simple ast transformer based on code template, and for advance case: - `Mangler` which is a simple ast transformer that mangle names in the ast. Example, let's try to make a simple version of the ``timeit`` magic, that run a code snippet 10 times and print the average time taken. Basically we want to run : .. code-block:: python from time import perf_counter now = perf_counter() for i in range(10): __code__ # our code print(f"Time taken: {(perf_counter() - now)/10}") __ret__ # the result of the last statement Where ``__code__`` is the code snippet we want to run, and ``__ret__`` is the result, so that if we for example run `dataframe.head()` IPython still display the head of dataframe instead of nothing. Here is a complete example of a file `timit2.py` that define such a magic: .. code-block:: python from IPython.core.magic import ( Magics, magics_class, line_cell_magic, ) from IPython.core.magics.ast_mod import ReplaceCodeTransformer from textwrap import dedent import ast template = template = dedent(''' from time import perf_counter now = perf_counter() for i in range(10): __code__ print(f"Time taken: {(perf_counter() - now)/10}") __ret__ ''' ) @magics_class class AstM(Magics): @line_cell_magic def t2(self, line, cell): transformer = ReplaceCodeTransformer.from_string(template) transformer.debug = True transformer.mangler.debug = True new_code = transformer.visit(ast.parse(cell)) return exec(compile(new_code, "", "exec")) def load_ipython_extension(ip): ip.register_magics(AstM) .. code-block:: python In [1]: %load_ext timit2 In [2]: %%t2 ...: import time ...: time.sleep(0.05) ...: ...: Time taken: 0.05435649999999441 If you wish to ran all the code enter in IPython in an ast transformer, you can do so as well: .. code-block:: python In [1]: from IPython.core.magics.ast_mod import ReplaceCodeTransformer ...: ...: template = ''' ...: from time import perf_counter ...: now = perf_counter() ...: __code__ ...: print(f"Code ran in {perf_counter()-now}") ...: __ret__''' ...: ...: get_ipython().ast_transformers.append(ReplaceCodeTransformer.from_string(template)) In [2]: 1+1 Code ran in 3.40410006174352e-05 Out[2]: 2 Hygiene and Mangling -------------------- The ast transformer above is not hygienic, it may not work if the user code use the same variable names as the ones used in the template. For example. To help with this by default the `ReplaceCodeTransformer` will mangle all names staring with 3 underscores. This is a simple heuristic that should work in most case, but can be cumbersome in some case. We provide a `Mangler` class that can be overridden to change the mangling heuristic, or simply use the `mangle_all` utility function. It will _try_ to mangle all names (except `__ret__` and `__code__`), but this include builtins (``print``, ``range``, ``type``) and replace those by invalid identifiers py prepending ``mangle-``: ``mangle-print``, ``mangle-range``, ``mangle-type`` etc. This is not a problem as currently Python AST support invalid identifiers, but it may not be the case in the future. You can set `ReplaceCodeTransformer.debug=True` and `ReplaceCodeTransformer.mangler.debug=True` to see the code after mangling and transforming: .. code-block:: python In [1]: from IPython.core.magics.ast_mod import ReplaceCodeTransformer, mangle_all ...: ...: template = ''' ...: from builtins import type, print ...: from time import perf_counter ...: now = perf_counter() ...: __code__ ...: print(f"Code ran in {perf_counter()-now}") ...: __ret__''' ...: ...: transformer = ReplaceCodeTransformer.from_string(template, mangling_predicate=mangle_all) In [2]: transformer.debug = True ...: transformer.mangler.debug = True ...: get_ipython().ast_transformers.append(transformer) In [3]: 1+1 Mangling Alias mangle-type Mangling Alias mangle-print Mangling Alias mangle-perf_counter Mangling now Mangling perf_counter Not mangling __code__ Mangling print Mangling perf_counter Mangling now Not mangling __ret__ ---- Transformed code ---- from builtins import type as mangle-type, print as mangle-print from time import perf_counter as mangle-perf_counter mangle-now = mangle-perf_counter() ret-tmp = 1 + 1 mangle-print(f'Code ran in {mangle-perf_counter() - mangle-now}') ret-tmp ---- ---------------- ---- Code ran in 0.00013654199938173406 Out[3]: 2 T)NodeTransformerStoreLoadNameExprAssignModuleN)DictOptionalc|dvrdSdS)N)__ret____code__FTnames =/usr/lib/python3/dist-packages/IPython/core/magics/ast_mod.pyrs4+B#B%cZeZdZUdZdZeed<dZeed<dZddZ d Z d Z d Z d Z d Zy)Manglerzb Mangle given names in and ast tree to make sure they do not conflict with user code. TenabledFdebugc4|jr t|i|yyN)rprint)selfargskwargss rlogz Mangler.logs :: 4 "6 " rNc|d}||_y)Nc$|jdS)N___) startswithrs rrz"Mangler.__init__..sT__U%;r predicate)rr%s r__init__zMangler.__init__s  ;I"rc|j|jr2|jd|jd|jz|_|S|jd|j|S)NManglingmangle- Not mangling)r%idrrnodes r visit_NamezMangler.visit_NamesV >>$'' " HHZ ) $'')DG  HH^TWW - rc|j|jr1|jd|jd|jz|_n|jd|j|jjD]j}|j|jr1|jd|jd|jz|_O|jd|jl|j |S)Nr(r)r*zMangling function argzNot mangling function arg)r%rrrarg generic_visit)rr-r0s rvisit_FunctionDefzMangler.visit_FunctionDefs >>$)) $ HHZ +!DII-DI HH^TYY /99>> ?C~~cgg&0#'':#cgg-4cgg>  ? !!$''rc$|j|Sr_visit_Import_and_ImportFromr,s rvisit_ImportFromzMangler.visit_ImportFrom0066rc$|j|Srr4r,s r visit_ImportzMangler.visit_Importr7rc|jD]r}|j |jn |j}|j|rd|z}|j d|||_W|j d|jt|S)Nr)zMangling AliaszNot mangling Alias)namesasnamerr%r)rr-aliasr<new_names rr5z$Mangler._visit_Import_and_ImportFromssZZ =E#(<<#7UZZU\\F~~f% )F 2)84' -u||< = rr)__name__ __module__ __qualname____doc__rbool__annotations__rrr&r.r2r6r9r5rrrrrs@ GTE4##  (77 rrc~eZdZUdZeed<dZeed<eed< d dede e fd Z e d de de e fd Zd Zd Zy)ReplaceCodeTransformerTrFrmanglerNtemplatemappingct|ttdfsJt|tdtdfsJt|tjsJ||_t ||_|i}||_y)Ncyrrrrrrz1ReplaceCodeTransformer.__init__..srr$) isinstancedicttypeastr rHrrGrI)rrHrImangling_predicates rr&zReplaceCodeTransformer.__init__su'D$t*#5666,tDz4 ;M.NOOO(CJJ///  );< ?G rc>|tj|||S)N)rIrP)rOparse)clsrHrIrPs r from_stringz"ReplaceCodeTransformer.from_string s" IIh EW  rc |js|S|jd}t|tr|jj |jj t tdtg|jtj|ttdt}tj|}||jd<n/tjdjd|jd<|j|jd<tj|j}t!j"|}|j$j'|}|j)|}tj|}|j*r4t-d t-tj.|t-d |S) Nzret-tmp)ctx)valuer Nonerrz---- Transformed code ----z---- ---------------- ----)rbodyrLrpopappendrrrrXrOfix_missing_locationsrrIrRrHcopydeepcopyrGvisitr1rrunparse)rcodelastrettpltxr-node_2s r visit_Modulez#ReplaceCodeTransformer.visit_ModulesU||Kyy} dD ! IIMMO II  VT)%A$B$**U V  % %d +T)89C++C0C&)DLL #&)ii&7&<&rqsLobIHH !M :o:zD(_D(r