o h< @sddlZddlmZmZmZddlmZedddiedejd ejd ejd ejdejf d d ZdddZ ddZ edejd ejd ejdejd ejf ddZ dddZ ddZ dddZGdddejjZGdddZdS) N)cdiv heuristicsjit)languageEVEN_KcCs|d|ddkS)NKTILE_Kr)nargsr r S/var/www/html/ai/venv/lib/python3.10/site-packages/triton/ops/blocksparse/matmul.pyr TILE_MTILE_Nr BLOCKc*Cs<td|}||d7}td}t|d}t|d}||td||}td|}||||||dddf||dddf|}t|d}||td||}td|} ||||||dddf| | dddf| }!tj||ftjd}"t|d| D]F}#|rt|}$t|!}%n tj||dddf|#kdd}$tj|!| dddf|#kdd}%|"tj|$|%tjd7}"|||7}|!|| 7}!q|"|j j }&td||}'td||}(||| || |'dddf| |(dddf|})tj |)|&d d dS) Nrrdtypeg)maskother out_dtypeTr) tl program_idloadarangezerosfloat32rangedottor element_tystore)*ABC stride_za stride_ha stride_ma stride_ak stride_zb stride_hb stride_bk stride_nb stride_zc stride_hc stride_mc stride_ncr grid_offsetlutrrr rrblock_idoff_zoff_hstart_amoffs_amoffs_aka_ptrsstart_bnoffs_bnoffs_bkb_ptrsacckabcoffs_cmoffs_cnpcr r r _sdd_kernelsf         rJc Cs|ddkr|ddkr|}|ddkr$|ddkr$|}|r2||}}| | }}|r6dnd} |rdnd|j} } | }|rLn| }|rR| n}| durftj| |||f| |jd}n | j| |||fksqJ| }d}fdd}t|||||d|d||rdnd||rdnd|d|d||rdnd||rdnd|d|d||rdnd||rdnd| |f||t |d |d d d d |S) NrrrrrNcst|dgS)Nr)r)metaBS0BS3widthr r r rzdsd_matmul..rPrQ)rrr rrRrSrm) rTrUsizerrXrYrOrVrmin)rDrErZr[r\r]r^r6rr`AS1BS1rCS0CS1CS2CS3rFrrer rr dsd_matmuls<    000 rcCst||rdnd}t|jdd\}}|}||}|r&|jdd} n |ddjdd} | d} t|} tj|dddd | dd<t | | dt| } | dddf|} | } | dd| dd8<||}| dd d|} || ddddf<| dddf|d|8<| | |dk| | |dkdf<| d} |rtj | |jd }nLtjgtj|jd }d}t|dD]8}||ddddf }|}dtj ||jd ||dk<t|||j|jdkdf}||7}q|||}|dd|dd||8<| dd d|}|rF||ddddf<|dddf|d|8<n|||ddddf<|dddf|d||8<|| |dk|| |dkdf<| d}|d}| d|d |} ||}tj| |||fdd  d}tj| |fdd  d}tjd |j|jd}t||f}t||f}|tj|}||fS)a Generates the look-up table for incrementing pointers in the DSD/DDS matmul. Example (BLOCK=32, STEP=16) [[1, 0, 0, 1, 0], [0, 1, 1, 0, 1], [1, 0, 1, 0, 0]] Then the offsets for A are [0 , 16, 32, 48] <- row 0 \----/ \----/ col=0 col=3 [64, 80, 96, 112, 128, 144] <- row 1 \----/ \----/ \------/ col=1 col=2 col=3 [160, 176, 192, 208] which leads to increments table [0, 16, 16, 16, || 64, 16, 16, 16, 16, 16, || 160, 16, 16, 16] Because B is dense, the offsets are [0, 16, 96, 112] <- row 0 [32, 48, 64, 80] <- row 1 [0, 16, 64, 80] <- row 2 rrTrgFrNrL)dim)rOrNrQ)rOr)rXsum ones_likeriflatten transposer zeros_likecumsumrcloneviewrepeatrrOtensorint64r!longcatTstackrUrrtypeint32r#)rkr^steptransrOsizeshead_idcol_idsegmentsnnz num_blocksoffsetsB_idxB_incsdivA_idxcurrent_offsetzlayoutwmsumA_incsrrincspadr6r r r dsd_lutsd     "  $"$    rc Cs"t||| | | ||||| d SN)r`)r) rDrErZr[r\r]r^r6rr`r r r dds_matmulZs"rc@s0eZdZeeedZeddZeddZ dS)_matmulsdddsdddsc Csxtj||||||||| | |d }|||| |_| |_| |_||_||_||_||_ ||_ ||_ ||_ |du|_ |Sr)rfnsave_for_backwardda_lutda_widthdb_lutdb_widthmoder]r^rZr[r\has_out)ctxrDrErZr[r\rr]r^c_lutc_widthrrrrr`rFr r r forwardfs"  z_matmul.forwardc Cs|j\}}d\}}|j}|jdr4|d|d|d}tj||||j|j |j|j|j |j |j }|jdr\|d|d|d}tj||||j |j|j|j|j |j |j }|jra|nd} ||dddddddddddd| fS)N)NNrrr) saved_tensorsrneeds_input_gradrrr\r[rZr]r^rrrrr) rdcrDrEdadbrmode_damode_dbdoutr r r backward{s$  " "  z_matmul.backwardN) __name__ __module__ __qualname__rfrrr staticmethodrrr r r r rbs   rc@s eZdZdddZdddZdS) matmulFc CsD|dvrtd||_||_||_||_||_||_|j|_t |d}|jdkrJt |||\|_ |_ t |||d|\|_|_t |||d|\|_|_|jdkrtt ||||j |\|_ |_ t |||\|_|_t ||||j|\|_|_|jdkrt ||||j|\|_ |_ t ||||j |\|_|_t |||\|_|_dSdS) Nrz"Supported modes are: sdd, dsd, ddsrPrTFrr)NotImplementedErrorr^rrZr[r\rkrVr]rrlrrrrrrr) selfrkr^rrOrZr[r\rr r r __init__s.    zmatmul.__init__NcCsBt|||j|j|j|j|j|j|j|j |j |j |j |j |}|SN)rapplyrZr[r\rr]r^rrrrrr)rrDrEr`rFr r r __call__szmatmul.__call__)FFFr)rrrrrr r r r rs rr)rXrrrrr constexprrJrfrlrrrrautogradFunctionrrr r r r sJ   @"  K& f1