o h@sdZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddlmZmZddlmZdd lmZmZd d d d dZddiddiddidZddiZddZeeZddZidddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:id;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\id]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrdsdtdudvdwdxdydzd{d|d}d~iddddddddddddddddddddddddddddddddddidddddddddddddddddddddddddddddddddd“ddddddddddddddddҜZiddԄeDdd/d1dddddKdKddddל Z ddgZ!GddۄdeZ"dd݄Z#ddd߄Z$ddZ%  ddee&de'de'de'fddZ(dee&fddZ)dee&fddZ*ddZ+dS)z!Tokenization classes for Whisper.N) lru_cache)ListOptionalTupleUnion) AddedTokenPreTrainedTokenizer)logging)BasicTextNormalizerEnglishTextNormalizerz vocab.jsonztokenizer.jsonz merges.txtznormalizer.json) vocab_filetokenizer_file merges_filenormalizer_filezopenai/whisper-basezBhttps://huggingface.co/openai/whisper-base/resolve/main/vocab.jsonzGhttps://huggingface.co/openai/whisper-base/resolve/main/merges_file.txtzGhttps://huggingface.co/openai/whisper-base/resolve/main/normalizer.json)rrricCstttdtddtttdtddtttdtdd}|dd}d }td D]}||vrI|||d ||d7}q3d d |D}tt||S) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~r ¡¬®ÿNrcSsg|]}t|qS)chr).0nrrf/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/whisper/tokenization_whisper.py Hz$bytes_to_unicode..)listrangeordappenddictzip)bscsrbrrrbytes_to_unicode4sL    r)cCs6t}|d}|ddD] }|||f|}q |S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rr N)setadd)wordpairs prev_charcharrrr get_pairsPs r0enenglishzhchinesedegermanesspanishrurussiankokoreanfrfrenchjajapanesept portuguesetrturkishplpolishcacatalannldutchararabicsvswedishititalianid indonesianhihindififinnishvi vietnamesehehebrewuk ukrainianelgreekmsmalayr'czechroromaniandadanishhu hungariantatamilno norwegianththaiururduhrcroatianbg bulgarianlt lithuanianlalatinmimaoriml malayalamcywelshskslovaktetelugufapersianlvlatvianbnbengalisrserbianaz azerbaijanisl slovenianknkannadaetestonianmk macedonianbrbretoneubasqueis icelandichyarmeniannenepalimn mongolianr&bosniankkkazakhsqalbanianswswahiliglgalicianmrmarathipapunjabisisinhalakmkhmersnshonayoyorubasosomaliaf afrikaansococcitankageorgianbe belarusiantgtajiksdsindhigugujaratiamamharicyiyiddishlolaouzuzbekfofaroesehtzhaitian creolepspashtotkturkmennnnynorskmtmaltesesanskrit luxembourgishmyanmartibetantagalogmalagasyassamesetatarhawaiianlingalahausabashkirjavanese sundanese cantonese)salbmybotlmgastthawlnhabajwsuyuecCi|]\}}||qSrr)rcodelanguagerrr rrr) burmese valencianflemishhaitian letzeburgeschpushtopanjabi moldavianmoldovan sinhalese castilianmandarin translate transcribecsDeZdZdZeZeZeZ ddgZ          dWfdd Z e d e fd d Zd dZddZdXdededefddZe d ee fddZdYd ee fddZ dZdee deee ded ee ffdd Zdd Zd!d"Zd#d$Zd%d&Zed[d'd(Zd\d efd*d+Zd]d,d-Z e!d]d.d/Z"d[d0efd1d2Z#d3d4Z$    )    d^d0ed5ed6ed7ed8ed9ed:ed effd;d< Z%    d_d=e&e ee fd0ed8ed9ed:ed ef d>d?Z'd@dAZ(dYdBedCeed e)efdDdEZ*d[dFdGZ+e dHdIZ,d`dKdLZ-dMdNZ.dadPefdQdRZ/ed=ee dSe dTe fdUdVZ0Z1S)bWhisperTokenizeraa Construct a Whisper tokenizer. This tokenizer inherits from [`PreTrainedTokenizer`] which contains some of the main methods. Users should refer to the superclass for more information regarding such methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. normalizer_file (`str`, *optional*): Path to the normalizer_file file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The beginning of sequence token. The `decoder_start_token_id` is used to set the first token as `"<|startoftranscript|>"` when generating. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*): The token used for padding, for example when batching sequences of different lengths. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. language (`str`, *optional*): The language of the transcription text. The corresponding language id token is appended to the start of the sequence for multilingual speech recognition and speech translation tasks, e.g. for Spanish the token `"<|es|>"` is appended to the start of sequence. This should be used for multilingual fine-tuning only. task (`str`, *optional*): Task identifier to append at the start of sequence (if any). This should be used for mulitlingual fine-tuning, with `"transcribe"` for speech recognition and `"translate"` for speech translation. predict_timestamps (`bool`, *optional*, defaults to `False`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. input_idsattention_maskNreplace <|endoftext|>Fc  st|trt|dddddn|}t|trt|dddddn|}t|tr.t|dddddn|}t|tr>t|dddddn|}t|dd}t||_Wdn1sWwYdd|jD|_||_ t |_ dd|j D|_ t|dd}| d d d }Wdn1swYd d |D}tt|tt||_i|_| |_|durt|dd}t||_Wdn1swYnd|_td|_td|_| |_tjd|||||| d| | |_| |_dS)NFT)lstriprstrip normalizedspecialutf-8encodingcSrrrrkvrrrr.rz-WhisperTokenizer.__init__..cSrrrrrrrr1r r cSsg|]}t|qSr)tuplesplit)rmergerrrr4z-WhisperTokenizer.__init__..zJ's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+z<\|(\d+\.\d+)\|>)errors unk_token bos_token eos_token pad_tokenadd_prefix_spacer) isinstancestrropenjsonloadencoderitemsdecoderrr) byte_encoder byte_decoderreadrr$r%r!len bpe_rankscacher#english_spelling_normalizerrecompilepat timestamp_patrsuper__init__taskpredict_timestamps)selfrrrrrr r!r"r#rr9r:kwargs vocab_handle merges_handle bpe_merges __class__rrr8sf   zWhisperTokenizer.__init__returncCs t|jSN)r/r)r;rrr vocab_sizeQs zWhisperTokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr)convert_ids_to_tokensrirDrrrVrz.WhisperTokenizer.get_vocab..)r!rEupdateadded_tokens_encoder)r;vocabrrDr get_vocabUs zWhisperTokenizer.get_vocabc sX|jvr j|St|}t|}|s|S t|fddd}|jvr'ny|\}}g}d}|t|krz|||} WntyO|||dYn?w|||| | }|||kr}|t|dkr}||d|kr}| |||d7}n | |||d7}|t|ks5t|}|}t|dkrnt|}qd |}|j|<|S) NTcsj|tdS)Ninf)r0getfloat)pairrDrresz&WhisperTokenizer.bpe..keyrr  ) r1rr0minr0r/index ValueErrorextendr#join) r;tokenr,r-bigramfirstsecondnew_wordrHjrrDrbpe[sJ     ,     zWhisperTokenizer.bperr9r:cCsF|dur|n|j|_|dur|n|j|_|dur||_dS|j|_dS)a Override the prefix tokens appended to the start of the label sequence. This method can be used standalone to update the prefix tokens as required when fine-tuning. Example: ```python >>> # instantiate the tokenizer and set the prefix token to Spanish >>> tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny", language="spanish") >>> # now switch the prefix token from Spanish to French >>> tokenizer.set_prefix_tokens(language="french") ``` Args: language (`str`, *optional*, defaults to `None`): The language of the transcription text. task (`str`, *optional*, defaults to `None`): Task identifier to append at the start of sequence (if any). predict_timestamps (`bool`, *optional*, defaults to `None`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. N)rr9r:)r;rr9r:rrrset_prefix_tokenssz"WhisperTokenizer.set_prefix_tokensc Cs<|d}|d}|d}|d}tt}|jdurZ|j|_|jtvr0t|j}n*|jtvr;|j}nt|jdk}t d|jd|rPt tnt td|j duro|j t vrot d |j d t |g}|jdur| |d |||j dur| |j d kr|n||js| ||S) N<|startoftranscript|>z <|translate|>z<|transcribe|><|notimestamps|>rTzUnsupported language: z. Language should be one of: .zUnsupported task: z. Task should be in: r r)convert_tokens_to_idsr LANGUAGESkeysrlowerTO_LANGUAGE_CODEvaluesr/rXr r9TASK_IDSr#rWr:) r; bos_token_idtranslate_token_idtranscribe_token_idnotimestamps_token_idlangs language_idis_language_code bos_sequencerrr prefix_tokenss8               zWhisperTokenizer.prefix_tokenscCs0|dur |j||jgS|j|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N)ru eos_token_id)r; token_ids_0 token_ids_1rrr build_inputs_with_special_tokenssz1WhisperTokenizer.build_inputs_with_special_tokensrwrxalready_has_special_tokenscsn|r tj||ddSdgt|j}dg}|dur%|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rwrxrzr Nr)r7get_special_tokens_maskr/ru)r;rwrxrz prefix_ones suffix_onesr@rrr{s$z(WhisperTokenizer.get_special_tokens_maskcsZg}tj|D]!}dfdd|dD}|dd|dDq |S)zTokenize a string.c3s|]}j|VqdSrC)r,)rr(rDrr s  z-WhisperTokenizer._tokenize..rcs|]}|VqdSrCr)r bpe_tokenrrrrrU)r3findallr5rZencoderYrar)r;text bpe_tokensr[rrDr _tokenizes"zWhisperTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r)rNr)r;r[rrr_convert_token_to_idsz%WhisperTokenizer._convert_token_to_idcCs|j|dS)z Converts an index (integer) in a token (str) using the vocab. Whisper's base tokenizer always decodes OOV tokens as "", thus we do not use the `unk_token` here. r~)r+rN)r;rWrrr_convert_id_to_tokensz%WhisperTokenizer._convert_id_to_tokencCst|j}||S)z Normalize a given string using the `EnglishTextNormalizer` class, which preforms commons transformation on english text. )r r2)r;r normalizerrrr _normalizes zWhisperTokenizer._normalizecCst|d}||S)z Normalize a given string using the `BasicTextNormalizer` class, which preforms commons transformation on multilingual text. remove_diacritics)r )rrrrrr_basic_normalizes z!WhisperTokenizer._basic_normalize{Gz?cs~jdd}gg}|D]#}||kr(d|||dd}|||gq |d|q fdd|D}d|S) z Timestamp tokens are above the special tokens' id range and are ignored by `decode()`. This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". rr z<|z.2fz|>cs(g|]}t|tr |nj|dqS)skip_special_tokens)r$r%decode)rsr;rrrrsz.r~)all_special_idsr#rZ)r; token_idsrtime_precisiontimestamp_beginoutputsr[ timestamprrr_decode_with_timestampss    z(WhisperTokenizer._decode_with_timestampsc CsZg}t|}|jddkrt|jdkrtd|jdd}||k}t|dd|dd@dd}|jddkrF|dkrFgSt|ddd|vrbt|t|ddd}t|dd}|D]=}|||} t| dkr| d |} | d |} | | } | | } | | } || | || |fd|}qm|S)a~ Compute offsets for a given tokenized input Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. time_precision (`float`, `optional`, defaults to 0.02): The time ratio to convert from token to time. rr z)Can only process a single input at a timerNrr) nparrayshaper/rXrwheresumr#item_preprocess_token_ids_decode_filter_timestamp_ids) r;rroffsetsrtimestamp_tokens consecutive last_slice current_slice sliced_tokensstart_timestamp_positionend_timestamp_positionrrrr_compute_offsets!s: &      z!WhisperTokenizer._compute_offsetscs|fddtdDS)a Compute the timestamp token ids for a given precision and save to least-recently used (LRU) cache. Args: time_precision (`float`, `optional`, defaults to 0.02): The time ratio to convert from token to time. csg|]}d|qS)z<|%.2f|>rrGrrrrZrz2WhisperTokenizer.timestamp_ids..i)rfr!)r;rrrr timestamp_idsQs zWhisperTokenizer.timestamp_idsrcCs*|r|d}|d}||||}|S)a Pre-process the token ids for decoding by removing the prompt tokens ids and timestamp token ids. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Typically, obtained using the `__call__` method of the tokenizer. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens from the token ids. If `True`, the prompt token ids will be removed. <|startofprev|>rc)rf _strip_prompt)r;rrprompt_token_iddecoder_start_token_idrrrr\s   z&WhisperTokenizer._preprocess_token_idscCst|jd|S)Nr~)r3subr6)r;rrrrrnz&WhisperTokenizer._filter_timestamp_idsclean_up_tokenization_spacesoutput_offsetsdecode_with_timestamps normalizebasic_normalizerc  sn|j||d} tj| f||||| d| } |r"|j| ||d} n|| } |r5|j||d} | | dS| S)a# Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). output_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output the offsets of the tokens. This should only be set if the model predicted timestamps. time_precision (`float`, `optional`, defaults to 0.02): The time ratio to convert from token to time. decode_with_timestamps (`bool`, *optional*, defaults to `False`): Whether or not to decode with timestamps included in the raw text. normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the English text normalizer to the decoded text. Only applicable when the target text is in English. Otherwise, the basic text normalizer should be applied. basic_normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the Basic text normalizer to the decoded text. Applicable to multilingual target text. remove_diacritics (`bool`, *optional*, defaults to `False`): Whether or not to remove diacritics when applying the Basic text normalizer. Removing diacritics may destroy information in the decoded text, hence it should be used with caution. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. r)rrrrr)rrr)rr)rr7rrrr)r;rrrrrrrrrr< filtered_idsrrr@rrrqs00   zWhisperTokenizer.decoderc Ks|dd|_|j||d}g}g} |D]&} |r| |jvrq| |jvr5| r/||| g} || q| | q| rE||| d|} |rS|| } | S|r^|j | |d} | S| S)Nuse_source_tokenizerFrr~r) pop_decode_use_source_tokenizerrFrrJr#convert_tokens_to_stringrZrr) r;rrrrrr<filtered_tokens sub_textscurrent_sub_textr[r clean_textrrrrs.      zWhisperTokenizer._decodecs0d|}tfdd|Djdjd}|S)z:Converts a sequence of tokens (string) in a single string.r~csg|]}j|qSr)r-)rcrDrrrrz=WhisperTokenizer.convert_tokens_to_string..r)r)rZ bytearrayrr)r;tokensrrrDrrs "z)WhisperTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixc Cstj|std|ddStj||r|dndtd}tj||r,|dndtd}tj||r=|dndtd}t|dd d }|t j |j d d d ddWdn1sewYd}t|dd d :}|dt |j dddD]!\} } || krtd|d| }|d| d|d7}qWdn1swY|jdurt|dd d }|t j |jd d d ddWdn1swY|||fS)NzVocabulary path (z) should be a directory-r~rrrwrrrTTF)indent sort_keys ensure_asciirrz#version: 0.2 cSs|dS)Nr r)kvrrrrQsz2WhisperTokenizer.save_vocabulary..rRzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rUr )ospathisdirloggererrorrZVOCAB_FILES_NAMESr&writer'dumpsr)sortedr0r*warningr2) r;rrr merge_filerfrWwriterr token_indexrrrsave_vocabularysF       z WhisperTokenizer.save_vocabularycKs&|d|j}|s |rd|}||fS)Nr#rU)rr#)r;ris_split_into_wordsr<r#rrrprepare_for_tokenizationsz)WhisperTokenizer.prepare_for_tokenizationcCstd|jjddS)zv A simple chat template that ignores role information and just concatenates messages with EOS tokens. zU No chat template is defined for this tokenizer - using the default template for the z class. If the default is not appropriate for your model, please set `tokenizer.chat_template` to an appropriate template. See https://huggingface.co/docs/transformers/main/chat_templating for more information. zM{% for message in messages %}{{ message.content }}{{ eos_token }}{% endfor %})r warning_oncerA__name__rDrrrdefault_chat_templates z&WhisperTokenizer.default_chat_templateTcCs6|j||| d|jdd}ddt|D}|S)N)r9rr:r cSsg|] \}}|d|fqS)r r)rrankr[rrrr/sz;WhisperTokenizer.get_decoder_prompt_ids..)rbru enumerate)r;r9r no_timestamps forced_tokensforced_decoder_idsrrrget_decoder_prompt_ids(sz'WhisperTokenizer.get_decoder_prompt_idscCst|||||dS)N)return_timestampsreturn_languager) _decode_asr)r; model_outputsrrrrrrr2szWhisperTokenizer._decode_asrrrcstdd|dd}|ddd}tfdd |Dd}|dur0|}td |d |j|d |dS) z`Converts prompt text to IDs that can be passed to [`~WhisperForConditionalGeneration.generate`].rrUF)add_special_tokensr r Nc3s"|] }|jdkr|VqdS)rN)r)rxrDrrrAs z2WhisperTokenizer.get_prompt_ids..zJEncountered text in the prompt corresponding to disallowed special token: re) tensor_type)stripnextrFrXconvert_to_tensors)r;rreturn_tensorsbatch_encodingprompt_text_idsspecial_token_idr[rrDrget_prompt_ids;s  zWhisperTokenizer.get_prompt_idsrrcCs@t|to |o |d|k}|r||vr|||dSgS|S)Nr)r$r rW)rrr has_promptrrrrIs zWhisperTokenizer._strip_prompt) Nr r r r NFNNF)NNNrC)NF)F)Fr)r)FNFrFFFF)FFFF)NNT)r)2r __module__ __qualname____doc__rvocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_mapMAX_MODEL_INPUT_SIZESmax_model_input_sizesmodel_input_namesr8propertyintrErLrar%boolrbrruryrr{rrrr staticmethodrrrrrrrrrrrrrrrrrrr __classcell__rrr@rr s(J*"     0     O *  '    $r c'sdfdd}g}|}d}|dd} g} g} d} d} t|j}t|D]\}}|dd }|d kr@|d d }d}| }d |vr||d \}}}||8}||} |r_||| }|r|t|D]}|| kr{|dury|| || kryn|}qeg}g}t|D]\}}||vr||g}|d d}t|d}|durЈr|kr|s| |t | }||}||d<| |g} g}|}||d<|q q|| krV|| ||} t | d } |r||krd} q| s| r||krd} q|dd dur| |dd <q| |dd krq| |dd<| ||d kr'| |t | | \}}!||}||d<|d krEt |||!|d<| |g} g}g} g}|}q| ||d krt |||d }"|dt |kr~t ||d|d }#nd}#| |"|#fqd |vr|||7}|r| ||d kr| |q'tdd| Ds|}g} g}g} g}q'| r|rtdt | | \}}!||}||d<|d krt |||!|d<| |ddd|D}$|s|r=|D]}|s|dnt|d|d<|s|dq|d kr5g}%|D] }|%|dq"d|%i}&|$|&fSd|i}&|$|&fSi}&|$|&fS)z Internal method meant to only be used by asr pipeline. Handles all the little quirks specific to whisper to handle the various options not allowed in other seq2seq models NcsddgddS)Nr~)rrrrr last_languagerr new_chunkirz_decode_asr..new_chunkrdr Frrr,token_timestampsstriderTrrTrwordscsrrCr)rprrrrrz_decode_asr..zWhisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.r~css|]}|dVqdS)rNr)rchunkrrrr schunks)rfr*rrtolistreversedrrgrNr#_find_longest_common_sequenceround_collate_word_timestampsr/anyrrrZrrrY)' tokenizerrrrrr rr time_offsetrprevious_tokensprevious_token_timestampsskipright_stride_startrchunk_idoutputrr last_timestampfirst_timestamp chunk_len stride_left stride_rightr[current_tokenscurrent_token_timestampsrHrrresolved_tokens resolved_texttimeresolved_token_timestamps start_timeend_time full_text new_chunksoptionalrrrrUs                                  rcCs|d}t|}g}|r|d}g}t|ddD]\}}d} ||ddf} t|} td|| D]]} | d} td|| }t||| | }t|||}td| |}t| | }t|||}t|t|krqtdt||k}|| | }|dkr|| kr|} ||||f} q1| \}}}}||d}||d}| |d|||d}t|}|r| |d|||d|d}q| ||dur|St|dkr| |||fS|gfS)Nrr r g@ziThere is a bug within whisper `decode_asr` function, please report it. Dropping to prevent bad inference.rT) r/rr!maxrVrr RuntimeErrorrrY) sequencestoken_timestamp_sequences left_sequence left_lengthtotal_sequenceleft_token_timestamp_sequencetotal_token_timestamp_sequenceseq_idxright_sequencemax_ max_indices right_lengthrHeps left_start left_stopleft right_start right_stoprightmatchesmatchingleft_mid right_midrrrr6sZ           rcs.t|||\}}}fddt||D}|S)Ncs6g|]\}}||dd|ddfdqS)rrr rr)rr,indicesr rrrs z,_collate_word_timestamps..)_combine_tokens_into_wordsr%)rrr rr_ token_indicestimingsrrKrrs  r "'“¡¿([{-"'.。,，!！?？:：”)]}、rrprepend_punctuationsappend_punctuationscCs^|dur|j}|dur d}|dvrt||\}}}nt||\}}}t||||||||fS)z Groups tokens by word. Returns a tuple containing a list of strings with the words, and a list of `token_id` sequences with the tokens making up each word. Nr2>rrmr4rr@r)r_split_tokens_on_unicode_split_tokens_on_spaces_merge_punctuations)rrrrRrSr word_tokensrNrrrrLs  rLc Cs|j|dd}d}g}g}g}g}g}d} t|D]=\} } || || |j|dd} || vs=|| | ||krV|| ||||g}g}| t| 7} q|||fS)zlCombine tokens into words by splitting at any position where the tokens are decoded as valid unicode points.T)ru�r)rrr#rWr/) rr decoded_fullreplacement_charrrWrNr&current_indicesunicode_offset token_idxr[decodedrrrrTs,       rTcCst||\}}}g}g}g}t|||D]I\}} } | d|jk} |d} |dv} | s7| s7| s7t|dkrG|||| || q|d||d<|d| |d| q|||fS)zLCombine tokens into words by splitting at whitespace and punctuation tokens.rrUz !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~r)rTr%rv startswithrr/r#rY)rrsubwordssubword_tokens_listsubword_indices_listrrWrNsubwordsubword_tokenssubword_indicesr with_space punctuationrrrrUs       rUcCst|d}t|d}|dkrT||drJ|||vrJ||||||<||||||<||||||<d||<g||<g||<n|}|d8}|dksd}d}|t|kr||ds|||vr||||7<||||7<||||7<d||<g||<g||<n|}|d7}|t|ks^dd|D|dd<d d|D|dd<d d|D|dd<dS) z1Merges punctuation tokens with neighboring words.rTr rrUr~cSg|]}|r|qSrr)rr,rrrrrz'_merge_punctuations..NcSrgrr)rr[rrrrrcSrgrr)ridxrrrrr)r/r^rendswith)rrrJ prependedappendedrHr`rrrrVs:        rVrC)NrPrQ),rr'r functoolsrtypingrrrrnumpyrregexr3tokenization_utilsrr utilsr english_normalizerr r rrrr) get_loggerrrr0rgr*rjrlr rrrrr%rLrTrUrVrrrrs           !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVh bi