o h]@sdZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl Z ddl m Z ddl mZmZmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'e$(e)Z*dZ+dZ,ddgZ-gdZ.eGdddeZ/ddZ0Gddde j1Z2Gddde j1Z3Gddde j1Z4Gddde j1Z5Gdd d e j1Z6Gd!d"d"e j1Z7Gd#d$d$e j1Z8Gd%d&d&e j1Z9Gd'd(d(e j1Z:Gd)d*d*e j1Z;Gd+d,d,e j1ZGd1d2d2e j1Z?Gd3d4d4eZ@d5ZAd6ZBe"d7eAGd8d9d9e@ZCe"d:eAGd;d<dd?d?e@ZEe"d@eAGdAdBdBe@ZFe"dCeAGdDdEdEe@ZGdS)Fz PyTorch CANINE model.N) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutput ModelOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) CanineConfigzgoogle/canine-srzgoogle/canine-r)+;=Iagqc@sZeZdZUdZdZejed<dZejed<dZ e e ejed<dZ e e ejed<dS)CanineModelOutputWithPoolinga Output type of [`CanineModel`]. Based on [`~modeling_outputs.BaseModelOutputWithPooling`], but with slightly different `hidden_states` and `attentions`, as these also include the hidden states and attentions of the shallow Transformer encoders. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model (i.e. the output of the final shallow Transformer encoder). pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Hidden-state of the first token of the sequence (classification token) at the last layer of the deep Transformer encoder, further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the input to each encoder + one for the output of each layer of each encoder) of shape `(batch_size, sequence_length, hidden_size)` and `(batch_size, sequence_length // config.downsampling_rate, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial input to each Transformer encoder. The hidden states of the shallow encoders have length `sequence_length`, but the hidden states of the deep encoder have length `sequence_length` // `config.downsampling_rate`. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of the 3 Transformer encoders of shape `(batch_size, num_heads, sequence_length, sequence_length)` and `(batch_size, num_heads, sequence_length // config.downsampling_rate, sequence_length // config.downsampling_rate)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlast_hidden_state pooler_output hidden_states attentions) __name__ __module__ __qualname____doc__r.torch FloatTensor__annotations__r/r0rrr1r9r9`/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/canine/modeling_canine.pyr-As r-c Cs*zddl}ddl}ddl}Wn tytdwtj|}t d||j |}g}g} |D] \} } t d| d| |j || } | | | | q6t|| D]5\} } | d} tdd | Dr{t d d| q\| dd krd | d<n>| d dkr| | d n0| d dkrd| d <n%| d dkrdg| dd} n| ddkr| d dvrdg| d d} |} | D]x}|d|rd|vr|d|}n|g}|ddks|ddkrt| d} n?|ddks|ddkrt| d} n,|dd krt| d} nz t| |d} Wnty.t d d| Yqwt|d!kr@t|d }| |} q|d"dd#krPt| d} n |d$dd%d&td'Dvrft| d} n |dkrp|| } | j| jkrtd(| jd)| jd*t d+| t| | _q\|S),z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /css|]}|dvVqdS))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepclsautoregressive_decoderchar_output_weightsNr9).0nr9r9r: s  z,load_tf_weights_in_canine..z Skipping bertencoderr embeddingssegment_embeddingstoken_type_embeddingsinitial_char_encoderchars_to_moleculesfinal_char_encoder) LayerNormconv projectionz [A-Za-z]+_\d+Embedderz_(\d+)kernelgammaweight output_biasbetabiasoutput_weightsi _embeddingsicSsg|]}d|qS) Embedder_r9)rDir9r9r: sz-load_tf_weights_in_canine..zPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzipsplitanyjoinremove fullmatchgetattrAttributeErrorlenintrange transposeshape ValueErrorr6 from_numpydata)modelconfigtf_checkpoint_pathranptftf_path init_varsnamesarraysnamer{arraypointerm_name scope_namesnumr9r9r:load_tf_weights_in_canineds                        rc seZdZdZfddZdedefddZdededefd d Z dd ee j d ee j dee j dee j de j f ddZ Z S)CanineEmbeddingsz