Triton layer normalization kernels. This kernel implements layers normalization using Triton. This kernel is from the flash attention project. Functions Function layer norm (x: torch.Tensor, weight: torch.Tensor, bias: torch.Tensor, residual: Optional[torch.Tensor] = None, x1: Optional[torch.Tensor] = None, weight1: Optional[torch.Tensor] = None, bias1: Optional[torch.Tensor] = None, eps: float = 1e 06, dropout p: float = 0.0, rowscale=None, prenorm: bool = False, residual in fp32: bool = False, is rms norm: bool = False, return dropout mask: bool = False, out: Optional[torch.Tensor] = None, residual out: Optional[torch.Tensor] = None) Apply layer normalization to the input tensor with Triton acceleration. Parameters x ( torch.Tensor ) Input tensor to normalize. weight ( torch.Tensor ) Scale parameter for normalization. bias ( torch.Tensor ) Shift parameter for normalization. residual ( torch.Tensor , optional ) Optional residual tensor to add to the input before normalization. x1 ( torch.Tensor , optional ) Optional second input tensor to combine with x . When provided, the function first adds x1 to x and then applies normalization. weight1 ( torch.Tensor , optional ) Scale parame…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy