I installed gtn by pip install gtn and tested STC like below

from criterions.STC import STC
import random
import torch

stcloss = STC(blank_idx=0, p0=1, plast=1, thalf=1, reduction="none")

batch_size = 2
mel_len = 115
text_len = 25
num_token = 64
inputs = torch.randn(mel_len, batch_size, num_token)
targets = [[random.randint(1, text_len) for _ in range(random.randint(1, num_token))] for _ in range(batch_size)]
loss = stcloss(inputs, targets)

But when I trying to run code like above. The Error Like below occurs.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-b0ecfb513efd> in <module>
      7 inputs = torch.randn(mel_len, batch_size, num_token)
      8 targets = [[random.randint(1, text_len) for _ in range(random.randint(1, num_token))] for _ in range(batch_size)]
----> 9 loss = stcloss(inputs, targets)

~/anaconda3/envs/leecho/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

/data/leecho/xi-stt/xi-stt/model/STCLoss.py in forward(self, inputs, targets)
    212             # concatenate (tokens, <star>, <star>\tokens)
    213             log_probs = torch.cat([log_probs, lse, neglse], dim=2)
--> 214         return STCLoss(log_probs, targets, prob, self.reduction)

/data/leecho/xi-stt/xi-stt/model/STCLoss.py in forward(ctx, inputs, targets, prob, reduction)
     96             emissions_graphs[b] = g_emissions
     97 
---> 98         gtn.parallel_for(process, range(B))
     99 
    100         ctx.auxiliary_data = (losses, scales, emissions_graphs, inputs.shape)

/data/leecho/xi-stt/xi-stt/model/STCLoss.py in process(b)
     71             # create emission graph
     72             g_emissions = gtn.linear_graph(
---> 73                 T, Cstar, gtn.Device(gtn.CPU), inputs.requires_grad
     74             )
     75             cpu_data = inputs[b].cpu().contiguous()

AttributeError: module 'gtn' has no attribute 'Device'

Is there any way to solve this problem?

Hello, I'm training ASR model with STC Loss and letter-to-word encoder like below. But when I progress training, STC Loss ascended and became 'Inf' after 12000 step.

Is there any miss in my implementation? Any help would be appreciated. Thank you.

training args:

• num_gpu : 4
• audio_length_per_gpu : 160s
• lr : 0.0001~0.001
• use FullyShardedDataParallel, mix_precision=off

#   max_word_length : 10
#   n_letter_symbols : 69 (blank + pad + korean)
#   n_word_symbols : 12158 (blank + korean_morph 97% in corpus)

#   blank_idx=0, p0=0.05, plast=0.15, thalf=16000

self.criterion = STC(
    blank_idx=self.cfg.blank_idx, 
    p0=self.cfg.p0, 
    plast=self.cfg.plast, 
    thalf=self.cfg.thalf, reduction="mean"
)

#   model_output : Tensor[batch_size, max_frame_length, n_letter_symbols*max_word_length]
#   self.l2w_matrix : Tensor[n_letter_symbols*max_word_length, n_word_symbols]

word_level_output = model_output @ self.l2w_matrix

#   word_level_output : Tensor[batch_size, max_frame_length, n_word_symbols]

word_level_output =  F.log_softmax(word_level_output.transpose(1, 0), dim=-1)

loss = self.criterion(word_level_output, word_labels)

Hello, I'm trying to apply STC for my ASR model training.

Before proceeding with the training, I have a question to ask about STC training mentioned in STC paper [1] If anyone has experimented with the case I posited, please give me advise.

1. Does STC valid for pDrop=0.01~0.02 data? or do I just have to use CTC?
    data - The 99% reliable data that may contain some typos or sometimes the business name is erased.

    1-1. If STC is valid for 1., Is it sufficient to set p_0=0.01, p_max=0.03 for this case?

2.  Adam is not allowed for STC/(WFST)?

3. For future work, I am thinking of using pseudo labeled YouTube data for ASR training.
    In this case, data could have much incorrect labels in it.
    Does STC perform better than CTC even in case of incorrect labeled data training?

Thank you.

[1] Star Temporal Classification: Sequence Classification with Partially Labeled Data.

RNN model's default parameters contain stride. There are 2 convolutional layers each with a stride of 2. So for num_feature = 80, get reduced to 80 ->40 ->20. Updated the code to integrate it.

Summary

• Create criterions, models directory and organize files appropriately

• Make __init__.py files empty so that model training works. Otherwise, there were issues with module loading.
• Will send a follow-up PR to add documentation to STC code

Test Plan:

Ran IamDB training and make sure it runs an epoch.

When should we set the arc_sort(true) ? For example, in https://github.com/facebookresearch/gtn_applications/blob/eb1cb83dda3d3887f980dbd6b697c2c2b6fd1d45/transducer.py#L265 arc_sort was given parameter True. If my understanding is correct, for FSA target, arc_sort(true) and arc_sort(false) would give the exact same result? How do we decide to set it true or false? (When we prefer sort it in olable and when ilabel? )

It seems that in model.viterbi(self, outputs), only one recognition results will be returned for each sample. Is it possible to return multiple alternates from this decoding method, like beam search? If yes, what would be the recommended decoding algorithm? Will that be time consuming?

Thanks!

• Update IAM download script. Previous scripts were not working - https://stackoverflow.com/q/64715260
• Create a new directories dataset/download and dataset/utils and move the files appropriately

Consider this as V0. TODO

• Test all the recipes again
• More details to README. Possibly share ltr, WP based tokens, lexion, transition graph directly via S3 ...

Hi. I am planning to try asg to replace the Wav2Vec2 with LM.

Compairing to the tutorial on CTC at Pytorch, https://pytorch.org/audio/main/tutorials/asr_inference_with_ctc_decoder_tutorial.html

I am thinking how to do the same thing with asg to replace the CTC? If it is using the LM model with asg, is it better than wav2vec2LM with CTC? How much the difference?

Cloud you show me how to do automatic speech recognition with ASG? I hope I would write the decoder in javascript if I understand how it is going.

Hello, I'm trying to implement ASR model proposed in Star Temporal Classification. But I have some trouble for implementing my first 'word level output ASR model'.

When I use simple word-to-encoder's one hot tensor(E matrix), STC loss ascends. So I made several modifications to solve this problem.

(1). I view x : [T, B, A_L × l_max] to x : [T, B, l_max, A_L], apply F.log_softmax(dim=-1) and view it to original shape x : [T, B, A_L × l_max]

(2). And I apply letter-to-word encoder e_matrix : [A_L × l_max, A_W] by x = x @ e_matrix, and F.log_softmax(torch.exp(x), dim=-1) for STC input.

The reason I applied softmax for A_L is make x to probability of the appearance of alphabets at each location in the word. And log -> e_matrix -> exp is to convert the one hot sum by e_matrix operation into a probability product.

1.Is it right for letter-to-word encoder?

After applying this, STC loss starts from 3.5 and fall to 2.1~2.5 for 15 epoch. But the viterbi decoded(implemented in gtn_applications/criterions/ctc.py) output is always BLANK while checking WER for every predicted output.

Cause CTC loss and word level classification output has the same result, I assume that this is a problem with the properties of CTC training and word-level output ASR.

2. Is this an ordinary result?
3. How many epochs are needed to get results other than blank usually?
4. How many epochs are needed to reach the highest performance usually?

I'm sorry to bother you every time.

I want to compose two gtn, one is for lexicon, and the second is for grammar (LM) which is created from lm_arpa.py file. I am getting "MemoryError: std::bad_alloc" while doing with 250 GB RAM. I am not sure whether this is on the expected lines or not. PFA of both the gtn and code for reproducibility. gu-G.txt gu-L.txt

Code: gtn.savetxt('gu-LG.txt', gtn.compose(gtn.loadtxt('gu-L.txt'), gtn.loadtxt('gu-G.txt')).arc_sort())