This is a pytorch implementation of the transformer model. If you'd like to understand the model, or any of the code better, please refer to <ahref=https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec>my tutorial</a>.
Using the Europarl dataset plus the dataset in the data folder, I was able to achieve a BLEU score of 0.39 on the test set (current SOTA is around 0.42), after 4/5 days of training on a single 8gb GPU. For more results see the tutorial again.
# Train the model immediately on FloydHub
[](https://floydhub.com/run)
Launch a [FloydHub Workspace](https://www.floydhub.com/product/build) to start training this model with 1 click. Workspace is a GPU-enabled cloud IDE for machine learning. It provides a fully configured environment so you can start hacking right away, without worrying about dependencies, data sets, etc.
Once you've started the workspace, run the 'start_here' notebook or type 'floyd run' into the workspace terminal. This will begin to train the model on the sample dataset.
# Usage
Two text files containing parallel sentences (seperated by '\n' characters) in two languages are required to train the model. See an example of this in the data/ folder (french.txt and english.txt).
The spacy tokenizer is used to tokenize the text, hence only languages supported by spacy are supported by this program. The languages supported by Spacy and their codes are:
English : 'en'<br/>
French : 'fr'<br/>
Portugese : 'pt'<br/>
Italian : 'it'<br/>
Dutch : 'nl'<br/>
Spanish : 'es'<br/>
German : 'de'<br/>
For example, to train tan English->French translator on the datasets provided in the data folder, you would run the following:
```
python train.py -src_data data/english.txt -trg_data data/french.txt -src_lang en -trg_lang fr
```
Additional parameters:<br/>
-epochs : how many epochs to train data for (default=2)<br/>
-batch_size : measured as number of tokens fed to model in each iteration (default=1500)<br/>
-n_layers : how many layers to have in Transformer model (default=6)<br/>
-heads : how many heads to split into for multi-headed attention (default=8)<br/>
-no_cuda : adding this will disable cuda, and run model on cpu<br/>
-SGDR : adding this will implement stochastic gradient descent with restarts, using cosine annealing<br/>
-d_model : dimension of embedding vector and layers (default=512)<br/>
-dropout' : decide how big dropout will be (default=0.1)<br/>
-printevery : how many iterations run before printing (default=100)<br/>
-lr : learning rate (default=0.0001)<br/>
-load_weights : if loading pretrained weights, put path to folder where previous weights and pickles were saved <br/>
-max_strlen : sentenced with more words will not be included in dataset (default=80)<br/>
-checkpoint : enter a number of minutes. Model's weights will then be saved every this many minutes to folder 'weights/'<br/>
