malarinv
/
tacotron2
mirror of https://github.com/malarinv/tacotron2
1
0
Fork 0
Code Issues Releases Wiki Activity
121 Commits
10 Branches
0 Tags
3.3 MiB
Python 96.9%
Makefile 3.1%
 
 
Go to file
Malar Kannan 505af768a0 1. create a class for the tts api 
2. implement a grpc server for tts
3. add a demo client for the grpc service
4. update gitignore and requriements
5. cleanup
 		2019-07-01 14:47:55 +05:30
text text/symbols.py: updating symbols 2018-11-25 22:34:26 -08:00
waveglow@4b1001fa33 adding waveglow submodule 2018-11-27 11:53:20 -08:00
.gitignore 1. create a class for the tts api 2019-07-01 14:47:55 +05:30
.gitmodules adding waveglow submodule 2018-11-27 11:53:20 -08:00
LICENSE Update license such that it appears on repo fron tpage 2018-05-03 23:18:34 -07:00
README.md Fixed link to download waveglow from inference.py 2019-04-19 15:21:09 +09:00
WORKFLOW.md add WORKFLOW.md and update final.py 2019-06-28 14:24:36 +05:30
audio_processing.py adding python files 2018-05-03 15:16:57 -07:00
corpus.txt add a function to cache all responses from corpus.txt 2019-06-28 18:44:24 +05:30
data_utils.py data_utils.py: rewrite 2018-11-25 22:32:47 -08:00
demo_client.py 1. create a class for the tts api 2019-07-01 14:47:55 +05:30
distributed.py Fixing concatenation error for fp16 ditributed training 2019-02-01 09:55:59 +01:00
final.ipynb add a function to cache all responses from corpus.txt 2019-06-28 18:44:24 +05:30
final.py 1. create a class for the tts api 2019-07-01 14:47:55 +05:30
hparams.py 1. create a class for the tts api 2019-07-01 14:47:55 +05:30
layers.py layers.py: rewrite 2018-11-25 22:33:32 -08:00
logger.py logger.py: using new pytorch api 2019-04-03 13:35:04 -08:00
loss_function.py adding python files 2018-05-03 15:16:57 -07:00
loss_scaler.py clean-up for cpu-only inference 2019-06-28 09:46:46 +05:30
model.py model.py: renaming variables, removing dropout from lstm cell state, removing conversions now handled by amp 2019-04-03 13:36:35 -08:00
multiproc.py adding python files 2018-05-03 15:16:57 -07:00
plotting_utils.py adding python files 2018-05-03 15:16:57 -07:00
requirements.txt clean-up for cpu-only inference 2019-06-28 09:46:46 +05:30
requirements_dev.txt 1. create a class for the tts api 2019-07-01 14:47:55 +05:30
server.py 1. create a class for the tts api 2019-07-01 14:47:55 +05:30
stft.py clean-up for cpu-only inference 2019-06-28 09:46:46 +05:30
train.py clean-up for cpu-only inference 2019-06-28 09:46:46 +05:30
tts.py 1. create a class for the tts api 2019-07-01 14:47:55 +05:30
utils.py clean-up for cpu-only inference 2019-06-28 09:46:46 +05:30

README.md

Tacotron 2 (without wavenet)

PyTorch implementation of Natural TTS Synthesis By Conditioning Wavenet On Mel Spectrogram Predictions.

This implementation includes distributed and automatic mixed precision support and uses the LJSpeech dataset.

Distributed and Automatic Mixed Precision support relies on NVIDIA's Apex and AMP.

Visit our website for audio samples using our published Tacotron 2 and WaveGlow models.

Alignment, Predicted Mel Spectrogram, Target Mel Spectrogram

Pre-requisites

  1. NVIDIA GPU + CUDA cuDNN

Setup

  1. Download and extract the LJ Speech dataset
  2. Clone this repo: git clone https://github.com/NVIDIA/tacotron2.git
  3. CD into this repo: cd tacotron2
  4. Initialize submodule: git submodule init; git submodule update
  5. Update .wav paths: sed -i -- 's,DUMMY,ljs_dataset_folder/wavs,g' filelists/*.txt
    • Alternatively, set load_mel_from_disk=True in hparams.py and update mel-spectrogram paths
  6. Install PyTorch 1.0
  7. Install Apex
  8. Install python requirements or build docker image
    • Install python requirements: pip install -r requirements.txt

Training

  1. python train.py --output_directory=outdir --log_directory=logdir
  2. (OPTIONAL) tensorboard --logdir=outdir/logdir

Training using a pre-trained model

Training using a pre-trained model can lead to faster convergence
By default, the dataset dependent text embedding layers are ignored

  1. Download our published Tacotron 2 model
  2. python train.py --output_directory=outdir --log_directory=logdir -c tacotron2_statedict.pt --warm_start

Multi-GPU (distributed) and Automatic Mixed Precision Training

  1. python -m multiproc train.py --output_directory=outdir --log_directory=logdir --hparams=distributed_run=True,fp16_run=True

Inference demo

  1. Download our published Tacotron 2 model
  2. Download our published WaveGlow model
  3. jupyter notebook --ip=127.0.0.1 --port=31337
  4. Load inference.ipynb

N.b. When performing Mel-Spectrogram to Audio synthesis, make sure Tacotron 2 and the Mel decoder were trained on the same mel-spectrogram representation.

Related repos

WaveGlow Faster than real time Flow-based Generative Network for Speech Synthesis

nv-wavenet Faster than real time WaveNet.

Acknowledgements

This implementation uses code from the following repos: Keith Ito, Prem Seetharaman as described in our code.

We are inspired by Ryuchi Yamamoto's Tacotron PyTorch implementation.

We are thankful to the Tacotron 2 paper authors, specially Jonathan Shen, Yuxuan Wang and Zongheng Yang.