2. update streamlit dep to 1.0
3. add dev optional dep key
4. implement mono diarized dataset generation script
5. enable gpu support on asr transformers inference pipeline
6. use typer logging
7. clean-up annotation ui with everything other than asr-data keys as optional(including plots)
8. implement chunk_transcribe_meta_gen abstraction for asr chunking logic
9. make ui_persist compatibility change for streamlit 1.0
10. add diarize commands(bugfix)
11. add notebooks for diarization
2. set sample width to 1 for wav2vec2 training data export from jasper
3. add 'audio_seg' key to asr_manifest_reader
4. add alpha rules
5. bugfixes and tests
2. added training script
fix module packaging issue
implement call audio data recycler for asr
1. added streamlit based validation ui with mongodb datastore integration
2. fix asr wrong sample rate inference
3. update requirements
1. refactored streamlit code
2. fixed issues in data manifest handling
refresh to next entry on submit and comment out mongo clearing code for safety :P
add validation ui and post processing to correct using validation data
1. added a tool to extract asr data from gcp transcripts logs
2. implement a funciton to export all call logs in a mongodb to a caller-id based yaml file
3. clean-up leaderboard duration logic
4. added a wip dataloader service
5. made the asr_data_writer util more generic with verbose flags and unique filename
6. added extendedpath util class with json support and mongo_conn function to connect to a mongo node
7. refactored the validation post processing to dump a ui config for validation
8. included utility functions to correct, fill update and clear annotations from mongodb data
9. refactored the ui logic to be more generic for any asr data
10. updated setup.py dependencies to support the above features
unlink temporary files after transcribing
1. clean-up unused data process code
2. fix invalid sample no from mongo
3. data loader service return remote netref
1. added training utils with custom data loaders with remote rpyc dataservice support
2. fix validation correction dump path
3. cache dataset for precaching before training to memory
4. update dependencies
1. implement dataset augmentation and validation in process
2. added option to skip 'incorrect' annotations in validation data
3. added confirmation on clearing mongo collection
4. added an option to navigate to a given text in the validation ui
5. added a dataset and remote option to trainer to load dataset from directory and remote rpyc service
1. added utility command to export call logs
2. mongo conn accepts port
refactored module structure
1. enabled silece stripping in chunks when recycling audio from asr logs
2. limit asr recycling to 1 min of start audio to get reliable alignments and ignoring agent channel
3. added rev recycler for generating asr dataset from rev transcripts and audio
4. update pydub dependency for silence stripping fn and removing threadpool hardcoded worker count
1. added support for mono/dual channel rev transcripts
2. handle errors when extracting datapoints from rev meta data
3. added suport for annotation only task when dumping ui data
cleanup rev recycle
added option to disable plots during validation
fix skipping null audio and add more verbose logs
respect verbose flag
don't load audio for annotation only ui and keep spoken as text for normal asr validation
1. refactored wav chunk processing method
2. renamed streamlit to validation_ui
show duration on validation of dataset
parallelize data loading from remote
skipping invalid data points
1. removed the transcriber_pretrained/speller from utils
2. introduced get_mongo_coll to get the collection object directly from mongo uri
3. removed processing of correction entries to remove space/upper casing
refactor validation process arguments and logging
1. added a data extraction type argument
2. cleanup/refactor
1. using dataname args for update/fill annotations
2. rename to dump_ui
added support for name/dates/cities call data extraction and more logs
handling non-pnr cases without parens in text data
1. added conv data generator
2. more utils
1. added start delay arg in call recycler
2. implement ui_dump/manifest writer in call_recycler itself
3. refactored call data point plotter
4. added sample-ui task-ui on the validation process
5. implemented call-quality stats using corrections from mongo
6. support deleting cursors on mongo
7. implement multiple task support on validation ui based on task_id mongo field
fix 11st to 11th in ordinal
stripping silence on call chunk
1. added option to strip silent chunks
2. computing caller quality based on task-id of corrections
1. fix update-correction to use ui_dump instead of manifest
2. update training params no of checkpoints on chpk frequency
1. split extract all data types in one shot with --extraction-type all flag
2. add notes about diffing split extracted and original data
3. add a nlu conv generator to generate conv data based on nlu utterances and entities
4. add task uid support for dumping corrections
5. abstracted generate date fn
1. added a test generator and slu evaluator
2. ui dump now include gcp results
3. showing default option for more args validation process commands
added evaluation command
clean-up
Malar Kannan
wabi_sabi004