Produced checkpoints are using absolute path where they should really be using relative ones.
Checkpoints FR STT 0.9
# ./best_dev_checkpoint
model_checkpoint_path: "/mnt/checkpoints/best_dev-221133"
all_model_checkpoint_paths: "/mnt/checkpoints/best_dev-221133"
Here is English checkpoint for reference.
# ./best_dev_checkpoint
model_checkpoint_path: "best_dev-3663881"
all_model_checkpoint_paths: "best_dev-3663881"
Fix should probably go into package.sh I think.
Related : coqui-ai/STT#2338
^ STT's lm_optimizer throws a NotFoundError from TF, since the path points to nowhere in transfer-learning configuration, as checkpoint path is /transfer-checkpoint/ instead of /mnt/checkpoints/.
|
To use an existing checkpoint, just ensure the `docker run` includes a mount such as: |
|
`type=bind,src=PATH/TO/CHECKPOINTS,dst=/transfer-checkpoint`. Upon running, data |
|
will be copied from that place. |
|
if [ -f "/transfer-checkpoint/checkpoint" -a ! -f "/mnt/models/output_graph.pb" ]; then |
|
echo "Using checkpoint from ${TRANSFER_CHECKPOINT}" |
|
cp -a /transfer-checkpoint/* /mnt/checkpoints/ |
|
fi; |
That was for DS but with STT, we have access to --load_checkpoint_dir and --save_checkpoint_dir flags instead.
if [ -f "/transfer-checkpoint/checkpoint" -a ! -f "/mnt/models/output_graph.tflite" ]; then
echo "Using checkpoint from ${TRANSFER_CHECKPOINT}"
# use --load_checkpoint_dir for transfer learning
LOAD_CHECKPOINT_FROM="--load_checkpoint_dir /transfer-checkpoint --save_checkpoint_dir /mnt/checkpoints"
else
LOAD_CHECKPOINT_FROM="--checkpoint_dir /mnt/checkpoints/"
fi;
Example from my branch stt140-cv9
Produced checkpoints are using absolute path where they should really be using relative ones.
Checkpoints FR STT 0.9
Here is English checkpoint for reference.
Fix should probably go into
package.shI think.Related : coqui-ai/STT#2338
^ STT's
lm_optimizerthrows aNotFoundErrorfrom TF, since the path points to nowhere in transfer-learning configuration, as checkpoint path is/transfer-checkpoint/instead of/mnt/checkpoints/.commonvoice-fr/DeepSpeech/CONTRIBUTING.md
Lines 76 to 78 in 5699e59
commonvoice-fr/DeepSpeech/train.sh
Lines 14 to 17 in 5699e59
That was for DS but with STT, we have access to
--load_checkpoint_dirand--save_checkpoint_dirflags instead.Example from my branch stt140-cv9