The only was to download the vagrant box faster is to do it manually. Before doing vagrant up, just do the following steps.

Add box manually

The box download is often slow. Download the box from the URL the official download is using for example:

wget "https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20200701.0.0/providers/virtualbox.box"

Use the following command to add the box:

vagrant box add ubuntu/bionic64 bionic-server-cloudimg-amd64-vagrant.box

Change the name ubuntu/bionic64 and the file name according to the box.