Faster Download of PeopleSoft Images April 15, 2015Posted by Duncan in SelectiveAdoption, TW.
With the advent of Selective Adoption, many more people will be downloading the huge PeopleSoft Images every 10 weeks or so. They’re large (circa 35GB) and that’s going to take a while even with a decent connection.
What makes matters worse is that the default method (clicking the Download All button) runs the downloads in serial. Even on a 1MB/sec connection that’s going to take ~10 hours to download all 35GB.
In addition, the download seems to be throttled somewhat, I’m not sure why. The speed reported in the above window varied wildly between 100KB/s and 500KB/s. Even at the top end of that range, only downloading one at a time it’s going to take almost 24 hours for the complete set.
An alternative is to run the downloads in parallel. Instead of clicking Download All, click WGET Options and download the script instead. After a little modification so that it ran on Windows I was able to run 4 copies of the script side-by-side, giving gains not only by downloading in parallel, but the downloads ran faster too:
You can click for a bigger version, but basically the screenshot is of four downloads with a combined download speed of over 4MB/s! All downloads completed in a touch over 2 hours (and this is on a home broadband connection).