Speeding up Disk scans in FTP by Zaweba

Scanning a large hosting account over FTP is slow by nature. The client has to ask for each folder's listing in turn, wait for the answer, then ask for the folders inside it. On a site with tens of thousands of files that adds up. The Disk view in FTP by Zaweba has three ways to cut the wait, from the simple and always available to the faster but more specialised.

Scan connections

The first is to scan over more than one connection. Scan connections is a per-site setting in Site Manager, and 3 is described as safe on shared hosts. With several connections, the scan lists several folders at once rather than one after another.

If a host refuses an extra connection with a 421 error, the scan backs off and carries on with fewer, so a setting that is a little high will not stop the scan. Pause, Resume and Stop are there while it runs, which is useful on a big site when you need the connection for something else for a moment.

One find command over SFTP

The second applies to SFTP sites where the account has shell access. Rather than asking for folder after folder, the app runs one find command that lists the whole tree in a single go. On a big site that is a large saving, because the waiting between requests largely disappears.

Not every host allows it, and you do not have to know in advance. If the find command is not available, the scan falls back automatically to the normal crawl, so the worst case is simply the speed you would have had anyway.

The optional PHP scan

The third is an option for sites that run PHP. The app uploads a small, read-only listing script with a random name, calls it once over HTTPS with a one-time token, and deletes it straight away. The script lists the files from the server's side, which is usually far quicker than listing them over FTP.

Each part of that design has a reason. Read-only means the script cannot change anything. The random name makes it hard for anyone to find in the moment it exists. The one-time token means even someone who did find it could not use it. And deleting it straight away means nothing is left behind on the server.

It needs the site's public address, set in Site Manager, because the app has to reach the script through the website, not through FTP. If the address is missing, fill it in first.

Choosing between them

For most sites, the default crawl with scan connections at 3 is enough, and it works everywhere. If you use SFTP and your host gives shell access, you get the find command without doing anything special. Turn to the PHP scan when a site is very large, runs PHP and the normal scan takes longer than you would like.

Whichever route the scan takes, the result is the same Disk view: a treemap, a sortable file list and the biggest 100 files, coloured by type or by age, with the web-server lens pointing out space hogs and security problems.

Things to keep in mind

Be considerate on shared hosting. A host that limits connections usually does so for a reason, and raising scan connections well beyond what it accepts only produces more refusals. Leave it at 3 unless you know the host is happy with more.

If you are cautious about scripts running on a client's server, simply leave the PHP scan off. It is an option, not a requirement, and the other two methods give the same results with more patience.

Faster scans matter because Disk is most useful when you run it often. If a scan takes a few minutes rather than half an hour, you are far more likely to check a server's space before it fills up, not after.