Copy Progress Issue

Lubuntu Support
1

When copying a file from ssd to flash-drive (say an .iso file) the copy progress seems to jump straight to 90-100% and sit there while it copies. Is there any way to view the actual progress of the writing progress instead of what’s copied into the buffer as it’s not very informative with the default behaviour.

2 Likes
2

I don’t know any file manager that will do that. Most can only give the write speed. You can probably write a script to do it, the formula is something like,

Time to finish = ( size - copied )/write_speed.

The downside is that a script is inconvenient because you have to enter both source and destination.

2 Likes
3

This is the problem with most linux file managers, they are not informative enough to show real figures when copying or a correctly timed progress bar.

4

I had a possibly related issue: when using rsync --progress to copy a file from SSD to flash drive, the initial write speed was enormous, and then suddenly dropped. I received an answer on the IRC #linux channel, that Linux firsts buffers the file to RAM, then copies it to the external drive. Maybe this is the issue: the initial jump in progress is simply loading the file into RAM. I.e., the file is being written, but to RAM not to the USB drive.

1 Like
5

Yep, that’s the problem, the progress bar zips along to fill up the buffer instead of using what’s been written, giving a more accurate reading.

1 Like
6

I think this is a flaw/bug. Maybe you can submit a ticket to rsync (I presume pcmanfm-qt uses this) to have --progress query the destination about how much has been written, rather than the source.

7

It’s not really a flaw, in the sense that the OS ‘believes’ the file has been copied. If you do a listing, the file is already ‘there’ with it’s full size even though it hasn’t finished writing it.
IMO Buffered I/O smooths things out and is more apparent in linux but it’s not without some downsides.

1 Like
8

Introduction

The user’s tool has finished its task, when the data is copied to a buffer in RAM, but when the target drive is slow, there might remain a long time until the data is written from the buffer to the memory cells in the target device.

Two methods using sync

It is important to synchronize or flush the buffers, typically use the command sync to give priority to flushing the buffers. When the terminal window with sync returns to prompt, all data are copied to the memory cells.

I have used two ways to solve this problem and show the progress.

A specific method using dd

dd bs=1M status=progress oflag=dsync

This method syncs (flushes the buffer) after each batch of 1 mibibyte of data and tells you how much that is copied.

This method is used by mkusb-nox, and you can use it yourself via an alias or a small shellscript instead of typing the whole command line every time.

A general method that can be combined with all copying tools

You can watch the size of the buffer alias content of ‘dirty’ data, and from its size and change after a time interval estimate the remaining time.

In mkusb there is a shellscript for this purpose, watch-flush. The shellscript is in text mode. In mkusb its output is put into zenity windows to be shown graphically, but watch-flush can be used manually in text mode, I would recommend to start it from another terminal window right after starting sync.

There is a general description at this link.

5 Likes
9

That’s a much better method than what I was working on, it targets the write stream and is controllable.

The best I could come up with was using vmstat.
The problem I had was that it was for ‘all’ disk activity and that the copying was still buffered, so couldn’t be stopped;

#!/bin/bash
# File copy with disk(s) write summary per second

SRCE="testfile.bin"
DEST="/media/auser/SDCARD16G/testout.bin"
FSIZE=$(stat -c%s "$SRCE")
FSECTS=$(( FSIZE/512 ))
S0=$(eval vmstat -D | grep written | cut -c1-13)

cp $SRCE $DEST &

echo ctrl-c to stop monitoring
SPEED=0
CTR=0
while (( COPIED<FSECTS)); do
    sleep 1
    S2=$(eval vmstat -D | grep written | cut -c1-13)
    CTR=$(( ++CTR ))
    COPIED=$(( S2-S0 ))
    SPEED=$(( COPIED/CTR ))
    printf "\r$(tput el)"
    if (( SPEED>0 )); then
        printf $(( SPEED*512 )) | numfmt --to=iec
        printf "/s"
        printf " ETA: $(( (FSECTS-COPIED)/SPEED ))s"
    fi
done
echo;echo Finished.
1 Like
10

@humpty, I tested your method, made a shellscript (with some minor modifications), and it works. Using vmstat the way you do is an alternative to my second method with watch-flush.

But I agree, that it is a good idea to target the write stream directly, when possible.

1 Like
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.