How to Transfer Large Amounts of Data on a PC or Server

Here are several examples of tools available for DIYers and IT professionals.  These tools are used for transferring from server to server or PC to PC.  We begin with the basic tools and end with a more elaborate, somewhat confusing to a new person but ultimately, the best, fastest, and most reliable tool out there for an individual to transfer large amounts of data between two computers or servers.  This is actually step one of a two or three step process.

 

DirSycPro

I would use DirSync Pro.  It is flexible, easy and no need to bother with scripting.  Simply specify which directories to transfer and turn on verification then let it do its job.

http://www.dirsyncpro.org/

7zip

Remember zipping files?  It’s still a very common approach to taking large amounts data, compressing, and zipping it into one file.  We zip files because the standard copy option on Windows will break when trying to copy large amounts of data.

https://www.7-zip.org/download.html

Transfer Large Amounts of Data

 

 

 

 

 

The path of the saved 7zip file is c:\users\user_name\documents.7z if you are referencing the above illustration.

 

Robocopy

You know the person assisting is a true IT professional if they are utilizing robocopying for copying large amounts of data.

The number of files you’re copying can make a HUGE difference in how long it takes.  Copying two 30GB files is much, much faster than copying 30,000 2MB files, even though they are both the same total size.

The command below will copy all contents of \\NAS1\Clients to \\NAS2\Clients, including all subdirectories (even empty ones).  The /R:1 and /W:1 parameters tell it to retry once on any failures to copy, and wait for 1 second between each retry.

robocopy \\NAS1\Clients \\NAS2\Clients /e /r:1 /w:1

After the command is complete, run the exact same command again and it will automatically copy over any new/changed files, and complete much, much faster.

This is what I use pretty much anytime I need to copy a large amount of data to one place or another.  Run it once, takes a while, run it a second time to catch anything that changed during the hours the first one ran.

——————————————————————————-
ROBOCOPY     ::     Robust File Copy for Windows
——————————————————————————-

Started : Tue Feb 02 15:06:27 2016

Usage :: ROBOCOPY source destination [file [file]…] [options]

source :: Source Directory (drive:\path or \\server\share\path).
destination :: Destination Dir  (drive:\path or \\server\share\path).
file :: File(s) to copy  (names/wildcards: default is “*.*”).

::
:: Copy options :
::
/S :: copy Subdirectories, but not empty ones.
/E :: copy subdirectories, including Empty ones.
/LEV:n :: only copy the top n LEVels of the source directory tree.

/Z :: copy files in restartable mode.
/B :: copy files in Backup mode.
/ZB :: use restartable mode; if access denied use Backup mode.
/EFSRAW :: copy all encrypted files in EFS RAW mode.

/COPY:copyflag[s] :: what to COPY for files (default is /COPY:DAT).
(copyflags : D=Data, A=Attributes, T=Timestamps).
(S=Security=NTFS ACLs, O=Owner info, U=aUditing info).

/DCOPY:T :: COPY Directory Timestamps.

/SEC :: copy files with SECurity (equivalent to /COPY:DATS).
/COPYALL :: COPY ALL file info (equivalent to /COPY:DATSOU).
/NOCOPY :: COPY NO file info (useful with /PURGE).

/SECFIX :: FIX file SECurity on all files, even skipped files.
/TIMFIX :: FIX file TIMes on all files, even skipped files.

/PURGE :: delete dest files/dirs that no longer exist in source.
/MIR :: MIRror a directory tree (equivalent to /E plus /PURGE).

/MOV :: MOVe files (delete from source after copying).
/MOVE :: MOVE files AND dirs (delete from source after copying).

/A+:[RASHCNET] :: add the given Attributes to copied files.
/A-:[RASHCNET] :: remove the given Attributes from copied files.

/CREATE :: CREATE directory tree and zero-length files only.
/FAT :: create destination files using 8.3 FAT file names only.
/256 :: turn off very long path (> 256 characters) support.

/MON:n :: MONitor source; run again when more than n changes seen.
/MOT:m :: MOnitor source; run again in m minutes Time, if changed.

/RH:hhmm-hhmm :: Run Hours – times when new copies may be started.
/PF :: check run hours on a Per File (not per pass) basis.

/IPG:n :: Inter-Packet Gap (ms), to free bandwidth on slow lines.

/SL :: copy symbolic links versus the target.

/MT[:n] :: Do multi-threaded copies with n threads (default 8).
n must be at least 1 and not greater than 128.
This option is incompatible with the /IPG and /EFSRAW options.
Redirect output using /LOG option for better performance.

::
:: File Selection Options :
::
/A :: copy only files with the Archive attribute set.
/M :: copy only files with the Archive attribute and reset it.
/IA:[RASHCNETO] :: Include only files with any of the given Attributes set.
/XA:[RASHCNETO] :: eXclude files with any of the given Attributes set.

/XF file [file]… :: eXclude Files matching given names/paths/wildcards.
/XD dirs [dirs]… :: eXclude Directories matching given names/paths.

/XC :: eXclude Changed files.
/XN :: eXclude Newer files.
/XO :: eXclude Older files.
/XX :: eXclude eXtra files and directories.
/XL :: eXclude Lonely files and directories.
/IS :: Include Same files.
/IT :: Include Tweaked files.

/MAX:n :: MAXimum file size – exclude files bigger than n bytes.
/MIN:n :: MINimum file size – exclude files smaller than n bytes.

/MAXAGE:n :: MAXimum file AGE – exclude files older than n days/date.
/MINAGE:n :: MINimum file AGE – exclude files newer than n days/date.
/MAXLAD:n :: MAXimum Last Access Date – exclude files unused since n.
/MINLAD:n :: MINimum Last Access Date – exclude files used since n.
(If n < 1900 then n = n days, else n = YYYYMMDD date).

/XJ :: eXclude Junction points. (normally included by default).

/FFT :: assume FAT File Times (2-second granularity).
/DST :: compensate for one-hour DST time differences.

/XJD :: eXclude Junction points for Directories.
/XJF :: eXclude Junction points for Files.

::
:: Retry Options :
::
/R:n :: number of Retries on failed copies: default 1 million.
/W:n :: Wait time between retries: default is 30 seconds.

/REG :: Save /R:n and /W:n in the Registry as default settings.

/TBD :: wait for sharenames To Be Defined (retry error 67).

::
:: Logging Options :
::
/L :: List only – don’t copy, timestamp or delete any files.
/X :: report all eXtra files, not just those selected.
/V :: produce Verbose output, showing skipped files.
/TS :: include source file Time Stamps in the output.
/FP :: include Full Pathname of files in the output.
/BYTES :: Print sizes as bytes.

/NS :: No Size – don’t log file sizes.
/NC :: No Class – don’t log file classes.
/NFL :: No File List – don’t log file names.
/NDL :: No Directory List – don’t log directory names.

/NP :: No Progress – don’t display percentage copied.
/ETA :: show Estimated Time of Arrival of copied files.

/LOG:file :: output status to LOG file (overwrite existing log).
/LOG+:file :: output status to LOG file (append to existing log).

/UNILOG:file :: output status to LOG file as UNICODE (overwrite existing log).
/UNILOG+:file :: output status to LOG file as UNICODE (append to existing log).

/TEE :: output to console window, as well as the log file.

/NJH :: No Job Header.
/NJS :: No Job Summary.

/UNICODE :: output status as UNICODE.

::
:: Job Options :
::
/JOB:jobname :: take parameters from the named JOB file.
/SAVE:jobname :: SAVE parameters to the named job file
/QUIT :: QUIT after processing command line (to view parameters).
/NOSD :: NO Source Directory is specified.
/NODD :: NO Destination Directory is specified.
/IF :: Include the following Files.

 

 

Additional Resources:

Need to more products out there to transfer large amounts of data?

RoboCopy vs RichCopy vs SyncBack vs EMCopy vs XXCopy vs XCopy vs KillCopy vs FastCopy vs Rsync vs TeraCopy vs 7zip

http://mfctips.com/2013/11/01/robocopy-vs-richcopy-vs-syncback-vs-emcopy/

 

Transfer Large Amounts of Data Summary

 

This is step one of a two or three step process.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *