How does gsutil rsync determine whether a file is old or new when syncing two folders?
google-compute-engine
Solution
From gsutil rsync --help:
CHANGE DETECTION ALGORITHM To determine if a file or object has changed gsutil rsync first checks whether the source and destination sizes match. If they match, it next checks if their checksums match, using checksums if available (see below).
Unlike the Unix rsync command, gsutil rsync does not use timestamps to determine if the file/object changed, because the GCS API does not permit the caller to set an object's timestamp (hence, timestamps of identical files/objects cannot be made to match).
Checksums will not be available in two cases:
When synchronizing to or from a file system. By default, gsutil does not checksum files, because of the slowdown caused when working with large files. You can cause gsutil to checksum files by using the gsutil rsync -c option, at the cost of increased local disk I/O and run time when working with large files. You should consider using the -c option if your files can change without changing sizes (e.g., if you have files that contain fixed width data, such as timestamps).
When comparing composite GCS objects with objects at a cloud provider that does not support CRC32C (which is the only checksum available for composite objects). See 'gsutil help compose' for details about composite objects.
Regards, Paolo
Problem
I'm rsyncing files to a DRA bucket, and I need to make sure that when a file is newer in the source folder it must be synced to the destination folder. Right now I am using MD5 checksum to be 100% sure, but this is too slow on a data set of 8TB with a very large amount of files. If I disable the MD5 checking, how does gsutil rsync determine whether a file should get synced or not?