Page 1 of 1

incremental snapshot

Posted: Wed Apr 19, 2017 8:55 pm
by jeremydunn
I have purchased and am using Automaticloud, and generally very happy with it. We are using the option to copy snapshot to another region. However the copy-snapshot always copies a NEW snapshot, and the data-transfer costs are very high: 4 EC2 volumes totalling 350gb x ~30 day/month = 10.5TB/month @ .02 / GB for data transfer between regions = US$210/month for TRANSFER alone, which is about the same price as the EC2 instances themselves! (and much more expensive than the S3 storage).

Can you suggest a way to make use of Amazon incremental-snapshot feature, while at the same time transferring snapshot to another region, from within Automaticloud ?

http://docs.aws.amazon.com/AWSEC2/lates ... pshot.html
"If you make periodic snapshots of a volume, the snapshots are incremental so that only the blocks on the device that have changed after your last snapshot are saved in the new snapshot."

But this behavior is not happening when we use Automaticloud snapshot + copy to another region.

http://www.n2ws.com/how-to-guides/copyi ... ve-ha.html

"It is very important to note that the snapshot copy functionality follows the incremental backup model of a snapshot. The first snapshot copy of a volume is always a full copy. Each subsequent snapshot copy is incremental. "

From these statements it appears that if we *re-used* the local snapshot, then when we copied-to-another-region, it would be incremental, which is the desired result.

==> So is there a way to re-use the same snapshot ID when making the snapshot which will then be copied to another region?

Again the goal is to reduce inter-region data-transfer costs by making use of incremental-snapshot-copy feature of AWS.

Thanks!
Jeremy

Re: incremental snapshot

Posted: Wed Apr 19, 2017 10:57 pm
by admin
AWS documentation says:
The first snapshot copy to another region is always a full copy. Each subsequent snapshot copy is incremental (which makes the copy process faster), meaning that only the blocks in the snapshot that have changed after your last snapshot copy to the same destination are transferred. Support for incremental snapshots is specific to a region pair where a previous complete snapshot copy of the source volume is already available in the destination region, and it is limited to the default EBS CMK for encrypted snapshots. For example, if you copy an unencrypted snapshot from the US East (N. Virginia) region to the US West (Oregon) region, the first snapshot copy of the volume is a full copy and subsequent snapshot copies of the same volume transferred between the same regions are incremental.
This process seems to happen on storage level and I don't see a way to control (or break) it.

Can you please post a snapshot of your job definiton. Are you sure that the transfer volume is not caused by changed blocks on your volumes? Do you see the full size of your volumes in your transfer volume?

Stephan

Re: incremental snapshot

Posted: Thu Apr 20, 2017 1:52 pm
by jeremydunn
> Can you please post a snapshot of your job definiton.
see attached. note that we are deleting the snapshot after 1 day. I wonder if this deletion is causing the problem. But when we did _not_ delete, then the snapshots accumulated (new snapshot ID daily).

In either case it is not behaving as the documentation states, doing incremental copies.

> Are you sure that the transfer volume is not caused by changed blocks on your volumes? Do you see the full size of your volumes in your transfer volume?

yes, I am sure. we are doing snapshot-copy-to-other-region for (4) instances with total 350MB volume. 350 x 30 days = 10,500 GB.
We are currently about 1/2 way thru our monthly billing cycle, and total month-to-date transfer is 5684GB. So it seems the entire disk volume of all four instances is transferred daily.

Re: incremental snapshot

Posted: Thu Apr 20, 2017 2:00 pm
by jeremydunn
> meaning that only the blocks in the snapshot that have changed after your last snapshot copy to the same destination are transferred.

the problem is, it's a _new_ snapshot each day. If it were the same snapshot, I think the incremental-snapshot-copy would be working.

Is there any way to configure Automaticloud to re-use or update the same snapshot, instead of taking a new snapshot?

Re: incremental snapshot

Posted: Tue May 02, 2017 10:29 pm
by admin
I have set up a test case with a "Create Snapshot Job" in AutomatiCloud for a daily snapshot of an 8 GB volume. The initial snapshot is created in US-West2 and then copied to region US-East1. The transfer volume for the first snapshot was 0,96 GB. The second and third were 0,00006315 GB.
This is exactly what we should expect because the first snapshot contains a compressed set of all blocks and all subsequent snapshots are only compressed deltas.
Please try to keep the snapshots for a few days and see if this helps.
AWS does not offer public tools to monitor the snapshot mechanism. The only way to find out what is happening with your volumes and snapshots will be to open a support case with Amazon.

Stephan

Re: incremental snapshot

Posted: Sat May 13, 2017 3:11 am
by jeremydunn
admin wrote:I have set up a test case with a "Create Snapshot Job" in AutomatiCloud for a daily snapshot of an 8 GB volume. The initial snapshot is created in US-West2 and then copied to region US-East1. The transfer volume for the first snapshot was 0,96 GB. The second and third were 0,00006315 GB. This is exactly what we should expect because the first snapshot contains a compressed set of all blocks and all subsequent snapshots are only compressed deltas.
Thank you for taking time to do this trial. I appreciate your help very much.

> Please try to keep the snapshots for a few days and see if this helps.

I started this experiment on 6-May, and have been monitoring the daily snapshot transfer volume. See attached screenshot. It seems that making the change to keep 3 days on the remote location, fixed the problem on one instance (20GB linux); but not on the other instance (125 GB windows). Any insight into why?
DailySnapshotTransferBytes.PNG
DailySnapshotTransferBytes.PNG (101.97 KiB) Viewed 4333 times

Re: incremental snapshot

Posted: Sat May 13, 2017 10:15 am
by admin
Unfortunately Amazon offers no tools to sees what's happening behind the scenes of their replication. You should open a support case and ask them for help. As these numbers are relevant for the invoicing they should handle it with priority even if you don't have a support contract.

Stephan

Re: incremental snapshot

Posted: Thu May 18, 2017 5:43 pm
by jeremydunn
admin wrote:Unfortunately Amazon offers no tools to sees what's happening behind the scenes of their replication. You should open a support case and ask them for help. As these numbers are relevant for the invoicing they should handle it with priority even if you don't have a support contract.

Stephan
we opened a support case, and found out the reason. Posting it here in case it helps others:

AWS Support wrote:
> the snapshot is encrypted using customer's KMS key. The snapshot then copied to (remote region) and re-encrypted using Ebs default KMS key. In this case a full copy will be performed each time by design. Note in the document http://docs.aws.amazon.com/AWSEC2/lates ... pshot.html , this is pointed out already.

> However, changing the encryption status of a snapshot or using a non-default EBS CMK during a copy operation always results in a full copy (not incremental), which may incur greater data transfer and storage charges." You have two kinds of snapshots, a small one 20GB and a big one 125GB. The small one is incremental because it is not encrypted. The big one is always full copy because of the above reason. So if the customer want incremental copy, there are two ways: 1) not encrypt the snapshot, or 2) only use ebs default KMS key to encrypt both source snapshot and the snapshots in the remote location.

=====
I've looked at the options, and option (1) "not encrypt the snapshot" does not appear to be possible, either from within AWS/EC2 console, or from Automaticloud. So our only options are (2) change to use default EBS KMS key (which would mean re-encrypting the whole volume); or else (3) just accept high data transfer costs / full snapshot each time.