Monday 17 September 2012

VMware ESXi fastcopy.py to quickly copy .vmdk files


Overview


fastcopy.py is a python script for VMware ESXi 4.0 - 5.x that can be used as a replacement for "cp -R" to copy Virtual Machines.
fastcopy use cp to copy normal files but use vmkfstools to copy .vmk files.
Because vmkfstools is 10 to 20 time faster to copy such files, you will see a big difference.
When copying .iso files you will not notice any difference !

The typical use of fastcopy.py is to copy multiple directories containing VMs to a target directory.
fastcopy.py don't handle recursive copy or simple file copy. Sources and target must be directories.

Usage

For example top copy one VM to a new directory :

./fastcopy.py  /vmfs/volumes/SATA1/windows2008 /vmfs/volumes/NEWSATA

or if you want to copy multiple VMs, just list all VM directories. The last one must be the target one.

./fastcopy.py  /vmfs/volumes/SATA1/windows2008 /vmfs/volumes/SATA1/linux /vmfs/volumes/NEWSATA


Installation

Cut&Paste script below to one of your datastore and don't  forget to make it executable using chmod :

chmod a+x

Here is fastcopy.py

#!/bin/python
# fastcopy.py
# copy VMware virtual machine directories using cp and vmkfstools
# (c) alain.spineux@gmail.com
# released under GPL
# usage: fastcopy.py vm_dir1 vm_dir2 ... target_dir
#
# ATTN: don't copy recursively

import sys, os, subprocess, re

def copy(srcdir, dstdir):
    skipped=0
    for filename in os.listdir(srcdir):
        fullfilename=os.path.join(srcdir, filename)
        if os.path.isdir(fullfilename):
            print fullfilename, 'SKIPPED'
            skipped+=1
            continue
        print filename
        if filename.endswith('-flat.vmdk'):
            # this is the data file, it will be copied by the .vmdk
            continue
        if re.match('.*-s[0-9]{3}.vmdk$', filename):
            # this is part of a sparse file, it will be copied by the .vmdk
            continue

        # dont use vmkfstools for snapshot files
        if filename.endswith('.vmdk') and \
            not re.match('.*-[0-9]{6}-delta.vmdk$', filename) and \
            not re.match('.*-[0-9]{6}.vmdk$', filename):
            args=['vmkfstools', '-i', fullfilename, os.path.join(dstdir, filename) ]
        else:
            args=['cp', fullfilename, os.path.join(dstdir, filename) ]
        # print args
        subprocess.call(args)
    return skipped

if len(sys.argv)<3:
    print 'Usage: fastcopy.py src_dir... target_dir'
    sys.exit(1)

srcdirs=map(lambda x:x.rstrip('/'), sys.argv[1:-1])
dstdir=sys.argv[-1].rstrip('/')

if not os.path.isdir(dstdir):
    print 'dst_dir must be a directories'
    sys.exit(1)

for srcdir in srcdirs:
    if not os.path.isdir(srcdir):
        print 'not a directory:', srcdir
        sys.exit(1)
    targetdir=os.path.join(dstdir, os.path.basename(srcdir))

    if os.path.exists(targetdir):
        print 'target dir already exists:', targetdir
        sys.exit(1)

skipped=0
for srcdir in srcdirs:
    targetdir=os.path.join(dstdir, os.path.basename(srcdir))
    os.mkdir(targetdir)
    skipped+=copy(srcdir, targetdir)

if skipped>0:
    print "SKIPPED:", skipped

8 comments:

  1. Thanks a lot for your script! I just did my migration in 30 min, previously it was taking hours...

    ReplyDelete
  2. Thanks heaps for this script too! It saved heaps of pain.
    We've just slightly modified it to be able to backup entire datastore with some smart logic around overwriting previously backup VMs.

    If you're interested in, you can get our version from here: http://pastebin.com/da7HxVug

    ReplyDelete
  3. python3 version for 6.5
    https://pastebin.com/kJjbSTFT

    ReplyDelete
  4. i think it is a good idea to copy it into a thin disk


    i had changed the comman to
    args=['vmkfstools', '-i', fullfilename, '-d', 'thin' , os.path.join(dstdir, filename) ]


    python3 with this change:
    https://pastebin.com/RFC0U4Ud

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Why did I not read the comments before doing this myself in vim which I barely know how to use.

      Thanks for sharing, wish I found it sooner!

      Delete
  5. Why is vmware giving me an error that

    -sh: ./fastcopy.py: not found


    It is there in the system as well

    ReplyDelete
  6. i've these days commenced a blog, the information you pay for more or less this website online has helped me substantially. thanks for every considered one of certainly one of of a while & discharge loyalty. photocopy machine

    ReplyDelete