TestNG does not explicitly support Contract Tests, as far as I can tell, but it’s fairly easy to make it work using the little trick I describe below.

For an interface or abstract class called FooBar, make a test class called FooBarInterfaceTest or FooBarAbstractTest or whatever.

For an implementation FooBarImpl, we could make a test FooBarImplTest that just inherits from FooBarInterfaceTest. But what if FooBarImpl implements multiple interfaces, or extends an abstract class in addition to implementing one or more interfaces?

I thought about making one inner class (inside FooBarImplTest) per interface, each inheriting from the appropriate InterfaceTest. But, that doesn’t work because TestNG doesn’t recognize tests inside inner classes (or even static inner classes).

My solution is to use the TestNG @Factory annotation, which marks a method that returns a bunch of test cases in an object array. We can put such a @Factory in the implementation test class, and use it to return Contract Tests for each of the implemented interfaces or extended abstract classes.

Since the abstract tests will need to create instances of the concrete implementation being tested, we’ll need to provide the abstract test case with a factory for the test instances. I use a simple generic interface to describe the factory:

public interface TestInstanceFactory<T>
    {
    T createInstance() throws Exception;
    }

Then the Contract Test looks like this:

public abstract class FooBarInterfaceTest
    {
    private TestInstanceFactory<? extends FooBar> tif;

    public FooBarInterfaceTest(TestInstanceFactory<? extends FooBar> tif)
        {
        this.tif = tif;
        }

    @Test
    public void someFooBarMethodTest
        {
        FooBar testInstance = tif.createInstance();

        ... 
        }
    }

That creates the problem that the only constructor for the FooBarInterfaceTest requires a TestInstanceFactory argument. If FooBarInterfaceTest were a regular class, then TestNG (or at least the IntelliJ IDEA plugin) would try to instantiate it as a regular test, failing because there’s no zero-arg constructor. The trick there is to make the Contract Test class actually abstract, as indicated above, and to concretize it inline in the @Factory method in the implementation test (see below).

I put some logic for collecting all the Contract Tests relevant to a given implementation test into an abstract class (from which the implementation test will inherit):

public abstract class ContractTestAware<T>
    {
    public abstract void addContractTestsToQueue(Queue<Object> theContractTests);

    @Factory
    public Object[] instantiateAllContractTests()
        {
        Set<Object> result = new HashSet<Object>();
        Queue<Object> queue = new LinkedList<Object>();

        addContractTestsToQueue(queue);

        // recursively find all applicable contract tests up the tree
        while (!queue.isEmpty())
            {
            Object contractTest = queue.remove();
            result.add(contractTest);
            if (contractTest instanceof ContractTestAware)
                {
                ((ContractTestAware) contractTest).addContractTestsToQueue(queue);
                }
            }

        return result.toArray();
        }
    }

And finally we make the implementation test, thus:

public class FooBarImplTest extends ContractTestAware<FooBarImpl>
        implements TestInstanceFactory<FooBarImpl>
    {
    public FooBarImpl createInstance() throws Exception
        {
        return new FooBarImpl();
        }

    public void addContractTestsToQueue(Queue<Object> theContractTests)
        {
        theContractTests.add(new FooBarInterfaceTest(this){});  // this is the trick
        }
    }

Two problems remain:

1. although FooBarImplTest extends ContractTestAware, TestNG doesn’t find the inherited @Factory method. So we have to override it in FooBarImplTest:

   @Factory
   public Object[] instantiateAllContractTests()
       {
       return super.instantiateAllContractTests();
       }
   

2. As written above, the FooBarImplTest @Factory method still isn’t found because there is no @Test method present. So, in the unfortunate case that you don’t have any real tests for the implementation class, you can just do this:

   @Test
   public void bogusTest()
       {
       }
   

   that will cause the @Factory method to be found so all the Contract Tests will be run.

That’s it! Note you can add as many Contract Tests as you want for each implementation test.

Also, although I’ve emphasized using this approach to test interfaces and abstract classes, there’s no reason you couldn’t use it for regular classes as well. You could do this if you’re in a situation where a subclass ought to pass all of the tests for its superclass (perhaps in addition to tests specific to the subclass).

Note too that this solution is chainable, so the test interface/class hierarchy can mirror the real interface/class hierarchy. For instance, if the FooBar interface extends another interface, say Baz, then FooBarInterfaceTest can itself extend ContractTestAware in order to provide a BazInterfaceTest (initialized with the provided concrete factory).

public abstract class FooBarInterfaceTest extends ContractTestAware<FooBar>
    {
    private TestInstanceFactory<? extends FooBar> tif;

    public FooBarInterfaceTest(TestInstanceFactory<? extends FooBar> tif)
        {
        this.tif = tif;
        }

    public void addContractTestsToQueue(Queue<Object> theContractTests)
        {
        theContractTests.add(new BazInterfaceTest(tif){});
        }
    }

Obviously, any change to a Contract Test will be automatically applied wherever it’s appropriate. That is, by following this methodology, all concrete classes implementing the interface will be always be tested with the current version of the Contract Test.

In searching for a way to back up one of my Linux boxes to Amazon S3, I was surprised to find that none of the many backup methods and scripts I found on the net did what I wanted, so I wrote yet another one.

The design requirements were:

• Occasional full backups, and daily incremental backups
• Stream data to S3, rather than making a local temp file first (i.e., if I want to archive all of /home at once, there’s no point in making huge local tarball, doing lots of disk access in the process)
• Break up large archives into manageable chunks
• Encryption

As far as I could tell, no available backup script (including, e.g. s3sync, backup-manager, s3backup, etc. etc.) met all four requirements.

The closest thing is js3tream, which handles streaming and splitting, but not incrementalness or encryption. Those are both fairly easy to add, though, using tar and gpg, as suggested by the js3tream author. However, the s3backup.sh script he provides uses temp files (unnecessarily), and does not encrypt. So I modified it a bit to produce s3backup-gpg-streaming.sh.

That’s not the end of the story, though, since it leaves open the problem of managing the backup rotation. I found the explicit cron jobs suggested on the js3tream site too messy, especially since I sometimes want to back up a lot of different directories. Some other available solutions will send incremental backups to S3, but never purge the old ones, and so use ever more storage.

Finally, I wanted to easily deal with MySQL and Subversion dumps.

The solution

I wrote s3napback, which wraps js3tream and solves all of the above issues by providing:

• Dead-simple configuration
• Automatic rotation of backup sets
• Alternation of full and incremental backups (using “tar -g”)
• Integrated GPG encryption
• No temporary files used anywhere, only pipes and TCP streams (optionally, uses smallish temp files to save memory)
• Integrated handling of MySQL dumps
• Integrated handling of Subversion repositories, and of directories containing multiple Subversion repositories.

It’s not rocket science, just a wrapper that makes things a bit easier.

Check out the project page for more info and to download it!

I have two 3ware 8506-4 controllers with three disks each connected to them. I’ve been getting high iowait on my box whenever there’s an even slightly disk-intensive task going on. The weird thing is that even processes that don’t use the high-usage disk get stalled in iowait. As a result, load skyrockets and the system becomes unresponsive.

Inducing disk load with bonnie++ and watching things with “iostat -x” revealed that even disks with extremely low bandwidth usage and very few i/o requests end up with 100% utilization. However, only those disks on the same controller as the high-usage disk are affected. The observed total throughput is around 50 Mb/sec, e.g., the limit of the high-usage disk. It’s nowhere near the bandwidth limit of the controller, the SATA channel, or the PCI bus. So, there’s something wrong with the controller that effectively limits it to control one disk at a time with reasonable performance.

The kernel is the current CentOS 3.6 one, 2.4.21-37.0.1.ELsmp.

The write cache is enabled on the controllers.

This seems to be related to at least some of the posts on the infamous RHEL vs. 3ware performance bug.

The Diagnosis

Apparently, what is happening is that the four SATA channels on the controller share a single request queue, with a depth of 256 requests. When requests to one disk arrive at a high rate, they fill the queue, thereby starving the other disks.

Controllers from other manufacturers typically limit the number of requests to a single disk to some number smaller than the total queue depth for the controller. Thus, the queue for a single disk can get full without blocking access to the other disks. Why 3ware made the default per-disk queue depth the same as the depth for the whole controller is beyond me. Fortunately, this is correctable.

The Solution

I googled around for some hints.

In 2.6 kernels, the queue depth can be adjusted using the sysfs mechanism (though that may require a patch to the 3w-xxxx driver). However, I need to continue running a 2.4 kernel for a while. So, the solution is to rebuild the driver with a different queue depth (and the kernel image, since I need to boot from these disks).

First I ran a bunch of tests to get a performance baseline using various readahead and elvtune settings:

3wareIssueTests.sh

Then I downloaded the kernel source, configured it, and added

CONFIG_3W_XXXX_CMD_PER_LUN=32

to /usr/src/linux-2.4/.config

Then, bypassing much of the standard stuff for a custom kernel since this is a simple change:

rm -f drivers/scsi/3w-xxxx.o
make modules
mv /lib/modules/2.4.21-37.0.1.ELsmp/kernel/drivers/scsi/3w-xxxx.o /lib/modules/2.4.21-37.0.1.ELsmp/kernel/drivers/scsi/3w-xxxx.o.orig
cp /usr/src/linux-2.4/drivers/scsi/3w-xxxx.o /lib/modules/2.4.21-37.0.1.ELsmp/kernel/drivers/scsi
mkinitrd /boot/initrd-2.4.21-37.0.1.ELsmp-3ware32b.img 2.4.21-37.0.1.ELsmp 

Then add a config with the new image to /etc/grub.conf, using the old image as a fallback:

default 0
fallback 1

And reboot.

Then run 3wareIssueTests.sh again and watch the load average. Hmm, that didn’t work !? The original symptoms persist, and /proc/scsi/3w-xxxx/1 shows Max commands posted: 254, which shouldn’t happen.

Added

printk(KERN_WARNING "3w-xxxx: set cmd_per_lun to %d.\n", tw_host->cmd_per_lun);

at line 1177 of 3w-xxxx.c

Aha, it’s still 254. The .config change didn’t get picked up (maybe I needed make dep or something) so we can just add the compile flag manually:

gcc -D__KERNEL__ -I/usr/src/linux-2.4.21-37.0.1.EL/include -Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing -fno-common  -Wno-unused -fomit-frame-pointer -pipe -freorder-blocks -mpreferred-stack-boundary=2 -march=i686 -DMODULE -DMODVERSIONS -include /usr/src/linux-2.4.21-37.0.1.EL/include/linux/modversions.h  -nostdinc -iwithprefix include -DKBUILD_BASENAME=3w_xxxx -DCONFIG_3W_XXXX_CMD_PER_LUN=32 -c -o 3w-xxxx.o 3w-xxxx.c

Then build a new image, reboot, and run the tests again as above.

Awesome, that worked. iowait is still kinda high during bonnie, but that’s to be expected. Load goes to 3, but not 12. Disks on the same controller as the test disk appear to be unaffected.

The various readahead and elvtune settings seem to have minimal effect; I’ll just leave them as:

blockdev --setra 120 /dev/sde
elvtune -r 2048 -w 8192 /dev/sde

Finally, other i/o performance tidbits:

• The disks are mounted with noatime
• Apache uses a single access log file
• Added echo 100 5000 640 2560 150 30000 5000 1884 2 > /proc/sys/vm/bdflush to /etc/rc.d/rc.local, as proposed on this handy tuning page

So my solution is:

1. Confirm that my latest daily Retrospect backup worked.
2. Delete old & large files; DiskInventoryX helps a lot to identify them.
3. Quick Defragment in iDefrag (oops, superfluous given later steps). Holy crap, it crashed in the middle of defragmenting! How nervewracking is that!? OK, no more iDefrag. Sheesh.
4. Optimize/repair in Disk Utility
5. Carbon Copy Cloner the disk to an image on another drive (using Target Disk Mode so the drive is not the boot disk)
6. Repartition to make a 2Gb Swap partition at the beginning of the disk (Maybe I could have done this without the cloning step using iPartition, but that’s expensive).
7. Carbon Copy Cloner from the image back to the (now slightly smaller) main partition. This should automatically defrag everything too.
8. Actually move the swap files to that partition using Xupport
9. Quit Mail.app, Delete ~/Mail/Envelope Index, start it up again and rebuild
10. Full Spotlight reindex of the disk (automatic, since CCC deletes the index)
11. Disable unneeded services… e.g., I had MySQL running but I don’t really use it
12. Mail.app prefs: delete messages immediately; etc.