Jonas Bandi writes in his CLOSED-LOOP blog a nice entry on Restarting the embedded OpenEJB container between each test. Jonas documents the use of a newer property "openejb.embedded.initialcontext.close" which allows you to destroy the embedded container system after each test.

Note that Maven users can use Surefire Forking to achieve a similar result that will also clear out any embedded database state as well. That approach basically gives you a fresh VM for every test and the approach we use internally in our build. Haven't done any speed comparisons to see how they stack up speed wise.