From jeremiah.foster at ericsson.com Mon May 5 09:35:01 2008 From: jeremiah.foster at ericsson.com (Jeremiah C. Foster) Date: Mon, 05 May 2008 11:35:01 +0200 Subject: [tap-l] Potential Perl Review article Message-ID: <1209980101.21246.9.camel@seasc0551.dyn.rnd.as.sw.ericsson.se> Hello! I am about to send an article to brian d. foy for the Perl Review on TAP. I thought I would post it here for comments and criticism. I am posting it in chunks hopefully making it more likely to be read. First chunk: Introduction ____________ "Unit testing is not optional because working software is not optional."[0] -- From a job posting Why should you care about testing? Well you shouldn't. Unless, of course, you want your software to work. If you hack a little code, run the code to check its output, then you are testing. So why not preserve those small tests in code and run it against your new module or program? If you do that, you will find that you catch a lot of unexpected bugs and learn more about how things work. Testing is an integral part of perl, as anyone who has programmed in perl for a while will tell you. When you add a new module from CPAN and you see all those "ok" messages shoot by, those are from various tests built into the module you are downloading. In fact the "ok" is the fundamental unit of perl testing showing how simple the testing syntax of perl is. If you contribute to CPAN, your software will enter a matrix of testing; from CPANTS (The CPAN testing service), to tests measuring "kwalitee", to CPAN testers at perl.org, to end users who will leave reviews on CPAN ratings - your code will be very public and well tested. Of course, you may be a seasoned software designer, forced by employers and habit to run tests. You want to run your tests in an automated fashion but you don't want to sift through the voluminous output of the test log - you want just the juicy bits; did my tests pass or fail? Whether you're a n00b or seasoned vet, TAP is the perfect thing for your toolkit. TAP is the "Test Anything Protocol" designed to run your tests and report back to you the relevant info you want - and only that. It is flexible enough to give you a complete picture of what is going on, but terse enough not to weigh you down with a novel thick enough to stun an ox every time you run your tests. TAP is a reporting layer between you and your tests. This reporting layer is agnostic as it were: it does not care what language your test is written in, as long as your program produces proper TAP output. So you can use the best tool for the job at hand. You can use a variety of programming languages and mix them together and TAP will still be able to check and report back the output. In fact we are going to use both perl and shell script (bash) to demonstrate TAP in this article showing how flexible it can be. TAP is useless of course without a test to report on, so lets create our first test. Starting with something simple for the purpose of this article we'll write a perl script that checks which operating system we are operating under. This type of test might be useful for your perl script since resources can be located in different places depending on operating system. To get the operating system, we are going to use Perl's built in variable $^0. Using Perl's built in variable for the OS helps ensure that we are going to be as cross-platform as possible since there are perl porters and other wizards working to ensure perl works on as much hardware as possible, let them do the heavy lifting for us. Jeremiah From publiustemp-perlqa3 at yahoo.com Mon May 12 09:32:22 2008 From: publiustemp-perlqa3 at yahoo.com (Ovid) Date: Mon, 12 May 2008 02:32:22 -0700 (PDT) Subject: [tap-l] TODO Tests In-Reply-To: <20080512111538.e5z8uvg9w0g88wgw@horde.wizbit.be> Message-ID: <196908.96779.qm@web65707.mail.ac4.yahoo.com> --- Bram wrote: > { > local $TODO = "test that foo() returns 1"; > local $WAS = 3; > my $s1 = foo(); > is($s1, "1"); > } I really haven't thought through all of the ramifications of this, but I like it. I was recently complaining to Schwern about the issue with TODO tests but didn't see this obvious solution :) Case in point: working on a codebase once where my only unusual test output was a TODO test and I happened to notice that the failure changed after some refactoring. This in turn led me to discover that my underlying code had a bug which my TODO test was masking. Alternatively, persistent TAP could potentially track TODO results and handle the $WAS for you, but this is quite a ways off and has the problem that we cannot always identify which tests are which. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From qa at perl.wizbit.be Mon May 12 09:15:38 2008 From: qa at perl.wizbit.be (Bram) Date: Mon, 12 May 2008 11:15:38 +0200 Subject: [tap-l] TODO Tests Message-ID: <20080512111538.e5z8uvg9w0g88wgw@horde.wizbit.be> [This idea was first submitted to p5p... See: http://www.nntp.perl.org/group/perl.perl5.porters/2008/05/msg136540.html ] While adding some todo tests (for t/op/range.t) I ran into some limitations (IMHO). Assume the following TODO test: { local $TODO = "test that foo() returns 1"; my $s1 = foo(); is($s1, "1"); } At the moment foo() returns 3. Time passes and code changes. Now there are 3 options: foo() returns 1, this will result in 'unexpected todo test passed' being outputted; foo() returns 3, no special output is produced; foo() returns 4, no special output is produced; To me the last one seems like a limitation. The result of foo() is now different then when the TODO test was written but there is nothing in the output that indicates this. (The different output could be (or most likely is) accidental or a side effect of another change.) My proposal would be to add syntax that catches this and produces an output like: Test Summary Report ------------------- op/range.t (Wstat: 0 Tests: 135 Failed: 0) TODO unexpected output: 122 (output 4 - expected 3) Files=1, Tests=135, 0 wallclock secs ( 0.05 usr 0.00 sys + 0.04 cusr 0.02 csys = 0.11 CPU) Result: PASS The important thing in this is that the result is PASS. It still is a TODO test and therefore should never produce FAIL. Regarding the syntax: David Nikol suggested: { local $TODO = "test that foo() returns 1"; local $WAS = 3; my $s1 = foo(); is($s1, "1"); } I initially suggested another syntax but I'm now in favor of David's suggestion. (Obviously this would also be applied to isn't, like, unlike, ....) Comments? Ideas? Would a patch that adds this behavior be considered? Kind regards, Bram From pagaltzis at gmx.de Mon May 12 12:32:46 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Mon, 12 May 2008 14:32:46 +0200 Subject: [tap-l] TODO Tests In-Reply-To: <196908.96779.qm@web65707.mail.ac4.yahoo.com> References: <20080512111538.e5z8uvg9w0g88wgw@horde.wizbit.be> <196908.96779.qm@web65707.mail.ac4.yahoo.com> Message-ID: <20080512123245.GS4173@klangraum.plasmasturm.org> * Ovid [2008-05-12 11:35]: > Alternatively, persistent TAP could potentially track TODO > results and handle the $WAS for you, but this is quite a ways > off and has the problem that we cannot always identify which > tests are which. Plus, you still need a way to specify which results conform to expectations and which don?t. Regards, -- Aristotle Pagaltzis // From marvin at rectangular.com Sat May 24 02:46:13 2008 From: marvin at rectangular.com (Marvin Humphrey) Date: Fri, 23 May 2008 19:46:13 -0700 Subject: [tap-l] Integrating TAP::Parser into an existing project Message-ID: <865F2DC9-C50F-4C21-BB63-F864BAA2B3D6@rectangular.com> Greets, I have an XS distro which currently does nearly all of its testing via Perl. Since I would like to build bindings for this project to other languages, it would be best if the tests could be migrated to C but still produce TAP -- that way, TAP consumers for each binding language could work off of the TAP output from the common test files. Wandering through the docs for TAP::Parser and its satellites, it wasn't clear to me how to integrate a TAP::Parser-based test instance into an existing build process. Right now, './Build test' runs everything, and I would like that regime to continue as test files get migrated one-by-one from Perl to C. I suppose that the obvious solution is to hack something like this: use Test::More; use TAP::Parser; my $test_output = qx|$c_test_executable|; my $parser = TAP::Parser->new( { tap => $test_output } ); my $plan = $parser->next; die "no plan" unless $plan && $plan->is_plan; plan( tests => $plan->tests_planned ); while ( my $result = $parser->next ) { ok( $result->is_ok, $result->as_string ); } However, the output from that is so messed up that it must not be the right way to go about things. Is there a recipe for this strategy somewhere that I managed to overlook? Marvin Humphrey Rectangular Research http://www.rectangular.com/