From david at kineticode.com Wed Jun 11 18:53:52 2008 From: david at kineticode.com (David E. Wheeler) Date: Wed, 11 Jun 2008 11:53:52 -0700 Subject: [tap-l] pgTAP Message-ID: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> Hey all, I just wanted to mention that I've "ported" Test::More to PostgreSQL. Yes, you heard right. Details here: http://justatheory.com/computers/databases/postgresql/introducing_pgtap.html The project page for it should appear here in a day or two: http://pgfoundry.org/projects/pgtap/ It includes a Perl app, pg_prove, that works just like prove, but use psql under the hood, and has relevant options for it. Thanks to Ovid and Andy for TAP::Harness -- it made it really easy! But a quick question: I'm going to start writing SQL unit tests in an Catalyst app I'm building. I'm thinking that the SQL tests will be right next to the Perl tests. So they'd be t/*.s to complement the t/ *.t Perl tests. So the question: How do I modify `make test` and/or `./ Build test` so that they can use a single TAP::Harness to process both types of tests, but send *.t tests to Perl and *.s tests to psql? Thanks! David From shlomif at iglu.org.il Wed Jun 11 21:57:35 2008 From: shlomif at iglu.org.il (Shlomi Fish) Date: Thu, 12 Jun 2008 00:57:35 +0300 Subject: [tap-l] pgTAP In-Reply-To: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> References: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> Message-ID: <200806120057.36106.shlomif@iglu.org.il> On Wednesday 11 June 2008, David E. Wheeler wrote: > Hey all, > > I just wanted to mention that I've "ported" Test::More to PostgreSQL. > Yes, you heard right. Details here: > > > http://justatheory.com/computers/databases/postgresql/introducing_pgtap.htm >l > > The project page for it should appear here in a day or two: > > http://pgfoundry.org/projects/pgtap/ > > It includes a Perl app, pg_prove, that works just like prove, but use > psql under the hood, and has relevant options for it. Thanks to Ovid > and Andy for TAP::Harness -- it made it really easy! > > But a quick question: I'm going to start writing SQL unit tests in an > Catalyst app I'm building. I'm thinking that the SQL tests will be > right next to the Perl tests. So they'd be t/*.s to complement the t/ > *.t Perl tests. So the question: How do I modify `make test` and/or `./ > Build test` so that they can use a single TAP::Harness to process both > types of tests, but send *.t tests to Perl and *.s tests to psql? > You can do that with Test::Run: http://search.cpan.org/dist/Test-Run-Plugin-AlternateInterpreters/ What I normally do is create a "runtest" target that runs tests using Test::Run: http://search.cpan.org/src/SHLOMIF/Test-Run-Plugin-AlternateInterpreters-0.0105/inc/Test/Run/Builder.pm But you can also overwrite the built-in Module::Build test targets to do something else. Regards, Shlomi Fish ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ The Case for File Swapping - http://xrl.us/bjn7i The bad thing about hardware is that it sometimes works and sometimes doesn't. The good thing about software is that it's consistent: it always does not work, and it always does not work in exactly the same way. From sjn at pvv.org Thu Jun 12 13:04:50 2008 From: sjn at pvv.org (Salve J Nilsen) Date: Thu, 12 Jun 2008 15:04:50 +0200 Subject: [tap-l] Test Anything Protocol as an IETF draft/RFC Message-ID: <48511EF2.6080908@pvv.org> Hi, Chris & Lisa I had a talk with Harald T. Alvestrand a while back, and he recommended me to contact you guys/the IETF Apps area about finding a way to get TAP (Test Anything Protocol) through the IETF standards track. Quickly, about TAP: It's basically a unidirectional, single-channel, transport-independent protocol for communicating test successes/failures and metadata between any TAP producer and a consumer which aggregates and reports the results. http://testanything.org/ http://testanything.org/wiki/index.php/TAP_Producers http://testanything.org/wiki/index.php/TAP_Consumers Earlier versions of TAP has grown and been used in the Perl/CPAN community for many years now, and currently there is a consensus in the Perl QA crowd that TAP probably would be useful for other communities too. Therefore this mail. Can you guys give us a few pointers on how to proceed on getting an IETF stamp of approval? :) Feel free to ask any technical and/or philosophical questions related to TAP, I'm sure the guys on (CC'd) can help you with those. ;-) Kind regards, - Salve J. Nilsen (informally on behalf of tap-l) From pagaltzis at gmx.de Sat Jun 14 04:18:12 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Sat, 14 Jun 2008 06:18:12 +0200 Subject: [tap-l] pgTAP In-Reply-To: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> References: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> Message-ID: <20080614041812.GO25433@klangraum.plasmasturm.org> * David E. Wheeler [2008-06-11 21:00]: > The project page for it should appear here in a day or two: > > http://pgfoundry.org/projects/pgtap/ > > It includes a Perl app, pg_prove, that works just like prove, > but use psql under the hood, and has relevant options for it. I think a separate utility is not necessary at all. `prove` has an `-e` switch which can be used to specify an interpreter to run the given scripts with, so all you should need to say is roughly prove -epsql t/*.s > So the question: How do I modify `make test` and/or `./Build > test` so that they can use a single TAP::Harness to process > both types of tests, but send *.t tests to Perl and *.s tests > to psql? I just spent an hour poring over the docs and code of M::B but found no easy way to do what you want. There?s the following sample code in the section on the `testall` target: my $mb = Module::Build->subclass( code => q( sub ACTION_testspecial { shift->generic_test(type => 'special'); } sub ACTION_testauthor { shift->generic_test(type => 'author'); } ) ); $mb->new( # ... test_types => { special => '.st', author => '.at', }, # ... However, there is no existing way to get harness switches passed in based on the filename extension, and the factorisation of the `generic_test` method and its helpers is such that you can?t really use any of the bits in isolation. It feels like this should be much easier? which I have to say is a feeling I often get when I try to extend M::B in any direction it doesn?t already explicitly accomodate. Of course the alternative is EU::MM where I have no desire to even try doing any interesting stuff at all. Regards, -- Aristotle Pagaltzis // From publiustemp-tapx at yahoo.com Sat Jun 14 09:22:48 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Sat, 14 Jun 2008 10:22:48 +0100 (BST) Subject: [tap-l] pgTAP In-Reply-To: <20080614041812.GO25433@klangraum.plasmasturm.org> Message-ID: <514202.23407.qm@web65709.mail.ac4.yahoo.com> --- Aristotle Pagaltzis wrote: > Of course the alternative is EU::MM where I have no desire to > even try doing any interesting stuff at all. Ironically, despite so many competent hackers discovering how difficult EU::MM is (for the non-Perl folks: http://search.cpan.org/dist/ExtUtils-MakeMaker/), the masses get very upset when folks try to use an alternative to it. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From david at kineticode.com Sat Jun 14 21:02:40 2008 From: david at kineticode.com (David E. Wheeler) Date: Sat, 14 Jun 2008 14:02:40 -0700 Subject: [tap-l] pgTAP In-Reply-To: <20080614041812.GO25433@klangraum.plasmasturm.org> References: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> <20080614041812.GO25433@klangraum.plasmasturm.org> Message-ID: <35B5DE7F-87CF-49DB-9A54-AFE2FCA18E91@kineticode.com> On Jun 13, 2008, at 21:18, Aristotle Pagaltzis wrote: > I think a separate utility is not necessary at all. `prove` has > an `-e` switch which can be used to specify an interpreter to run > the given scripts with, so all you should need to say is roughly > > prove -epsql t/*.s Yeah, but I wanted it to have the ability to specify username, password, database name, etc. So pg_prove looks a lot like other pg_* tools. > However, there is no existing way to get harness switches passed > in based on the filename extension, and the factorisation of the > `generic_test` method and its helpers is such that you can?t > really use any of the bits in isolation. Well, if I could start with some way to have one harness handle both kinds of tests, I can probably break Module::Build to make it do my bidding. Maybe someone would be willing to hack on this problem with me at YAPC this week? > It feels like this should be much easier? which I have to say is > a feeling I often get when I try to extend M::B in any direction > it doesn?t already explicitly accomodate. Heh, yeah. I've sent in many patches to get it to do my bidding (though not in a while). > Of course the alternative is EU::MM where I have no desire to > even try doing any interesting stuff at all. No, nor do I. thanks for looking into this, Aristotle. Best, David From scratchcomputing at gmail.com Sat Jun 14 21:37:02 2008 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Sat, 14 Jun 2008 14:37:02 -0700 Subject: [tap-l] pgTAP In-Reply-To: <35B5DE7F-87CF-49DB-9A54-AFE2FCA18E91@kineticode.com> References: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> <20080614041812.GO25433@klangraum.plasmasturm.org> <35B5DE7F-87CF-49DB-9A54-AFE2FCA18E91@kineticode.com> Message-ID: <200806141437.02572.ewilhelm@cpan.org> # from David E. Wheeler # on Saturday 14 June 2008 14:02: >> However, there is no existing way to get harness switches passed >> in based on the filename extension, and the factorisation of the >> `generic_test` method and its helpers is such that you can?t >> really use any of the bits in isolation. As the guy responsible for the test_types thing, I'll say that I have since repented and decided that organizing extra tests (named with ".t" extensions) under xt/ is probably a better route in the long run. This might still be true for the pgTAP case. Perhaps a shebang or something allows them to be '.t' files? The problem with changing file extensions and supporting it in only one tool is that various other tools don't work. >Well, if I could start with some way to have one harness handle both ? >kinds of tests, I can probably break Module::Build to make it do my ? >bidding. Maybe someone would be willing to hack on this problem with ? >me at YAPC this week? A good first step might be to discuss it on the module-build at perl.org list. Some way to use features from the new harness without breaking compatibility would be interesting. Perhaps that just takes the form of enabling the features if TAP::Harness is found. Or the tap-x list, where shebangs have been discussed before. There was once some form of config file which switched executables based on filenames, but I think that got scrapped. Shebang plugins? All the test has to do is emit TAP, right? > ... but use psql under the hood, and has relevant options for it. I suspect the complicated bits come in when you start trying to pass options to the executable/interpreter and adding DWIM bits. --Eric -- But you can never get 3n from n, ever, and if you think you can, please email me the stock ticker of your company so I can short it. --Joel Spolsky --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From david at kineticode.com Sun Jun 15 15:34:11 2008 From: david at kineticode.com (David E. Wheeler) Date: Sun, 15 Jun 2008 08:34:11 -0700 Subject: [tap-l] pgTAP In-Reply-To: <200806141437.02572.ewilhelm@cpan.org> References: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> <20080614041812.GO25433@klangraum.plasmasturm.org> <35B5DE7F-87CF-49DB-9A54-AFE2FCA18E91@kineticode.com> <200806141437.02572.ewilhelm@cpan.org> Message-ID: On Jun 14, 2008, at 14:37, Eric Wilhelm wrote: > As the guy responsible for the test_types thing, I'll say that I have > since repented and decided that organizing extra tests (named with > ".t" > extensions) under xt/ is probably a better route in the long run. > > This might still be true for the pgTAP case. Perhaps a shebang or > something allows them to be '.t' files? The problem with changing > file > extensions and supporting it in only one tool is that various other > tools don't work. They're not executables. One needs to know where psql is, plus database connectivity information (host, username, password, dbname, etc.). So the harness will have to have that knowledge built into it. What I imagined was something like this: TAP::Harness->new({ verbosity => $opts->{verbose} || $ENV{TEST_VERBOSE}, timer => $opts->{timer}, color => $opts->{color}, exec => { '[.]t$' => 'perl', '[.]s%' => [qw(psql -U postgres -d try -f)] } }); IOW, a way to map file name suffixes to the commands that should execute them. > A good first step might be to discuss it on the module-build at perl.org > list. Some way to use features from the new harness without breaking > compatibility would be interesting. Perhaps that just takes the form > of enabling the features if TAP::Harness is found. Yes. I have little doubt that M::B can be bent to my will. The question is, can TAP::Harness? > Or the tap-x list, where shebangs have been discussed before. There > was > once some form of config file which switched executables based on > filenames, but I think that got scrapped. Shebang plugins? All the > test has to do is emit TAP, right? Shebangs? That has the bias of assuming the files are executable, no? > I suspect the complicated bits come in when you start trying to pass > options to the executable/interpreter and adding DWIM bits. Yep. Best, David From david at kineticode.com Sun Jun 15 15:35:12 2008 From: david at kineticode.com (David E. Wheeler) Date: Sun, 15 Jun 2008 08:35:12 -0700 Subject: [tap-l] pgTAP In-Reply-To: References: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> <20080614041812.GO25433@klangraum.plasmasturm.org> <35B5DE7F-87CF-49DB-9A54-AFE2FCA18E91@kineticode.com> <200806141437.02572.ewilhelm@cpan.org> Message-ID: On Jun 15, 2008, at 08:34, David E. Wheeler wrote: > TAP::Harness->new({ > verbosity => $opts->{verbose} || $ENV{TEST_VERBOSE}, > timer => $opts->{timer}, > color => $opts->{color}, > exec => { > '[.]t$' => 'perl', > '[.]s%' => [qw(psql -U postgres -d try -f)] Uh, s/%/$/. Sorry. Best, David From scratchcomputing at gmail.com Sun Jun 15 16:12:23 2008 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Sun, 15 Jun 2008 09:12:23 -0700 Subject: [tap-l] pgTAP In-Reply-To: References: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> <200806141437.02572.ewilhelm@cpan.org> Message-ID: <200806150912.23751.ewilhelm@cpan.org> # from David E. Wheeler # on Sunday 15 June 2008 08:34: >On Jun 14, 2008, at 14:37, Eric Wilhelm wrote: >What I imagined was something like this: > >TAP::Harness->new({ > verbosity => $opts->{verbose} || $ENV{TEST_VERBOSE}, > timer => $opts->{timer}, > color => $opts->{color}, > exec => { > '[.]t$' => 'perl', > '[.]s$' => [qw(psql -U postgres -d try -f)] > } >}); > >IOW, a way to map file name suffixes to the commands that should >execute them. I still think the suffix thing tends to hurt. In any case, you have to find a way to configure the switches for psql from a few packages away to make it work with prove. >>... Shebang plugins? All the test has to do is emit TAP, right? > >Shebangs? That has the bias of assuming the files are executable, no? No. It doesn't even need to assume that the platform supports shebags. (trivia: perl5 automagically interprets shebangs if given one which doesn't contain 'perl' -- Test::Harness 2 (ab)used that feature, but this is not the path I suggest because it also passes your perl switches to whatever that interpreter is.) Some languages may not even support a comment on the first line or a shebang-compatible comment, etc. The idea is to inspect the file for shebangs (and shebang-like things (or any distinguishing directive/identifier on some line in the file.)) The harness would then choose the associated executable+switches (or Source subclass) to process it. These would be configured via plugins, which might register themselves with a shebang regexp or grep callback. --Eric -- software: a hypothetical exercise which happens to compile. --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From david at kineticode.com Sun Jun 15 20:25:37 2008 From: david at kineticode.com (David E. Wheeler) Date: Sun, 15 Jun 2008 13:25:37 -0700 Subject: [tap-l] pgTAP In-Reply-To: <200806150912.23751.ewilhelm@cpan.org> References: <98064F56-4F27-477A-B486-D16C0723F9DF@kineticode.com> <200806141437.02572.ewilhelm@cpan.org> <200806150912.23751.ewilhelm@cpan.org> Message-ID: <0722C04D-DB73-4B30-BAC3-33DB933E4C31@kineticode.com> On Jun 15, 2008, at 09:12, Eric Wilhelm wrote: > I still think the suffix thing tends to hurt. In any case, you have > to > find a way to configure the switches for psql from a few packages away > to make it work with prove. Yes, that's why I wrote pg_prove. But Module::Build doesn't use prove, it uses Test::Harness, IIRC. > No. It doesn't even need to assume that the platform supports > shebags. > (trivia: perl5 automagically interprets shebangs if given one which > doesn't contain 'perl' -- Test::Harness 2 (ab)used that feature, but > this is not the path I suggest because it also passes your perl > switches to whatever that interpreter is.) Yeah, that's not really desirable. > Some languages may not even support a comment on the first line or a > shebang-compatible comment, etc. Right. These are SQL scripts. You can have -- or /* */ comments, but not #!. > The idea is to inspect the file for shebangs (and shebang-like things > (or any distinguishing directive/identifier on some line in the > file.)) > The harness would then choose the associated executable+switches (or > Source subclass) to process it. These would be configured via > plugins, > which might register themselves with a shebang regexp or grep > callback. That would be fine, especially if there were more than one way to map test files with plugins, such as shebang, file name suffix, or whatever. Could they all be put in there so that we can see what sorts of cow paths develop? Best, David From lisa at osafoundation.org Mon Jun 16 18:49:28 2008 From: lisa at osafoundation.org (Lisa Dusseault) Date: Mon, 16 Jun 2008 11:49:28 -0700 Subject: [tap-l] Test Anything Protocol as an IETF draft/RFC In-Reply-To: <48511EF2.6080908@pvv.org> References: <48511EF2.6080908@pvv.org> Message-ID: Hi Salve, Since you mention the IETF Standards Track specifically, I guess you're looking for a Proposed Standard RFC that has been published through the IETF consensus process, which typically involves a Working Group (WG). Off the standards track is the other option, which doesn't involve consensus-building or a WG. This might mean publishing the protocol specification through the RFC Editor's individual submission stream. This gets you an RFC which is marked as "Informational" status and with a disclaimer saying it's not a product of IETF Consensus. You can, however, document not just the technical details but also some notes on implementation and deployment, to show people that it's really in use. An informational RFC is a moderately useful stamp of approval and is great for getting a stake in the ground. If you're looking to build a WG and build community awareness and participation, I think TAP would be an excellent topic and I will definitely help. The rough steps for forming a WG: - Create a mailing list under IETF contribution rules (https://datatracker.ietf.org/list/request/ ) - Advertise the mailing list and the effort, and get participation -- one might get people talking early about the proposals by publishing them as Internet-Drafts and asking for input on the new list - Prepare a draft charter for a WG and discuss it on the new list - Ask me for a BOF slot about 2 months in advance of the IETF meeting you want to have the BOF at. It's also sometimes possible to publish individual submissions, without a WG, on the Standards Track. This is less certain and it doesn't have the side effect of building a wider community around the new standard (the standards community helps maintain, revise or extend the standard, but also participation in the WG usually encourages wider implementation, deployment and interoperability). The lack of certainty is in part because different ADs have different personal requirements before they agree to sponsor an individual submission on the Standards Track. One of my requirements, for example, is to see demonstrated interoperability even before Proposed Standard, as well as showing wide appeal. Lisa On Jun 12, 2008, at 6:04 AM, Salve J Nilsen wrote: > Hi, Chris & Lisa > > I had a talk with Harald T. Alvestrand a > while back, and he recommended me to contact you guys/the IETF Apps > area about finding a way to get TAP (Test Anything Protocol) through > the IETF standards track. > > Quickly, about TAP: It's basically a unidirectional, single-channel, > transport-independent protocol for communicating test successes/ > failures and metadata between any TAP producer and a consumer which > aggregates and reports the results. > > http://testanything.org/ > http://testanything.org/wiki/index.php/TAP_Producers > http://testanything.org/wiki/index.php/TAP_Consumers > > Earlier versions of TAP has grown and been used in the Perl/CPAN > community for many years now, and currently there is a consensus in > the Perl QA crowd that TAP probably would be useful for other > communities too. Therefore this mail. > > Can you guys give us a few pointers on how to proceed on getting an > IETF stamp of approval? :) > > Feel free to ask any technical and/or philosophical questions > related to TAP, I'm sure the guys on (CC'd) > can help you with those. ;-) > > > Kind regards, > > - Salve J. Nilsen (informally on behalf of tap-l)