Subject: Re: [tap] TAP Diagnostics

From: Ovid <[email-masked]>
Date: Thu, 21 Aug 2008 11:53:27 -0700 (PDT)

--- On Thu, 21/8/08, chromatic <[email-masked]> wrote:

> If TAP goes any further than saying "Yes, you *can*
> stuff a big wad of custom 
> data in this ganky old potato sack, but it's ugly and
> covered in spiders and 
> there may be a hole in the bottom, so if there's any
> other way, we encourage 
> you to do that", I fear for the safety of my lawn,
> metaphorically speaking.

That's actually all TAP will be doing.  You *can* stuff that big wad of diagnostic data there in a machine-readable format, but there are two overriding concerns:

1.  It's entirely optional.
2.  It must not alter the interpretation of core TAP.

In fact, Schwern may very well add a feature to TB2 to disable the structured diagnostics on demand (I haven't asked him, but people might want that).

But at the end of the day, this is a feature people have been wanting for a long time, even if (for most cases), it's merely a case of "file, line, have, want".

Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Eric Wilhelm <[email-masked]>
Date: Thu, 21 Aug 2008 13:07:22 -0700

# from Ovid
# on Thursday 21 August 2008 11:53:

>> stuff a big wad of custom data in this ganky old potato sack
>> ... there may be a hole in the bottom ...
>> so if there's any other way, we encourage you to do that" ...

What would this "any other way" be?  And why would the sack have a hole 
in it?  Are we delivering a package or just kicking a sack down the 
street?

>That's actually all TAP will be doing.  You *can* stuff that big wad
> of diagnostic data there in a machine-readable format, but there are
> two overriding concerns:
>
>...
>But at the end of the day, this is a feature people have been wanting
> for a long time, even if (for most cases), it's merely a case of
> "file, line, have, want".

Well, exactly how are we defining "sack", "potato", and "wad" here, and 
how does it have anything to do with what people want?

If diagnostics are a data block, the only thing the TAP stream must do 
is keep the data from getting corrupted and/or leaking into the rest of 
the stream.  This says the consumer has to know where it ends, which 
doesn't work if it's a sack with a hole in it.

But none of that provides anybody with the "file, line, have, want" 
thing.  That would be a special, identified, form of data block with 
parsable, meaningful content.

If I have a GUI harness displaying the 'have' and 'want' graphics 
(pretend they are jpg or svg, or even one of each) for a failed test 
next to each other, would this be a custom consumer only in that it 
does something particular in interpreting the 'have' and 'want' values?

Or, is it a custom consumer in that it takes the $diagnostic block (with 
an $identifier which says it is JSON), deserializes that, and looks 
for 'have', 'want', and etc values in the resultant structure?

If the former, then the TAP spec must define the structure of the 
diagnostics, or at least some part of it, and still needs to provide a 
sack somehow.  If the latter, then TAP just defines a sack of bytes (it 
could still go forward from there to claim some particular sacks as 
reserved and/or recommended usage, right?)

--Eric
-- 
"...our schools have been scientifically designed to
prevent overeducation from happening."
--William Troy Harris
---------------------------------------------------
    http://scratchcomputing.com
---------------------------------------------------
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: "David E. Wheeler" <[email-masked]>
Date: Thu, 21 Aug 2008 14:14:23 -0700

On Aug 21, 2008, at 13:58, chromatic wrote:

> I wonder why anyone wants a test so complex that its diagnostic  
> requires you
> to serialize and deserialize objects and/or nested data structures  
> to and
> from custom TAP producers and consumers, and, if you really need to  
> do that,
> if you should start with a testing zeitgeist which doesn't consider  
> a strict
> separation of producer and consumer a necessary and sufficient  
> condition.

So this begs the question: What is this serialization thingy for,  
exactly? What will it be used for? Why does anything need to be  
serialized? TAP is a protocol for test results, not an object broker  
or remote execution protocol. Is it?

Best,

David

---

From: Ovid <[email-masked]>
Date: Thu, 21 Aug 2008 14:29:54 -0700 (PDT)

--- On Thu, 21/8/08, chromatic <[email-masked]> wrote:

> That part is the wad.

No, for the most part, it's a simple list of key/value pairs.

> (You might find it amusing to note that the SOAP folks
> didn't use the 
> sack/potato/wad metaphor when naming their failure.)

SOAP has the problem that the acronym has nothing to do with the reality, so they started with a huge mess o' confusion to start with.
 
For the most part, we're just looking for is taking the information that's ALREADY in diagnostics and making it machine readable.  Remember, this is mostly just a list of key/value pairs :)

> I wonder why anyone wants a test so complex that its
> diagnostic requires you 
> to serialize and deserialize objects and/or nested data
> structures to and 
> from custom TAP producers and consumers ...

Testing in large scale environments has different needs than CPAN modules.  People who don't work in those environments often don't see it.  Remember the brouhaha over Module::Build?  Many people said it was useless because EUMM does everything they need.  Obviously, as consumers, their needs are different from authors and they didn't appreciate concerns in domains they don't experience.

Remember, all of this is OPTIONAL.  If you don't want it, don't use it.  And don't forget to drop an email to Yahoo! telling them that their silly work with tagging TAP streams is over-engineering.

> Me, I'll be over here, trying to make my tests
> sufficiently easy to debug that 
> I don't have to worry about these things.

Wish I had that luxury.

Cheers,
Ovid

---

From: Ovid <[email-masked]>
Date: Thu, 21 Aug 2008 14:52:31 -0700 (PDT)

--- On Thu, 21/8/08, chromatic <[email-masked]> wrote:

> I've written a handful of TAP producers and consumers
> myself, as well as 
> software which other people have used in ways I never
> intended.  The more 
> complexity you add to a system to reduce the complexity
> people have to manage 
> to use your system as you never intended, the more
> complexity everyone has to 
> manage to use your system as you always intended.

Except that core TAP is not changing and all older TAP consumers will be just fine.

> I can deal with a "simple list of key/value
> pairs", but the "mostly" and "for 
> the most part" bother me.

Fair enough.  We'll just have to see how this shakes out.

Cheers,
Ovid

---

From: Ovid <[email-masked]>
Date: Thu, 21 Aug 2008 13:16:16 -0700 (PDT)

> how does it have anything to do with what people want?

To be fair, those were chromatic's words, not mine, so I'm not sure exactly what he meant by them and I don't want to put words in his mouth.

The rest addresses specific issues for the IETF process so I'll pull that out of this discussion and put it on the IETF list.  Time to stop spamming folks.  If they want the discussion, they can join the list:

Cheers,
Ovid

---

From: chromatic <[email-masked]>
Date: Thu, 21 Aug 2008 13:58:32 -0700

> how does it have anything to do with what people want?

Specifying a serialization format in TAP diagnostics says "Here's a 
serialization format you can use to store a complex data structure, perhaps 
an object, perhaps with type information, to encode into a text-based 
protocol intended for humans to read and interpret, hoping that something on 
the other end will understand the precise semantics of the serialized object 
and do something sane with it... oh, and there's something in that stream 
which tells you that the test passed, if you can dig out the important one 
bit of information."

That part is the wad.

The sack is the big arbitrary bag of out-of-band information potentially 
attached to the TAP stream which can contain anything (which tends to 
encourage people to put anything in it).

The potato is the arbitrary anything which people may want to put into the 
sack.

(You might find it amusing to note that the SOAP folks didn't use the 
sack/potato/wad metaphor when naming their failure.)

I wonder why anyone wants a test so complex that its diagnostic requires you 
to serialize and deserialize objects and/or nested data structures to and 
from custom TAP producers and consumers, and, if you really need to do that, 
if you should start with a testing zeitgeist which doesn't consider a strict 
separation of producer and consumer a necessary and sufficient condition.

Me, I'll be over here, trying to make my tests sufficiently easy to debug that 
I don't have to worry about these things.

-- c

---

From: chromatic <[email-masked]>
Date: Thu, 21 Aug 2008 14:41:36 -0700

On Thursday 21 August 2008 14:29:54 Ovid wrote:

> Remember, all of this is OPTIONAL.  If you don't want it, don't use it.

I'm not sure that line of thinking works well for standards, OOXML 
notwithstanding.  Acme::Calendar::Mayan might want to define a vigesimal type 
to encode time intervals; why not chuck a canonical representation of 
alternate base systems in the OPTIONAL section of the spec as well while 
we're talking about it?

I've written a handful of TAP producers and consumers myself, as well as 
software which other people have used in ways I never intended.  The more 
complexity you add to a system to reduce the complexity people have to manage 
to use your system as you never intended, the more complexity everyone has to 
manage to use your system as you always intended.

I can deal with a "simple list of key/value pairs", but the "mostly" and "for 
the most part" bother me.

-- c

---

From: Andy Armstrong <[email-masked]>
Date: Thu, 21 Aug 2008 23:32:40 +0100

> manage to use your system as you always intended.

Indeed - but there's a sweet spot somewhere between specifying too  
little and specifying too much. It's entirely proper that there should  
be a debate about where that sweet spot is - and parsimony should be a  
guiding principle - but the value of having inline, per-test metadata  
has already been demonstrated.

IMO we don't need a complete serialisation protocol - just somewhere  
we can make notes and have them pass unmolested through the harness.  
If someone needs arbitrary serialisation they can use a protocol that  
makes sense to them and either inline their serialised data as a  
Base64 encoded string or have the diagnostic refer to an external  
file. The important thing is to open up a conduit through which user  
defined data can flow from a test script.

Now, that could just be an opaque block of inline data - but you'd  
want that to be delimited in some way so the TAP parser knows how to  
step over it. Having to wrap it up as a minimal JSON or YAML is not a  
significantly greater burden than, say, MIME multipart encoding it and  
using a structured representation increases the chances that common  
usage will emerge. Cowpaths to pave.

I favour JSON because it's the simplest solution that fits those  
criteria.

-- 
Andy Armstrong, Hexten

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Fri, 22 Aug 2008 05:19:07 +0200

* Andy Armstrong <[email-masked]> [2008-08-22 00:35]:
> Indeed - but there's a sweet spot somewhere between specifying
> too little and specifying too much. It's entirely proper that
> there should be a debate about where that sweet spot is - and
> parsimony should be a guiding principle - but the value of
> having inline, per-test metadata has already been demonstrated.

Fully and strongly agreed, ++ on all counts.

> IMO we don't need a complete serialisation protocol - just
> somewhere we can make notes and have them pass unmolested
> through the harness. If someone needs arbitrary serialisation
> they can use a protocol that makes sense to them and either
> inline their serialised data as a Base64 encoded string or have
> the diagnostic refer to an external file. The important thing
> is to open up a conduit through which user defined data can
> flow from a test script. […] I favour JSON because it's the
> simplest solution that fits those criteria.

Exactly. We want to make it easy to annotate tests with any sort
of testing-relevant metadata that people might have, but the
purpose of the protocol is supposed to be conveying test results,
not one-way messaging of arbitrary data. Eg. someone who uses TAP
in testing a web app will probably have use for an HTTP response
status key and also for a hash of response headers.

The killer app example for diagnostics that I always come back to
is statical analysis over an archive of your TAP streams over
time. It makes sense to preserve as much test-relevant metadata
as possible to be able to examine trends. For this purpose, the
data does have to be machine readable, and it does have to be
structured at least a little – *just* a flat key-value bag won’t
cut it. However, no intricate structure is necessary.

This is a point where the APIs can help. If the API encourages
people to indiscriminately dump complex data structures into the
TAP stream, that’s what people will do. If the API instead makes
it very easy to use diagnostics in the way we intend for them to
be used, and doesn’t go out of its way to accomodate the potato
bag usage, fewer people will be tempted.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap