annathepiper: (Final Test)
[personal profile] annathepiper

On a previous job interview loop, one of the people I spoke with gave me good feedback about the work I’ve been putting up on my Github repos. He observed at the time that I had been hitting the low-hanging fruit: i.e., the test cases dealing with good, expected data.

He was right. So I’ve gone back and updated my repos for the WordPress REST API tests to also include negative test cases, i.e., known bad data, to test the error behavior of the endpoints. So here’s what I focused on to do that.

First, several of the endpoints ask for IDs of this, that, and the other thing: post IDs, category IDs, etc. For those sorts of cases, I did these negative tests:

  • IDs that could be valid (i.e., were legit integers), but which did not actually exist as posts.
  • IDs that were specifically not valid, i.e., things that were not integers. E.g., “aaaaaaa”.
  • Using MAXINT as a post ID, just to use a GIGANTIC integer. Practically speaking I’d usually expect this to also be a “this post does not exist” scenario. But I’ve tested things in the past where throwing a thing a value slightly above a limit behaved, but throwing it a value WAY bigger than the limit did not. So I wanted to do this scenario too.
  • Also using MININT as a post ID. This is not only to test using a gigantic thing, but ALSO to test using a gigantic thing that could not actually be a valid post ID (i.e., because it’s a negative number).

Secondly, several of the rest of the endpoints didn’t use IDs, but rather, slug/tags. So for those endpoints, I did negative test cases that were very similar.

  • Slugs/tags that could be valid, i.e., strings, but which did not actually exist in the database (such as using “pancakes” for a category tag).
  • Slugs/tags that could not be valid. E.g., using “pancakes” with additional non-alphabetic characters.
  • Using MAXINT and MININT again.

In all of these cases, I threw the bad data at the endpoints and looked for specific error codes and error messages I was expecting to get in response. I also looked for 404 response codes.

I got the expected behavior by testing the various endpoints manually with bad data, in Postman, and seeing how they responded. That gave me the basic JSON response structure I would need to expect: that there would be both an error code and an error message, and an additional data object that would contain the response code within it. So my test cases looked for all three of these things.

As of this writing, I’ve implemented the negative test cases in Java, Python, and C#. Once I had them working in Java, it was reasonably easy to port them over to the other languages. In all three languages, this has now brought the test case count for the REST API suite from 20 up to 60.

In addition to learning the error behavior of the various endpoints, the most useful thing I’ve picked up on in this part of the project is how to get at MAXINT and MININT in the various languages.

In Java, it’s Integer.MAX_VALUE and Integer.MIN_VALUE. In Python, it’s sys.maxsize or -sys.maxsize–and this is Python 3, specifically. (Apparently, in Python 2, it was sys.maxint, but it changed; if I understand the change correctly, it was because Python 3 doesn’t have a hard and fast integer size limit, because it automatically recasts your number to a long on the fly if you go over the theoretical int limit.) And in C#, it’s int.MaxValue and int.MinValue.

Meanwhile, over in the Selenium test suites, there are fewer opportunities there for negative cases. But I did add some there that throw bad data at the search box, to verify that I get the appropriate messaging if there are no matching results. There’s some room for expansion there as well, to try to test the limits of what the search box can accept. So that’ll be the next place I add more negative cases.

Date: 2019-04-13 02:52 am (UTC)
alatefeline: Painting of a cat asleep on a book. (Default)
From: [personal profile] alatefeline
That was indeed useful feedback! And a productive response on your part! I don't follow the programming per se, but your explanation of test cases made sense. Big wrong numbers, small wrong numbers, plausible wrong numbers, and wrong not-even-a-number aka screaming - honestly possible everything that takes a number and does a thing to it shpuld be given that sort of a test.

Date: 2019-04-13 10:10 am (UTC)
vatine: Generated with some CL code and a hand-designed blackletter font (Default)
From: [personal profile] vatine
Yep, writing good (and complete) unit tests is shockingly complicated. One of the things I use as a rough rule of thumb is "test 0, test 1, test close to the edges of well-defined, test just outside the edges of well-defined, test WAAAAY outside, and test at infinity".

Which, for values of strings, seems to be pretty much what you're doing. I would possibly also throw in a "what happens if you pass an empty string/don't actually pass an ID or slug/tag", for completeness.

Meanwhile, I find myself writing code that generates sliderule scales in SVG, maybe it'll end up going through a laser cutter at some point. Even if it doesn't, it keeps me amused.

Date: 2019-04-14 12:15 pm (UTC)
vatine: Generated with some CL code and a hand-designed blackletter font (Default)
From: [personal profile] vatine
Yep, one of those cases where "sometimes, this is useful" doesn't extrapolate to "in this case, this is useful".

And now I'm wondering what http://.../posts/ ACTUALLY does. It should do the same as without the terminal slash, though.

Profile

annathepiper: (Default)
Anna the Piper

April 2019

S M T W T F S
 123456
7891011 1213
14151617181920
21222324252627
282930    

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Apr. 20th, 2019 09:06 am
Powered by Dreamwidth Studios