Negative test cases

[annathepiper]

On a previous job interview loop, one of the people I spoke with gave me good feedback about the work I’ve been putting up on my Github repos. He observed at the time that I had been hitting the low-hanging fruit: i.e., the test cases dealing with good, expected data.

He was right. So I’ve gone back and updated my repos for the WordPress REST API tests to also include negative test cases, i.e., known bad data, to test the error behavior of the endpoints. So here’s what I focused on to do that.

First, several of the endpoints ask for IDs of this, that, and the other thing: post IDs, category IDs, etc. For those sorts of cases, I did these negative tests:

• IDs that could be valid (i.e., were legit integers), but which did not actually exist as posts.
• IDs that were specifically not valid, i.e., things that were not integers. E.g., “aaaaaaa”.
• Using MAXINT as a post ID, just to use a GIGANTIC integer. Practically speaking I’d usually expect this to also be a “this post does not exist” scenario. But I’ve tested things in the past where throwing a thing a value slightly above a limit behaved, but throwing it a value WAY bigger than the limit did not. So I wanted to do this scenario too.
• Also using MININT as a post ID. This is not only to test using a gigantic thing, but ALSO to test using a gigantic thing that could not actually be a valid post ID (i.e., because it’s a negative number).

Secondly, several of the rest of the endpoints didn’t use IDs, but rather, slug/tags. So for those endpoints, I did negative test cases that were very similar.

• Slugs/tags that could be valid, i.e., strings, but which did not actually exist in the database (such as using “pancakes” for a category tag).
• Slugs/tags that could not be valid. E.g., using “pancakes” with additional non-alphabetic characters.
• Using MAXINT and MININT again.

In all of these cases, I threw the bad data at the endpoints and looked for specific error codes and error messages I was expecting to get in response. I also looked for 404 response codes.

I got the expected behavior by testing the various endpoints manually with bad data, in Postman, and seeing how they responded. That gave me the basic JSON response structure I would need to expect: that there would be both an error code and an error message, and an additional data object that would contain the response code within it. So my test cases looked for all three of these things.

As of this writing, I’ve implemented the negative test cases in Java, Python, and C#. Once I had them working in Java, it was reasonably easy to port them over to the other languages. In all three languages, this has now brought the test case count for the REST API suite from 20 up to 60.

In addition to learning the error behavior of the various endpoints, the most useful thing I’ve picked up on in this part of the project is how to get at MAXINT and MININT in the various languages.

In Java, it’s Integer.MAX_VALUE and Integer.MIN_VALUE. In Python, it’s sys.maxsize or -sys.maxsize–and this is Python 3, specifically. (Apparently, in Python 2, it was sys.maxint, but it changed; if I understand the change correctly, it was because Python 3 doesn’t have a hard and fast integer size limit, because it automatically recasts your number to a long on the fly if you go over the theoretical int limit.) And in C#, it’s int.MaxValue and int.MinValue.

Meanwhile, over in the Selenium test suites, there are fewer opportunities there for negative cases. But I did add some there that throw bad data at the search box, to verify that I get the appropriate messaging if there are no matching results. There’s some room for expansion there as well, to try to test the limits of what the search box can accept. So that’ll be the next place I add more negative cases.

Comments

[alatefeline]

That was indeed useful feedback! And a productive response on your part! I don't follow the programming per se, but your explanation of test cases made sense. Big wrong numbers, small wrong numbers, plausible wrong numbers, and wrong not-even-a-number aka screaming - honestly possible everything that takes a number and does a thing to it shpuld be given that sort of a test.

[annathepiper]

Thank you. :)

And yeah, this is applying pretty standard QA strategy to testing a thing. Whatever that thing might happen to be.

[vatine]

Yep, writing good (and complete) unit tests is shockingly complicated. One of the things I use as a rough rule of thumb is "test 0, test 1, test close to the edges of well-defined, test just outside the edges of well-defined, test WAAAAY outside, and test at infinity".

Which, for values of strings, seems to be pretty much what you're doing. I would possibly also throw in a "what happens if you pass an empty string/don't actually pass an ID or slug/tag", for completeness.

Meanwhile, I find myself writing code that generates sliderule scales in SVG, maybe it'll end up going through a laser cutter at some point. Even if it doesn't, it keeps me amused.

[annathepiper]

I thought about the empty variable scenario, yeah, but in these particular test cases that's kinda irrelevant just because of how the endpoints work.

The Posts endpoint for example looks like this:

http://mywordpresssite.com/wp-json/wp/v2/posts

If I want to get at a specific post, I just stick a post ID on the end of that:

http://mywordpresssite.com/wp-json/wp/v2/posts/23434

If I leave off the post ID the behavior just falls back to "give me all the posts". And since I already have a test case to hit that version of the endpoint, that's kind of redundant. ;)

Now when I get to the point of this project where I'll be doing endpoint requests that actually need payloads, THEN I can get more interesting with trying to set empty variables in those payloads.

[vatine]

Yep, one of those cases where "sometimes, this is useful" doesn't extrapolate to "in this case, this is useful".

And now I'm wondering what http://.../posts/ ACTUALLY does. It should do the same as without the terminal slash, though.

[annathepiper]

Whst it does is give you a big ol' JSON payload of information about the posts on the site in question. Since this is an endpoint you can hit without authentication, what that actually means is "information about all the posts you can see as a visitor to the site", the same as if you were just browsing the site via normal websurfing.

This includes, say, the post ID, the title, the body, its date, its category and tags, etc.

If you want more deets you can see the various endpoints in the Wordpress REST API docs over here. This is the reference I'm using for the various endpoints I'm hitting in my tests.