Run Diferencia

Assumptions:

  • You’ve installed Diferencia using any of the methods explained there.

  • Have curl installed.

  • You have checked that curl 'http://rest.ensembl.org/info/ping' -H 'Content-type:application/json' returns a JSON document.

  • You have checked that curl http://now.httpbin.org/ returns a JSON document.

Starting Diferencia

To start diferencia you only need to run next command from your terminal:

~$ diferencia start -c http://rest.ensembl.org -p http://rest.ensembl.org

This command just starts Diferencia proxy, configuring it to use http://rest.ensembl.org (-p) as the primary (old version) and http://rest.ensembl.org (-c) as the candidate (new version).

For sake of simplicity now both versions point to te same host, but in a real use case, hosts would be different.

Then run next curl command:

curl 'http://localhost:8080/info/ping' -H 'Content-type:application/json' -i

And the response is HTTP/1.1 200 OK because obviously, both are the same version so they are producing the same output.

What is happening under the cover when you do that curl operation is that Diferencia proxy replaces the host with primary host and sends the request to it (http://rest.ensembl.org/info/ping) and then do the same with the candidate host, sending the request to it (http://rest.ensembl.org/info/ping).

After that, both responses are compared and if they are strictly equal then an HTTP 200 OK method is returned from the proxy.

By default Diferencia just skip any unsafe operation like PUT, POST, DELETE and PATCH, because they are not safe to run them blindly without taking into consideration their side effects.

You can remove this restriction by starting Diferencia with --unsafe flag.

Running Modes

In the previous example, both responses must be strictly equal (this means that JSON schema and each key/value must be equal).

About Headers

By default Diferencia only takes into consideration status code and body content and headers, for example, are ignored.

You can enable header verification by setting --headers flag.

You can also set which header values should be ignored by using --ignoreHeadersValues Accept, Accept-Charset

Notice that this only apply to header value, not the key. So for the previous example, it is checked that both primary and candidate contains the same headers (pair key/value) but in case of Accept and Accept-Charset only it is checked that there are the keys present, but they don’t need to have the same value.

But there are other modes that we are going to describe in next sections:

Strict

Strict mode checks that every key/value is strictly equal.

Subset

Subset mode checks if candidate response is a subset of primary response. For example:

V1
{
    "name": "Alex",
    "surname": "Soto"
}

and

V2
{
    "name": "Alex",
    "surname": "Soto",
    "fullname": "Alex Soto"
}

V2 document is a subset of V1, so in this case, Diferencia will say that both documents are equal.

Noise Detection

If you have Diferencia started from Starting Diferencia section, stop it and start it again with next command:

diferencia start -c http://now.httpbin.org/ -p http://now.httpbin.org/

Now we are configuring Diferencia proxy to redirect requests to http://now.httpbin.org/ which basically returns the current time.

So you can run next curl operation:

curl 'http://localhost:8080/' -i

And the response is HTTP/1.1 412 Precondition Failed. This is happening because although you are sending the request to the same service, it is returning the current time which of course it is not the same in both calls. This thing that it is pretty obvious makes that Diferencia result is a failure, saying that both services are not compatible (so there is a regression), when the truth is that they are equally valid, but having a value different.

To fix this there is a noise cancellation algorithm that detects these dynamic values and suppress them from JSON document. This algorithm sends the request to not two instances but three. Two are the old instances (primary and secondary) and one to the new instance (candidate).

The request to primary and seconday returns a valid request (since old instances are the correct ones), then from their responses, Diferencia checks for the differences between both responses.

Since both responses are valid (because the source is the same but different instances), the differences that are between them are just noise that should not take into consideration for response validation.

Then detected noise is removed from primary and candidate response, and they are compared, if they are equal, then you know that everything is fine.

This can be summarized in next schema:

noise
Figure 1. Noise Cancellation

Now stop Diferencia and start it again with next command:

diferencia start -c http://now.httpbin.org/ -p http://now.httpbin.org/ -s http://now.httpbin.org/ -n

You are setting URLs for primary (-c), secondary (-s), candidate (-c) and finally that you want to enable noise reduction (-n).

Now just execute the same curl command as before.

curl 'http://localhost:8080/' -i

And now the response is totally different returning a 200 OK.

Manual Noise Cancellation

Sometimes autoamtic noise cancellation is not enough. Usually this might happen when you have random values or when you know beforehand the fields that produces noise and you don’t want any automatic detection to save some execution time (1 extra call).

About Random Values

Suppose that you have a service where one field is a random value from 0 to 9. It could happen that primary and secondary field contains the same value. In this case this would not be considered as noise.

But then candidate field could be a different number of the one calculated in primary and secondary. In this case the request would be considered failed, since there were no noise detection algorithm applied, but you as a API designer knows that this field should be considered noise.

To avoid this problem, to flags are provided and they are enabled when noisedetection (-n) flag is enabled. These flags are ignoreValues and ignoreValuesFile.

ignoreValues

list (in CSV) of JSON pointers of elements where their values should be ignored in comparision.

ignoreValuesFile

path of a file where each line is a JSON pointer of element where their values should be ignored in comparision.

You need to have noise detection enabled to be able to specify manual noise cancellation.

Configuration

In next table you can see all possible configuration arguments that you can pass to Differentia:

start

Option Purpose Format Default

--serviceName

Sets service name under test

string

Hostname of candidate

--primary (-p)

Sets primary URL

URL

<mandatory>

--secondary (-s)

Sets secondary URL, only valid in case of Noise Reduction

URL

--candidate (-c)

Sets candidate URL

URL

<mandatory>

--noisedetection (-n)

Enable noise detection

boolean

false

--difference (-d)

Sets differencia mode

Strict,Subset

Strict

--logLevel (-l)

Set log level

debug, info, warn, error, fatal, panic

error

--headers

Enable Http headers comparision

boolean

false

--ignoreHeadersValues

List of headers key where its value should be ignored for comparision purposes

CSV

--ignoreValues

List of JSON Pointers of values that must be ignored for comparision purposes

CSV

--ignoreValuesFile

File location where each line is a JSON pointers definition for ignoring values

File

--unsafe (-u)

Allow none safe operations like PUT, POST, PATCH, ..

boolean

false

--storeResults

Directory where the output is set. If not specified then nothing is stored. Useful for local development.

File

--port

Sets port where the proxy is started

integer

8080

--prometheus

Enables Prometheus endpoint

boolean

false

--prometheusPort

Prometheus endpoint mort

integer

8081