Web Request Task Properties Window

The Web Request Task was substantially enhanced beginning with adTempus 6. If you are connected to an older version of the adTempus server the new options are not available and you will see a different window; see Web Request Task Properties (Legacy).

The Web Request Task Properties window is displayed when you edit a job step that runs a Web Request Task.

Version Compatibility: Server version 6.0 or later Console version 6.0 or later.

Versions of adTempus prior to 6.0 could only send a GET request and save the response to a file. The request method, body, parameters, headers, authentication types, response processing, and retry options described below require version 6.0 or later.

Property Pages

In addition to the standard response events, this task defines the following events:

Event	Description
The Web request was executed (regardless of the outcome)	Occurs when the server has returned a response, regardless of the status code.
The HTTP request returns the following status code	Occurs if the request returns the specified HTTP status code. For example, a status code of 404 is returned if the page or file is not found. Specify RequestFailed to respond when no response was received at all (for example, if the server could not be reached).

Request

Method

Select or type the HTTP method to use: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, or any other method supported by the server. The default is GET.

A request body cannot be sent with the GET or HEAD method. When either of these methods is selected, the body type on the Body page is set to None, and any Body parameters are sent in the query string instead.

URL

Enter the URL to request. For example, https://api.example.com/orders. If you do not include the protocol, http:// is assumed. The URL may contain Job Variables.

Query parameters can be included in the URL, or entered in the Parameters list, where they are encoded for you.

Paste curl

Click Paste curl to set up the task from a curl command. This is a convenient way to copy a request from API documentation, from a tool such as Postman, or from a Web browser's developer tools ("Copy as cURL"). Commands formatted for bash, for the Windows command prompt, and for PowerShell are supported.

The method, URL, query parameters, headers, body, and authentication are taken from the command. The following curl options are recognized:

• -X/--request, -H/--header, -A/--user-agent, -e/--referer, -b/--cookie, --url
• -d/--data, --data-raw, --data-binary, --data-urlencode, --json, -G/--get, -I/--head
• -F/--form (fields of the form name=@path become file uploads)
• -u/--user, --oauth2-bearer, --ntlm/--negotiate, and Authorization headers using the Basic or Bearer scheme
• -k/--insecure, -L/--location, -m/--max-time, --retry, --retry-delay, -o/--output

Options that cannot be represented in the task (such as proxy and client certificate options, or data read from a file with @filename) are listed in a message after the import. Header and parameter values that appear to be credentials (such as an X-Api-Key header or an api_key parameter) are automatically marked as Secret. If the command includes a user ID but no password, enter the password on the Authentication page.

Copy as curl

Click Copy as curl to display the task's request as a curl command (bash format), which you can copy to the clipboard. This can be helpful for troubleshooting a request outside of adTempus or sharing it with an API provider.

Variable tokens in the request are not expanded. Secret values that have been saved cannot be retrieved, so they appear as <secret>; check Include secret values entered but not yet saved to include secret values you have typed since opening the window.

Test Request

Click Test Request to send the request using the current settings (including changes you have not yet saved) and see the response. See Test Request Window below.

Parameters

Enter the parameters to send with the request. For each parameter, specify:

Column	Description
Name	The parameter name.
Value	The parameter value, which may contain Job Variables. For a File parameter, the full path of the file to upload.
Send In	Query to add the parameter to the URL's query string, or Body to send it in the request body. Body parameters are used when the body type on the Body page is Form, JSON from parameters, or Multipart; otherwise they are added to the query string.
Type	Text for a normal value, or File to upload the file named in the Value column. File parameters are used only with the Multipart body type. The file is read using the job's user account.
Secret	Check this box for values such as API keys and passwords. See Secret Values below.
Enabled	Uncheck this box to keep a parameter in the list without sending it.

Headers

Enter any HTTP headers to send with the request, such as Accept or X-Api-Key. Header values may contain Job Variables. As with parameters, check Secret for headers that contain credentials or other secrets, and uncheck Enabled to keep a header without sending it.

Do not enter an Authorization header for Basic, Bearer, or OAuth 2.0 authentication; use the Authentication page instead. A Content-Type header entered here overrides the content type on the Body page.

Body

Select the type of body to send. A body cannot be sent with the GET or HEAD method.

Body Type	Description
None	No body is sent.
Raw	The body is sent exactly as entered, after Job Variables are expanded. Select or type the Content type (for example, application/json or application/xml); if none is specified, text/plain is used. Click Format JSON/XML to indent the body for readability.
Form (URL-encoded parameters)	The Body parameters from the Request page are sent as an HTML form (application/x-www-form-urlencoded).
JSON from parameters	The Body parameters are sent as a JSON object with a string property for each parameter, for example {"name":"Widget","quantity":"5"}. To send nested objects, arrays, or non-string values, use a Raw body instead.
Multipart (parameters and files)	The Body parameters are sent as multipart/form-data, which is used to upload files. Text parameters become form fields and File parameters are uploaded.

Authentication

Authentication type

Select how the request is authenticated.

Type	Description
Windows authentication using the job's account	If the Web server requests Windows authentication (NTLM or Kerberos), the credentials of the account the job runs under are sent.
Windows authentication using selected credentials	The credentials from the selected Credential Profile are used for Windows authentication.
None (anonymous)	No credentials are sent.
Basic	A user ID and password are sent in an Authorization: Basic header. Select Credential profile to use the user ID and password from a Credential Profile, or Enter a user ID and password to enter them directly. Entered values may contain Job Variables; values from a Credential Profile are used exactly as stored.
Bearer token	The Token is sent in an Authorization: Bearer header. The token is stored encrypted, and may be a Job Variable (for example, a token obtained by a previous step).
OAuth 2.0 client credentials	adTempus obtains an access token from the Token URL using the Client ID, Client secret, and optional Scope, and sends it in an Authorization: Bearer header. Tokens are cached and reused until shortly before they expire. If the server rejects a cached token, a new token is requested and the request is sent again.

Passwords, tokens, and client secrets are stored encrypted and cannot be viewed after they are saved. To change one, type the new value over the masked value. All of these values may contain Job Variables, including Job Variables whose values are hidden.

Response

Use these options to choose what to do with the response returned by the server. The response is saved to a file, captured, and stored in variables even if the status code indicates failure. Values are extracted and the response script is run only if the request succeeds.

Save the response as a captured file in the job history

Check this option to save the response as a captured file in the job history. The file name is taken from the server's Content-Disposition header or the URL if possible, or is based on the content type of the response.

Save the response to file

Check this option and enter a file name to save the response to a file. The file name may contain Job Variables. If the file cannot be written, the task fails.

Store the response in variable

Check this option and enter or select a Job Variable to store the response body in that variable. The response is converted to text using the character set specified by the server (UTF-8 if none is specified).

Store response headers in variables

Check this option to store the response headers in Job Variables named WebRequest.ResponseHeader.name. For example, the ETag header is stored in WebRequest.ResponseHeader.ETag. If a header occurs more than once, the values are combined, separated by commas.

To store only some headers, enter a list in the box next to the option (see Include/Exclude Lists below). For example, ETag, Location stores only those two headers, and -Set-Cookie stores all headers except Set-Cookie.

Create variables from JSON/XML

Check this option to automatically create a Job Variable for each value in a JSON or XML response. The format is determined from the response's content type or, if that is not JSON or XML, from the response itself.

Each variable is named with the Variable prefix (by default, WebRequest.Response.) followed by the path to the value:

• For JSON, property names are separated by periods and array elements are numbered from 0. For example, given the response {"order":{"id":17,"lines":[{"sku":"A1"},{"sku":"B2"}]}}, the variables WebRequest.Response.order.id (17), WebRequest.Response.order.lines[0].sku (A1), and WebRequest.Response.order.lines[1].sku (B2) are created.
• For XML, element names are separated by periods, starting with the root element, and attributes are prefixed with "@". Elements are numbered from 0 only when an element contains more than one child element with the same name. For example, given <order id="17"><line sku="A1"/><line sku="B2"/></order>, the variables WebRequest.Response.order.@id, WebRequest.Response.order.line[0].@sku, and WebRequest.Response.order.line[1].@sku are created. Namespace prefixes are not included in the names.

To create variables for only part of the response, enter a list of paths in the box next to the option (see Include/Exclude Lists below). A path also selects everything beneath it: for example, order;-order.lines creates variables for the order but not its lines, and order.lines[*].sku creates variables for the SKU of every line.

If the response cannot be parsed as JSON or XML, a warning is logged and no variables are created but the task does not fail.

Include/Exclude Lists

The header and path lists use the following rules:

• Separate names with commas or semicolons. Names are not case-sensitive.
• Prefix a name with "-" to exclude it.
• Use "*" as a wildcard to match any characters.
• If the list is empty, everything is included.
• If the list contains only exclusions, everything that is not excluded is included.
• If the list contains any names that are not exclusions, only those names are included, except for any that are excluded. Exclusions always override inclusions, regardless of the order they are listed.

Extract values into variables

Use this list to set specific Job Variables from the response. For each row, specify the Variable to set, the Type of expression, and the Expression:

Type	Expression
JsonPath	A JSONPath expression, such as $.order.id or $.order.lines[0].sku. If the expression selects an object or array, the variable is set to its JSON text. If it selects more than one value, the variable is set to a JSON array of the values.
XPath	An XPath expression, such as /order/@id or count(//line). If the expression selects more than one node, the value of the first is used. Namespace prefixes are not supported; for XML that uses namespaces, use local-name(), for example //*[local-name()='order']/@id.
Header	The name of a response header, such as Location.

If an expression is invalid, or does not find a value in the response, the task fails. (To store a header only if it is present, use Store response headers in variables instead.)

Run a script to process the response

Check this option and select or create a .NET script to process the response. The script is run only if the request succeeds. If the script fails, the task fails.

The response is saved to a temporary file, which is deleted after the script runs. The script's Parameters object provides:

• ResponseText: the response body as text.
• ResponseFileName: the name of the temporary file containing the response, for processing binary or very large responses.
• StatusCode, ReasonPhrase, and ContentType.
• ResponseHeaders: the response headers.
• RequestURL: the URL that was requested.

Treat any HTTP status code as success

By default, the task fails if the server returns a status code outside the range 200-299. Check this option if the task should succeed regardless of the status code; for example, if a 404 status is an expected result that a later step will handle. The status code is always available in the WebRequest.StatusCode Job Variable, and you can use the "HTTP request returns the following status code" response event to respond to specific codes.

Maximum variable size

The maximum number of characters stored in any Job Variable set by the task (from the response body, headers, extracted values, or values created from JSON/XML). Longer values are truncated and a warning is logged. The full response is still available in the saved file, captured file, and response script. Specify 0 for no limit. The default is 1,048,576 characters.

Options

Follow redirects

If checked (the default), redirect responses from the server (such as 301 or 302) are followed automatically, and the task processes the final response. If cleared, the redirect response itself is returned.

Ignore server certificate errors

Check this option to connect to an HTTPS server even if its certificate is not valid (for example, a self-signed certificate on a test server). Because this removes protection against impersonation of the server, use it only for testing or for servers you trust.

Retry options

The request can be retried automatically if it fails due to a temporary problem.

• Number of retries: the maximum number of times to retry the request. The default is 0 (no retries).
• Delay between retries: the number of seconds to wait before each retry. If the server's response includes a Retry-After header requesting a longer delay, the longer delay is used.
• Retry on HTTP status codes: the status codes that cause a retry, separated by commas. Ranges such as 500-599 may be used. The default is 408,429,500,502,503,504.

The request is also retried if the server cannot be reached or the connection fails. A retry is not attempted if it would cause the step to exceed its maximum run time. Each retry is recorded in the Job Log, and the number of attempts is stored in the WebRequest.AttemptCount Job Variable.

Secret Values

Header and parameter values marked as Secret, and the password, token, or client secret on the Authentication page, are stored encrypted. Secret values:

• Cannot be viewed after they are saved. They are shown as "••••••••"; type a new value to replace one.
• Are masked in the Job Log, the step's detail log, the WebRequest.RequestURL variable, and the results of Test Request.
• Are included in exported job definitions in encrypted form.
• May contain Job Variables. Values that come from Job Variables whose values are hidden are also masked in logs.

Variable tokens in values that are not secret are visible to anyone who can view the job. For API keys and other credentials, mark the value as Secret or use a Job Variable whose value is hidden.

Test Request Window

The Test Web Request window, displayed when you click Test Request, sends the request from the adTempus server using the task's current settings, including changes you have not yet saved, and displays the result.

The Job Variables used by the request are listed at the top of the window, with their current values, including changes you have not yet saved. Variables whose values are set only when the job runs (for example, by a previous step) are listed with no value.

To supply a test value, edit the variable. Test values are not saved with the task. The values of hidden variables are not displayed. Unless you override them, their saved values are used.

Click Send Request to send the request. The results are shown on these tabs:

• Summary: the status code, whether the task would succeed, the time taken, the number of attempts, the Windows account used (for Windows authentication), and any errors or warnings.
• Response Body: the response, with JSON and XML responses indented for readability. Very large responses are truncated.
• Response Headers: the headers returned by the server.
• Variables: the Job Variables the task would set from this response, including variables from Extract values into variables and Create variables from JSON/XML. Use this tab to check your extraction expressions.
• Request Sent: the method, URL, and headers that were sent, with secret values masked.

When testing:

• The response is not saved to a file or captured, the response script is not run, and no Job Variables are changed.
• For Windows authentication using the job's account, the job's user account is used if the job has been saved; otherwise, the adTempus service account is used.
• The test is limited to 2 minutes, or the step's maximum run time if that is shorter.
• Testing an existing job requires permission to modify the job.