Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Revision History

2.1.0

2017-01-26T18:35

Preface

The Trafodion REST server is packaged within the trafodion-2.1.0.tar.gz file on the Trafodion download site. This document describes server version 2.1.0. Herein you will find either the definitive documentation on a REST server topic as of its standing when the referenced REST server version shipped, or it will point to the location in javadoc, where the pertinent information can be found.

About This document

This document is a work in progress. The source can be found in the core/rest/src/main/ascidoc directory.

Getting Started

1. Introduction

Quick Start will get your REST server up and running quickly. Configuration describes setup of REST in more detail.

2. Quick Start

This chapter describes setup of a Trafodion REST server. It leads you through the starting up and shutting down of your REST server. The Trafodion installer modifies the configuration files for you and starts/stops the server as part of the Trafodion sqstart and sqstop scripts so the following sections are for those times when you need a non-standard configuration or wish to start/stop the server independently.

Is Trafodion installed and running?

The server presumes a Trafodion instance is installed and running on your machine and available on your path; i.e. the TRAF_HOME environment variable is set and when you type sqcheck, you see output that confirms Trafodion is running. If this is not the case, the server may start but you’ll see many errors.

At this point, you are ready to start the server.

2.1. Starting

Now start your server:

 $ bin/start-rest.sh
 starting rest, logging to /logs/rest-user-1-rest-hostname.out

You should now have a running server. Logs can be found in the logs subdirectory. Peruse them especially if the server had trouble starting.

2.2. Stopping

Stop your server by running the stop script.

 $ ./bin/stop-rest.sh
 stopping rest..

Architecture

3. Overview

3.1. REST

The Trafodion REST server, see figure Figure 1, is a process that give access to cluster, nodes, applications, transactions and ODBC/JDBC Type4 connection information.

The REST server provides the following:

A lightweight standalone server.
Simple configuration and startup.
Well known REST API.
Embedded Jetty server.
Embedded JDBC Type 2/Type 4 driver.
Embedded ZooKeeper client.
Automatic script change detection and recompile.
HTTPS encryption.
100% Java implementation.

Figure 1: Trafodion REST Sever Architecture

4. Client

Http clients connect to the server using the default port 4200 for HTTP and 4201 if HTTPS is setup in the embedded Jetty server.

5. TrafodionRest

TrafodionRest is the implementation of the server. Using its embedded Jetty server it services Http client HTTP and HTTPS requests. It’s a 100% java implementation.

5.1. Startup Behavior

The server is started/stopped via the scripts found in the $REST_INSTALL_DIR/bin directory. During startup it opens a connection to ZooKeeper.

5.2. Threads

The server runs several background threads:

5.2.1. Embedded Jetty

An embedded Jetty multi-threaded server services all client requests. A default port is configured but this may be changed in the configuration by modifying the rest.port and rest.https.port properties.

5.2.2. ScriptManager

The script manager thread is responsible for reading and compiling the Python scripts found in $REST_INSTALL_DIR/bin/scripts directory. It can detect a change in any script found there. If any script changes it will recompile it.

5.2.3. JDBC Type 4

An embedded driver is available to execute Trafodion SQL queries by connecting to the DCS service.

5.2.4. JDBC Type 2

An embedded driver is available to execute Trafodion SQL queries.

Configuration

This chapter describes the server configuration. Even though the Trafodion installer prepares the configuration files for you a thorough read of this chapter will help with non-standard configurations. Please read this chapter carefully and ensure that all requirements have been satisfied. Failure to do so will cause you grief debugging strange errors.

Trafodion REST uses the same configuration mechanism as Apache Hadoop. All configuration files are located in the $REST_INSTALL_DIR/conf/ directory.

Be careful editing XML. Make sure you close all elements. Run your file through xmllint or similar to ensure well-formedness of your document after an edit session.

Keep Configuration In Sync Across the Cluster

After you make an edit to any configuration file, make sure you copy the content of the $REST_INSTALL_DIR/conf directory to all nodes of the cluster. The server will not do this for you. Use rsync, scp, or another secure mechanism for copying the configuration files to your nodes. A restart is needed for servers to pick up changes.

6. ZooKeeper

The server depends on a running ZooKeeper cluster to retrieve DCS server, client connection status and metrics.

Set the ensemble servers and client port in rest-site.xml using the rest.zookeeper.quorum and rest.zookeeper.property.clientPort properties. This rest.zookeeper.quorum property defaults to a single ensemble member at localhost.

<configuration>
  ...
  <property>
    <name>rest.zookeeper.property.clientPort</name>
    <value>2181</value>
    <description>The port at which ZooKeeper server is listening for clients.
    </description>
  </property>
  <property>
    <name>rest.zookeeper.quorum</name>
    <value>
    host1.example.com,host2.example.com,host3.example.com,host4.example.com,host5.example.com
    </value>
    <description>Comma separated list of servers in the ZooKeeper Quorum.
    For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
    By default this is set to localhost. For a multi-node setup, this should be set to a full
    list of ZooKeeper quorum servers.
    </description>
  </property>
...
</configuration>

7. Configuration Files

7.1. rest-site.xml and rest-default.xml

You add site-specific configuration to the file $REST_INSTALL_DIR/conf/rest-site.xml. For the list of configurable properties, see REST Default Configuration below or view the raw rest-default.xml source file in the source code at src/main/resources.

Not all configuration options make it out to rest-default.xml. Configuration that it is thought rare anyone would change can exist only in code; the only way to discover such configurations is via a reading of the source code itself.

Currently, changes here will require a restart for the server to notice the change.

7.2. REST Default Configuration

The documentation below is generated using the default rest configuration file, rest-default.xml, as source.

rest.dns.interface: Description

The server uses the local host name for reporting its IP address. If your machine has multiple interfaces the server will use the interface that the primary host name resolves to. If this is insufficient, you can set this property to indicate the primary interface e.g., "eth1". This only works if your cluster configuration is consistent and every host has the same network interface configuration.

Default

default

zookeeper.session.timeout: Description

ZooKeeper session timeout. The server passes this to the ZooKeeper quorum as suggested maximum time for a session (This setting becomes ZooKeeper’s 'maxSessionTimeout'). See http://hadoop.apache.org/ZooKeeper/docs/current/ZooKeeperProgrammers.html#ch_zkSessions "The client sends a requested timeout, the server responds with the timeout that it can give the client. " In milliseconds.

Default

180000

zookeeper.znode.parent: Description

Root znode in ZooKeeper. The server will look for DCS znodes under this znode and will create any REST server specific znodes here as well.

Default

/${user.name}

rest.zookeeper.quorum: Description

Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". By default this is set to localhost. For a fully-distributed setup, this should be set to a full list of ZooKeeper quorum servers.

Default

localhost

rest.zookeeper.property.clientPort: Description

The port at which ZooKeeper is listening for clients.

Default

2181

rest.port: Description

The http port for the REST server.

Default

4200

rest.https.port: Description

The https port for the REST server.

Default

4201

rest.readonly: Description

Defines the mode the server will be started in. Possible values are: false: All HTTP methods are permitted - GET/PUT/POST/DELETE. true: Only the GET method is permitted.

Default

false

rest.threads.max: Description

The maximum number of threads of the server thread pool. Threads in the pool are reused to process requests. This controls the maximum number of requests processed concurrently. It may help to control the memory used by the server to avoid out of memory issues. If the thread pool is full, incoming requests will be queued up and wait for some free threads. The default is 100.

Default

100

rest.threads.min: Description

The minimum number of threads of the server thread pool. The thread pool always has at least these number of threads so the server is ready to serve incoming requests. The default is 2.

Default

2

7.3. rest-env.sh

Set server environment variables in this file. Examples include options to pass the JVM on start of the server such as heap size and garbarge collector configs. You can also set configurations for server configuration, log directories, niceness, ssh options, where to locate process pid files, etc. Open the file at $REST_INSTALL_DIR/conf/rest-env.sh and peruse its content. Each option is fairly well documented. Add your own environment variables here if you want them read by the server on startup.

Changes here will require a restart for the server to notice the change.

7.4. log4j.properties

Edit this file to change rate at which the server log files are rolled and to change the level at which the server logs messages.

Changes here will require a restart for the server to notice the change.

7.5. rest-keystore

This file will only be present if the property rest.https.password has been set in $REST_INSTALL_DIR/conf/rest-site.xml. Please see Important Configurations for more detail.

8. The Important Configurations

Below we list the important Configurations.

8.1. `rest.port`

The default value is 4200. This is the port the embedded Jetty server binds to waiting for client HTTP connections. The value may need to be changed if this port number conflicts with other ports in use on your cluster.

To change this configuration, edit $REST_INSTALL_DIR/conf/rest-site.xml, copy the changed file around the cluster and restart.

8.2. `rest.https.port`

The default value is 4201. This is the port the embedded Jetty server binds to waiting for client HTTPS connections. The value may need to be changed if this port number conflicts with other ports in use on your cluster.

8.3. `rest.ssl.password`

This property is not present by default in $REST_INSTALL_DIR/conf/rest-site.xml. Typically the Trafodion installer sets this property to an obfuscated password string. If this property is present then the server will setup HTTPS in the embedded Jetty server. To create an obfuscated password for this property use the following command:

java -classpath $REST_INSTALL_DIR/lib/jetty-9.2.10.v20150310.jar:$REST_INSTALL_DIR/lib/jetty-util-9.2.10.v20150310.jar org.eclipse.jetty.util.security.Password}

The output of this command is similar to:

OBF:{obfuscated password string}
MD5:{obfuscated password string}

Copy/paste the entire string including "OBF:" into this property. The server will use this property when creating the $REST_INSTALL_DIR/conf/rest-keystore directory.

To change this configuration, edit $REST_INSTALL_DIR/conf/rest-site.xml, copy the changed file around the cluster and restart.

To decrypt the obfuscated string in rest.ssl.password do the following:

java -classpath $REST_INSTALL_DIR/lib/jetty-9.2.10.v20150310.jar:$REST_INSTALL_DIR/lib/jetty-util-9.2.10.v20150310.jar org.eclipse.jetty.util.security.Password {obfuscated password string}

The output of this command is similar to:

{your unobfuscated password}
OBF:{obfuscated password string}
MD5:{obfuscated password string}
CRYPT:{obfuscated password string}

8.4. `dcs.master.port`

The default value is 23400. This is the port the embedded JDBC Type 4 driver uses to connect to Trafodion DCS.

To change this configuration, edit $REST_INSTALL_DIR/conf/rest-site.xml, copy the changed file around the cluster and restart.

APIs

9. REST APIs

The URIs for the REST-based web services have the following syntax:

http://{hostname:port}/{version}/{resourcepath}

The elements in this syntax are as follows:

{hostname} - The http address of the service to get information about.
{version} - The version of the APIs. In this release, the version is v1.
{resourcepath} - A path that defines a singleton resource or a collection of resources.

There are many ways/languages to use the REST API’s. The following examples use the curl command line interface to do the REST calls.

EndPoint

HTTP Verb

Description

Example

/servers

GET

Displays the status of various Trafodion components

Status of Trafodion components

/dtm

GET

Displays the status of Distributed Transaction Manager(DTM)

Status of DTM

/rms

GET

Displays the status of Runtime Management Statistics(RMS)

Status of RMS

/dcs

GET

Displays the status of Database Connectivity Servers(DCS)

Status of DCS

/dcs/summary

GET

Displays the configuration information of Database Connectivity Servers(DCS)

DCS configuration information

/dcs/connections

GET

Displays the client/server information of Database Connectivity Servers(DCS) from zookeeper

Client-Server information

/nodes

GET

Displays the status of Trafodion nodes

Status of nodes

/pstack

GET

Displays the call stack for a collection of Trafodion processes

Various call stack information

/pstack/program/{program}

GET

Displays call stack for a specific Trafodion process

Call stack of a specific process

/jstack/program/{program}

GET

Displays call stack for a specific java process

Call stack of java process

/transactions

GET

Displays the status of Distributed Transaction manager(DTM)

Status of transactions

/transactions/stats

GET

Displays the status of Distributed Transaction manager(DTM) service statistics

Service statistics of transactions

9.1. Displays the status of various components

In this example the user wishes to know the status of all Trafodion components. This a jsonized version of the sqcheck -c all command.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/

Response

{
  "STATE":"UP",
  "SUBSTATE":"OPERATIONAL",
  "PROCESSES":
  [
    {
      "PROCESS":"DTM",
      "CONFIGURED":2,
      "ACTUAL":2,"DOWN":""
    },
    {
      "PROCESS":"RMS",
      "CONFIGURED":4,
      "ACTUAL":4,"DOWN":""
    },
    {
      "PROCESS":"DCSMASTER",
      "CONFIGURED":2,
      "ACTUAL":2,
      "DOWN":""
    },
    {
      "PROCESS":"MXOSRVR",
      "CONFIGURED":4,
      "ACTUAL":0,
      "DOWN":"4"
    }
  ]
}

9.2. Displays the status of Distributed Transaction Manager(DTM)

In this example the user wishes to know the status of Trafodion DTM. This a jsonized version of the sqcheck -c dtm command.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/dtm

Response

  "STATE":"UP",
  "SUBSTATE":"OPERATIONAL",
  "PROCESSES":
  [
    {
      "PROCESS":"DTM",
      "CONFIGURED":2,
      "ACTUAL":2,
      "DOWN":""
    },
  ]
}

9.3. Status of RMS

In this example the user wishes to know the status of Trafodion RMS. This a jsonized version of the sqcheck -c rms command.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/rms

Response

{
  "STATE":"UP",
  "SUBSTATE":"OPERATIONAL",
  "PROCESSES":
  [
    {
      "PROCESS":"RMS",
      "CONFIGURED":4,
      "ACTUAL":4,
      "DOWN":""
    },
  ]
}

9.4. Status of Database Connectivity Service (DCS)

In this example the user wishes to know the status of Trafodion DCS components. This a jsonized version of the sqcheck -c dcs command.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/dcs

Response

{
  "STATE":"UP",
  "SUBSTATE":"OPERATIONAL",
  "PROCESSES":
  [
    {
      "PROCESS":"DCSMASTER",
      "CONFIGURED":2,
      "ACTUAL":2,
      "DOWN":""
    },
    {
      "PROCESS":"MXOSRVR",
      "CONFIGURED":4,
      "ACTUAL":4,
      "DOWN":""
    },
  ]
}

9.5. Summary of DCS Configuration

In this example the user wishes to see a summary of the Trafodion DCS configuration. The server retrieves this information by executing dcscheck script.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/dcs/summary

Response

{
  "Cluster Configuration": "HA",
  "Zookeeper listen port": "2181",
  "DcsMaster listen port": "23400",
  "Configured Primary DcsMaster": "\"node1\"",
  "Configured Backup DcsMasters": "\"node2 node3\"",
  "Active DcsMaster": "\"node1\", pid 8526"
}

9.6. Displays the client/server information

In this example the user wishes to see the Trafodion DCS server/client connection information. The server retrieves this information from ZooKeeper.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/dcs/connections

Response

[
  {
    "HOSTNAME":"hostname",
    "INSTANCE":"1",
    "START_TIME":"Wed Mar 25 18:58:20 UTC 2015",
    "REGISTERED":"YES",
    "STATE":"AVAILABLE",
    "NID":"0",
    "PID":"21132",
    "PROCESS_NAME":"$Z000H8S",
    "IP_ADDRESS":"16.235.163.124",
    "PORT":"36176",
    "CLIENT_NAME":"",
    "CLIENT_APPL":"",
    "CLIENT_IP_ADDRESS":"",
    "CLIENT_PORT":""
  },
  {
    "HOSTNAME":"hostname",
    "INSTANCE":"1",
    "START_TIME":"Wed Mar 25 18:58:20 UTC 2015",
    "REGISTERED":"YES",
    "STATE":"AVAILABLE",
    "NID":"0",
    "PID":"20642",
    "PROCESS_NAME":"$Z000GUS",
    "IP_ADDRESS":"16.235.163.124",
    "PORT":"36174",
    "CLIENT_NAME":"",
    "CLIENT_APPL":"",
    "CLIENT_IP_ADDRESS":"",
    "CLIENT_PORT":""
  }
]

9.7. Displays status of Trafodion nodes

In this example the user wishes to know the status of Trafodion nodes. This is a jsonized version of the sqnodestatus command.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/nodes

Response

[
  {
    "NODE":"n013",
    "STATUS":"UP"
  },
  {
    "NODE":"n014",
    "STATUS":"UP"
  },
  {
    "NODE":"n015",
    "STATUS":"UP"
  },
  {
    "NODE":"n016",
    "STATUS":"UP"
  }
]

9.8. Gets the collection of stack for various Trafodion process

In this example the user wishes to see the call stack for a collection of Trafodion processes. This is a jsonized version of the sqpstack command. Newlines are added to all lines in the response so clients can recognize each end of line.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/pstack

Response

[
  {
  "PROGRAM":"pstack 6332\n
  #0  0x00000034c10df218 in poll () from \/lib64\/libc.so.6\n
  #1  0x00000034c243c655 in ?? () from \/lib64\/libglib-2.0.so.0\n
  #2  0x00000034c243cd55 in g_main_loop_run () from \/lib64\/libglib-2.0.so.0\n
  #3  0x00000000004105f1 in ?? ()\n
  #4  0x00000034c101ecdd in __libc_start_main () from \/lib64\/libc.so.6\n
  #5  0x0000000000407359 in ?? ()\n
  #6  0x00007fffffffe0b8 in ?? ()\n
  #7  0x000000000000001c in ?? ()\n
  #8  0x0000000000000001 in ?? ()\n
  #9  0x00007fffffffe3f8 in ?? ()\n
  #10 0x0000000000000000 in ?? ()\n"
  },
  {
  "PROGRAM":"pstack 6334\n
  #0  0x00000034c10df218 in poll () from \/lib64\/libc.so.6\n
  #1  0x00000034c243c655 in ?? () from \/lib64\/libglib-2.0.so.0\n
  #2  0x00000034c243cd55 in g_main_loop_run () from \/lib64\/libglib-2.0.so.0\n
  #3  0x0000000000406611 in ?? ()\n
  #4  0x00000034c101ecdd in __libc_start_main () from \/lib64\/libc.so.6\n
  #5  0x00000000004044a9 in ?? ()\n
  #6  0x00007fffffffe0b8 in ?? ()\n
  #7  0x000000000000001c in ?? ()\n
  #8  0x0000000000000001 in ?? ()\n
  #9  0x00007fffffffe3f0 in ?? ()\n
  #10 0x0000000000000000 in ?? ()\n"
  },
  {
  "PROGRAM":"pstack 6336\n
  Thread 2 (Thread 0x7ffff213a700 (LWP 6337)):\n
  #0  0x00000034c10acb8d in nanosleep () from \/lib64\/libc.so.6\n
  #1  0x00000034c10aca00 in sleep () from \/lib64\/libc.so.6\n
  #2  0x00000034c3c02600 in ?? () from \/usr\/lib64\/libusbmuxd.so.1\n
  #3  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #4  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 1 (Thread 0x7ffff7fcc7a0 (LWP 6336)):\n
  #0  0x00000034c10df253 in poll () from \/lib64\/libc.so.6\n
  #1  0x00000034c243c655 in ?? () from \/lib64\/libglib-2.0.so.0\n
  #2  0x00000034c243cd55 in g_main_loop_run () from \/lib64\/libglib-2.0.so.0\n
  #3  0x0000000000405101 in ?? ()\n
  #4  0x00000034c101ecdd in __libc_start_main () from \/lib64\/libc.so.6\n
  #5  0x0000000000403ee9 in ?? ()\n
  #6  0x00007fffffffe0b8 in ?? ()\n
  #7  0x000000000000001c in ?? ()\n
  #8  0x0000000000000001 in ?? ()\n
  #9  0x00007fffffffe3f8 in ?? ()\n
  #10 0x0000000000000000 in ?? ()\n"
  },
  {
  "PROGRAM":"pstack 11059\n
  #0  0x00000034c10df218 in poll () from \/lib64\/libc.so.6\n
  #1  0x00000034c243c655 in ?? () from \/lib64\/libglib-2.0.so.0\n
  #2  0x00000034c243cd55 in g_main_loop_run () from \/lib64\/libglib-2.0.so.0\n
  #3  0x00000000004105f1 in ?? ()\n
  #4  0x00000034c101ecdd in __libc_start_main () from \/lib64\/libc.so.6\n
  #5  0x0000000000407359 in ?? ()\n
  #6  0x00007fffffff1fb8 in ?? ()\n
  #7  0x000000000000001c in ?? ()\n
  #8  0x0000000000000001 in ?? ()\n
  #9  0x00007fffffff2868 in ?? ()\n
  #10 0x0000000000000000 in ?? ()\n"
  },
  {
  "PROGRAM":"pstack 11066\n
  #0  0x00000034c10df218 in poll () from \/lib64\/libc.so.6\n
  #1  0x00000034c243c655 in ?? () from \/lib64\/libglib-2.0.so.0\n
  #2  0x00000034c243cd55 in g_main_loop_run () from \/lib64\/libglib-2.0.so.0\n
  #3  0x0000000000406611 in ?? ()\n
  #4  0x00000034c101ecdd in __libc_start_main () from \/lib64\/libc.so.6\n
  #5  0x00000000004044a9 in ?? ()\n
  #6  0x00007fffffff1fb8 in ?? ()\n
  #7  0x000000000000001c in ?? ()\n
  #8  0x0000000000000001 in ?? ()\n
  #9  0x00007fffffff2860 in ?? ()\n
  #10 0x0000000000000000 in ?? ()\n"
  },
  {
  "PROGRAM":"pstack 11068\n
  Thread 2 (Thread 0x7ffff2139700 (LWP 11070)):\n
  #0  0x00000034c10acb8d in nanosleep () from \/lib64\/libc.so.6\n
  #1  0x00000034c10aca00 in sleep () from \/lib64\/libc.so.6\n
  #2  0x00000034c3c02600 in ?? () from \/usr\/lib64\/libusbmuxd.so.1\n
  #3  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #4  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 1 (Thread 0x7ffff7fcb7a0 (LWP 11068)):\n
  #0  0x00000034c10df253 in poll () from \/lib64\/libc.so.6\n
  #1  0x00000034c243c655 in ?? () from \/lib64\/libglib-2.0.so.0\n
  #2  0x00000034c243cd55 in g_main_loop_run () from \/lib64\/libglib-2.0.so.0\n
  #3  0x0000000000405101 in ?? ()\n
  #4  0x00000034c101ecdd in __libc_start_main () from \/lib64\/libc.so.6\n
  #5  0x0000000000403ee9 in ?? ()\n
  #6  0x00007fffffff1fb8 in ?? ()\n
  #7  0x000000000000001c in ?? ()\n
  #8  0x0000000000000001 in ?? ()\n
  #9  0x00007fffffff2868 in ?? ()\n
  #10 0x0000000000000000 in ?? ()\n"
  },
  {
  "PROGRAM":"pstack 19573\n
  Thread 8 (Thread 0x7ffff7726700 (LWP 19578)):\n
  #0  0x00000034c10e8f03 in epoll_wait () from \/lib64\/libc.so.6\n
  #1  0x000000000045fe8e in CRedirector::redirectThread() ()\n
  #2  0x00000000004605b5 in redirect(void*) ()\n
  #3  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #4  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 7 (Thread 0x7ffff6d04700 (LWP 19581)):\n
  #0  0x00000034c140b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from \/lib64\/libpthread.so.0\n
  #1  0x00000000004635b0 in CLock::timedWait(timespec*) ()\n
  #2  0x0000000000479f86 in CHealthCheck::healthCheckThread() ()\n
  #3  0x000000000047a57b in healthCheck(void*) ()\n
  #4  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #5  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 6 (Thread 0x7ffff6303700 (LWP 19583)):\n
  #0  0x00000034c140e84d in accept () from \/lib64\/libpthread.so.0\n
  #1  0x000000000041df72 in CCluster::AcceptSock(int) ()\n
  #2  0x000000000041dcf5 in CCluster::AcceptCommSock() ()\n
  #3  0x000000000047da7f in CCommAccept::commAcceptorSock() ()\n
  #4  0x000000000047d716 in CCommAccept::commAcceptor() ()\n
  #5  0x000000000047dd6d in commAccept(void*) ()\n
  #6  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #7  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 5 (Thread 0x7ffff39df700 (LWP 19584)):\n
  #0  0x00000034c1033ad7 in sigwaitinfo () from \/lib64\/libc.so.6\n
  #1  0x000000000044d7c3 in serialRequestThread(void*) ()\n
  #2  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #3  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 4 (Thread 0x7ffff2fde700 (LWP 19586)):\n
  #0  0x00000034c140b43c in pthread_cond_wait@@GLIBC_2.3.2 () from \/lib64\/libpthread.so.0\n
  #1  0x0000000000463735 in CLock::wait() ()\n
  #2  0x0000000000450e05 in SQ_LocalIOToClient::waitForNoticeWork() ()\n
  #3  0x000000000044dafc in pendingNoticeThread(void*) ()\n
  #4  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #5  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 3 (Thread 0x7ffff25dd700 (LWP 19587)):\n
  #0  0x00000034c1033ad7 in sigwaitinfo () from \/lib64\/libc.so.6\n
  #1  0x000000000044dc8c in lioBufCleanupThread(void*) ()\n
  #2  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #3  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 2 (Thread 0x7ffff1bcb700 (LWP 19591)):\n
  #0  0x00000034c140b43c in pthread_cond_wait@@GLIBC_2.3.2 () from \/lib64\/libpthread.so.0\n
  #1  0x0000000000463735 in CLock::wait() ()\n#2  0x0000000000489d2a in CReqQueue::getRequest() ()\n
  #3  0x000000000047e3ca in CReqWorker::reqWorkerThread() ()\n
  #4  0x000000000047e6d8 in reqWorker(void*) ()\n
  #5  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #6  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 1 (Thread 0x7ffff7b38b40 (LWP 19573)):\n
  #0  0x00000034c10e8f03 in epoll_wait () from \/lib64\/libc.so.6\n
  #1  0x0000000000417ee9 in CCluster::AllgatherSock(int, void*, char*, int, MPI_Status*) ()\n
  #2  0x0000000000417103 in CCluster::Allgather(int, void*, char**, int, MPI_Status*) ()\n
  #3  0x000000000041c48a in CCluster::exchangeNodeData() ()\n
  #4  0x0000000000409c9c in main ()\n"}]

9.9. Displays call stack information for a specific process

In this example the user wishes to see the call stack for Trafodion process id 20642. This is a jsonized version of the sqpstack [<program>] command. Newlines are added to all lines in the response so clients can recognize each end of line.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/pstack/program/20642

Response

[
  {
  "PROGRAM":"pstack 20642\n
  Thread 8 (Thread 0x7fffecb17700 (LWP 20660)):\n
  #0  0x00000034c10e94cd in accept () from \/lib64\/libc.so.6\n
  #1  0x00007ffff77859a5 in SB_Trans::Sock_Listener::accept() () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #2  0x00007ffff778c4f6 in SB_Trans::Sock_Stream_Accept_Thread::run() () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #3  0x00007ffff778c223 in sock_stream_accept_thread_fun(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #4  0x00007ffff53b7b0f in SB_Thread::Thread::disp(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #5  0x00007ffff53b7f67 in thread_fun(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #6  0x00007ffff53bb1dc in sb_thread_sthr_disp(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #7  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #8  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 7 (Thread 0x7fffec116700 (LWP 20664)):\n
  #0  0x00000034c140b43c in pthread_cond_wait@@GLIBC_2.3.2 () from \/lib64\/libpthread.so.0\n
  #1  0x00007ffff53ba5a2 in SB_Thread::CV::wait() () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #2  0x00007ffff53ba67e in SB_Thread::CV::wait(bool) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #3  0x00007ffff777ec57 in SB_Sig_Queue::remove() () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #4  0x00007ffff778c987 in SB_Trans::Sock_Stream_Helper_Thread::run() () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #5  0x00007ffff778c24a in sock_helper_thread_fun(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #6  0x00007ffff53b7b0f in SB_Thread::Thread::disp(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #7  0x00007ffff53b7f67 in thread_fun(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #8  0x00007ffff53bb1dc in sb_thread_sthr_disp(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #9  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #10 0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 6 (Thread 0x7fffe97f2700 (LWP 20671)):\n
  #0  0x00000034c103399d in sigtimedwait () from \/lib64\/libc.so.6\n
  #1  0x00007ffff774a5d4 in local_monitor_reader(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #2  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #3  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 5 (Thread 0x7fffe8df1700 (LWP 20677)):\n
  #0  0x00000034c10df253 in poll () from \/lib64\/libc.so.6\n
  #1  0x00007ffff6691482 in do_io () from trafodion\/git\/core\/sqf\/export\/lib64d\/libzookeeper_mt.so.2\n
  #2  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #3  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 4 (Thread 0x7fffe83f0700 (LWP 20679)):\n
  #0  0x00000034c140b43c in pthread_cond_wait@@GLIBC_2.3.2 () from \/lib64\/libpthread.so.0\n
  #1  0x00007ffff669126b in do_completion () from trafodion\/git\/core\/sqf\/export\/lib64d\/libzookeeper_mt.so.2\n
  #2  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #3  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 3 (Thread 0x7fffe79ef700 (LWP 20680)):\n
  #0  0x00000034c1032d85 in sigwait () from \/lib64\/libc.so.6\n
  #1  0x00007ffff7793aaf in SB_Timer_Thread::run() () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #2  0x00007ffff77938b7 in sb_timer_thread_fun(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #3  0x00007ffff53b7b0f in SB_Thread::Thread::disp(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #4  0x00007ffff53b7f67 in thread_fun(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #5  0x00007ffff53bb1dc in sb_thread_sthr_disp(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #6  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #7  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 2 (Thread 0x7fffe6f90700 (LWP 20685)):\n
  #0  0x00000034c10e14f3 in select () from \/lib64\/libc.so.6\n
  #1  0x00000000004c47a7 in CNSKListenerSrvr::tcpip_listener(void*) ()\n
  #2  0x00007ffff53bb1dc in sb_thread_sthr_disp(void*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #3  0x00000034c1407851 in start_thread () from \/lib64\/libpthread.so.0\n
  #4  0x00000034c10e890d in clone () from \/lib64\/libc.so.6\n
  Thread 1 (Thread 0x7fffecdab2e0 (LWP 20642)):\n
  #0  0x00000034c140b43c in pthread_cond_wait@@GLIBC_2.3.2 () from \/lib64\/libpthread.so.0\n
  #1  0x00007ffff53ba5a2 in SB_Thread::CV::wait() () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #2  0x00007ffff53ba623 in SB_Thread::CV::wait(bool) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbutil.so\n
  #3  0x00007ffff775c7d6 in SB_Ms_Event_Mgr::wait(long) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #4  0x00007ffff777d17e in XWAIT_com(short, int, bool) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #5  0x00007ffff777cf2f in XWAIT(short, int) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbms.so\n
  #6  0x00007ffff79e932a in fs_int_fs_file_awaitiox(short*, void**, int*, long*, int, short*, bool, bool) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbfs.so\n
  #7  0x00007ffff79e2a09 in BAWAITIOX(short*, void**, int*, long*, int, short*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbfs.so\n
  #8  0x00007ffff79e5a9d in XAWAITIOX(short*, void**, unsigned short*, long*, int, short*) () from trafodion\/git\/core\/sqf\/export\/lib64d\/libsbfs.so\n
  #9  0x00000000004c4d85 in CNSKListenerSrvr::runProgram(char*, long, int) ()\n
  #10 0x00000000005a12bb in runCEE(char*, long, int) ()\n
  #11 0x00000000005a35f3 in main ()\n"
  }
]

9.10. Displays the call stack for a specific java process

In this example the user wishes to see the call Java stack for Trafodion process id 20642. This is a jsonized version of the jstack [<program>] command. Newlines are added to all lines in the response so clients can recognize each end of line.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/servers/jstack/program/8439

Response

[
  {
  "PROGRAM":"2015-04-02 20:47:48
  Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):

  "org.eclipse.jface.text.reconciler.MonoReconciler" daemon prio=10 tid=0x0000000005c55800 nid=0x62e7 in Object.wait() [0x00007fffe600e000]
     java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00000000ecd27e50> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)
      at org.eclipse.jface.text.reconciler.AbstractReconciler$BackgroundThread.run(AbstractReconciler.java:179)
      - locked <0x00000000ecd27e50> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)

  "Worker-295" prio=10 tid=0x000000000479b800 nid=0x6036 in Object.wait() [0x00007fffe2e6d000]
     java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
      at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
      - locked <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
      at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
      at org.eclipse.core.internal.jobs.Worker.run(Worker.java:51)

  "Worker-293" prio=10 tid=0x0000000005822800 nid=0x6034 in Object.wait() [0x00007fffdec7e000]
     java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
      at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
      - locked <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
      at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
      at org.eclipse.core.internal.jobs.Worker.run(Worker.java:51)

  "Worker-291" prio=10 tid=0x0000000005742000 nid=0x3a60 in Object.wait() [0x00007fffe2850000]
     java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
      at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
      - locked <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
      at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
      at org.eclipse.core.internal.jobs.Worker.run(Worker.java:51)

  "org.eclipse.jdt.internal.ui.text.JavaReconciler" daemon prio=10 tid=0x000000000100f000 nid=0x2f8d in Object.wait() [0x00007fffe5244000]
     java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00000000eca355f0> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)
      at org.eclipse.jface.text.reconciler.AbstractReconciler$BackgroundThread.run(AbstractReconciler.java:179)
      - locked <0x00000000eca355f0> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)

  "Worker-288" prio=10 tid=0x0000000004618800 nid=0x6558 in Object.wait() [0x00007fffe244c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
    at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
    - locked <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
    at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:51)

  "Worker-284" prio=10 tid=0x0000000004148800 nid=0x31e8 in Object.wait() [0x00007fffe5e1c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
    at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
    - locked <0x00000000e1614038> (a org.eclipse.core.internal.jobs.WorkerPool)
    at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:51)

  ...intentionally deleted lines for brevity

  "main" prio=10 tid=0x0000000000618800 nid=0x20f8 runnable [0x00007ffff6f46000]
   java.lang.Thread.State: RUNNABLE
    at org.eclipse.swt.internal.gtk.OS.Call(Native Method)
    at org.eclipse.swt.widgets.Display.sleep(Display.java:4294)
    at org.eclipse.ui.application.WorkbenchAdvisor.eventLoopIdle(WorkbenchAdvisor.java:368)
    at org.eclipse.ui.internal.ide.application.IDEWorkbenchAdvisor.eventLoopIdle(IDEWorkbenchAdvisor.java:918)
    at org.eclipse.ui.internal.Workbench$3.eventLoopIdle(Workbench.java:498)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1155)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
    at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
    at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
    at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:135)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1438)

  "VM Thread" prio=10 tid=0x000000000080c000 nid=0x2104 runnable

  "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000062e000 nid=0x20f9 runnable

  "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000000630000 nid=0x20fb runnable

  "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000000631800 nid=0x20fc runnable

  "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000633800 nid=0x20fd runnable

  "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000000635800 nid=0x20fe runnable

  "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000000637000 nid=0x20ff runnable

  "GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000000639000 nid=0x2100 runnable

  "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000000063b000 nid=0x2101 runnable

  "GC task thread#8 (ParallelGC)" prio=10 tid=0x000000000063d000 nid=0x2102 runnable

  "GC task thread#9 (ParallelGC)" prio=10 tid=0x000000000063e800 nid=0x2103 runnable

  "VM Periodic Task Thread" prio=10 tid=0x0000000000852800 nid=0x210b waiting on condition

  JNI global references: 799

  }
]

9.11. Displays status of transactions

In this example the user wishes to know the status of Trafodion DTM service. This is a jsonized version of the dtmci status tm command.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/transactions

Response

[
  {
    "node":0,
    "isLeadTM":true,
    "state":"UP",
    "sys_recovery_state":"END",
    "tmshutdown_level":"RUNNING",
    "number_active_txns":0
  },
  {
    "node":1,
    "isLeadTM":false,
    "state":"UP",
    "sys_recovery_state":"END",
    "tmshutdown_level":"RUNNING",
    "number_active_txns":0
  }
]

9.12. Displays the transaction service statistics

In this example the user wishes to know the Trafodion DTM service statistics. This is a jsonized version of the dtmci stats command.

Http command

curl  -X GET -H "Accept: application/json" http://{hostname}:4200/v1/transactions/stats

Response

[
  {
    "node": 0,
    "txnStats":
    {
      "txnBegins": 17,
      "txnAborts": 0,
      "txnCommits": 13
    }
  },
  {
    "node": 1,
    "txnStats":
    {
      "txnBegins": 0,
      "txnAborts": 0,
      "txnCommits": 0
    }
  }
]

Troubleshooting and Debugging

10. Logs

The key process logs are as follows…(replace <user> with the user that started the service, <instance> for the server instance and <hostname> for the machine name)

$REST_INSTALL_DIR/logs/rest-<user>-<instance>-rest-<hostname>.log
$REST_INSTALL_DIR/logs/rest-<user>-<instance>-rest-<hostname>.out

The logs are your best resource when troubleshooting the REST server.

11. Tools

11.1. zkcli

zkcli is a very useful tool for investigating ZooKeeper-related issues. To invoke:

./rest zkcli -server host:port <cmd> <args>

The commands (and arguments) are:

	connect host:port
	get path [watch]
	ls path [watch]
	set path data [version]
	delquota [-n|-b] path
	quit
	printwatches on|off
	create [-s] [-e] path data acl
	stat path [watch]
	close
	ls2 path [watch]
	history
	listquota path
	setAcl path acl
	getAcl path
	sync path
	redo cmdno
	addauth scheme auth
	delete path [version]
	setquota -n|-b val path

Appendix

Appendix A: Contributing to Documentation

A.1. Reference Guide Style Guide and Cheat Sheet

This Reference Guide is written in Asciidoc and built using AsciiDoctor. The following cheat sheet is included for your reference. More comprehensive documentation is available in the AsciiDoctor user manual.

Table 1. AsciiDoc Cheat Sheet
Element Type	Desired Rendering	How to do it
A paragraph	a paragraph	Just type some text with a blank line at the top and bottom.
Add line breaks within a paragraph without adding blank lines	Manual line breaks	This will break + at the plus sign. Or prefix the whole paragraph with a line containing `[%hardbreaks]`
Give a title to anything	Colored italic bold differently-sized text	`.MyTitle` (no space between the period and the words) on the line before the thing to be titled
In-Line Code or commands	monospace	`text`
In-line literal content (things to be typed exactly as shown)	bold mono	`typethis`
In-line replaceable content (things to substitute with your own values)	bold italic mono	_typesomething_
Code blocks with highlighting	monospace, highlighted, preserve space	[source,java] ---- myAwesomeCode() { } ----
Code block included from a separate file	included just as though it were part of the main file	[source,ruby] ---- include\::path/to/app.rb[] ----
Include only part of a separate file	Similar to Javadoc	See http://www.asciidoctor.org/docs/user-manual/#by-tagged-regions
Filenames, directory names, new terms	italic	_rest-default.xml_
External naked URLs	A link with the URL as link text	link:http://www.google.com
External URLs with text	A link with arbitrary link text	link:http://www.google.com[Google]
Create an internal anchor to cross-reference	not rendered	[[anchor_name]]
Cross-reference an existing anchor using its default title	an internal hyperlink using the element title if available, otherwise using the anchor name	<<anchor_name>>
Cross-reference an existing anchor using custom text	an internal hyperlink using arbitrary text	<<anchor_name,Anchor Text>>
A block image	The image with alt text	image::sunset.jpg[Alt Text] (put the image in the src/main/site/resources/images directory)
An inline image	The image with alt text, as part of the text flow	image:sunset.jpg [Alt Text] (only one colon)
Link to a remote image	show an image hosted elsewhere	image::http://inkscape.org/doc/examples/tux.svg[Tux,250,350] (or `image:`)
Add dimensions or a URL to the image	depends	inside the brackets after the alt text, specify width, height and/or link="http://my_link.com"
A footnote	subscript link which takes you to the footnote	Some text.footnote:[The footnote text.]
A note or warning with no title	The admonition image followed by the admonition	NOTE:My note here WARNING:My warning here
A complex note	The note has a title and/or multiple paragraphs and/or code blocks or lists, etc	.The Title [NOTE] ==== Here is the note text. Everything until the second set of four equals signs is part of the note. ---- some source code ---- ====
Bullet lists	bullet lists	* list item 1 (See http://asciidoctor.org/docs/user-manual/#unordered-lists)
Numbered lists	numbered list	. list item 2 (See http://asciidoctor.org/docs/user-manual/#ordered-lists)
Checklists	Checked or unchecked boxes	Checked: - [*] Unchecked: - [ ]
Multiple levels of lists	bulleted or numbered or combo	. Numbered (1), at top level * Bullet (2), nested under 1 * Bullet (3), nested under 1 . Numbered (4), at top level * Bullet (5), nested under 4 ** Bullet (6), nested under 5 - [x] Checked (7), at top level
Labelled lists / variablelists	a list item title or summary followed by content	Title:: content Title:: content
Sidebars, quotes, or other blocks of text	a block of text, formatted differently from the default	Delimited using different delimiters, see http://asciidoctor.org/docs/user-manual/#built-in-blocks-summary. Some of the examples above use delimiters like ...., ----,====. [example] ==== This is an example block. ==== [source] ---- This is a source block. ---- [note] ==== This is a note block. ==== [quote] ____ This is a quote block. ____ If you want to insert literal Asciidoc content that keeps being interpreted, when in doubt, use eight dots as the delimiter at the top and bottom.
Nested Sections	chapter, section, sub-section, etc	= Book (or chapter if the chapter can be built alone, see the leveloffset info below) == Chapter (or section if the chapter is standalone) === Section (or subsection, etc) ==== Subsection and so on up to 6 levels (think carefully about going deeper than 4 levels, maybe you can just titled paragraphs or lists instead). Note that you can include a book inside another book by adding the `:leveloffset:+1` macro directive directly before your include, and resetting it to 0 directly after. See the index.adoc source for examples, as this is how this guide handles chapters. Don’t do it for prefaces, glossaries, appendixes, or other special types of chapters.
Include one file from another	Content is included as though it were inline	include::[/path/to/file.adoc] For plenty of examples. see book.adoc.
A table	a table	See http://asciidoctor.org/docs/user-manual/#tables. Generally rows are separated by newlines and columns by pipes
Comment out a single line	A line is skipped during rendering	`// This line won’t show up`
Comment out a block	A section of the file is skipped during rendering	//// Nothing between the slashes will show up. ////
Highlight text for review	text shows up with yellow background	Test between #hash marks# is highlighted yellow.

A.2. Auto-Generated Content

Some parts of the REST Reference Guide, most notably REST default configuration, are generated automatically, so that this area of the documentation stays in sync with the code. This is done by means of an XSLT transform, which you can examine in the source at src/main/xslt/configuration_to_asciidoc_chapter.xsl. This transforms the rest-common/src/main/resources/rest-default.xml file into an Asciidoc output which can be included in the Reference Guide. Sometimes, it is necessary to add configuration parameters or modify their descriptions. Make the modifications to the source file, and they will be included in the Reference Guide when it is rebuilt.

It is possible that other types of content can and will be automatically generated from REST source files in the future.

A.3. Images in the REST Reference Guide

You can include images in the REST Reference Guide. It is important to include an image title if possible, and alternate text always. This allows screen readers to navigate to the image and also provides alternative text for the image. The following is an example of an image with a title and alternate text. Notice the double colon.

.My Image Title
image::sunset.jpg[Alt Text]

Here is an example of an inline image with alternate text. Notice the single colon. Inline images cannot have titles. They are generally small images like GUI buttons.

image:sunset.jpg[Alt Text]

When doing a local build, save the image to the src/main/site/resources/images/ directory. When you link to the image, do not include the directory portion of the path. The image will be copied to the appropriate target location during the build of the output.

A.4. Adding a New Chapter to the REST Reference Guide

If you want to add a new chapter to the REST Reference Guide, the easiest way is to copy an existing chapter file, rename it, and change the ID (in double brackets) and title. Chapters are located in the src/main/asciidoc/_chapters/ directory.

Delete the existing content and create the new content. Then open the src/main/asciidoc/book.adoc file, which is the main file for the REST Reference Guide, and copy an existing include element to include your new chapter in the appropriate location. Be sure to add your new file to your Git repository before creating your patch.

When in doubt, check to see how other files have been included.

A.5. Common Documentation Issues

The following documentation issues come up often. Some of these are preferences, but others can create mysterious build errors or other problems.

Syntax Highlighting

The REST Reference Guide uses coderay for syntax highlighting. To enable syntax highlighting for a given code listing, use the following type of syntax:

[source,xml]
----
<name>My Name</name>
----

Several syntax types are supported. The most interesting ones for the REST Reference Guide are java, xml, sql, and bash.

Trafodion REST Server Reference Guide

Preface

Getting Started

1. Introduction

2. Quick Start

2.1. Starting

2.2. Stopping

Architecture

3. Overview

3.1. REST

4. Client

5. TrafodionRest

5.1. Startup Behavior

5.2. Threads

5.2.1. Embedded Jetty

5.2.2. ScriptManager

5.2.3. JDBC Type 4

5.2.4. JDBC Type 2

Configuration

6. ZooKeeper

7. Configuration Files

7.1. rest-site.xml and rest-default.xml

7.2. REST Default Configuration

7.3. rest-env.sh

7.4. log4j.properties

7.5. rest-keystore

8. The Important Configurations

8.1. rest.port

8.2. rest.https.port

8.3. rest.ssl.password

8.4. dcs.master.port

APIs

9. REST APIs

9.1. Displays the status of various components

9.2. Displays the status of Distributed Transaction Manager(DTM)

9.3. Status of RMS

9.4. Status of Database Connectivity Service (DCS)

9.5. Summary of DCS Configuration

9.6. Displays the client/server information

9.7. Displays status of Trafodion nodes

9.8. Gets the collection of stack for various Trafodion process

9.9. Displays call stack information for a specific process

9.10. Displays the call stack for a specific java process

9.11. Displays status of transactions

9.12. Displays the transaction service statistics

Troubleshooting and Debugging

10. Logs

11. Tools

11.1. zkcli

Appendix

Appendix A: Contributing to Documentation

A.1. Reference Guide Style Guide and Cheat Sheet

A.2. Auto-Generated Content

A.3. Images in the REST Reference Guide

A.4. Adding a New Chapter to the REST Reference Guide

A.5. Common Documentation Issues

8.1. `rest.port`

8.2. `rest.https.port`

8.3. `rest.ssl.password`

8.4. `dcs.master.port`