Cassandra: nodetool: Failed to connect to 'IP-ADDRESS:7199' - ConnectException: 'Connection refused (Connection refused)'.

By default, Cassandra JMX listens to localhost only. When Cassandra started up, port 7199 was brought up on localhost. This issue has been bothering me for nearly 2 weeks until I read everything hint in the cassandra-env.sh and refer to this documentation  https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/configuration/secureJmxAuthentication.html. 

 Hopefully, this blog will help someone with a similar issue. The ConnectException can be vague and refer to other issues. As far as I can tell, the same error message has 4 variants of different issues.

 

 

Note: anything connecting to JMX monitoring will fail with ConnectException

cassandra@cassandra01:/etc/cassandra$ nodetool -u cassandra -pw casspass -h 192.168.1.32 status

nodetool: Failed to connect to '102.10.10.91:7199' - ConnectException: 'Connection refused (Connection refused)'.

 

 

cassandra@cassandra01:/etc/cassandra$ nodetool -h 102.10.10.91 -p 7199  decommission

nodetool: Failed to connect to '102.10.10.91:7199' - ConnectException: 'Connection refused (Connection refused)'.

 

Further research showing Cassandra will only be brought up to localhost.

 

cassandra@cassandra01:/etc/cassandra$ lsof -i :7199

COMMAND  PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

java    1778 cassandra   95u  IPv4  28317      0t0  TCP localhost:7199 (LISTEN)

 

 

 

 

Here is how to resolve the issue.

 

 

in Cassandra-env.sh

Enable and set the port for JMS as 7199

 

Uncomment and insert the ip address

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<Pubilc Address>"

TO

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=102.10.10.91"

 

 

Change the yes as local to no, so it enables JMX remote connectivity.

 

if [ "x$LOCAL_JMX" = "x" ]; then

    LOCAL_JMX=yes

fi

 

TO 

 

if [ "x$LOCAL_JMX" = "x" ]; then

    LOCAL_JMX=no

fi

 

 

 

Change the remote authentication from false to true. All these have to be done in the set.  Changing on without the others will not work.

if [ "$LOCAL_JMX" = "yes" ]; then

  JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"

  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"

else

 

TO

 

if [ "$LOCAL_JMX" = "yes" ]; then

  JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"

  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"

else

 

 

Create and insert the following. This will allow the jmx to go to the file to grab the username and password. 

 

sudo vi /etc/cassandra/jmxremote.password

      cassandra casspass

 

chown cassandra:cassandra /etc/cassandra/jmxremote.password

sudo chown cassandra:cassandra /etc/cassandra/jmxremote.password

sudo chmod 400 /etc/cassandra/jmxremote.password

 

 

Restart Cassandra

cassandra@cassandra01:/etc/cassandra$ sudo service cassandra restart

 

 

Test the dedicated host connectivity with username and password

cassandra@cassandra01:/etc/cassandra$ nodetool -u cassandra -pw casspass -h 102.10.10.91 status

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address       Load       Tokens  Owns (effective)  Host ID                               Rack 

UN  102/10/10.92  24.42 MiB  16      51.2%             d9d37072-817a-4095-87da-c4fc7a4686f8  rack1

UN  102.10.10.91  23.31 MiB  16      48.8%             37562f38-b9da-4e77-b44d-6572248ec7d1  rack1

 

 

It is no longer listening to only localhost.

cassandra@cassandra01:/etc/cassandra$ lsof -i :7199

COMMAND  PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

java    3293 cassandra   95u  IPv4  58397      0t0  TCP *:7199 (LISTEN)

 

 

Test without username and password will work as well.

cassandra@cassandra01:/etc/cassandra$ nodetool -h 102.10.10.91 cleanup

All JMX connectivities are working now.