Cassandra: nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'

 

I encountered a Cassandra start-up issue after adding a node, specifically after inserting the seed. Restoring the cassandra.yaml couldn't return to its original situation as things have been cached. The VMs had to be restored with snapshots. The node stays up on its own only briefly and then died down with the following error "nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'"

cassandra@cassandra:~$ nodetool status

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address    Load        Tokens  Owns (effective)  Host ID                               Rack 

UN  127.0.0.1  228.44 KiB  16      100.0%            37562f38-b9da-4e77-b44d-6572248ec7d1  rack1

As soon as seeds are added to the cassandra.yaml,  the nodes died.

cassandra@cassandra02:~$ nodetool status

nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'

system.log show the following.

Gossiper.java:2142 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown

I did not clear out the node2 Cassandra data I cloned from node1. Once deleted the data and restarted Cassandra, it came back up. Since node2 was cloned from node1 and its data, it caused a synching issue and refused to start up.

cassandra@cassandra02:/var/log/cassandra$ ls -las /var/lib/cassandra/

total 24

4 drwxr-xr-x 6 cassandra cassandra 4096 Aug 15 16:39 .

4 drwxr-xr-x 42 root root 4096 Aug 15 16:39 ..

4 drwxr-xr-x 2 cassandra cassandra 4096 Sep 29 16:36 commitlog

4 drwxr-xr-x 8 cassandra cassandra 4096 Aug 17 21:18 data

4 drwxr-xr-x 2 cassandra cassandra 4096 Aug 15 16:39 hints

4 drwxr-xr-x 2 cassandra cassandra 4096 Sep 28 16:31 saved_caches

cassandra@cassandra02:/var/log/cassandra$ sudo service cassandra stop 

cassandra@cassandra02:/var/log/cassandra$ sudo rm -rf /var/lib/cassandra/*

cassandra@cassandra02:/var/log/cassandra$ sudo service cassandra start

cassandra@cassandra02:/var/log/cassandra$ nodetool status

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

-- Address Load Tokens Owns (effective) Host ID Rack 

UN 192.168.1.32 265 KiB 16 100.0% 37562f38-b9da-4e77-b44d-6572248ec7d1 rack1

cassandra@cassandra02:/var/log/cassandra$ nodetool status

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

-- Address Load Tokens Owns (effective) Host ID Rack 

UN 192.168.1.35 79.81 KiB 16 51.2% d9d37072-817a-4095-87da-c4fc7a4686f8 rack1

UN 192.168.1.32 265 KiB 16 48.8% 37562f38-b9da-4e77-b44d-6572248ec7d1 rack1

Had tried the following through google, but I need help with it.

It isn't the JMX issue. 

1.$JVM_OPTS -Djava.rmi.server.hostname=127.0.0.1 in cassandra-env.sh

It isn't the Ubuntu firewall UFW issue. It is disabled.

https://www.javatpoint.com/ubuntu-disable-firewall#:~:text=Status%3A%20active-,Disable%20the%20Firewall,%24%20sudo%20UFW%20disable

sudo UFW disable 

It isn't the 2 gig memory issue.

https://stackoverflow.com/questions/43828168/why-am-i-seeing-nodetool-status-connection-refused

Cassandra: How to tell where the data resides and on which nodes.

 

cqlsh:linkedin> select * from location;

 vehicle_id | date       | time                            | latitude | longtitude

------------+------------+---------------------------------+----------+------------

   ME100AAS | 2014-05-19 | 2014-05-19 08:50:00.000000+0000 | 44.74941 |   -67.2507

   ME100AAS | 2014-05-19 | 2014-05-19 08:40:00.000000+0000 | 44.74648 |  -67.26444

   ME100AAS | 2014-05-19 | 2014-05-19 08:30:00.000000+0000 | 44.74258 |  -67.34272

   ME100AAS | 2014-05-19 | 2014-05-19 08:20:00.000000+0000 | 44.72795 |  -67.40177

   ME100AAS | 2014-05-19 | 2014-05-19 08:10:00.000000+0000 | 44.69965 |  -67.47043

   ME100AAS | 2014-05-19 | 2014-05-19 08:00:00.000000+0000 | 44.61909 |   -67.8462

   WA063JXD | 2014-05-19 | 2014-05-19 08:50:00.000000+0000 | 47.70144 | -117.01791

   WA063JXD | 2014-05-19 | 2014-05-19 08:40:00.000000+0000 | 47.69589 | -117.04126

   WA063JXD | 2014-05-19 | 2014-05-19 08:30:00.000000+0000 | 47.68711 |  -117.0701

   WA063JXD | 2014-05-19 | 2014-05-19 08:20:00.000000+0000 | 47.68017 | -117.08932

   WA063JXD | 2014-05-19 | 2014-05-19 08:10:00.000000+0000 | 47.67093 | -117.10924

   WA063JXD | 2014-05-19 | 2014-05-19 08:00:00.000000+0000 | 47.67547 | -117.23619

  CA6AFL218 | 2014-05-19 | 2014-05-19 08:50:00.000000+0000 | 36.11959 | -115.17258

  CA6AFL218 | 2014-05-19 | 2014-05-19 08:40:00.000000+0000 | 36.04423 | -115.18112

  CA6AFL218 | 2014-05-19 | 2014-05-19 08:30:00.000000+0000 | 35.91153 | -115.20631

  CA6AFL218 | 2014-05-19 | 2014-05-19 08:20:00.000000+0000 | 35.88326 | -115.22528

  CA6AFL218 | 2014-05-19 | 2014-05-19 08:10:00.000000+0000 | 35.76301 | -115.33514

  CA6AFL218 | 2014-05-19 | 2014-05-19 08:00:00.000000+0000 | 35.69166 |  -115.3681

cqlsh:linkedin> desc table location

CREATE TABLE linkedin.location (

vehicle_id text,

date text,

time timestamp,

latitude double,

longtitude double,

PRIMARY KEY ((vehicle_id, date), time)

) WITH CLUSTERING ORDER BY (time DESC)

AND additional_write_policy = '99p'

AND bloom_filter_fp_chance = 0.01

AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

AND cdc = false

AND comment = ''

AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}

AND compression = {'chunk_length_in_kb': '16', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

AND memtable = 'default'

AND crc_check_chance = 1.0

AND default_time_to_live = 0

AND extensions = {}

AND gc_grace_seconds = 864000

AND max_index_interval = 2048

AND memtable_flush_period_in_ms = 0

AND min_index_interval = 128

AND read_repair = 'BLOCKING'

AND speculative_retry = '99p';

Provide the token function with the compound keys. Compound keys are hashed and stored in corresponding nodes.

cqlsh:linkedin> select token(vehicle_id,date) from location;

system.token(vehicle_id, date)

--------------------------------

-7657837382140274291

-7657837382140274291

-7657837382140274291

-7657837382140274291

-7657837382140274291

-7657837382140274291

7624412873298128873

7624412873298128873

7624412873298128873

7624412873298128873

7624412873298128873

7624412873298128873

8294848196898204914

8294848196898204914

8294848196898204914

8294848196898204914

8294848196898204914

8294848196898204914

(18 rows)

Use nodetool getendpoint function to figure out where they reside.

cassandra@cassandra02:/var/log/cassandra$ nodetool getendpoints linkedin location '8294848196898204914'

192.168.1.35

cassandra@cassandra02:/var/log/cassandra$ nodetool getendpoints linkedin location '-7657837382140274291'

192.168.1.32

Cassandra: Adding Cassandra node

A quick guide on how to add a Cassandra node on Ubuntu OS.

Specification

VMware Fusion Professional 13.0.2

/var/log/cassandra$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 22.04.3 LTS

Release: 22.04

Codename: jammy

cqlsh cassandra01

Connected to Test Cluster at cassandra01:9042

[cqlsh 6.1.0 | Cassandra 4.1.3 | CQL spec 3.4.6 | Native protocol v5]

Adding Cassandra node

Requirement

1 - a cluster name:  Test Cluster (default and making sure all nodes have the same name)

2 - a static IP for both nodes: 192.168.1.32, 192.168.1.35

3 - delete the cloned nodes' data

Steps

Step 1: Clone the primary node to other nodes

Step 2: Set both nodes IP address to static IP. In my lab, I simply set them to 2 fictitious IPs - 192.168.1.32, 192.168.1.35

Test them both can be ping and ssh.

Step 3: change hostname

sudo vim /etc/hostname

cassandra1

Step 4: Add IP and hostname to /etc/hosts

192.168.1.32 cassandra01

192.168.1.35 cassandra02

Step 5: Reboot the VM

Perform the same for all other nodes and reboot

Step 6: Change the listen_address and rpc_address properties in the cassandra.yaml for each node.

set the IP address as the corresponding node had. If node1 have 192.168.1.1, the listener_address and rpc_address should also have 192.168.1.1 and if node2 has 192.168.1.2 then the listener and rpc should also have 192.1.168.2

Step7: Need at least one seed node for others to join the cluster

Step 8: Restart Cassandra

sudo service cassandra restart

Step 9:  Edit the Cassandra seed section within the cassandra.yaml file and replace a primary IP address on it or all nodes. Perform the same for node2 with the primary IP address.

Step 10: Delete node2 data directory and restart

cassandra@cassandra02:/var/log/cassandra$ ls -las /var/lib/cassandra/

total 24

4 drwxr-xr-x 6 cassandra cassandra 4096 Aug 15 16:39 .

4 drwxr-xr-x 42 root root 4096 Aug 15 16:39 ..

4 drwxr-xr-x 2 cassandra cassandra 4096 Sep 29 16:36 commitlog

4 drwxr-xr-x 8 cassandra cassandra 4096 Aug 17 21:18 data

4 drwxr-xr-x 2 cassandra cassandra 4096 Aug 15 16:39 hints

4 drwxr-xr-x 2 cassandra cassandra 4096 Sep 28 16:31 saved_caches

cassandra@cassandra02:/var/log/cassandra$ sudo service cassandra stop 

cassandra@cassandra02:/var/log/cassandra$ sudo rm -rf /var/lib/cassandra/*

cassandra@cassandra02:/var/log/cassandra$ ps auwx | grep cassandra

root 2048 0.0 0.1 18368 9636 ? Ss 16:02 0:00 sshd: cassandra [priv]

cassand+ 2114 0.0 0.0 18500 6460 ? S 16:02 0:00 sshd: cassandra@pts/0

cassand+ 4616 0.0 0.0 6024 2008 pts/0 R+ 16:47 0:00 grep --color=auto cassandra

cassandra@cassandra02:/var/log/cassandra$ sudo service cassandra start

cassandra@cassandra02:/var/log/cassandra$ nodetool status

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

-- Address Load Tokens Owns (effective) Host ID Rack 

UN 192.168.1.32 265 KiB 16 100.0% 37562f38-b9da-4e77-b44d-6572248ec7d1 rack1

cassandra@cassandra02:/var/log/cassandra$ nodetool status

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

-- Address Load Tokens Owns (effective) Host ID Rack 

UN 192.168.1.35 79.81 KiB 16 51.2% d9d37072-817a-4095-87da-c4fc7a4686f8 rack1

UN 192.168.1.32 265 KiB 16 48.8% 37562f38-b9da-4e77-b44d-6572248ec7d1 rack1

In summary, it adds the listen_address, rpc_address, seeds, deletes the added node's directory, and starts node2 Cassandra.

With the help of the following guide.

https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/initialize/initSingleDS.html