Losing connection to Insight server + node @ 100%


(Akira Takizawa) #1

Good afternoon BitPay

In the past two weeks I’ve set up a Insight for an X11 POW/POS altcoin, using:

  • Insight v0.2.5
  • Insight API v0.2.12
  • Bitcore v0.1.18

Insight successfully loaded the whole blockchain via RPC and I can query every aspect of the altcoin.

Unfortunately, I face a connection problem every 10 to 15 minutes. Insight show “Can’t connect to Insight server. Attempting to reconnect”:

Screenshot: [http://s14.postimg.org/kedcns901/6d624a18_fd99_11e4_8ee7_f8fd76538000.png]

While this error is on screen, I see the follow messages in my log:

info:   Unconfirming 2 Txs
info: Resuming sync from block: 0b08a3ecf5a540693be7e8ae73fc0dc219884de8f80369876b4bef3931dc385d #1558116
info: Got 1558116 blocks in current DB, out of 1558117 blocks at startcoind
info: syncing from RPC (slow)
info: Starting from:  0b08a3ecf5a540693be7e8ae73fc0dc219884de8f80369876b4bef3931dc385d
info: status: [100%]
info: NEW TIP: 0b08a3ecf5a540693be7e8ae73fc0dc219884de8f80369876b4bef3931dc385d NEED REORG (old tip: 2e80f5add1f77861baf99a60f1aeb1557cb4b7906d5a3d14061f20b5456f4072 #1558116)
info: Reorg Case 1) OldNext: 0b08a3ecf5a540693be7e8ae73fc0dc219884de8f80369876b4bef3931dc385d NewHeight: 1558115
info:   Unconfirming 2 Txs
info:   Unconfirming 2 Txs
info: status: [100%]
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }

No new { ‘0’: ‘livenet’, ‘1’: ‘1 of 1 peers connected’ } lines are printed.

It looks as if somehow it can’t manage to connect to the coin’s wallet anymore. The wallet itself is still operational and accepts RPC commands using the command line or a a homebrew daemon management tool written in Java.

I made sure that the right protocol version, matching the wallet is used (60013 iso 70000 or 70002) and client subversion to match my altcoin’s “/Satoshi:1.1.0/”

While Insight doesn’t react anymore and the browser is waiting for connection, the process “node” consumes 100% of the CPU and memory usage keeps rising, as you can see on the graph below:

Screenshot: [https://cloud.githubusercontent.com/assets/8864848/7704460/ba833b2c-fe3e-11e4-873a-4774bd99f53a.png]

Each spike above 1.20 approximately is when it gets connection problems. The prolonged period on Tuesday is when I let it run overnight to see if it would recover. Unfortunately, the only thing it did was effectively crash:

{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
{ '0': 'livenet', '1': '1 of 1 peers connected' }
FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory
Aborted
npm ERR! weird error 134
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian

npm ERR! not ok code 

Could this be due to a misconfiguration or typing mistake in one of the file on my part or is there a known issue with Bitcore v0.1.18? Would it be possible to get an indication what to look for?

I want to tackle this problem first before even attempting to deal with a few other quirks I ran into, working with a POS coin (coinstake transactions (unparsed address and/or NaN), unregistered spent coins and detection of double spending).

Sorry for the links, but I’m not allowed to post a screenshot as a new user.


(V6) #2

// , Did you ever resolve this? I am looking for ways to troubleshoot connections to bitcored in general, as a matter of fact.