NodeLink crash in the middle of the night

[mblitz]

I've been running NodeLink for a couple of days.  It has crashed in the middle of the night for the last 2 nights.  This started after I configured it to send data to my local emoncms, running on the same RPi2.  I didn't get a stack trace the first time, but was running it from a terminal yesterday/last night, so I have the recent trace.  I am running backups every 4 hours via rsnapshot, which appears to cause the "Emoncms send Errors".  However, if my memory serves me right (it does sometimes), those error were from yesterday evening, before I went to bed.  So I'm not sure the errors are related to the crash.  I will disable backups and see if I get a crash again.

 

2017-01-13 03:00:20 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:00:30 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:00:40 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:00:50 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:01:00 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:01:10 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:01:20 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:01:30 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:01:40 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:04:32 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:04:32 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:04:32 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:05:50 - Emoncms Send Error - The request timed out [gem1]

2017-01-13 03:06:00 - Emoncms Send Error - The request timed out [gem1]

Stacktrace:

 

  at <unknown> <0xffffffff>

  at System.Collections.Concurrent.ConcurrentQueue`1<NodeLink.ThreadSafeList/QueueData>.TryDequeue (NodeLink.ThreadSafeList/QueueData&) <0x000cb>

  at NodeLink.ThreadSafeList.a () <0x000bf>

  at System.Threading.Thread.StartInternal () <0x0008f>

  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>

 

Native stacktrace:

 

 

Debug info from gdb:

 

[New LWP 16271]

[New LWP 16270]

[New LWP 8423]

[New LWP 8416]

[New LWP 3746]

[New LWP 3745]

[New LWP 3742]

[New LWP 3741]

[New LWP 3740]

[New LWP 3739]

[New LWP 3738]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

 

[paulbates]

Do you know the basic state of the RPi2, either at normal runtime or before it happens in terms of memory or CPU? While its a pretty substantial box for running Nodelink, its worth starting there. There are a number of tools, nmon is probably the simplest to install and use for a basic diagnostic. 

 

I would also simplify the integration, and add components from the basic OS, one day at a time. Start with Emoncms, wait a few days, add nodelink, wait a few days and then final configure the backups

 

Paul

[mblitz]

I suspect my hard drive may have been the culprit.  It was making a clicking sound occasionally, and I saw a few other corrupted files.  I replaced the drive and reinstalled everything (except the backups).  Things went much smoother, and NodeLink didn't crash last night.  I'll wait until tomorrow to start backups again.

 

Marty

[larryllix]

I suspect my hard drive may have been the culprit.  It was making a clicking sound occasionally, and I saw a few other corrupted files.  I replaced the drive and reinstalled everything (except the backups).  Things went much smoother, and NodeLink didn't crash last night.  I'll wait until tomorrow to start backups again.

 

Marty

Interesting. It seems strange, to me, to her of people using a HDD with a RPi.

 

Usually we think of then as little, no moving parts, controller type workhorses.

 

Do you run this as  a larger system for desktop work?

[mblitz]

Interesting. It seems strange, to me, to her of people using a HDD with a RPi.

 

Usually we think of then as little, no moving parts, controller type workhorses.

 

Do you run this as  a larger system for desktop work?

 

 

I picked up a 1TB portable USB drive on sale at Best Buy for $35.  Thought it would give me a little more space. 

 

Just using the RPi as an extension server for my automation.  Running NodeLink and emoncms.  Don't know how much space I'll need for emoncms database.

 

Also I've been playing with Alexa skill development, and thought I could integrate it as a skill server.

 

And I can use it as an NFS server for archiving stuff on my MacBook Pro.

 

Got a second drive for backing up the first drive, which came in handy since the first drive started failing.  I suspect its just a power problem, since it only came with a simple micro USB cable.  The second one came with a micro-b cable, which supplies additional power.  Ordered an additional micro-b from Amazon to see if that fixes the first drive.

[paulbates]

I would guess it was power. I hooked a USB drive to mine and it wasn't happy until I put a USB hub in the middle to power it. Then its been fine.

 

My pi runs nodelink with 4 nodes. It uses very little processor, 5%-ish hi average. I'm guessing the emoncms won't add much to the memory overhead to Nodelink.. not sure how much processor it would use. NFS should be ok too, i think its loaded and on by default (?). 

 

Not sure about the alexa skill development environment. Does it need the gui/desktop/xwindows? Running X has a noticeable impact on the pi1 & pi2. 

 

Paul

[larryllix]

Yeah, My RPi 3 b? came with a 2.5A power supply, but you really have to wonder about those microUSB connectors to power anything with that current. USB was spec'ed at 500mA current limited originally and it seems like the industry has stretched it passed it's capacity.

 

Now I see USB chargers have mostly gone to 5.25 volts in an attempt to compensate for the voltage loss in those micro sized conductors and micro sized connections.

[mblitz]

I also have a powered usb hub in the middle.  I don't think I was getting enough power to the disk through the standard micro usb cable.  Got my micro-b cable this morning, and hooked up the old drive as my archive drive.  No clicking so far.  We will see.  So far using the new drive for root has been working.  No more crashes or corruption.

 

Now NodeLink.exe and emoncms seem to be working ok (except for the random wh values, another post).  Now it runs long enough that I'm seeing problems on my isy.  See yet another post.

 

Marty

[larryllix]

Sounds like an 'old drives tale.'

 

Sent from a tiny keyboard. Response may be brief.