Apex Troubleshooting

[+KimP]

Still trying to diagnose the problem with my Apex classic. My display is broken, I know that. I can now reach the Apex through fusion and it acts like everything's working, but when I go to control an outlet nothing happens. I'm still using the eb8 to power my 2 return pumps. Can it supply power but not have any logic control?  I have an ALD connected to the eb8 and that doesn't work. Any thoughts on what to try next?

[jolt]

Maybe try a different eb8 to see if the problem is the controller or the energy bar? Also maybe swap out the aquabus cable between the controller and the energy bar?

[+KimP]

Good call on the cable. I'll try that first. I don't have another energy bar. I wonder if the club still has an extra to loan out?

[jolt]

I think mike mentioned he had an extra one

[Timfish]

1 hour ago, KimP said:

Good call on the cable. I'll try that first. I don't have another energy bar. I wonder if the club still has an extra to loan out?

I've got one you can borrow.

[mhart032]

Can you change anything on the fusion dashboard? have you tried to access it through the local network see if that will work? 

Correct me if im wrong but it should be (IP address)/Apex/Local for instance mine is 192.168.1.17/apex/dash.

might also check the access while your in there to make sure you can edit/modify not just view. as well as the lock not enabled. you should also be able to go to diagnostics and read all the info on the apex system i.e serial numbers and all that. 

[mhart032]

the ALDS might need to be setup via the dashboard under modules as well. 

[+KimP]

26 minutes ago, mhart032 said:

Can you change anything on the fusion dashboard? have you tried to access it through the local network see if that will work? 

Correct me if im wrong but it should be (IP address)/Apex/Local for instance mine is 192.168.1.17/apex/dash.

might also check the access while your in there to make sure you can edit/modify not just view. as well as the lock not enabled. you should also be able to go to diagnostics and read all the info on the apex system i.e serial numbers and all that. 

I can change things through the fusion dashboard, as if everything is working fine. As far as accessing it through the local network, I can't get in. I always used to be able to and I know what the password is, but I can't get in anymore. I'm planning on doing a factory reset this evening and seeing if I can get it updated.

I appreciate the ideas!

[+KimP]

I think this is puzzling. I had turned off my 2 return pumps through fusion to see if it was working last night. It didn't work and I got distracted, so I left it as is and went about my night. Today I go to fiddle with it and when I unplug the display from the head unit, the 2 return pumps shut off. Plugged the display back in and they turn back on.

I hope a reset and update does the trick.

[mhart032]

If the router rebooted it might have jumped to a different local ip address. You can find that on the fusion under settings and network. but it seems things are working via the dashboard. so might just be the cable, i had a  issue kind of like that before on my older tank a few years ago when the classic wasn't classic lol. Apex replaced the EB8 well Mark Callahan got it replaced for me as he got it for me. but i think it got wet, or moisture in it. but i think only 2 outlets worked. 

[Isaac]

its best to debug directly on local dashboard, not through fusion.  Download "net analyzer" and scan your network.  neptune classic apex network hardware will show up as "nipon dempa co."  that will give you its current IP.

https://techet.net/netanalyzer/

[+KimP]

I finally decided to send the Apex up to my internal tech support and got this report:

(tl;dr The head unit should be replaced, so I need to decide whether it's worth it to buy another of the same used, or upgrade to the new one)

Able to update firmware just fine.  Not able to update pages.
Followed all of the ceremonious precautions:  Disabled AV, Disabled firewall, hardwired network,
adjacent on same network switch (`traceroute apex` would show zero intermediary hops),
deactivated all other network adapters, Safe Mode w/ Networking, ran Flash Utility as Administrator,
following the firmware update instructions even reinstalled the prescribed Microsoft .Net Framework 3.5.

No matter what, just cannot get the update pages to succeed.  Each time I try I would get one of 4
possible outcomes:  (500) Internal Server Error, (413) Error, Network timeout, or (
about 1/30 chance) a single file `aihost.bin` would successfully upload but would receive an error for the following `aihydra.bin`

The following examples pretty much resemble what was reported here:
https://forum.neptunesystems.com/showthread.php?5433-Apex-Failure-Where-do-I-go-from-here&p=35219&viewfull=1#post35219
(a few comments later there is a "me too" with a disappointing "even had a guy from neptune take a look and he could't figure it out. ")

Here are some examples of the result output:

=====

Uploading file: aihost.bin...

Operation failed with the following error information:

The remote server returned an error: (413) Error.

====

Uploading file: aihost.bin...
Success
Uploading file: aihydra.bin...

Operation failed with the following error information:

The remote server returned an error: (413) Error.

====

Uploading file: aihost.bin...

Operation failed with the following error information:

The remote server returned an error: (500) Internal Server Error.

====

Uploading file: aihost.bin...

Operation failed with the following error information:

The remote server returned an error: (413) Error.

====

I had a thought that if it takes 30 attempts to get one file to successfully update then maybe I can manipulate the firmware
`www` directory to contain only 1 file at a time and painstakingly get all 57 files uploaded.  But that didn't work
as I couldn't get the next `aihydra.bin` to upload.  Just in case it is a clue, `aihydra.bin` is 12KB whereas `aihost.bin` was 6KB
so if there were a storage defect (flash storage degrade?) maybe the device struggles to find even a small usable chunk.

Able to telnet into Controller.  Followed advice from https://forum.neptunesystems.com/showthread.php?3673-Webpages-update-failed-on-high-js-error-413
to delete log files as a possible solution to a 413 error.

Purging logs using `logdel` as well as `del logs/<logname>` worked for
removing some but eventually no longer had any effect.  Some logs simply couldn't
be deleted.  Further suggesting a filesystem/storage problem.  
`dir` command shows sufficient free space to accommodate all pages (which should overwrite instead of duplicate anyway)

`dir` command at the shell root would sometimes show duplicate filenames
at different timestamps and sizes like 3 separate `syslog.txt` and `aihost.bin`
differing by days or minutes.  Usually one of them would be 0 bytes.  The limited
shell capability doesn't make directory structure clear but presumably these were all
contents of the same directory.  As such it would be baffling that there could exist
files of the same name within the same directory.  Further evidence of a filesystem/storage problem.

As stated in this posting https://forum.neptunesystems.com/showthread.php?23229-Web-Pages-update-fail
Sometimes additional strange things were listed like `aihost.bin        Directory`  with an
absurd filesize.  Couldn't delete any of the files.  As stated in the above referenced forum, maybe this is something that can be fixed remotely.

Tried flashing prior versions of firmware + pages but same problem occurs.  
One would wonder how a firmware update can succeed and pages fail, but I presume that
they have different storage mechanisms (EEPROM vs Flash).

Anyone know of super secret commands you can issue over telnet to reformat the Controller's filesystem?

[Jimbo662]

The new version is so much easier to work with...especially when it comes to updating and having wireless built in...just sayin'... 🙂

[+KimP]

5 minutes ago, Jimbo662 said:

The new version is so much easier to work with...especially when it comes to updating and having wireless built in...just sayin'... 🙂

Thanks for the feedback. That's useful to know. I haven't had a chance to look into it yet.

[mhart032]

You can use most of legacy modules on the Apex as well. It's a big investment but in my eyes completely worth it. When building the 20gal I was thinking of just using the classic, but I'm really glad I made the investment on the new one. 

[Isaac]

i've actually had to modify the html prior to uploading because it broke their android app.  so, yea, its possible to edit things prior to upload, they have no crc/md5 checks.  if you only wanna upload certain files, just remove the files you dont wanna upload and upload that partial directory. seems like a pain in the arse, but might work.

if you use the initialize command, its supposed to clear it out.  then flash firmware, reboot, flash web, reboot.  if you are getting timeouts mid-flash,  i'd check the simplest things first.. are you doing it over wifi, is your ethernet cable going bad, is your switch port going bad, is the ethernet port on the apex going bad, etc,etc.

if you end up buying a new one, ill buy the old probes off ya.  my backup base unit wants to work on my second tank, but has no probes, its lonely.

[+KimP]

On 2/14/2019 at 2:44 PM, Jimbo662 said:

The new version is so much easier to work with...especially when it comes to updating and having wireless built in...just sayin'... 🙂

Now that I've had a chance to work with the new one I see what you're talking about. I can't believe how much easier everything is on the 2016!

[+KimP]

15 hours ago, Isaac said:

i've actually had to modify the html prior to uploading because it broke their android app.  so, yea, its possible to edit things prior to upload, they have no crc/md5 checks.  if you only wanna upload certain files, just remove the files you dont wanna upload and upload that partial directory. seems like a pain in the arse, but might work.

if you use the initialize command, its supposed to clear it out.  then flash firmware, reboot, flash web, reboot.  if you are getting timeouts mid-flash,  i'd check the simplest things first.. are you doing it over wifi, is your ethernet cable going bad, is your switch port going bad, is the ethernet port on the apex going bad, etc,etc.

if you end up buying a new one, ill buy the old probes off ya.  my backup base unit wants to work on my second tank, but has no probes, its lonely.

Thanks for the advice!

I did get a new one a couple days ago. I'm still working on setting everything up so I'll let you know what all I end up having left over.