Software update on the 4th of April - Bug fixes, Alerts and some stability fixes

seems yes , but as I say before, I reset power, I turn on car and turn off … device seems not reply

and now how can I do ?

There is a few ways to figure out what is going on.
At first you can try to leave it powered on for a while, because maybe it failed a step while updating, and if that’s the case, it will automatically retry the update, and usually it will fix itself.

If it has been a while and it still does not work, you can go through the following steps to diagnose the issue further.

  1. Check if the sim card has any data left.
    No data = no connection to the cloud.

  2. Check if the green light comes on.
    If it does not light up when powered up, it could be an issue with the SD card.
    NOTE: It can be quite difficult to see the light.

  3. Is the autopi hotspot accessible?
    If the hotspot it available, then the device is running, but something could still be interfering with it’s ability to connect to the cloud.

  4. Is the device seen as online on my.autopi.io?

If none of the above fixed the issue, the usual way to debug this, is to get access to the logs in the device.

If you can connect to the hotspot of the device, then it can be retrieved that way. If not, you may need to either connect a screen and keyboard to the device, or take out the SD card and insert it into a linux pc to browse the files and see the logs (linux because windows can not mount the primary EXT4 rootfs filesystem).

Best regards
/Malte

Working again for me.
Last today update work well
Also good news with real time SoC !
Just need to refresh but real-time.
image image

3 Likes

1 /2 /3 is ok
4 Not (says 2 days ago online )
terminal not work propely

do you ve and address to send logs files on /var/logs/ ?

At now , my autopi not work with 4g sim , sim is working and has data .
i falsi sd but nothing work … I think there are something I not understand

Hi Alexandro

Can you try running the commands from the troubleshooting guide? and send me the output?

You can send it in a zip file to malte@autopi.io

Also, I have to mention this, are you sure the SIM card is oriented the right way?

image

Image taken from https://www.autopi.io/getting-started

Best regards
/Malte

Having the same issues as Alexandro.
SIM is 100% correctly oriented, i have 4G, i have internet connectivity, i can use it as a hotspot just fine.

It just seems to have no connectivity to the cloud.

The first time this happened was out of the box, as soon as i first installed it. I believed back then it was due to update that kept getting interrupted by the sleep. It eventually updated, and became accessible from the cloud after another reboot.

I left it in the parked car, with no modem hibernation overnight, so i can wake it up via SMS. It ate through the battery in 16 hours (from 12.6+V to 12.2V) just from the interval wakeups and the semi-active modem, went into hibernation, and now it’s back to cloud not seeing it at all.

I let it go to sleep, i drove the car around for a few hours, i left the car parked and on for a while - nothing. Each time it’d go to sleep, i’d wake it up with an SMS or by power cycling the car, i’d verify it has 4G connectivity (it did every time), and each time - no communication with the cloud.

Another interesting thing… This is the last entry in the event log:

56

yet this is what the power status pane says:

30

Emphasis on the selected text.

I’ve extracted the var/log folder from the Autopi.
I can upload it somewhere if that’d help in debugging…

There’s quite a few exceptions and errors in the minion log, no clue which of them are expected and which of them are not.

This seems to be the point where things started going south…

2019-04-11 19:18:53,984 [cloud_cache      :184 ][WARNING ][785] Temporarily unable to upload batch with 195 entries from queue 'pend': ('Connection aborted.', BadStatusLine("''",))                                                                                                
2019-04-11 19:18:57,600 [messaging        :232 ][ERROR   ][783] Recurring exception (207 times) in worker thread 'readout_1sec' while running workflow for message: {'filter': 'alternating_readout', 'handler': 'query', 'returner': 'cloud', 'args': ['RPM'], 'kwargs': {'protocol': 'auto', 'force': True, 'verify': True}}
2019-04-11 19:19:00,337 [salt.pillar      :116 ][ERROR   ][385] Exception getting pillar:                                                 
Traceback (most recent call last):                                                                                                        
  File "/usr/lib/python2.7/dist-packages/salt/pillar/__init__.py", line 113, in compile_pillar                                            
    dictkey='pillar',                                                                                                                     
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run                                                               
    value = future.result()                                                                                                               
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result                                                      
    raise_exc_info(self._exc_info)                                                                                                        
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1021, in run                                                               
    yielded = self.gen.throw(*exc_info)                                                                                                   
  File "/usr/lib/python2.7/dist-packages/salt/transport/zeromq.py", line 188, in 
crypted_transfer_decode_dictentry                        
    tries=tries,                                                                                                                          
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run                                                               
    value = future.result()                                                                                                               
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result                                                      
    raise_exc_info(self._exc_info)                                                                                                        
  File "<string>", line 3, in raise_exc_info                                                                                              
SaltReqTimeoutError: Message timed out                                                                                                    
2019-04-11 19:19:00,400 [salt.minion      :901 ][ERROR   ][385] Error while bringing up minion for multi-master. Is master at hub01.autopi.io responding?

(i omitted some seemingly irrelevant log entries - “unable to determine GNSS location”, RPM missing etc.)

From the above, it seems like minion can’t contact salt master, even though 4G is up and running and otherwise working fine. NTP works fine too, i can see it syncing with 81.7.4.128:123 in the logs.

So my thoughts are:

  1. either minion is borked, or
  2. hub01.autopi.io is borked, or
  3. domain name resolution is borked, or
  4. something’s wrong with the iptables config and salt ports are blocked.

I’ve never worked with salt stack, though, so… no clue.

Edit: As an unrelated sidenote, I’ve just noticed hub01.autopi.io runs on Linode. I’d strongly recommend moving away to a more serious VPS provider, or bare metal. The company i work for used to run on Linode as well. We eventually bailed on them after years of unexpected downtimes due to their incompetence, issues with noisy neighbours, issues with their (physical) networking, week-long ping-pong games with support where they wouldn’t admit for days that they were wrong, when they were clearly in the wrong, etc. Honestly, save your nerves and hair - switch to something else :slight_smile:

1 Like

I think a new image needed with correction … I can execute commands because device go sleep in few minutes , if is not linked to cloud/ configured please not use timer to sleep , only critical check .
it’s so difficult use it in this situation .

Need a img with correction bug

image

i’ve reset img but nothing and now is unusable … I don’t know if is something related to Italian operator or boh ??

Hm.

Mine’s having issues connecting to 4G now too. Or, rather, to the cell modem.
ec2x.cell_signal reports:

[Errno 2] could not open port /dev/ttyUSB2: [Errno 2] No such file or directory: '/dev/ttyUSB2'

Going to open it up and reseat the LTE modem in case it’s a contact issue.

Try executing lsusb command via ssh, it should list all USB devices.

Opened it up, reseated the modem, works fine now.
Hotspot works and everything.

Still no connection to the cloud though, so original problem remains.

Edit: pinging hub01 (salt master) from autopi

pi@autopi-###########:~ $ ping hub01.autopi.io
PING hub01.autopi.io (139.162.152.129) 56(84) bytes of data.
64 bytes from li1419-129.members.linode.com (139.162.152.129): icmp_seq=1 ttl=249 time=76.3 ms
64 bytes from li1419-129.members.linode.com (139.162.152.129): icmp_seq=2 ttl=249 time=60.5 ms
64 bytes from li1419-129.members.linode.com (139.162.152.129): icmp_seq=3 ttl=249 time=57.1 ms 
64 bytes from li1419-129.members.linode.com (139.162.152.129): icmp_seq=4 ttl=249 time=60.2 ms
64 bytes from li1419-129.members.linode.com (139.162.152.129): icmp_seq=5 ttl=249 time=68.3 ms
64 bytes from li1419-129.members.linode.com (139.162.152.129): icmp_seq=6 ttl=249 time=103 ms
^C
--- hub01.autopi.io ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5007ms
rtt min/avg/max/mdev = 57.184/71.054/103.520/15.854 ms

So the problem definitely isn’t in the 4G network or connectivity.
I guess the only remaining thing to do without further input from the autopi crew is to reflash it?

Edit: reflashed, gonna give it a try tomorrow, as it’s almost 04:00 here already.

2 Likes

Aaaand back to not being able to update before it goes to sleep.

Also this:

image

Edit: Okay, so hooked it up to a bench power supply (messing with it in the car became old real quick, and it doesn’t help anyways, since it keeps going to sleep due to missing RPM PID anyways). Then kept rebooting it until it connected to the cloud. About 3/4 boots i get the error from the screenshot above, and in 1/4 it actually manages to connect to the cloud.

Then, i’ve managed to get the power settings to save before it goes to sleep (after further 4-5 reboots). And now i’m running the update, at least i hope i am, as there is no way for me to confirm it’s running. Because minionutil returns a timeout.

This is a right mess…

Edit2: this is getting ridiculous. Update finished, it rebooted, minionutil works now, it’s connecting to cloud (finally!), BUT now the wifi hotspot is gone. What on earth is wrong with this thing?!

Edit3:

autopi

¯_(ツ)_/¯

Wifi hotspot working after a reboot.

@Malte after a lot of faffing about, here’s my hypothesis… if the LTE network takes a bit before it starts letting data through (i guess it happens with some providers), minion tries to contact master and fails. And if it fails, it keeps on failing for some reason (same BadStatusLine error) forever.

Solution - ping master and wait for it to respond (indicating LTE network is finally letting data through) before bringing minion up.

Additionally, the power timers don’t get suspended during updates, causing issues on EVs or when running on a bench. Device should be always on while the update is running.

All of this is just guesswork, though, so i’m probably completely off.
I’ve resolved all my issues except the original one - i need to reboot the device multiple times until i luck out and have the cloud working.

My is unstable , I think it can be useful try to use an old image ?? But It can not work with cloud … so how can do ??? It’s 7days of died

Still no go for SMS wakeup after last update. Modem power save is off, connection to cloud and wifi OK. Is there a way to rollback to an older update without doing a reflash?

/Mats

I’m seeing the gauge is doing averaging - IMHO, a gauge needs to be “the last known single valid value”.

2 Likes

You can change it to mix, max, sum or average. Not very useful though :frowning:
Last value would be great.

1 Like