{"id":3166,"date":"2022-12-11T00:26:19","date_gmt":"2022-12-11T05:26:19","guid":{"rendered":"https:\/\/blog.jonesling.us\/?p=3166"},"modified":"2022-12-24T09:43:52","modified_gmt":"2022-12-24T14:43:52","slug":"cant-call-method-stat-on-an-undefined-value","status":"publish","type":"post","link":"https:\/\/blog.jonesling.us\/?p=3166","title":{"rendered":"Can’t call method “stat” on an undefined value"},"content":{"rendered":"\n

I use Rexify<\/a> to manage my servers. (I’m learning to treat my servers like cattle, not pets.) I have it distributing standardized config files, running updates, and more.<\/p>\n\n\n\n

The Problem<\/h2>\n\n\n\n

After adding a Raspberry Pi-based system to manage my backups, I was plagued with this error:<\/p>\n\n\n

\n[2022-12-10 23:41:13] INFO - Running task Config:System:standard on backup\n[2022-12-10 23:41:15] ERROR - Error executing task:\n[2022-12-10 23:41:15] ERROR - Can't call method "stat" on an undefined value at \/usr\/lib64\/perl5\/vendor_perl\/5.34\/Rex\/Interface\/Fs\/OpenSSH.pm line 70, &lt;> line 144.\n[2022-12-10 23:41:15] ERROR - 1 out of 1 task(s) failed: \n[2022-12-10 23:41:15] ERROR - Config:System:standard failed on backup\n[2022-12-10 23:41:15] ERROR - Can't call method "stat" on an undefined value at \/usr\/lib64\/perl5\/vendor_perl\/5.34\/Rex\/Interface\/Fs\/OpenSSH.pm line 70, &lt;> line 144.\n<\/pre><\/div>\n\n\n
The error didn’t make much sense, and wasn’t consistent. Not all Rex commands triggered it.  When I did run into it, the same command executed directly on the server worked just fine.<\/p>\n\n\n\n
Just to be clear:<\/strong> the problem had nothing to do with the system architecture.  I.e. the fact that it was a Raspberry Pi, or an ARM device generally, was a red herring, but this was the first RPi that I had managed with Rex, and the only system with this problem, so I didn’t rule it out right away.<\/p>\n\n\n\n
The Diagnosis<\/h2>\n\n\n\n
At first glance it appeared to be a bug in Rex, because that was the only thing that made sense.  Why would it be complaining that it can’t call “stat” on an undefined file path, when the files are all clearly defined in my Rexfile configuration?<\/p>\n\n\n\n
The line number referred to a place where Rex calls stat using SFTP.  The actual value of $path was unknown to me, as I didn’t want to immediately jump into someone else’s code and start outputting debug info, but it seemed reasonable that it would either be the local or remote file, and the files should be clearly defined.<\/p>\n\n\n
\nmy $attr = $sftp->stat($path);\n<\/pre><\/div>\n\n\n
That seems pretty straight-forward.  If this is<\/em> a bug in Rex then other people should be encountering it.<\/p>\n\n\n\n
Googling the error was rather unhelpful.  That specific error, with quotes to search exact matches, returned no hits.  An inexact match returned nothing helpful.  Nothing about Rex, just a bunch of other applications doing other things.  Let’s set that assumption aside and look elsewhere.<\/p>\n\n\n\n
I could run simple commands using Rex, like ‘uptime’, to the server but more complicated things were a problem.  The only common thread so far involved putting files.  Could it be an issue with sending and receiving files?<\/p>\n\n\n\n
Rex is configured to connect as root, there aren’t other users on the system (besides default and service accounts) to accidentally connect as, and the paths are all regular filesystem paths (not \/dev or similar), so it’s almost certainly not a permissions issue.<\/p>\n\n\n\n
Maybe a second SSH connection couldn’t be established; SFTP establishes an SSH connection and tunnels across it, after all.<\/p>\n\n\n\n
I don’t set a common ControlMaster and ControlPath setting in my config, but I know from experience that if you do, and you force the first (“master”) to close, you’ll knock out other connections to the same host.<\/p>\n\n\n\n
Well, I could connect via SSH just fine, using multiple simultaneous connections and opening \/ closing independently, so this clearly<\/em> wasn’t an SSH connection problem.  Time to look somewhere else.<\/p>\n\n\n\n
Except I was wrong.<\/p>\n\n\n\n
By ruling out SSH I missed a critical, based troubleshooting step: connect to my new server via SFTP and issue a couple of commands. When I finally tried it, after beating around the bush for far too long, I had my “duh!” moment. <\/p>\n\n\n\n
Lo and behold, SFTP failed.  I didn’t expect that could happen if SSH was fine.<\/p>\n\n\n\n
Of course, it failed with an inscrutable error in batch mode:<\/p>\n\n\n
\n$ echo "ls \/" | sftp -b - backup\nConnection closed.\nConnection closed\n<\/pre><\/div>\n\n\n
Using the -v option was hardly more informative:<\/p>\n\n\n
\n$ echo "ls \/" | sftp -v -b - backup\nOpenSSH_9.0p1, OpenSSL 1.1.1q  5 Jul 2022\ndebug1: Reading configuration data ...\n<snip>\nsubsystem request failed on channel 0\nConnection closed.\nConnection closed\n<\/pre><\/div>\n\n\n
Getting out of batch mode and trying to get a prompt gave me a little bit more information, but only just:<\/p>\n\n\n
\n$ sftp -v backup\nOpenSSH_9.0p1, OpenSSL 1.1.1q  5 Jul 2022\ndebug1: Reading configuration data ...\n<snip>\ndebug1: Sending subsystem: sftp\ndebug1: client_input_channel_req: channel 0 rtype exit-status reply 0\ndebug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0\ndebug1: channel 0: free: client-session, nchannels 1\nTransferred: sent 4528, received 4060 bytes, in 0.2 seconds\nBytes per second: sent 22612.3, received 20275.2\ndebug1: Exit status 127\ndebug1: compress outgoing: raw data 211, compressed 171, factor 0.81\ndebug1: compress incoming: raw data 918, compressed 764, factor 0.83\nConnection closed.  \nConnection closed\n<\/pre><\/div>\n\n\n
Most disconcerting.  Fortunately, my Google-fu still had some juice today, and I learned something new from the search results.<\/p>\n\n\n\n
The best clue was the exit status 127: https:\/\/serverfault.com\/questions\/770541\/can-connect-via-ssh-but-not-via-sftp-exit-status-127<\/a><\/p>\n\n\n\n
The Solution<\/h2>\n\n\n\n
SSH has something that appears to be a “legacy” option for configuring SFTP: the SFTP server subsystem.<\/p>\n\n\n\n
Subsystem sftp <subsystem><\/code><\/pre>\n\n\n\n
You have three options:<\/p>\n\n\n\n