Exceptions thrown when stop_on_errors=False

[tiagom62]

Bug reports

Steps to reproduce:
Using the native client run a command on a remote host that times out.

1. Example code that produces error.

from pssh.clients import ParallelSSHClient
import os

client = ParallelSSHClient(['a.example.com', 'b.example.com'], allow_agent=False, user='ubuntu', pkey=os.path.expanduser('~/.ssh/key'), proxy_host='proxy.example.com', proxy_user='user', proxy_pkey=os.path.expanduser('~/.ssh/id_rsa'))

cmd_response = client.run_command('lsb_release -r', stop_on_errors=False, shell='bash -c')

for node in cmd_response:
    print(cmd_response[node]['host'])

2. Stack trace or error messages.

Exception in thread Thread-360:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/tunnel.py", line 123, in run
    self._init_tunnel_client()
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/tunnel.py", line 119, in _init_tunnel_client
    timeout=self.timeout)
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/clients/native/single.py", line 120, in __init__
    THREAD_POOL.apply(self._init)
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/gevent/pool.py", line 324, in apply
    return func(*args, **kwds)
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/clients/native/single.py", line 158, in _init
    return self._connect_init_retry(retries)
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/clients/native/single.py", line 148, in _connect_init_retry
    return self._init(retries=retries)
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/clients/native/single.py", line 160, in _init
    raise SessionError(msg, self.host, self.port, ex)
SessionError: ('Error connecting to host %s:%s - %s', 'proxy.example.com', 22, Timeout())

Exception in thread Thread-17:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/tunnel.py", line 123, in run
    self._init_tunnel_client()
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/tunnel.py", line 119, in _init_tunnel_client
    timeout=self.timeout)
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/clients/native/single.py", line 119, in __init__
    self._connect(self._host, self.port)
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/clients/native/single.py", line 189, in _connect
    return self._connect(host, port, retries=retries+1)
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/clients/native/single.py", line 194, in _connect
    self.num_retries,)
ConnectionErrorException: ("Error connecting to host '%s:%s' - %s - retry %s/%s", 'a.example.com', 22, 'Operation timed out', 2, 2)

Expected behaviour:
Exceptions not thrown.

Actual behaviour:
Exceptions above. I don't get this with the paramiko client.
My testing is specifically with a proxy. I did not test without a proxy.

Additional info: [Include version of ssh2-python and/or paramiko and any other relevant information.]
Parallel-ssh 1.6.1
ssh2-python 0.13.0.post2

[pkittenis]

Hi there,

Thank you for the interest and report! Will take a look at this, seems that tunnel initialisation failures are not being caught by stop_on_errors at first glance.

[pkittenis]

Resolved in 1.6.2.

Please report any other issues that you may find - tunnel tests on the native client have been lacking, as you found out, due to issues with running multiple SSH servers in same thread.

[tiagom62]

Upgraded to 1.6.2 and now i am getting the following error. This is running against ~500 hosts and a pool_size of 10

python(2033,0x700091185000) malloc: *** error for object 0x10aa1b4d700: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

[tiagom62]

This doesn't appear to be resolved.

Exception in thread Thread-127:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/Users/PATH_TO_APP/venv/lib/python2.7/site-packages/pssh/tunnel.py", line 143, in run
    self.fw_host, self.fw_port, '127.0.0.1', forward_addr[1])
  File "ssh2/session.pyx", line 331, in ssh2.session.Session.direct_tcpip_ex
    return handle_error_codes(c_ssh2.libssh2_session_last_errno(
  File "ssh2/utils.pyx", line 113, in ssh2.utils.handle_error_codes
    raise exceptions.Timeout
Timeout

[pkittenis]

That seems to be a different issue. Can you please show code to replicate with?

Edit - Found the issue, was different one yes. The implementation was bad and creating multiple tunnels. Also the cause for that crash you ran into.

[pkittenis]

This one will take a while - the proxying implementation for the native client needs some work. One of the issues is that there is currently one new client and tunnel being opened per host in a new thread. Beside the scaling issues, this also causes the seg faults you have seen.

Have a working implementation that creates multiple tunnels from one connection to the proxy host but is not ready yet, there are some race conditions that need ironing out along with better tests. Can watch #123 for this.