Python exception
Traceback (most recent call last):File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/srv/sbproxy/sandboxproxytest/framework/multiprocess.py", line 39, in run
res = self._function(*task)
.
.
.
return requests.request(method, url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='172.31.37.5', port=8080): Max retries exceeded with url: /jobs/list?after=2015-09-27 (Caused by <class 'socket.error'>: [Errno 99] Cannot assign requested address)
Check netstat | grep -c tcp
When the expection happens, the above commans show around 28000.
Reason can be seen here:
http://stackoverflow.com/questions/11190595/repeated-post-request-is-causing-error-socket-error-99-cannot-assign-reques
If the connections are in CONNECTION_WAIT then the server closed the connection earlier than should, but if the connection is in TIME_WAIT state, that is normal:
RFC 793 sets the TIME-OUT to be twice the Maximum Segment Lifetime, or 2MSL. Since MSL, the maximum time a packet can wander around Internet, is set to 2 minutes, 2MSL is 4 minutes. Since there is no ACK to an ACK, the active closer can't do anything but to wait 4 minutes if it adheres to the TCP/IP protocol correctly, just in case the passive sender has not received the ACK to its FIN (theoretically).
Some explanation:
http://serverfault.com/a/329846
If the connections are in CONNECTION_WAIT then the server closed the connection earlier than should, but if the connection is in TIME_WAIT state, that is normal:
RFC 793 sets the TIME-OUT to be twice the Maximum Segment Lifetime, or 2MSL. Since MSL, the maximum time a packet can wander around Internet, is set to 2 minutes, 2MSL is 4 minutes. Since there is no ACK to an ACK, the active closer can't do anything but to wait 4 minutes if it adheres to the TCP/IP protocol correctly, just in case the passive sender has not received the ACK to its FIN (theoretically).
Some explanation:
http://serverfault.com/a/329846
Solution
Set the tcp_tw_recycle to 1
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
Or add to /etc/sysctl.conf:
net.ipv4.tcp_tw_recycle = 1
and reload the configuration like this:
sysctl -p
Some documentation advise to use /proc/sys/net/ipv4/tcp_tw_reuse, but in my case it has not solved this issue.
