Limits of ssh-agent on Mac OS
When executing task using our custom SSH-based tool from some kind of (linux) jump box, I got "lost ssh-agent" error from about 1/3 hosts and the tool could not connect to those hosts. Found out that this happens only when using ssh-agent from Mac OS machine; it works fine on Ubuntu and Fedora machines. Logs and even running ssh-agent in debug mode don't show anything suspicious. Use of powerful weapons sudo dtruss -p `pgrep ssh-agent` was much more helpful: "accept from AUTH_SOCKET: Too many open files". So let's have look at Mac OS limits... Short version, there are three levels of limits: ulimit, launchd and sysctl.
$ ulimit -n
2560
$ launchctl limit maxfiles
maxfiles 256 unlimited
$ sysctl -a | grep files
kern.maxfiles: 12288
kern.maxfilesperproc: 10240
kern.num_files: 4681
All values looks fine, except launchctl's maxfiles. After increasing to reasonable value using sudo launchctl limit maxfiles 1024 unlimited (and restarting ssh-agent) it finally works! To persists this settings, use for example this:
cat >> EOF | sudo tee /Library/LaunchDaemons/limit.maxfiles.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxfiles</string>
<key>ProgramArguments</key>
<array>
<string>/bin/launchctl</string>
<string>limit</string>
<string>maxfiles</string>
<string>2048</string>
<string>unlimited</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>ServiceIPC</key>
<false/>
</dict>
</plist>
EOF
cat >> EOF | sudo tee /Library/LaunchDaemons/limit.maxfiles.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxfiles</string>
<key>ProgramArguments</key>
<array>
<string>/bin/launchctl</string>
<string>limit</string>
<string>maxfiles</string>
<string>2048</string>
<string>unlimited</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>ServiceIPC</key>
<false/>
</dict>
</plist>
EOF
(use of /etc/launchd.conf doesn't work in 10.10.2 anymore)