If you use GlusterFS then you are also using rpcbind. The linux man page provides the following description for rpcbind:
DESCRIPTION
The rpcbind utility is a server that converts RPC program numbers into universal addresses. It must be running on the host to be able to make RPC
calls on a server on that machine.
When an RPC service is started, it tells rpcbind the address at which it is listening, and the RPC program numbers it is prepared to serve. When a
client wishes to make an RPC call to a given program number, it first contacts rpcbind on the server machine to determine the address where RPC
requests should be sent.
GlusterFS (3.6.9) on Ubuntu 14.04 requires rpcbind. By default rpcbind is listening on all interfaces. The linux man page states that this can be changed using the -h option at startup:
-h Specify specific IP addresses to bind to for UDP requests. This option may be specified multiple times and is typically necessary when run‐
ning on a multi-homed host. If no -h option is specified, rpcbind will bind to INADDR_ANY, which could lead to problems on a multi-homed host
due to rpcbind returning a UDP packet from a different IP address than it was sent to. Note that when specifying IP addresses with -h,
rpcbind will automatically add 127.0.0.1 and if IPv6 is enabled, ::1 to the list.
You can list the interfaces rpcbind is listening on using netstat:
netstat -aunlp | grep rpcbind udp 0 0 0.0.0.0:111 0.0.0.0:* 8861/rpcbind udp 0 0 0.0.0.0:980 0.0.0.0:* 8861/rpcbind udp6 0 0 :::111 :::* 8861/rpcbind udp6 0 0 :::980 :::* 8861/rpcbind
The startup script in /etc/init.d/rpcbind looks for a configuration file in /etc/default/rpcbind or /etc/rpcbind.conf (Ubuntu 14.04) but these do not exist. A bug was filed for this with a suggested rpcbind configuration. I created my /etc/rpcbind.conf based on the suggestion:
# /etc/init.d/rpcbind
OPTIONS=""
# Cause rpcbind to do a "warm start" utilizing a state file (default)
# OPTIONS="-w "
# Uncomment the following line to restrict rpcbind to localhost only for UDP requests
OPTIONS="${OPTIONS} -h 192.168.166.108"
# Uncomment the following line to enable libwrap TCP-Wrapper connection logging
# OPTIONS="${OPTIONS} -l "
After restarting rpcbind with the new configuration file netstat listed the following:
netstat -aunlp | grep rpcbind udp 0 0 192.168.166.108:111 0.0.0.0:* 8890/rpcbind udp 0 0 127.0.0.1:111 0.0.0.0:* 8890/rpcbind udp 0 0 0.0.0.0:1009 0.0.0.0:* 8890/rpcbind udp6 0 0 ::1:111 :::* 8890/rpcbind udp6 0 0 :::1009 :::* 8890/rpcbind
You can also use rpcinfo -p from an external machine to check if rpcbind is open to connections (replace XXX.XXX.XXX.XXX with your server IP):
rpcinfo -p XXX.XXX.XXX.XXX
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 33218 status
100024 1 tcp 57664 status
That wasn’t good to see. I am not the only one that has come across this issue as reported here.
To secure rpcbind and restrict access to specific hosts you can add entries to /etc/hosts.allow as such (replace XXX.XXX.XXX.XXX with your local server IP):
# /etc/hosts.allow: list of hosts that are allowed to access the system. # See the manual pages hosts_access(5) and hosts_options(5). # # Example: ALL: LOCAL @some_netgroup # ALL: .foobar.edu EXCEPT terminalserver.foobar.edu # # If you're going to protect the portmapper use the name "rpcbind" for the # daemon name. See rpcbind(8) and rpc.mountd(8) for further information. # rpcbind: XXX.XXX.XXX.XXX rpcbind: 127.0.0.1
Then edit /etc/hosts.deny and add:
# /etc/hosts.deny: list of hosts that are _not_ allowed to access the system. # See the manual pages hosts_access(5) and hosts_options(5). # # Example: ALL: some.host.name, .some.domain # ALL EXCEPT in.fingerd: other.host.name, .other.domain # # If you're going to protect the portmapper use the name "rpcbind" for the # daemon name. See rpcbind(8) and rpc.mountd(8) for further information. # # The PARANOID wildcard matches any host whose name does not match its # address. # # You may wish to enable this to ensure any programs that don't # validate looked up hostnames still leave understandable logs. In past # versions of Debian this has been the default. # ALL: PARANOID rpcbind: ALL
Now you can try again from an external machine using rpcinfo:
rpcinfo -p 104.237.142.239 rpcinfo: can't contact portmapper: RPC: Authentication error; why = Client credential too weak
Success!
You can find some additional information about this new DDoS reflection attack on the Level 3 Communication blog.