Sfdwhf

I have weird problem with resolving hostnames on Ubuntu 16.04. I'm searching for solution from friday (4 days!), so I think it's totally not duplicate to similar questions.

I have a server in local network with two interfaces: external (to the internet) and internal (to local network). Both are static configured. I'll show config below.

It was working perfectly since installed in March. Several times I've successfully made 'apt update' with no problem. On friday I tried to update it again, and here the main story begins...

apt update

Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-93-generic x86_64)



~$ sudo apt update

Err:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease

  Temporary failure resolving 'us.archive.ubuntu.com'

Err:2 http://security.ubuntu.com/ubuntu xenial-security InRelease

  Temporary failure resolving 'security.ubuntu.com'

Err:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease

  Temporary failure resolving 'us.archive.ubuntu.com'

Err:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease

  Temporary failure resolving 'us.archive.ubuntu.com'

Reading package lists... Done

Building dependency tree

Reading state information... Done

195 packages can be upgraded. Run 'apt list --upgradable' to see them.

W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial/InRelease  Temporary failure resolving 'us.archive.ubuntu.com'

W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease  Temporary failure resolving 'us.archive.ubuntu.com'

W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease  Temporary failure resolving 'us.archive.ubuntu.com'

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease  Temporary failure resolving 'security.ubuntu.com'

W: Some index files failed to download. They have been ignored, or old ones used instead.

ping/telnet

~$ ping google.com

ping: unknown host google.com



~$ ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=3.87 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=3.93 ms

64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=3.88 ms



--- 8.8.8.8 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2003ms

rtt min/avg/max/mdev = 3.870/3.898/3.939/0.077 ms



~$ telnet 8.8.8.8 53

Trying 8.8.8.8...

Connected to 8.8.8.8.

Escape character is '^]'.

Connection closed by foreign host.

On previous updates there was no such problems with resolving hostnames. Server is primarily used in local network by ip address, so i don't know when this problem arise.
So here are some commands, that i executed (external ip address replaced with #):

ifconfig

~$ ifconfig

enp29s0   Link encap:Ethernet  HWaddr 00:10:18:25:cd:40

          inet addr:#.#.#.#  Bcast:#.#.#.#  Mask:255.255.255.248

          inet6 addr: fe80::210:18ff:fe25:cd40/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:145862 errors:0 dropped:0 overruns:0 frame:0

          TX packets:119991 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:14777641 (14.7 MB)  TX bytes:22823397 (22.8 MB)





enp3s0    Link encap:Ethernet  HWaddr 00:1a:64:c9:93:f8

          inet addr:10.0.35.115  Bcast:10.255.255.255  Mask:255.0.0.0

          inet6 addr: fe80::21a:64ff:fec9:93f8/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:779951 errors:0 dropped:0 overruns:0 frame:0

          TX packets:608340 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:514425482 (514.4 MB)  TX bytes:189891768 (189.8 MB)





lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:65536  Metric:1

          RX packets:2145438 errors:0 dropped:0 overruns:0 frame:0

          TX packets:2145438 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1

          RX bytes:1185976997 (1.1 GB)  TX bytes:1185976997 (1.1 GB)

cat /etc/network/interfaces

~$ cat /etc/network/interfaces

# This file describes the network interfaces available on your system

# and how to activate them. For more information, see interfaces(5).





source /etc/network/interfaces.d/*





# The loopback network interface

auto lo

iface lo inet loopback





# The primary network interface - Internal

auto enp3s0

iface enp3s0 inet static

        address 10.0.35.115

        netmask 255.0.0.0

        network 10.0.0.0

        broadcast 10.255.255.255

#       gateway 10.1.10.102

#       # dns-* options are implemented by the resolvconf package, if installed

#       dns-nameservers 10.1.10.102

        metric 20





# The secondary network interface - External

auto enp29s0

iface enp29s0 inet static

        address #.#.#.#

        netmask 255.255.255.248

#       network #.#.#.#

#       broadcast #.#.#.#

        gateway #.#.#.#

        dns-nameservers 8.8.8.8 8.8.4.4

        metric 10





#auto enp6s0

iface enp6s0 inet manual

/etc/resolv.conf

~$ ls -la /etc/resolv.conf

lrwxrwxrwx 1 root root 27 Oct 14 01:46 /etc/resolv.conf -> /run/resolvconf/resolv.conf





~$ cat /etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)

#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 8.8.8.8

nameserver 8.8.4.4

But nmcli didn't show any DNS configured:
nmcli

~$ nmcli dev show | grep 'DNS'



~$ nmcli dev show | grep 'IP4'

IP4.ADDRESS[1]:                         #.#.#.#/29

IP4.GATEWAY:                            #.#.#.#

What i did:

• several times restarted server.




• several times restarted systemd-resolved, NetworkManager.




• comment and uncomment "dns=dnsmasq" in /etc/NetworkManager/NetworkManager.conf (with restart service and server).




• found advice about switch off DNSSEC, but as i found it's already switched off.




• made /etc/resolv.conf static file (not symbolic link), get back to symbolic link - all with restarts.

Nothing of this helps...

Today 16 Oct

Today I've made 'apt update' and 'apt upgrade' with some trick: changed in /etc/apt/source.list domains names to their ip addresses. So system updated and become 16.04.3.
It doesn't help.

/etc/nsswitch.conf

Remove some settings for "hosts" and leave minimal values.

~$ cat /etc/nsswitch.conf

# /etc/nsswitch.conf

#

# Example configuration of GNU Name Service Switch functionality.

# If you have the `glibc-doc-reference' and `info' packages installed, try:

# `info libc "Name Service Switch"' for information about this file.



passwd:         compat

group:          compat

shadow:         compat

gshadow:        files



#hosts:          files mdns4_minimal [NOTFOUND=return] dns

hosts:          files dns

networks:       files



protocols:      db files

services:       db files

ethers:         db files

rpc:            db files



netgroup:       nis

nslookup on custom DNS server

This thing finally drives me crazy. I run nslookup on custom DNS server...

~$ nslookup yandex.ru 208.67.222.222

;; connection timed out; no servers could be reached

tcpdump on 53 port

...check tcpdump - it was showing too much records with "bad udp cksum"...

~# sudo tcpdump -vvv -B 4096 -i enp29s0 host 208.67.222.222 and port 53

tcpdump: listening on enp29s0, link-type EN10MB (Ethernet), capture size 262144 bytes

16:01:57.043326 IP (tos 0x0, ttl 64, id 30657, offset 0, flags [none], proto UDP (17), length 55)

#.#.#.#.55778 > 208.67.222.222.domain: [bad udp cksum 0x445e -> 0x6d74!] 47660+ A? yandex.ru. (27)

ethtool

...execute ethtool to avoid such records...

~$ sudo ethtool --offload enp29s0 rx off tx off

Actual changes:

rx-checksumming: off

tx-checksumming: off

    tx-checksum-ipv4: off

tcp-segmentation-offload: off

    tx-tcp-segmentation: off [requested on]

    tx-tcp-ecn-segmentation: off [requested on]

...and check tcpdump again:

~# cat tcpdump.log

15:26:08.451181 IP (tos 0x0, ttl 64, id 65257, offset 0, flags [none], proto UDP (17), length 55)

    #.#.#.#.54228 > 208.67.222.222.domain: [udp sum ok] 23065+ A? yandex.ru. (27)

15:26:08.498299 IP (tos 0x0, ttl 57, id 4778, offset 0, flags [DF], proto UDP (17), length 119)

    208.67.222.222.domain > #.#.#.#.54228: [udp sum ok] 23065 q: A? yandex.ru. 4/0/0 yandex.ru. [4m4s] A 77.88.55.80, yandex.ru. [4m4s] A 5.255.255.80, yandex.ru. [4m4s] A 77.88.55.50, yandex.ru. [4m4s] A 5.255.255.60 (91)

15:26:13.451133 IP (tos 0x0, ttl 64, id 537, offset 0, flags [none], proto UDP (17), length 55)

    #.#.#.#.54228 > 208.67.222.222.domain: [udp sum ok] 23065+ A? yandex.ru. (27)

15:26:13.498225 IP (tos 0x0, ttl 57, id 5523, offset 0, flags [DF], proto UDP (17), length 119)

    208.67.222.222.domain > #.#.#.#.54228: [udp sum ok] 23065 q: A? yandex.ru. 4/0/0 yandex.ru. [3m59s] A 5.255.255.80, yandex.ru. [3m59s] A 77.88.55.50, yandex.ru. [3m59s] A 5.255.255.60, yandex.ru. [3m59s] A 77.88.55.80 (91)

15:26:18.451231 IP (tos 0x0, ttl 64, id 1389, offset 0, flags [none], proto UDP (17), length 55)

    #.#.#.#.54228 > 208.67.222.222.domain: [udp sum ok] 23065+ A? yandex.ru. (27)

15:26:18.498305 IP (tos 0x0, ttl 57, id 6088, offset 0, flags [DF], proto UDP (17), length 119)

    208.67.222.222.domain > #.#.#.#.54228: [udp sum ok] 23065 q: A? yandex.ru. 4/0/0 yandex.ru. [3m54s] A 77.88.55.50, yandex.ru. [3m54s] A 5.255.255.60, yandex.ru. [3m54s] A 77.88.55.80, yandex.ru. [3m54s] A 5.255.255.80 (91)

So it seems that Ubuntu works well with DNS server, sending request, getting answers. But somehow cannot parse this answer...

For now I have no any more ideas how to get DNS resolving back to work.

Am I did all right, or something was not, and what else I can do to make it work?

Upd 17 Oct:

ip route output:

~$ ip route

default via x.x.x.41 dev enp29s0  metric 10 onlink 

10.0.0.0/8 dev enp3s0  proto kernel  scope link  src 10.0.35.115 

x.x.x.40/29 dev enp29s0  proto kernel  scope link  src x.x.x.42 

169.254.0.0/16 dev enp3s0  scope link  metric 1000 



~$ ip route get 8.8.8.8

8.8.8.8 via x.x.x.41 dev enp29s0  src x.x.x.42 

    cache 

traceroute to Google DNS:

~$ traceroute 8.8.8.8

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets

 1  x.x.x.41 (x.x.x.41)  1.413 ms  1.404 ms  1.420 ms

 2  89.221.63.15 (89.221.63.15)  3.451 ms  3.473 ms  3.463 ms

 3  195.208.208.232 (195.208.208.232)  3.360 ms  2.851 ms  2.843 ms

 4  108.170.250.129 (108.170.250.129)  3.582 ms 108.170.250.97 (108.170.250.97)  4.029 ms 108.170.250.33 (108.170.250.33)  3.737 ms

 5  108.170.226.91 (108.170.226.91)  3.679 ms 66.249.94.195 (66.249.94.195)  4.103 ms 108.170.227.65 (108.170.227.65)  3.828 ms

 6  8.8.8.8 (8.8.8.8)  4.506 ms  3.786 ms  3.979 ms

x.x.x.42 - server address.

x.x.x.40 and x.x.x.41 - "x" is the same as in server address.

Finally I found the problem place: it was iptables issue.
I didn't remember where I change this rule, but all UDP packets were set to DROP:

iptables -A INPUT -p udp -i $EXT_INTERFACE -j DROP

Setting to ACCEPT resolve problem:

iptables -A INPUT -p udp -i $EXT_INTERFACE -j ACCEPT

After doing a apt update & apt upgrade my server (Ubuntu 16.04) was unable to resolve hosts as well.

Quite annoying as it broke my delegate node's ability to generate blocks...

Anyways. I stumbled upon this topic.

Instead of accepting udp via iptables (I utilize ufw)...

I used this command: sudo ufw reset to clear out previous rules.

Then recreate my rules. I didn't have that many.

Then sudo ufw enable to restart the firewall.

Works again :)