Mac seems to ignore the result of local DNS for local hosts

Question

I have a server (running Ubuntu 16.04) on my local network, which amongst other things, runs DMSmasq. My local domain is njj.chickenkiller.com.

Running on my Mac, if you query the DNS, it returns the LAN address of various machines on my network. E.g.:

beethoven:~ nick$ dig serv2.njj.chickenkiller.com

; <<>> DiG 9.8.3-P1 <<>> serv2.njj.chickenkiller.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36867
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;serv2.njj.chickenkiller.com.   IN  A

;; ANSWER SECTION:
serv2.njj.chickenkiller.com. 0  IN  A   192.168.2.4

;; Query time: 82 msec
;; SERVER: 192.168.2.4#53(192.168.2.4)
;; WHEN: Fri Aug 11 14:22:39 2017
;; MSG SIZE  rcvd: 61

and in this example, my Mac can correctly resolve, using the default DNS, the address of serv2 on my LAN.

However, when I use other commands (e.g. ping) sometimes they correctly resolve:

beethoven:~ nick$ ping serv2.njj.chickenkiller.com
PING serv2.njj.chickenkiller.com (192.168.2.4): 56 data bytes
64 bytes from 192.168.2.4: icmp_seq=0 ttl=64 time=0.832 ms

other times, my Mac seems to ignore the result from the local DNS (which is the only name server it is configured to use) and somehow manages to use the WAN address (that you would get if you queried an other DNS):

beethoven:lib nick$ ping serv2
PING njj.chickenkiller.com (90.255.91.177): 56 data bytes
64 bytes from 90.255.91.177: icmp_seq=0 ttl=64 time=2.140 ms

As far as I can tell, the local DNS server works correctly -- it always returns the LAN address (192.168.2.4) and never returns the WAN address (90.255.91.177). I also have both Windows 7 and Linux machine running on the network when I use the ping command (or any other command) they always use the LAN address (192.168.2.4), so I assume that this is something specific to macOS.

Why is this important to me? The router I have from my ISP discards packets from the LAN attempting to access machines using the WAN address. This means that commands accessing other machines on the network fail. eg:

beethoven:lib nick$ ssh serv2
ssh: connect to host serv2 port 22: Connection refused

I am using macOS 10.12.6

Any suggestions as to how I should configure my Mac so it only uses the configured DNS to resolve hosts would be gratefully received.

EDIT: Whats my search domain?: On the network setting page the DNS server is shown as 192.168.2.4 and the search domain is shown as njj.chickenkiller.com

EDIT 2: Output of commands:

beethoven:bin nick$ networksetup -listallnetworkservices
An asterisk (*) denotes that a network service is disabled.
Wi-Fi
*Bluetooth PAN
*Thunderbolt Bridge
beethoven:bin nick$ networksetup -getsearchdomains  Wi-Fi
There aren't any Search Domains set on Wi-Fi.

EDIT 3: dnsmasq.conf (from ubuntu 16.04 host, "serv2")

# Use google open DNS name servers (avoids the risk of a router attack)
server=/#/8.8.8.8
server=/#/8.8.4.4

# Do not use /etc/hosts, but serve hostnames from /etc/hosts.dnsmasq
no-hosts
addn-hosts=/etc/hosts.dnsmasq
local-ttl=105

# expand unqualified hostnames to suffix njj.chickenkiller.com
expand-hosts
domain=njj.chickenkiller.com

# turn on dhcp (limit leases to 12hours so updates dont need a restart)
dhcp-range=192.168.2.10,192.168.2.99,12h

# and redirect all gateway requests through router
dhcp-option=option:router,192.168.2.1

# and offer a time server
dhcp-option=option:ntp-server,192.168.2.4

# and we are the *only* dns server
dhcp-option=option:dns-server,192.168.2.4

Answer 1

This has been driving me mad for ages, anyway, I think that I have stumbled on the solution.

The way I had set up my dynamic DNS was that serv2.njj.chickenkiller.com was an alias for njj.chickenkiller.com(probably not the best way of doing this, but it meant that serv2 handled all the requests for njj.chickenkiller.com.

So when you query the server:

beethoven:~ nick$ host serv2
serv2.njj.chickenkiller.com has address 192.168.2.4
serv2.njj.chickenkiller.com is an alias for njj.chickenkiller.com.

So what I think is happening is that mDNSresponder was clever enough to cache njj.chickenkiller.com as an alias (and I assume that this doesn't happen on windows and linux), so when it did a subsequent DNS lookup for serv2, it tried to save time by sending a query for njj.chickenkiller.com (not serv2...), which it has cached as the alias. My local DNS server didn't recognize njj.chickenkiller.com as part of the local domain and responded by giving the external IP address.

To solve this I added njj.chickenkiller.com as an alias for serv2 in the hosts file used by dnsmasq:

# Fixed IP machines
192.168.2.1 gateway
192.168.2.2 wifi1
192.168.2.3 wifi2
192.168.2.4 serv2   njj.chickenkiller.com