DHCPD and HTTPD services do not start





Last year, in 2020, I had a problem that HTTPD (the Apache web server) would not start on a reboot or cold boot on my server. At the time I added an override file, /etc/systemd/system/httpd.service.d/override.conf. That file contained the following statements that were designed to delay the startup of HTTPD until the network was started properly and on-line. It should be obvious by now that I use the NetworkManager and systemd not the old SystemV network service and start scripts.

# Trying to delay the startup of httpd so that the network is
# fully up and running so that httpd can bind to the correct
# IP address
#
# By David Both, 2020-04-16
[Unit]
After=network-online.target
Wants=network-online.target

This circumvention worked until recently when I not only needed to start HTTPD manually, and it also became necessary to start DHCPD manually. The wait for the network-online.target was no longer working for some reason.

The causes

After some more Internet searches and some digging around my /etc directory I think I have now discovered the true culprit. I found an ancient remnant from the SystemV and init days in the /etc/init.d directory. There was a copy of the old network startup file which should not have been there. It is probable that this file is a leftover from when I spent some time reverted to the old network program before I switched over to the NetworkManager. Apparently systemd did what it is supposed to do — it generated a target file from that SystemV start script on the fly and tried to start the network using both the SystemV start script and systemd target that it created. This caused systemd to attempt to start HTTPD and DHCPD before the network was ready and those services timed out and did not start.

I removed the /etc/init.d/network script from my server and it now reboots without the need for me to start the HTTPD and DHCPD services manually. This is a much better solution because it gets to the root cause and is not simple a circumvention. But this is still not the best solution. That file is owned by the network-scripts package and will be replaced if that package is updated. So I also removed that package from my server which ensures that this should not happen again. Can you guess how I discovered this?

So then I upgraded to Fedora 34 and DHCPD and HTTPD again would not start up. After some additional experimentation I found that the override.conf file also needed to have a couple additional lines added. These two new lines force those two services to wait until 60 seconds have passed before starting. That seems to solve the problem again — for now.

The revised override.conf file now looks like this. It not only sleeps for 60 seconds before starting the services it specifies that it is not supposed to start until after the network-online.target. This latter part is what seems to be broken at this time. But I figured I might as well do both things as one or the other usually seems to work.

# Delay the startup of any network service so that the 
# network is fully up and running so that httpd can bind to the correct 
# IP address.
#
# By David Both, 2020-04-28
#
################################################################################
#                                                                              #
#  Copyright (C) 2021 David Both                                               #
#  LinuxGeek46@both.org                                                        #
#                                                                              #
#  This program is free software; you can redistribute it and/or modify        #
#  it under the terms of the GNU General Public License as published by        #
#  the Free Software Foundation; either version 2 of the License, or           #
#  (at your option) any later version.                                         #
#                                                                              #
#  This program is distributed in the hope that it will be useful,             #
#  but WITHOUT ANY WARRANTY; without even the implied warranty of              #
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
#  GNU General Public License for more details.                                #
#                                                                              #
#  You should have received a copy of the GNU General Public License           #
#  along with this program; if not, write to the Free Software                 #
#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA   #
#                                                                              #
################################################################################

[Service]
ExecStartPre=/bin/sleep 60

[Unit]
After=network-online.target
Wants=network-online.target

The Ansible playbook

This is the type of problem that lends itself to an easy solution using Ansible. So I created the following playbook. This relatively simple playbook has two plays. The first play removes the network-scripts and then the /etc/init.d/network script because if the script is there and the package is not, the script won’t get removed. At least one of my systems had that circumstance. This play is run against all the hosts regardless of whether they are workstations or servers.

The second play runs only against the server and installs the override.conf files.

################################################################################
#                                 fix-network                                  #
#                                                                              #
# This Ansible playbook removes the network-scripts package and the            #
# /etc/rc.d/init.d/network SystemV start script. The /etc/init.d/network       #
# script which conflicts with NetworkManager and causes some network services  #
# such as DHCPD and HTTPD to fail to start.                                    #
#                                                                              #
# This playbook also installs override files for httpd and dhcpd which causes  #
# them to wait 60 seconds before starting.                                     #
#                                                                              #
# All of these things taken together seem to resolve or circumvent the issues  #
# that seem to stem from multiple causes.                                      #
#                                                                              #
# NOTE: The override file is service neutral and can be used with any service. #
#       I have found that using the systemctl edit command does not work as    #
#       it is supposed to according to the documenation.                       #
#                                                                              #
#                                                                              #
# From the network-scripts package info:                                       #
#                                                                              #
# : This package contains the legacy scripts for activating & deactivating of most
# : network interfaces. It also provides a legacy version of 'network' service.
# : 
# : The 'network' service is enabled by default after installation of this package,
# : and if the network-scripts are installed alongside NetworkManager, then the
# : ifup/ifdown commands from network-scripts take precedence over the ones provided
# : by NetworkManager.
# : 
# : If user has both network-scripts & NetworkManager installed, and wishes to
# : use ifup/ifdown from NetworkManager primarily, then they has to run command:
# :  $ update-alternatives --config ifup
# : 
# : Please note that running the command above will also disable the 'network'
# : service.
#                                                                              #
#                                                                              #
#------------------------------------------------------------------------------#
#                                                                              #
# Change History                                                               #
# 2021/04/26 David Both V01.00 New code.                                       #
# 2021/04/28 David Both V01.10 Revised to also remove network-scripts package. #
#                              Also install an override file to do a 60 second #
#                              timeout before the services start.              #                                                                              #                                                                              #
################################################################################
---
################################################################################
# Play 1: Remove the /etc/init.d/network file
################################################################################
- name: Play 1 - Remove the network-scripts legacy package on all hosts
  hosts: all

  tasks:
    - name: Remove the network-scripts package if it exists
      dnf:
        name: network-scripts
        state: absent

    - name: Remove /etc/init.d/network file if it exists but the network-scripts package is not installed
      ansible.builtin.file:
        path: /etc/init.d/network
        state: absent


- name: Play 2 - Install override files for the server services
  hosts: server

  tasks:

    - name: Install the override file for DHCPD
      copy:
        src: /root/ansible/BasicTools/files/override.conf
        dest: /etc/systemd/system/dhcpd.service.d
        mode: 0644
        owner: root
        group: root

    - name: Install the override file for HTTPD
      copy:
        src: /root/ansible/BasicTools/files/override.conf
        dest: /etc/systemd/system/httpd.service.d
        mode: 0644
        owner: root
        group: root

This Ansible playbook removed that bit of cruft from 2 additional hosts on my network and one host on another network that I support. All the hosts that still had the SystemV network script and the network-scripts package had not been re-installed from scratch for several years. They have all been upgraded using dnf-upgrade. I never circumvented NetworkManager on my newer hosts so they don’t have this problem.

This playbook also installed the override files for both services. Note that the override file has no reference to the service for which it provides the configuration override. For this reason it can be used for any service that does not start because the attempt to start them has not allowed the NetworkManager service to finish starting up.

Final thoughts

Although this problem is related to systemd startup it cannot all be blamed on systemd. This is, partly at least, a self-inflicted problem that was mostly caused when I circumvented systemd. At the time I thought I was making things easier for myself but I have spent more time trying to locate the problem caused by my avoidance of NetworkManager than I ever saved because I had to learn it anyway. And — in reality — there are multiple causes for this problem all of which are addressed by the Ansible playbook.

Remember that the if* scripts like ifup and ifdown will no longer work. Of course, if your host uses NetworkManager and DHCP those scripts and the ifcfg-* (interface configuration) files no longer exist in /etc/sysconfig/network-scripts. That directory is empty on all of my DHCP client hosts.