Upstart

linux

https://www.digitalocean.com/community/tutorials/the-upstart-event-system-what-it-is-and-how-to-use-it
http://upstart.ubuntu.com/getting-started.html
https://www.linux.com/learn/manage-system-startup-and-boot-processes-linux-upstart
https://www.experts-exchange.com/questions/27118232/Amazon-EC2-inittab-not-spawning-process.html

http://upstart.ubuntu.com/cookbook/ - printed
http://www.zohaib.me/how-to-reliably-start-a-service-with-upstart/ - printed
https://askubuntu.com/questions/62434/why-does-upstart-keep-respawning-my-process - printed
https://serverfault.com/questions/777682/does-upstart-have-a-built-in-way-to-respawn-a-process-that-is-not-stopped-using - done reading
https://serversforhackers.com/video/process-monitoring-with-upstart - printed
http://www.alexreisner.com/code/upstart - printed
https://unix.stackexchange.com/questions/84252/how-to-start-a-service-automatically-when-ubuntu-starts - printed, contains good explanation of both "System V" init style and upstart
https://www.keypressure.com/blog/a-few-notes-on-upstart/ - printed
https://newcome.wordpress.com/2012/02/26/running-programs-as-linux-daemons-using-upstart/ - printed
http://stackoverflow.com/questions/19862285/have-process-automatically-restart-with-upstart-and-initctl - done reading

Init is the first process that is started. It is responsible for starting the rest of the processes that need to be started when the system boots. Some distribution use "System V" init system, and some distribution use a simpler init system known as "BSD init" system.

Modern distributions now use a more modern replacement for init. Such modern replacements includes upstart, systemd.

Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.

Using upstart, we only need to worry about a single configuration file, which by convention, lives in /etc/init and is named "service".conf, where "service" is the name of the service that this file describes. An example of a service configuration file:

description "my service description"
author "Me <myself@i.com>"

# Stanzas
#
# Stanzas control when and how a process is started and stopped
# See a list of stanzas here: http://upstart.ubuntu.com/wiki/Stanzas#respawn

# When to start the service
start on runlevel [2345]

# When to stop the service
stop on runlevel [016]

# Automatically restart process if crashed
respawn

# Essentially lets upstart know the process will detach itself to the background
expect fork

# Specify working directory
chdir /path/to/some/directory

# Run before process
pre-start script
    [ -d /var/run/myservice ] || mkdir -p /var/run/myservice
    echo "Put bash code here"
end script

# Start the process
exec myprocess

Another example:

description     "uShare UPnP A/V & DLNA Media Server"
author          "Roman Yepishev <rtg@rtg.in.ua>"

start on filesystem
stop on runlevel [016]

respawn

setuid nobody
setgid nogroup

exec /usr/bin/ushare
start on filesystem
stop on runlevel [!2345]

respawn
respawn limit 5 30

chdir /home/minecraft/bukkit

expect daemon
kill timeout 30

pre-start script
    test -x /home/minecraft/bukkit/craftbukkit-0.0.1-SNAPSHOT.jar || { stop; exit 0; }
end script

pre-stop script
    tmux send -t bukkit "stop"
    tmux send -t bukkit "Enter"
    sleep 10  # Wait for server to shut down properly
end script

exec tmux new-session -d -s minecraft -n bukkit "sudo -u minecraft -- 
  /home/minecraft/java/jre1.6.0_27/bin/java -Xincgc -Xmx1G -jar 
  /home/minecraft/bukkit/craftbukkit-0.0.1-SNAPSHOT.jar"
description "A stupid golang http listener"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]

setuid www-data
setgid www-data

respawn
respawn limit 5 2

exec /opt/listen
description "A stupid golang http listener"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]

setuid www-data
setgid www-data

env API_KEY=abcdefgh

respawn
respawn limit 5 2

chdir /opt

pre-start script
    # Source File
    . /path/to/env/file
end script

script
    # Start Listener
    /opt/listen
end script
start on runlevel [2345]

script  
  # Change the working directory to where the realase is
  chdir /path/to/release

  # Start the phoenix server
  exec ./hello_phoenix foreground
end script

# Respawn and stop respawning if it got respawned 10 time in 10 seconds
respawn limit 10 10

See more upstart-examples

Where can we find the log file for our service?

Upstart maintains a log file for our service at: /var/log/upstart/serviceName.log. If there is any issue with the service, look at this log file. This has the stdout from our service.

How can we start a service after another service had been started?

Use the "start on started serviceName".

start on started rc

How can we start a service that is managed by upstart?

start serviceName

If we are using Amazon AMI, to start this service:

initctl start serviceName

How can we stop a service?

stop serviceName
initctl stop serviceName

How can we see the status of a a service?

status serviceName
initctl status serviceName

How can we list all the services managed by upstart?

initctl list

How can we reload / restart (stop and start) a service?

reload serviceName

How can we use the respawn feature?

respawn
respawn limit 10 5
respawn limit COUNT INTERVAL

In order to control the respawn limit, we need to provide both the "respawn" and the "respawn limit" stanzas. The "respawn limit" alone does not enable the respawn behavior. The "respawn limit" stanza above limits the respawn to 10 respawns per 5 seconds. If the application respawn faster than that (basically it cannot start successfully), then init will say "…respawning too fast, stopped". This limit does not necessarily mean that it will spread the number of respawn evenly. It may respawn as quickly as possible upto that limit if the time interval has not expired. Check the official documentation for details. For example

respawn limit 10 90

means to give up if restart occurs 10 times in 90 seconds.

The difficulty here is the combination of 'respawn' with the pre-stop script that tells the process to stop. From init(5), respawn:

A service or task with this stanza will be automatically started if it should stop abnormally. All reasons for a service stopping except for the stop(8) command itself are considered as abnormal. Tasks may exit with a zero exit status to prevent being respawned.

The documentation is a little unclear on the point of whether exiting with a zero status should cause a respawn. Later version of upstart fixed this bug. However, if we have to work around this bug, we can use the "normal" stanza:

normal exit STATUS|SIGNAL

Additional exit statuses or even signals may be added if the job process terminates with any of these, it will not be considered to have failed and will not be respawned.

normal exit 0 1 TERM HUP

It may be more robust to stop a process by sending it a signal instead of using a pre-stop process that issues commands, but of course this is not always possible if the service does not support a clean shutdown upon receipt of a signal.

How can we monitor another service?

description "Notify root about ushare respawn limit reached"
author "Roman Yepishev <rtg@rtg.in.ua>"

# This is a short-lived job
task

start on stopped ushare PROCESS=respawn

script
    echo "uShare reached respawn limit" | mail -s "uShared crashed" root
end script

This is another type of Upstream job, a task. It does not keep running forever and once it has finished it will not be respawned. What we just did is we created a simple monitoring task that will be executed once ushare reaches stopped state and a special PROCESS variable will have respawn as the reason why the service stopped.

Creating a service to monitor another service indicates that there is a real problem with the original service. Before monitoring another service this way, consider if we can fix the root cause in the original service, really think about what we want to accomplish, see if what we are about to do is the only solution, and check whether there are existing solutions already.

How can we disable an upstart task?

By default when we install a service in Ubuntu, it will be immediately started. Sometimes, however, we might want to start the service manually. Upstart has a concept of overrides that basically let you override any stanza in the original job file by creating a "job".override file in /etc/init and placing the stanzas there. Disabling a task is trivial. We just need to add the "manual" stanza:

manual

If we must override the original service configuration file, we should create the .override file, so that future upgrading to a new version does not overwrite our overrides.

In the above code, because we do not want the service to be started automatically, we add the "manual" stanza to the .override file. If we want to change the behavior of other stanzas, such as various options that are passed to the main program, copy the stanza from the original service configuration file into the .override file, and do the necessary changes in the .override file.

What are the various stages that upstart uses?

waiting -> starting -> security -> pre-start -> spawned -> post-started -> running -> pre-stop -> stopping -> killed -> post-stop

The states are self-explanatory but it can be confusing between a few (spawned vs post-started vs running). 'spawned' is the state before your script is executed to start the service, post-started is the state before the event signalling that the job has started is emitted, and running is when your job is considered as started.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License