https://www.digitalocean.com/community/tutorials/the-upstart-event-system-what-it-is-and-how-to-use-it
http://upstart.ubuntu.com/getting-started.html
https://www.linux.com/learn/manage-system-startup-and-boot-processes-linux-upstart
https://www.experts-exchange.com/questions/27118232/Amazon-EC2-inittab-not-spawning-process.html
http://upstart.ubuntu.com/cookbook/ - printed
http://www.zohaib.me/how-to-reliably-start-a-service-with-upstart/ - printed
https://askubuntu.com/questions/62434/why-does-upstart-keep-respawning-my-process - printed
https://serverfault.com/questions/777682/does-upstart-have-a-built-in-way-to-respawn-a-process-that-is-not-stopped-using - done reading
https://serversforhackers.com/video/process-monitoring-with-upstart - printed
http://www.alexreisner.com/code/upstart - printed
https://unix.stackexchange.com/questions/84252/how-to-start-a-service-automatically-when-ubuntu-starts - printed, contains good explanation of both "System V" init style and upstart
https://www.keypressure.com/blog/a-few-notes-on-upstart/ - printed
https://newcome.wordpress.com/2012/02/26/running-programs-as-linux-daemons-using-upstart/ - printed
http://stackoverflow.com/questions/19862285/have-process-automatically-restart-with-upstart-and-initctl - done reading
Init is the first process that is started. It is responsible for starting the rest of the processes that need to be started when the system boots. Some distribution use "System V" init system, and some distribution use a simpler init system known as "BSD init" system.
Modern distributions now use a more modern replacement for init. Such modern replacements includes upstart, systemd.
Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.
Using upstart, we only need to worry about a single configuration file, which by convention, lives in /etc/init and is named "service".conf, where "service" is the name of the service that this file describes. An example of a service configuration file:
description "my service description"
author "Me <myself@i.com>"
# Stanzas
#
# Stanzas control when and how a process is started and stopped
# See a list of stanzas here: http://upstart.ubuntu.com/wiki/Stanzas#respawn
# When to start the service
start on runlevel [2345]
# When to stop the service
stop on runlevel [016]
# Automatically restart process if crashed
respawn
# Essentially lets upstart know the process will detach itself to the background
expect fork
# Specify working directory
chdir /path/to/some/directory
# Run before process
pre-start script
[ -d /var/run/myservice ] || mkdir -p /var/run/myservice
echo "Put bash code here"
end script
# Start the process
exec myprocess
Another example:
description "uShare UPnP A/V & DLNA Media Server"
author "Roman Yepishev <rtg@rtg.in.ua>"
start on filesystem
stop on runlevel [016]
respawn
setuid nobody
setgid nogroup
exec /usr/bin/ushare
start on filesystem
stop on runlevel [!2345]
respawn
respawn limit 5 30
chdir /home/minecraft/bukkit
expect daemon
kill timeout 30
pre-start script
test -x /home/minecraft/bukkit/craftbukkit-0.0.1-SNAPSHOT.jar || { stop; exit 0; }
end script
pre-stop script
tmux send -t bukkit "stop"
tmux send -t bukkit "Enter"
sleep 10 # Wait for server to shut down properly
end script
exec tmux new-session -d -s minecraft -n bukkit "sudo -u minecraft --
/home/minecraft/java/jre1.6.0_27/bin/java -Xincgc -Xmx1G -jar
/home/minecraft/bukkit/craftbukkit-0.0.1-SNAPSHOT.jar"
description "A stupid golang http listener"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]
setuid www-data
setgid www-data
respawn
respawn limit 5 2
exec /opt/listen
description "A stupid golang http listener"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]
setuid www-data
setgid www-data
env API_KEY=abcdefgh
respawn
respawn limit 5 2
chdir /opt
pre-start script
# Source File
. /path/to/env/file
end script
script
# Start Listener
/opt/listen
end script
start on runlevel [2345]
script
# Change the working directory to where the realase is
chdir /path/to/release
# Start the phoenix server
exec ./hello_phoenix foreground
end script
# Respawn and stop respawning if it got respawned 10 time in 10 seconds
respawn limit 10 10
See more upstart-examples
Where can we find the log file for our service?
Upstart maintains a log file for our service at: /var/log/upstart/serviceName.log. If there is any issue with the service, look at this log file. This has the stdout from our service.
How can we start a service after another service had been started?
Use the "start on started serviceName".
start on started rc
How can we start a service that is managed by upstart?
start serviceName
If we are using Amazon AMI, to start this service:
initctl start serviceName
How can we stop a service?
stop serviceName
initctl stop serviceName
How can we see the status of a a service?
status serviceName
initctl status serviceName
How can we list all the services managed by upstart?
initctl list
How can we reload / restart (stop and start) a service?
reload serviceName
How can we use the respawn feature?
respawn
respawn limit 10 5
respawn limit COUNT INTERVAL
In order to control the respawn limit, we need to provide both the "respawn" and the "respawn limit" stanzas. The "respawn limit" alone does not enable the respawn behavior. The "respawn limit" stanza above limits the respawn to 10 respawns per 5 seconds. If the application respawn faster than that (basically it cannot start successfully), then init will say "…respawning too fast, stopped". This limit does not necessarily mean that it will spread the number of respawn evenly. It may respawn as quickly as possible upto that limit if the time interval has not expired. Check the official documentation for details. For example
respawn limit 10 90
means to give up if restart occurs 10 times in 90 seconds.
The difficulty here is the combination of 'respawn' with the pre-stop script that tells the process to stop. From init(5), respawn:
A service or task with this stanza will be automatically started if it should stop abnormally. All reasons for a service stopping except for the stop(8) command itself are considered as abnormal. Tasks may exit with a zero exit status to prevent being respawned.
The documentation is a little unclear on the point of whether exiting with a zero status should cause a respawn. Later version of upstart fixed this bug. However, if we have to work around this bug, we can use the "normal" stanza:
normal exit STATUS|SIGNAL
Additional exit statuses or even signals may be added if the job process terminates with any of these, it will not be considered to have failed and will not be respawned.
normal exit 0 1 TERM HUP
It may be more robust to stop a process by sending it a signal instead of using a pre-stop process that issues commands, but of course this is not always possible if the service does not support a clean shutdown upon receipt of a signal.
How can we monitor another service?
description "Notify root about ushare respawn limit reached"
author "Roman Yepishev <rtg@rtg.in.ua>"
# This is a short-lived job
task
start on stopped ushare PROCESS=respawn
script
echo "uShare reached respawn limit" | mail -s "uShared crashed" root
end script
This is another type of Upstream job, a task. It does not keep running forever and once it has finished it will not be respawned. What we just did is we created a simple monitoring task that will be executed once ushare reaches stopped state and a special PROCESS variable will have respawn as the reason why the service stopped.
Creating a service to monitor another service indicates that there is a real problem with the original service. Before monitoring another service this way, consider if we can fix the root cause in the original service, really think about what we want to accomplish, see if what we are about to do is the only solution, and check whether there are existing solutions already.
How can we disable an upstart task?
By default when we install a service in Ubuntu, it will be immediately started. Sometimes, however, we might want to start the service manually. Upstart has a concept of overrides that basically let you override any stanza in the original job file by creating a "job".override file in /etc/init and placing the stanzas there. Disabling a task is trivial. We just need to add the "manual" stanza:
manual
If we must override the original service configuration file, we should create the .override file, so that future upgrading to a new version does not overwrite our overrides.
In the above code, because we do not want the service to be started automatically, we add the "manual" stanza to the .override file. If we want to change the behavior of other stanzas, such as various options that are passed to the main program, copy the stanza from the original service configuration file into the .override file, and do the necessary changes in the .override file.
What are the various stages that upstart uses?
waiting -> starting -> security -> pre-start -> spawned -> post-started -> running -> pre-stop -> stopping -> killed -> post-stop
The states are self-explanatory but it can be confusing between a few (spawned vs post-started vs running). 'spawned' is the state before your script is executed to start the service, post-started is the state before the event signalling that the job has started is emitted, and running is when your job is considered as started.
How can we add a daemon to upstart?
To add your daemon to Upstart, add a file such as /etc/init/daemon-example.conf with contents:
start on startup
respawn
exec /opt/portal/daemon-example.py start
description "Manage the bqm processes and the submitter processes"
author "Khai Doan"
start on runlevel [3]
exec /download/SRM/submitters/startQueuesManager.sh
How can we check the syntax of the configuration file?
To check the syntax of the configuration file:
init-checkconf /etc/init/testjob.conf
Amazon AMI does not seem to have the init-checkconf command for some reason.
