Wether you're running 1800 production web servers or a single VPS, there will come a time where you need to do some automated log cleanup, update some configs or monitor the state of some process or network connection.
So you write your little script, and dump it on the machine, via puppet or salt. You add an entry to your cron tab and never worry about it again.
Yeah. That's never the end - Some new requirements come up, the script needs to check new things or the config files move because of an os upgrade, the damn thing keeps emailing you warnings. We come to a point where we need to patch our script and deploy the changes. And make sure the old version stops, and the new versions is running. Over and over.
This is the point where it's handy to have your bundle of maintenance scripts running from an auto-updating git working copy on your boxes.
The aim here is to build a script that runs and does it's work while keeping an eye on your git repos' state.
Setup clone the repo install your depends
I'm going to assume you want to add this to an existing project:
me@compy386:~ git clone your-github-project /usr/local/yourthing
me@compy386:~ cpanm -L /usr/local/yourthing/misc AnyEvent AnyEvent::HTTP
but if you want to just play with this without defacing one of your public projects:
me@compy386:~
mkdir -p restarter/misc
cd restarter
git init
<paste code in a file called computer_program>
git add .
git commit -am "Let's get this party started"
cpanm -L misc AnyEvent AnyEvent::HTTP
(maybe git ignore misc/*)
Supervise it, or just run it
Pop something like this in if your supervisor.conf, conf.d/computer_program.conf or whatever:
[program:computer_program]
command=/usr/local/yourthing/computer_program --loglevel=%(ENV_LOGLEVEL)s
# You'll also want to configure log rotation if you're doing this long term.
And fire it up
me@compy386:~ supervisorctl start computer_program
me@compy386:~ supervisorctl status
And
me@compy386:~ ps ax | grep computer[_]program
computer_program 4b825dc642cb6eb9a060e54bf8d69288fbee4904
If you're not convinced, you can just run it directly
me@compy386:~ /usr/local/yourthing/computer_program
... It'll just run.
See it at work
... since the point is to restart the script each time the git repo its running from changes, The best way to to test this is to just commit locally and see the script notice the change.
Open up another terminal and commit a change to the script, or a anything in the directory.
me@compy386:/usr/local/yourthing echo "# never mind" >> computer_program
me@compy386:/usr/local/yourthing git commit -am "Helpful docs"
In the other tab we get one of these:
2017-05-02 23:15:11.163089 +0200 trace main: restarter: first run, starting with /usr/local/yourthing at 4ffdd6cc2b36b6fd69553202b7db1c7288927521
2017-05-02 23:17:41.198546 +0200 trace main: restarter[4ffdd6cc2b36b6fd69553202b7db1c7288927521 cmp 5f0e2d68cc2fef915215490db7ad5f6a52c75a45]: not taking more jobs
Depending on what you set $CHECK_EVERY to, the script will bail out, and if supervised it'll be restarted.
The script itself
#! /usr/bin/perl
#ABSTRACT: wireframe AnyEvent program that exits when code changes.
use warnings; use strict;
my $MY_NAME = $0;
use Cwd qw(cwd abs_path);
use File::Basename qw(dirname);
use File::Spec::Functions qw(catdir);
our $NEST;
use lib catdir( $NEST= dirname( abs_path($0) ), 'misc/lib/perl5');
# ...for use with cpanm -L misc AnyEvent
use AnyEvent;
use AnyEvent::Util qw/ run_cmd /;
my $computer_program = AnyEvent->condvar;
# simple self-restart setup
my ( $CHECK_EVERY, $MURDER_AFTER) = ( 2.5*60, 150 );
my (
$WRAP_IT_UP, # flag, if true don't pick up any new jobs. (Timer for exit).
$RUNNING_JOBS, # guards for currently running jobs
$running_sha, # string, sha1 of the last commit on this directory
$restarter, # guard, runs git log.
);
my $timer = AnyEvent->timer(
after => 1,
interval => $CHECK_EVERY,
cb => sub {
# this will kill the previous run by destroying the guard.
$restarter = run_cmd [qw/ git log -1 --format=%H /, $NEST],
'>' => sub {
return unless @_; # If it doesn't run, just keep on keep'on.
my $output = shift;
chomp $output;
if (not defined $running_sha) {
$0 = join ' ', $MY_NAME, $running_sha=$output;
AE::log trace=> "restarter: first run, starting with %s at %s", $NEST, $output;
return;
}
my $we_good = (defined $running_sha and defined $output and $running_sha eq $output);
AE::log trace=> "restarter[%s cmp %s]: %s",$running_sha, $output,
$we_good ? 'seems fine' : 'not taking more jobs'
;
# life is fine, keep going.
return if $we_good;
# nothing going on, so just bail out.
if (not defined $RUNNING_JOBS or not @$RUNNING_JOBS) {
AE::log info=> "restarter[%s cmp %s]: New software while idle. Bailing out to upgrade.",$running_sha, $output,
return $computer_program->send
}
my $murder_in=$MURDER_AFTER;
AE::log trace=> "restarter[%s cmp %s]: New software while Busy. Murder in %ss",$running_sha, $output, $murder_in;
# busy. murder the jobs in a mintes time.
$WRAP_IT_UP = AnyEvent->timer(after => $murder_in, cb => sub {
AE::log info=> "restarter: %ss timer has passed, murdering %s jobs", $murder_in, 0+@{ $RUNNING_JOBS || [] };
undef $_ for @{ $RUNNING_JOBS || [] };
return $computer_program->send;
});
}
}
);
# start the loop:
$computer_program->recv;
__END__
Exercises for the reader
When first working on a script like this one can easily get in trouble, the need to commit a change in order to to test the restart behaviour makes for an exciting life. Fortunately the script will also notice if you use git commit --amend to change the sha of the current commit.
This won't be your usual work flow, but it makes testing much more fun.
Obviously you can only amend a commit if you haven't pushed it yet.
Have the script do some stuff
Use AnyEvent::HTTP to poll some web service, and check the response for some value.
use AnyEvent::Log info to log when the value is present so you can see your code working.
If you add the right callbacks to your request you should be able to see the event being cancelled when you commit a change to the script.
Have your script check for changes on your remote
Since we're already polling git for changes, we could go one step further and poll for changes to the remote too.
This is relatively straight forward if you have a passwordless http remote like the ones github provide.
Here you'd want to use git remote update and git rev-parse origin/HEAD to check if your working copy and/or script are at the lastest SHA for your repo.
Spawn an ssh-agent for private repos
If our git repo is secret, like on gitlab or paid github accounts, you'll need to use a "deploy key" to fetch changes from the repo.
You can use run_cmd to start up another process running ssh-agent, once you know the socket path it choses, you can pass that info on in the environment to your GIT_SSH for any commands that interact with your remotes.
Wrap up
There are some other obvious options for getting your changes out:
- Packaging your script and sticking it the package a mirror, restarting with hooks
- Have your config management drop the new copy and do the
init/rc.d/systemddance - put your script in a base container, and rebuild your farm with every change
Each have their tradeoffs, and I'm a simple fellow. I like to push a change, and watch graphs to see my changes kick in. After all, if your change doesn't have an impact on the graphs, was it really a change?
Being able to push changes to a git repo and have them go live automatically takes a lot of the sting out of having to deploy daemons to large groups of production boxes, the more pessimistic reader will ask "But if I push a broken version, won't the script stop doing the git-fetch?"
Yep. That'll happen. You'd likely want one script that just does the git pull, and nothing else, while the rest of your scripts just watch for the change and restart when they change.