[14:53] <jrwren> anyone else a little worried right now?
[14:53] <urulama> jrwren: ?
[14:53] <urulama> jrwren: irc server?
[14:54] <jrwren> not just irc server. www.jujucharms.com is down too. makes me think a whole DC is down
[14:54] <urulama> jrwren: oh!
[14:54] <rick_h_> boom
[14:55] <rick_h_> hmm, ubuntu.com, wiki works
[14:55] <bac> pow
[14:55] <jrwren> whew
[14:55] <rick_h_> ubuntu insights is down though
[14:55] <rick_h_> so yea, big boom
[14:55] <bac> launchpad i sup
[14:55] <bac> is up
[14:55] <jrwren> i wonder if ubuntu.com is cloudflair fronted :)
[14:56] <jrwren> oh, yeah lp!
[14:56] <rick_h_> LP seems up
[14:56] <bac> join freenode #canonical-sysadmin
[14:56] <rick_h_> and SSO
[14:56] <urulama> login.ubuntu.com as well
[14:58] <rick_h_> bac: or urulama can you please see about updating topics in juju related channels for the outage?
[14:58] <bac> rick_h_: yes
=== urulama changed the topic of #juju-gui to: Juju GUI - Mailing List  https://lists.ubuntu.com/mailman/listinfo/juju-gui - Note: jujucharms.com is down
[14:59] <rick_h_> urulama: rumor is "ps4.5 went boom"
[14:59]  * rick_h_ dies a little more inside
[14:59] <rick_h_> urulama: bac jcsackett we need to have a plan for a redo on mojo/prodstack just in case please. 
[15:00] <bac> rick_h_: i'm not sure what such a plan would look like.  our current mojo is fully deployable from scratch
[15:00] <jcsackett> so then, that's our plan. :p
[15:00] <bac> rick_h_: IOW, if we have new machines it should be good to go
[15:01] <urulama> let's see, it sounds it's a networking issue
[15:01] <rick_h_> jcsackett: bac urulama I guess maybe I mean more...let's have folks on deck ready to help walk/deal with any issues getting a rebuild
[15:01] <bac> rick_h_: certainly
[15:02] <rick_h_> urulama: rumor in telegram is "openvswitch auto upgrade"
[15:02] <urulama> lovely. so content should be fine
[15:02] <rick_h_> urulama: hopefully
[15:02] <urulama> but yeah, i'll be around ... it's been months since we had real issues :)
[15:02] <bac> uiteam: the topic in #canonical-sysadmin was changed to "network issues".  so cross fingers that the actual machines are still there
[15:02] <frankban_> yeah
[15:03]  * rick_h_ starts to open/air a bottle of wine
[15:03] <bac> urulama: haha, too bad you're not on telegram.  <ducks>
[15:03] <frankban_> urulama: on the other hand, great news in mail about cmr
[15:03] <urulama> bac: one more reason not to be there :)
[15:03] <bac> urulama: chicken
[15:03] <urulama> frankban_: \o/
[15:04] <jrwren> lol @ "auto upgrade"
[15:04] <bac> urulama: should we tweet a status message
[15:04] <urulama> bac: please do
[15:04]  * bac looks for credentials
[15:07] <bac> urulama: done
[15:07] <urulama> bac: ty much!
[15:08] <bac> incident report opened: https://wiki.canonical.com/IncidentReports/2015-12-17-ps45-networking-issues
[15:08] <bac> no content yet
[15:09] <urulama> we need to have 2 envs in two reagions
[15:09] <urulama> at least
[15:11] <jrwren> cross env relations are coming :p
[15:17] <urulama> heh, old clients (< 1.24) can still deploy
[15:17] <urulama> https://manage.jujucharms.com/api/3/charm/trusty/juju-gui
[15:17] <urulama> :)
[15:20] <frankban_> urulama: not without the legacy charm store
[15:20] <urulama> frankban_: hah, right
[15:21] <fabrice> urulama: let's ave our own region on aws 
[15:23] <bac> urulama: that's great news that the old charmworld is alive.  i'd hate to have to bring it back up
[15:24] <urulama> exactly! that's why i wanted to check it out
[15:24] <bac> mthaddon posted an update.  restarting neutron on affected machines
[15:25] <fabrice> humm neutrron update, sounds familiar
[15:27] <urulama> uiteam: irc is up
[15:27] <frankban_> urulama: the charm store is up
[15:27] <fabrice> jujucharms.com is up
[15:28] <jcsackett> \o/
=== fabrice is now known as fabrice|school
[15:32] <jrwren> nice recovery!
[15:33] <urulama> jrwren: it's still down
[15:33] <jrwren> oh.
[15:33] <urulama> or better, it's down again
=== urulama changed the topic of #juju-gui to: Juju GUI - Mailing List  https://lists.ubuntu.com/mailman/listinfo/juju-gui
[15:45] <rogpeppe1> fabrice|school: how can you say i'm not on vacation?! that's *fun. :)
[15:50] <frankban_> uiteam call in 10
[15:53] <urulama> hey there, rogpeppe1
[15:53] <urulama> how's it going?
[15:53] <rogpeppe1> urulama: yo!
[15:53] <rogpeppe1> urulama: good thanks
[15:53] <rogpeppe1> urulama: made some good progress with my hydro project this week
[15:53] <rogpeppe1> urulama: (and just got distracted with the letsencrypt thing :-])
[15:53] <rogpeppe1> urulama: how's it going in topazland?
[15:54] <urulama> rogpeppe1: all good apart from current outage :)
[15:54] <urulama> rogpeppe1: where are you at with hydro project? 
[15:55] <rogpeppe1> urulama: i have linked frontend and backend, so i can edit stuff in frontend and i see relays turn on and off (well, notionally, on my fake eth8020 relay server)
[15:55] <rogpeppe1> urulama: the frontend still looks shit tho
=== fabrice|school is now known as fabrice
[15:55] <urulama> cool!
[15:56] <urulama> will you make a snap in the end?
[15:56] <rogpeppe1> urulama: and i'm not quite sure what kind of approach to use for "intermediate values" (i.e. user is modifying stuff; it *will* be valid but isn't yet)
[15:56] <rogpeppe1> urulama: yeah, that's the idea
[15:56] <fabrice> rogpeppe1: we have fun everyday !
[15:56] <rogpeppe1> urulama: although i'm still not sure how to work the persistent data thing
[15:56] <rogpeppe1> fabrice: yeah, of course, forgot :)
[15:58] <jrwren> hydroponics?
[15:58] <rogpeppe1> jrwren: that's right. an automated control system for my 13 acre marijuana plot
[15:59] <rogpeppe1> jrwren: :)
[15:59] <rogpeppe1> jrwren: hydro-electric
[16:00] <jrwren> 13 acres! i wouldn't have to grow marijuana if I had that much land.
[16:00] <jrwren> err, I mean, I wouldn't want to.
[16:00] <rogpeppe1> lol