josehey jcastro, did you have that call with elmo?03:25
dholbachgood morning08:22
dholbachballoons, davidcalle, dpm, mhall119: who of us will do the Q&A today?09:11
dholbachpopey won't be around today09:11
davidcalledholbach: can do09:12
dpmI can as well09:13
dholbachI was in the one last week, but I can do it too09:17
dholbachjust thought it'd be good if we could plan and set up the event a bit earlier than last time :)09:17
dpmwow, snowing again here11:51
jcastrojose: not until thursday12:42
dpmhey jcastro, who can help us figuring out how to bring back ubucon.org online? We're planning UbuCon Europe and we really need it back live. I couldn't even get 'juju status' on the jumping station13:28
dpmballoons, are hangouts working for you today, and if so, are you up for the Q&A?13:29
dpmif not, mhall119 or dholbach, would you be up for it even though you were on last week?13:29
balloonsohh, so long as I'm not the only one hosting, I think it'll be fine13:29
balloonsbut I suppose if there are others availible it would be safer heh. Google loves me until it doesn't13:30
dholbachdpm, sure...13:30
josedpm: si necesitas, te puedo dar una mano en un momento13:30
dpmballoons, I'll be there too today13:31
dpmjose, sería genial. Tienes credenciales para entrar en la jumpstation?13:31
dpmballoons, awesome work with the core apps tests doc!13:35
balloonsyea, I thought it would be good to have a mapping for every test we touch, in case anyone wants that much detail :-)13:36
dpmI think it makes it much easier to see the overall picture of the QA status of core apps13:37
josedpm: no las tengo, pero podemos hacer un troubleshooting básico hasta que jorge aparezca13:37
dpmballoons, quick question: would it be possible to a) add the counts for the unit, integration and manual tests, in the same way you did for the acceptance tests? and b) have the conditional formatting be green if > 0?13:37
dpmjose, ok, podemos hacer las dos cosas, te añado las credenciales13:39
balloonsdpm, I considered adding counts for those. Sure, let me tewak13:41
dpmballoons, cool. Yeah, I'm thinking as we're going through this, it might be worth having some metrics13:41
dpmand then the formatting is consistent across all test columns13:42
jcastrodpm: marco is at MWC so mostly unavailable13:48
jcastrodpm: is it running in canonicstack?13:48
jcastroor in aws?13:48
dpmjcastro, aws13:49
jcastrook, I'll text him to let him know asap13:50
jcastrohas anyone been able to get a hold of him?13:50
dpmwe've got a shared jumpstation from where it's all managed13:50
dpmI e-mailed yesterday, but haven't heard back. I assumed he was traveling or at MWC13:50
dholbachballoons, so you and myself for the Q&A later on?13:53
josedpm: so, apparently jujud is down, should be as simple as starting it again, but what worries me is the fact that the service is not up... should be even if jujud is down13:58
dpmjcastro, and none of us can bring it up without the sudo password :/13:58
josethe other option would be launching a new instance and using that in the meanwhile13:59
jcastrohey so I telegrammed him, dpm, do you know anyone you can ping at MWC to get ahold of marco?14:02
jcastroaka. let's ping multiple people14:02
joseI can try and ping Amber14:02
dpmwillcooke or olli. Let me see who else14:02
jcastroLeft a voicemail too14:03
balloonsdholbach, so 1500 or 1600 for the q and a?14:06
* balloons is curious why we have 2 entries again in calendar ;-(14:06
joseblame timezones, it's scheduled to start in 53 mins14:07
dholbachno, 1h53m14:07
dholbach16 UTC14:07
josewhen did we move it?14:07
balloonsahh, indeed, joseeantonior@gmail.com created the one at 150014:07
balloonsblah, sorry14:08
joselast time you blamed me because it was at 16 and told me to move it to 15, now we back at 16?14:08
balloonswe've been as early as 1400 and as late as 160014:08
balloonsdepending on the wind, weather, and timezones14:08
joseok, set a time for the next months and let me know so I can set all of the events to that time14:10
jcastrojose: he's stuck at the booth, is spinning up a new instance an option?14:16
dholbachjose, it's because of DST I think14:18
josejcastro: I can spin up a new instance and we can change DNSs14:24
jcastrothat seems like a safe bet for now14:24
josedholbach: yup, but we need to set a UTC time and stick to it14:24
josedpm: ok, if you give me around an hour I can spin up a new instance and then we can copy stuff over14:25
dholbachjose, no, because that'd mean that the Europeans work an hour later14:26
dholbach(half of the year)14:26
joseoh, the q&a is your last hour of the day14:27
joseok, but let me know when those time changes please, that we tweets go off alright and people don't get confused with the cal :)14:27
josewhen those time changes happen*14:27
dholbachok :)14:41
dpmjose, sorry, just finished our call with Daniel. Let me get hold of willcooke to see if he can physically grab Marco. If it's just as simple as bringing juju back up, that'd save you quite a lot of work14:56
dholbachballoons, so you and myself for the Q&A later on?14:56
dholbachor did you set up the event already?14:56
balloonsI did not setup the event dholbach, no14:58
dholbachok... I'll do it then14:58
balloonsdholbach, thanks. And yes, we'll do it, and we have an hour before we're live14:58
dpmjose, jcastro, I guess another instance will mean a new IP address? I'm not sure what would be quicker, we'd need to get hold of Richard Gaskin to point the domain to the new IP, and I'm not sure if he's around today15:20
dpmand then the IP change to propagate?15:27
josedpm: yes, unfortunately it's our best bet right now because Marco is stuck at the booth15:57
joseif you want me to spin up that instance just lmk15:57
dpmjose, mhall119, can I leave it in your hands to decide on the best course of action to get ubucon.org back online? You guys are far more experts than I when it comes to deployments, and I trust you to take the best decision16:04
joseI can definitely work on it. if mhall119 or you want to have a call to discuss it/deploy it, I'm free all day long16:05
joseI know your EOD is approaching, but I'll be around all US day long in case Mike is available later16:05
mhall119jose: I'm available any time, what's our current status with the site?16:12
josemhall119: site is down, jujud in the instance is down and I can't reboot it - don't have the sudo password to reboot it, only Marco has it and we're blocked until he's free or out of MWC16:13
mhall119jcastro: do you happen to know what password marcoceppi used for the ubucon.org site?16:14
* dpm hugs jose16:15
* jose hugs dpm back16:15
czajkowskimhall119: upwithjujudownwithwiki16:16
* mhall119 tries it anyway16:18
mhall119dpm: tracks are setup in summit16:29
dholbachall right my friends - I call it a day - see you tomorrow!17:01
dpmthanks dholbach for running the Q&A!17:01
dpmit was a fun one today, have a nice rest of the day!17:01
dholbachyeah, you too17:02
josemhall119: have time for a call now?17:08
dpmmhall119, ah, I forgot to ask you, how did the Fairphone meeting with mariogrip go yesterday?17:10
=== zequence_ is now known as zequence
mhall119dpm: no breakthroughs on the Mir side, but he suspects it's related to the other kernel problems, same with wifi. He's documenting those so we can take them to the FP contacts you sent17:53
mhall119jose: sorry, was eating lunch, I'm available now17:53
josemhall119: sounds good, will send you a link in a bit?17:56
mhall119jose: hang on, I have to load Firefox17:59
joselol np17:59
dpmthanks for the update mhall11918:15
marcoceppijose: mhall119 what is the jumpstations18:33
marcoceppimhall119 jose I don't have much time before I have to pack up18:35
marcoceppimhall119 jose on the jumpstation, investigating18:36
mhall119marcoceppi: jose wanted to try and restart some stuff, or su to eco user, or something18:39
marcoceppimhall119: why? juju is /not/ running on this machine it's a client18:40
marcoceppiit looks like amazon killed those instances18:40
mhall119marcoceppi: can you first see if the ubucon.org node still exist?18:40
mhall119do we have backups?18:40
marcoceppii don't see them in the ec2 dashboard18:40
mhall119or block storage devices still available?18:40
marcoceppilet me check if put the backup script in plcae18:40
marcoceppiokay, while I try to do disaster recovery, I'm going to redeploy the charms18:41
mhall119marcoceppi: we set some EC2_ environment variables18:42
mhall119you'll probably want to unset those18:43
mhall119though that shouldn't cause the error there...18:43
marcoceppiwtf aws18:47
marcoceppithey took out like half of the deployments18:47
* mhall119 hopes marcoceppi isn't live-demoing anything this week18:51
marcoceppino, not my demos18:53
marcoceppibut like18:53
mhall119marcoceppi: did we get backups?18:53
marcoceppihalf ofthe Charm Developer Program accounts18:53
marcoceppinone, all the disks, instances, gone18:54
marcoceppiI'm checking the API logs18:54
mhall119I think we've officially ruined marcoceppi's evening18:55
dpmdoes it mean there is no way to recover the site's content?18:55
marcoceppino, you guys didn't do this18:55
marcoceppistupid clouds are ephemeral and I didn't put the backup cron in place to sync them off the servers18:56
marcoceppiI messed up you alls week18:56
marcoceppiOkay, so I've redeployed, I'm putting the backup cron in place, I'm so sorry18:56
marcoceppiI'm going to RCA the heck outta this18:57
marcoceppiI am not happy about 15 aws instances going missing18:57
marcoceppitraveling brb 1 hr18:58
dpmmarcoceppi, so just to be clear, there is absolutely no way to recover the content?18:58
marcoceppidpm: not that I'm aware of at this time18:58
marcoceppidpm: wayback machine and google cache migh thelp19:00
dpmno copies on archive.org either :/19:00
dpmdoes anyone have any other ideas how to get back some of the data at least? This was months of work19:01
marcoceppimhall119  dpm: so sorry :((((((19:02
svijdpm: https://web.archive.org/web/20151222184907/http://ubucon.org/en/19:03
svijseems to work?19:03
svijhm not all subpages19:04
dpmsvij, ah, nice find!19:04
marcoceppisome content > no19:04
* marcoceppi cries in corner19:04
dpmsvij, only the front page is online as far as I can tell19:05
svijhm, only talks available :/19:05
svijhttps://web.archive.org/web/20160112104741/http://ubucon.org/en/events/ubucon-summit-us/talks/ this one works aswell19:05
dpmoh, didn't see talks19:05
marcoceppisvij dpm please let me know if there's anyhting I can do to help19:05
* dpm starts saving pages19:05
marcoceppiI have to run to catch a bus19:05
marcoceppiI will be back online in a bit19:05
svijok, thanks so far marcoceppi19:06
mhall119thanks marcoceppi19:06
dpmmarcoceppi, we'll let you go, thanks. If there is _anything_  you can think of for us to recover at least some of the content, that'd be awesome19:06
marcoceppisvij dpm I don' thave a solution or RCA atm, you're not the only ones affected, you just the only one I didn't set up the cron for19:08
marcoceppithat won't happen again19:08
svijdpm: http://www.cachedview.com/index.php?lang=de putting the links over there does load the text-content, thats good!19:08
marcoceppiroot cause analysis - why did this happen and how to not have it happen again19:09
josethanks for your help marcoceppi19:10
svijah, didn't know that19:11
svijdpm: is there anything gone which was not publicly on the webpage?19:14
dpmsvij, all info was public, there were no private pages19:15
svijso we can use the data from google cache19:15
mhall119dpm: svij: jose: google cache has some content as well19:17
svijmhall119: that's what I just said^^19:17
dpmsvij, I'm saving the pages as soon as I find a cached version - so far I've found part of the front page, ubucon summit page, schedule19:17
mhall119svij: ah, I wasn't familiar with cachedview.com19:17
dpmit will still be quite a lot of work to format the pages, but if we can find as many as possible, then this will reduce it from a disaster to "a lot of work", which I can live with19:18
svijthere is every page accessible19:18
svijI'll c/p it here and you can save them up:19:19
svijSponsors: https://webcache.googleusercontent.com/search?q=cache:http://ubucon.org/en/events/ubucon-summit-us/sponsors/19:19
svijUbuCon LA https://webcache.googleusercontent.com/search?q=cache:http://ubucon.org/en/events/ubucon-latin-america/19:19
svijor any better ideas to collect the content?19:20
mhall119wget them and gzip them up?19:20
joseI can help migrate the site tomorrow or later, but need to do some stuff now19:21
svijmight better option19:21
josebut please assign me something19:21
mhall119thank sfor your help earlier jose19:21
joseand I'll look over it as soon as I have time19:21
* svij should go back to his slides and his exam tomorrow :/19:21
joseno worries mhall11919:21
josesvij: go study! real life first!19:21
josealso, it would be a great time to consider moving the site to IS hosting19:21
svijjose: I do now! Thanks for your help ;)19:22
dpmsvij, yeah, go back to slides + study, you've helped quite a lot already!19:22
mhall119jose: mojo :(19:22
josemhall119: afaik they only require the service to be charmed, which is already done19:22
dpmmhall119, or canonistack?19:22
mhall119jose: they require a mojo spec now too19:22
dpmjose, we wished19:22
mhall119dpm: canonistack is less reliable than AWS19:22
josewhat I was about to say19:23
joseif you want I can start working on that later today19:23
mhall119it's intentionally that way too, it's not for production hosting19:23
dpmso if I understand it correctly, the most immediate actions are:19:23
dpm- redeploy the site19:24
mhall119already done19:24
dpm- fetch the content19:24
dpm- upload the content19:24
mhall119 - change the DNS to point to
joseas soon as the content is uploaded, I can get ahold of Richard19:25
dpm- set up backups19:25
svij"set up backups" should be moved to point 1. ;)19:25
svij(or task 1)19:25
dpmlol, I agree19:25
dpmjose, mhall119, I guess we can already change the DNS even if the content is not there yet?19:26
svijno web page or broken webpage…19:26
svijok, really should close IRC now…19:26
dpmhave a nice evening svij!19:26
svij"nice evening" … need to study. :P but thanks!19:27
mhall119jose: may as well start the DNS change now, it'll take time and at least the new IP responds to request19:27
dpmjose, mhall119, do you know who can set up the backups, is it marcoceppi only?19:29
josedpm: he's already set them up afaik19:33
josemhall119: ok, as soon as I'm back I'll ping richard, need to run to the store19:34
dpmmarcoceppi, when you're back online, can you confirm they're set up and backed up offsite?19:34
dpmjose, mhall119, http://pad.ubuntu.com/ubucon-recovery19:35
dpmalso, another option could be to host the site at Digital Ocean. It works well with juju, and actually the first version of the site was hosted there19:36
mhall119it's already back up on AWS19:43
mhall119let's focus on gettign the content back there, we can talk about moving it elsewhere once marco is back to normal hours19:44
dpmmhall119, can you help me populating the cached pages list on the etherpad? I'll start creating the IA on the live site now, but I'll have to go soon and continue uploading tomorrow morning19:46
dpmok, I need to run, I'll follow up on e-mail20:21
=== philipballew is now known as Guest44851

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!