| jose | hey jcastro, did you have that call with elmo? | 03:25 |
|---|---|---|
| dholbach | good morning | 08:22 |
| dholbach | balloons, davidcalle, dpm, mhall119: who of us will do the Q&A today? | 09:11 |
| dholbach | popey won't be around today | 09:11 |
| davidcalle | dholbach: can do | 09:12 |
| dpm | I can as well | 09:13 |
| dholbach | I was in the one last week, but I can do it too | 09:17 |
| dholbach | just thought it'd be good if we could plan and set up the event a bit earlier than last time :) | 09:17 |
| dpm | wow, snowing again here | 11:51 |
| jcastro | jose: not until thursday | 12:42 |
| dpm | hey jcastro, who can help us figuring out how to bring back ubucon.org online? We're planning UbuCon Europe and we really need it back live. I couldn't even get 'juju status' on the jumping station | 13:28 |
| dpm | balloons, are hangouts working for you today, and if so, are you up for the Q&A? | 13:29 |
| dpm | if not, mhall119 or dholbach, would you be up for it even though you were on last week? | 13:29 |
| balloons | ohh, so long as I'm not the only one hosting, I think it'll be fine | 13:29 |
| balloons | but I suppose if there are others availible it would be safer heh. Google loves me until it doesn't | 13:30 |
| dholbach | dpm, sure... | 13:30 |
| jose | dpm: si necesitas, te puedo dar una mano en un momento | 13:30 |
| dpm | balloons, I'll be there too today | 13:31 |
| dpm | jose, sería genial. Tienes credenciales para entrar en la jumpstation? | 13:31 |
| dpm | balloons, awesome work with the core apps tests doc! | 13:35 |
| balloons | yea, I thought it would be good to have a mapping for every test we touch, in case anyone wants that much detail :-) | 13:36 |
| dpm | cool | 13:36 |
| dpm | I think it makes it much easier to see the overall picture of the QA status of core apps | 13:37 |
| jose | dpm: no las tengo, pero podemos hacer un troubleshooting básico hasta que jorge aparezca | 13:37 |
| dpm | balloons, quick question: would it be possible to a) add the counts for the unit, integration and manual tests, in the same way you did for the acceptance tests? and b) have the conditional formatting be green if > 0? | 13:37 |
| dpm | jose, ok, podemos hacer las dos cosas, te añado las credenciales | 13:39 |
| balloons | dpm, I considered adding counts for those. Sure, let me tewak | 13:41 |
| dpm | balloons, cool. Yeah, I'm thinking as we're going through this, it might be worth having some metrics | 13:41 |
| dpm | and then the formatting is consistent across all test columns | 13:42 |
| jcastro | dpm: marco is at MWC so mostly unavailable | 13:48 |
| jcastro | dpm: is it running in canonicstack? | 13:48 |
| jcastro | or in aws? | 13:48 |
| dpm | jcastro, aws | 13:49 |
| jcastro | ok, I'll text him to let him know asap | 13:50 |
| jcastro | has anyone been able to get a hold of him? | 13:50 |
| dpm | we've got a shared jumpstation from where it's all managed | 13:50 |
| jcastro | ack | 13:50 |
| dpm | I e-mailed yesterday, but haven't heard back. I assumed he was traveling or at MWC | 13:50 |
| dholbach | balloons, so you and myself for the Q&A later on? | 13:53 |
| balloons | Indeed | 13:55 |
| jose | dpm: so, apparently jujud is down, should be as simple as starting it again, but what worries me is the fact that the service is not up... should be even if jujud is down | 13:58 |
| dpm | jcastro, and none of us can bring it up without the sudo password :/ | 13:58 |
| jose | the other option would be launching a new instance and using that in the meanwhile | 13:59 |
| jcastro | hey so I telegrammed him, dpm, do you know anyone you can ping at MWC to get ahold of marco? | 14:02 |
| jcastro | aka. let's ping multiple people | 14:02 |
| jose | I can try and ping Amber | 14:02 |
| dpm | willcooke or olli. Let me see who else | 14:02 |
| jcastro | Left a voicemail too | 14:03 |
| balloons | dholbach, so 1500 or 1600 for the q and a? | 14:06 |
| * balloons is curious why we have 2 entries again in calendar ;-( | 14:06 | |
| jose | blame timezones, it's scheduled to start in 53 mins | 14:07 |
| dholbach | no, 1h53m | 14:07 |
| jose | ? | 14:07 |
| dholbach | 16 UTC | 14:07 |
| jose | when did we move it? | 14:07 |
| balloons | ahh, indeed, joseeantonior@gmail.com created the one at 1500 | 14:07 |
| balloons | blah, sorry | 14:08 |
| jose | last time you blamed me because it was at 16 and told me to move it to 15, now we back at 16? | 14:08 |
| balloons | we've been as early as 1400 and as late as 1600 | 14:08 |
| balloons | depending on the wind, weather, and timezones | 14:08 |
| jose | >.> | 14:09 |
| jose | ok, set a time for the next months and let me know so I can set all of the events to that time | 14:10 |
| jcastro | jose: he's stuck at the booth, is spinning up a new instance an option? | 14:16 |
| dholbach | jose, it's because of DST I think | 14:18 |
| jose | jcastro: I can spin up a new instance and we can change DNSs | 14:24 |
| jcastro | that seems like a safe bet for now | 14:24 |
| jose | dholbach: yup, but we need to set a UTC time and stick to it | 14:24 |
| jose | dpm: ok, if you give me around an hour I can spin up a new instance and then we can copy stuff over | 14:25 |
| dholbach | jose, no, because that'd mean that the Europeans work an hour later | 14:26 |
| dholbach | (half of the year) | 14:26 |
| jose | oh, the q&a is your last hour of the day | 14:27 |
| dholbach | yep | 14:27 |
| jose | ok, but let me know when those time changes please, that we tweets go off alright and people don't get confused with the cal :) | 14:27 |
| jose | when those time changes happen* | 14:27 |
| dholbach | ok :) | 14:41 |
| dpm | jose, sorry, just finished our call with Daniel. Let me get hold of willcooke to see if he can physically grab Marco. If it's just as simple as bringing juju back up, that'd save you quite a lot of work | 14:56 |
| dholbach | balloons, so you and myself for the Q&A later on? | 14:56 |
| dholbach | or did you set up the event already? | 14:56 |
| balloons | I did not setup the event dholbach, no | 14:58 |
| dholbach | ok... I'll do it then | 14:58 |
| balloons | dholbach, thanks. And yes, we'll do it, and we have an hour before we're live | 14:58 |
| dpm | jose, jcastro, I guess another instance will mean a new IP address? I'm not sure what would be quicker, we'd need to get hold of Richard Gaskin to point the domain to the new IP, and I'm not sure if he's around today | 15:20 |
| dpm | and then the IP change to propagate? | 15:27 |
| jose | dpm: yes, unfortunately it's our best bet right now because Marco is stuck at the booth | 15:57 |
| jose | if you want me to spin up that instance just lmk | 15:57 |
| dpm | jose, mhall119, can I leave it in your hands to decide on the best course of action to get ubucon.org back online? You guys are far more experts than I when it comes to deployments, and I trust you to take the best decision | 16:04 |
| jose | I can definitely work on it. if mhall119 or you want to have a call to discuss it/deploy it, I'm free all day long | 16:05 |
| jose | I know your EOD is approaching, but I'll be around all US day long in case Mike is available later | 16:05 |
| mhall119 | jose: I'm available any time, what's our current status with the site? | 16:12 |
| jose | mhall119: site is down, jujud in the instance is down and I can't reboot it - don't have the sudo password to reboot it, only Marco has it and we're blocked until he's free or out of MWC | 16:13 |
| mhall119 | jcastro: do you happen to know what password marcoceppi used for the ubucon.org site? | 16:14 |
| * dpm hugs jose | 16:15 | |
| * jose hugs dpm back | 16:15 | |
| czajkowski | mhall119: upwithjujudownwithwiki | 16:16 |
| mhall119 | lol | 16:17 |
| * mhall119 tries it anyway | 16:18 | |
| mhall119 | nope | 16:18 |
| mhall119 | dpm: tracks are setup in summit | 16:29 |
| dpm | \o/ | 16:29 |
| dholbach | all right my friends - I call it a day - see you tomorrow! | 17:01 |
| dpm | thanks dholbach for running the Q&A! | 17:01 |
| dpm | it was a fun one today, have a nice rest of the day! | 17:01 |
| dholbach | yeah, you too | 17:02 |
| jose | mhall119: have time for a call now? | 17:08 |
| dpm | mhall119, ah, I forgot to ask you, how did the Fairphone meeting with mariogrip go yesterday? | 17:10 |
| === zequence_ is now known as zequence | ||
| mhall119 | dpm: no breakthroughs on the Mir side, but he suspects it's related to the other kernel problems, same with wifi. He's documenting those so we can take them to the FP contacts you sent | 17:53 |
| mhall119 | jose: sorry, was eating lunch, I'm available now | 17:53 |
| jose | mhall119: sounds good, will send you a link in a bit? | 17:56 |
| mhall119 | jose: hang on, I have to load Firefox | 17:59 |
| jose | lol np | 17:59 |
| dpm | thanks for the update mhall119 | 18:15 |
| marcoceppi | jose: mhall119 what is the jumpstations | 18:33 |
| marcoceppi | mhall119 jose I don't have much time before I have to pack up | 18:35 |
| marcoceppi | mhall119 jose on the jumpstation, investigating | 18:36 |
| mhall119 | marcoceppi: jose wanted to try and restart some stuff, or su to eco user, or something | 18:39 |
| marcoceppi | mhall119: why? juju is /not/ running on this machine it's a client | 18:40 |
| marcoceppi | it looks like amazon killed those instances | 18:40 |
| mhall119 | marcoceppi: can you first see if the ubucon.org node still exist? | 18:40 |
| mhall119 | :( | 18:40 |
| mhall119 | do we have backups? | 18:40 |
| marcoceppi | i don't see them in the ec2 dashboard | 18:40 |
| mhall119 | or block storage devices still available? | 18:40 |
| marcoceppi | let me check if put the backup script in plcae | 18:40 |
| marcoceppi | okay, while I try to do disaster recovery, I'm going to redeploy the charms | 18:41 |
| mhall119 | marcoceppi: we set some EC2_ environment variables | 18:42 |
| mhall119 | you'll probably want to unset those | 18:43 |
| mhall119 | though that shouldn't cause the error there... | 18:43 |
| marcoceppi | :( | 18:47 |
| marcoceppi | wtf aws | 18:47 |
| marcoceppi | they took out like half of the deployments | 18:47 |
| * mhall119 hopes marcoceppi isn't live-demoing anything this week | 18:51 | |
| marcoceppi | no, not my demos | 18:53 |
| marcoceppi | but like | 18:53 |
| mhall119 | marcoceppi: did we get backups? | 18:53 |
| marcoceppi | half ofthe Charm Developer Program accounts | 18:53 |
| marcoceppi | none, all the disks, instances, gone | 18:54 |
| marcoceppi | I'm checking the API logs | 18:54 |
| dpm | oh | 18:55 |
| mhall119 | I think we've officially ruined marcoceppi's evening | 18:55 |
| dpm | does it mean there is no way to recover the site's content? | 18:55 |
| marcoceppi | no, you guys didn't do this | 18:55 |
| marcoceppi | stupid clouds are ephemeral and I didn't put the backup cron in place to sync them off the servers | 18:56 |
| marcoceppi | I messed up you alls week | 18:56 |
| marcoceppi | Okay, so I've redeployed, I'm putting the backup cron in place, I'm so sorry | 18:56 |
| marcoceppi | I'm going to RCA the heck outta this | 18:57 |
| marcoceppi | I am not happy about 15 aws instances going missing | 18:57 |
| marcoceppi | traveling brb 1 hr | 18:58 |
| dpm | marcoceppi, so just to be clear, there is absolutely no way to recover the content? | 18:58 |
| marcoceppi | dpm: not that I'm aware of at this time | 18:58 |
| marcoceppi | dpm: wayback machine and google cache migh thelp | 19:00 |
| dpm | no copies on archive.org either :/ | 19:00 |
| svij | ouch | 19:00 |
| dpm | does anyone have any other ideas how to get back some of the data at least? This was months of work | 19:01 |
| marcoceppi | mhall119 dpm: 52.90.179.155 so sorry :(((((( | 19:02 |
| svij | dpm: https://web.archive.org/web/20151222184907/http://ubucon.org/en/ | 19:03 |
| svij | seems to work? | 19:03 |
| svij | hm not all subpages | 19:04 |
| dpm | svij, ah, nice find! | 19:04 |
| marcoceppi | some content > no | 19:04 |
| * marcoceppi cries in corner | 19:04 | |
| dpm | svij, only the front page is online as far as I can tell | 19:05 |
| svij | hm, only talks available :/ | 19:05 |
| svij | https://web.archive.org/web/20160112104741/http://ubucon.org/en/events/ubucon-summit-us/talks/ this one works aswell | 19:05 |
| dpm | oh, didn't see talks | 19:05 |
| marcoceppi | svij dpm please let me know if there's anyhting I can do to help | 19:05 |
| * dpm starts saving pages | 19:05 | |
| marcoceppi | I have to run to catch a bus | 19:05 |
| marcoceppi | I will be back online in a bit | 19:05 |
| svij | ok, thanks so far marcoceppi | 19:06 |
| mhall119 | thanks marcoceppi | 19:06 |
| dpm | marcoceppi, we'll let you go, thanks. If there is _anything_ you can think of for us to recover at least some of the content, that'd be awesome | 19:06 |
| marcoceppi | svij dpm I don' thave a solution or RCA atm, you're not the only ones affected, you just the only one I didn't set up the cron for | 19:08 |
| marcoceppi | that won't happen again | 19:08 |
| svij | RCA? | 19:08 |
| svij | dpm: http://www.cachedview.com/index.php?lang=de putting the links over there does load the text-content, thats good! | 19:08 |
| marcoceppi | root cause analysis - why did this happen and how to not have it happen again | 19:09 |
| jose | thanks for your help marcoceppi | 19:10 |
| svij | ah, didn't know that | 19:11 |
| svij | dpm: is there anything gone which was not publicly on the webpage? | 19:14 |
| dpm | svij, all info was public, there were no private pages | 19:15 |
| svij | good | 19:15 |
| svij | so we can use the data from google cache | 19:15 |
| mhall119 | dpm: svij: jose: google cache has some content as well | 19:17 |
| svij | mhall119: that's what I just said^^ | 19:17 |
| dpm | svij, I'm saving the pages as soon as I find a cached version - so far I've found part of the front page, ubucon summit page, schedule | 19:17 |
| svij | https://webcache.googleusercontent.com/search?q=cache:http://ubucon.org/en/events/ubucon-summit-us/travel-and-accommodation/ | 19:17 |
| mhall119 | svij: ah, I wasn't familiar with cachedview.com | 19:17 |
| svij | https://webcache.googleusercontent.com/search?q=cache:http://ubucon.org/en/events/ubucon-summit-us/travel-and-accommodation/ | 19:17 |
| svij | arg | 19:18 |
| dpm | it will still be quite a lot of work to format the pages, but if we can find as many as possible, then this will reduce it from a disaster to "a lot of work", which I can live with | 19:18 |
| svij | there is every page accessible | 19:18 |
| svij | I'll c/p it here and you can save them up: | 19:19 |
| svij | Sponsors: https://webcache.googleusercontent.com/search?q=cache:http://ubucon.org/en/events/ubucon-summit-us/sponsors/ | 19:19 |
| svij | UbuCon LA https://webcache.googleusercontent.com/search?q=cache:http://ubucon.org/en/events/ubucon-latin-america/ | 19:19 |
| svij | or any better ideas to collect the content? | 19:20 |
| mhall119 | wget them and gzip them up? | 19:20 |
| jose | I can help migrate the site tomorrow or later, but need to do some stuff now | 19:21 |
| svij | might better option | 19:21 |
| jose | but please assign me something | 19:21 |
| mhall119 | thank sfor your help earlier jose | 19:21 |
| jose | and I'll look over it as soon as I have time | 19:21 |
| * svij should go back to his slides and his exam tomorrow :/ | 19:21 | |
| jose | no worries mhall119 | 19:21 |
| jose | svij: go study! real life first! | 19:21 |
| jose | also, it would be a great time to consider moving the site to IS hosting | 19:21 |
| svij | jose: I do now! Thanks for your help ;) | 19:22 |
| dpm | svij, yeah, go back to slides + study, you've helped quite a lot already! | 19:22 |
| mhall119 | jose: mojo :( | 19:22 |
| jose | mhall119: afaik they only require the service to be charmed, which is already done | 19:22 |
| dpm | mhall119, or canonistack? | 19:22 |
| mhall119 | jose: they require a mojo spec now too | 19:22 |
| dpm | jose, we wished | 19:22 |
| jose | blah | 19:22 |
| mhall119 | dpm: canonistack is less reliable than AWS | 19:22 |
| jose | what I was about to say | 19:23 |
| jose | if you want I can start working on that later today | 19:23 |
| mhall119 | it's intentionally that way too, it's not for production hosting | 19:23 |
| dpm | so if I understand it correctly, the most immediate actions are: | 19:23 |
| dpm | - redeploy the site | 19:24 |
| mhall119 | already done | 19:24 |
| dpm | - fetch the content | 19:24 |
| dpm | - upload the content | 19:24 |
| mhall119 | - change the DNS to point to 52.90.179.155 | 19:25 |
| jose | as soon as the content is uploaded, I can get ahold of Richard | 19:25 |
| dpm | - set up backups | 19:25 |
| svij | "set up backups" should be moved to point 1. ;) | 19:25 |
| svij | (or task 1) | 19:25 |
| dpm | lol, I agree | 19:25 |
| dpm | jose, mhall119, I guess we can already change the DNS even if the content is not there yet? | 19:26 |
| svij | no web page or broken webpage… | 19:26 |
| svij | ok, really should close IRC now… | 19:26 |
| svij | o/ | 19:26 |
| dpm | have a nice evening svij! | 19:26 |
| svij | "nice evening" … need to study. :P but thanks! | 19:27 |
| mhall119 | jose: may as well start the DNS change now, it'll take time and at least the new IP responds to request | 19:27 |
| dpm | jose, mhall119, do you know who can set up the backups, is it marcoceppi only? | 19:29 |
| jose | dpm: he's already set them up afaik | 19:33 |
| jose | mhall119: ok, as soon as I'm back I'll ping richard, need to run to the store | 19:34 |
| dpm | marcoceppi, when you're back online, can you confirm they're set up and backed up offsite? | 19:34 |
| dpm | jose, mhall119, http://pad.ubuntu.com/ubucon-recovery | 19:35 |
| dpm | also, another option could be to host the site at Digital Ocean. It works well with juju, and actually the first version of the site was hosted there | 19:36 |
| mhall119 | it's already back up on AWS | 19:43 |
| mhall119 | let's focus on gettign the content back there, we can talk about moving it elsewhere once marco is back to normal hours | 19:44 |
| dpm | mhall119, can you help me populating the cached pages list on the etherpad? I'll start creating the IA on the live site now, but I'll have to go soon and continue uploading tomorrow morning | 19:46 |
| dpm | ok, I need to run, I'll follow up on e-mail | 20:21 |
| dpm | cheers | 20:23 |
| === philipballew is now known as Guest44851 | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!