[00:03] <wallyworld> perrito666: i made some more comments
[00:05] <davecheney> how's LP, is it still crapped up ?
[00:13] <perrito666> wallyworld: ack, re registering in the shim, I just cargoculted that from other use of that pattern, ill change it
[00:16] <perrito666> wallyworld: I can do the per series dissallowance, that is free
[00:16] <perrito666> wallyworld: I answered
[00:21] <wallyworld> perrito666: so we copy whatever init files are relevant to that internal directory? upstart or systemd?
[00:23] <davecheney> thumper: when you skip a test you should use t.Skipf
[00:26] <davecheney> thumper: https://github.com/juju/juju/pull/5496/files#diff-758d572a629db5b9982ba1841b64dda5R14
[00:31] <davecheney> thumper: but, two thumbs up for not running the state tests on windows
[00:32] <perrito666> wallyworld: yup, its quite a clever thing that eric did, we create the init files there and are linked to the right path
[00:33] <wallyworld> perrito666: ok, so long as it all works :-)
[00:55] <perrito666> mmpf, trying to bootstrap trusty with --config enable-os-upgrade=false --config enable-os-refresh-update=false never ends :(
[00:56] <thumper> davecheney: ah, ok
[00:57] <thumper> will remember for next time
[01:02] <mup> Bug #1587689 opened: Can't upgrade controller with --upload-tools (due to rename) <juju-core:New for natefinch> <https://launchpad.net/bugs/1587689>
[01:02] <davecheney> thumper: s'ok, I had to rebase my branch anyway, so I fixed it for you
[01:03] <axw> wallyworld anastasiamac: I'm back
[01:03] <axw> free to chat now
[01:03] <anastasiamac> axw: wallyworld: tanzanite?
[01:04] <wallyworld> ok, one sec
[01:04] <axw> anastasiamac wallyworld: I'm there
[01:09] <davecheney> thumper: can we do the same thing we did to the windows state tests timing out to here
[01:09] <davecheney> https://bugs.launchpad.net/juju-core/+bug/1581157
[01:09] <mup> Bug #1581157: github.com/juju/juju/cmd/jujud test timeout on windows <blocker> <ci> <regression> <test-failure> <unit-tests> <windows> <juju-core:Triaged by dave-cheney> <https://launchpad.net/bugs/1581157>
[01:09] <davecheney> ?
[01:09] <davecheney> the underlying cause is the same
[01:10] <thumper> state was different because we don't run apiservers on windows
[01:10] <thumper> however jujud is needed
[01:10] <thumper> I'd be loathed to skip those on windwos
[01:10] <thumper> however
[01:10] <thumper> we could skip it for now until the npipe implementation is fixed
[01:11] <thumper> as long as we tag the skip with a bug number
[01:11] <thumper> and have the bug targetted to 2.0
[01:11] <thumper> thoughts?
[01:13] <davecheney> thumper: it ain't going to get fixed soon
[01:13] <thumper> davecheney: why? can't the windows npipe impl just use select?
[01:34]  * thumper taps his fingers waiting for tests
[01:37] <perrito666> wallyworld: around?
[01:37] <wallyworld> yes
[01:37] <perrito666> wallyworld: answer my priv then :p
[01:37] <wallyworld> perrito666: was getting coffee
[01:37] <wallyworld> perrito666: it's already committed
[01:37] <wallyworld> perrito666: but actually
[01:37] <wallyworld> it can't be used yet, i'll fix
[01:38] <wallyworld> after you land
[01:38] <wallyworld> so ignore me
[01:40] <perrito666> lol, ok, because I tried merging master and did not bring the change
[01:45]  * perrito666 ponders getting coffee then remembers the tim
[01:45] <perrito666> time
[01:55] <thumper> menn0: got a second?
[01:55] <menn0> thumper: yep
[01:55] <thumper> menn0: 1:1 hangout
[01:58] <davecheney> thumper: 'cos it's windows and select doesn't exist on windows
[02:19] <thumper> wallyworld: are you touching core/description with the Service -> Application rename at all?
[02:20] <wallyworld> thumper: yeah
[02:20] <thumper> ok
[02:20] <wallyworld> if you make changes between now and landing, i'll do the merge work
[02:20] <thumper> I'm not
[02:20] <thumper> but I was going to if you weren't
[02:21] <wallyworld> all good, i'm covering everything
[02:22] <davecheney> thumper: the metrics sender uses a global channel to send metrics ...
[02:22] <davecheney> one channel
[02:22] <davecheney> per process
[02:22] <davecheney> no matter how many workers or manifolds are running
[02:23] <thumper> I reader many writers?
[02:28] <davecheney> no, it's just another global that has to be patched everywhere
[02:31] <davecheney> thumper: oh, it gets worse
[02:31] <davecheney> now I've extracted that global
[02:31] <davecheney> it is celar nothing is draining that channel
[02:32] <davecheney> so shit is sending into it (it's not buffered of course) and nothing is reading from it
[02:32] <thumper> uh... wat?
[02:32] <davecheney> it's only used in tests
[02:32] <mup> Bug #1587701 opened: worker/metricworker: test timeout under race detector <juju-core:New> <https://launchpad.net/bugs/1587701>
[02:36] <davecheney> thumper: up, when not patched by tests, metricsworker.notify is nil
[02:38] <thumper> um...
[02:38] <thumper> isn't that ok though
[02:38] <thumper> ?
[02:40] <thumper> if notify is nil is it still sending?
[02:41] <davecheney> its only used as a hook during tests
[02:41] <davecheney> a nil channel is never ready to send or receive it is ignored in select
[02:43] <davecheney> looking into this a little further, almost everthing in that package is exported because tests
[02:43] <davecheney> nothing is called outside juju
[02:43] <davecheney> err, calling outside test scope
[02:45]  * thumper sighs
[02:46]  * thumper smacks himself in the face again with the lxd tests
[02:46] <thumper> FAIL	github.com/juju/juju/worker/terminationworker	1200.065s
[02:46] <thumper> failed to stop
[02:46] <thumper> 	/home/tim/go/src/github.com/juju/juju/worker/terminationworker/worker_test.go:63 +0x1ee
[02:46] <thumper> bah humbug
[02:57] <davecheney> thumper:         -n
[02:57] <davecheney>                 do not execute test binaries, compile only
[02:58] <davecheney> ^ I call this nope mode
[02:58] <davecheney> i just added to to gb to compile, but not run the tests
[03:07] <anastasiamac> axw: wallyworld:thumper: one less invalid login ... https://github.com/juju/juju/pull/5506
[03:08] <axw> looking
[03:19] <axw> anastasiamac: reviewed
[03:19] <axw> anastasiamac: and I don't know why, but I reviewed on github
[03:20] <anastasiamac> axw: i saw.. thank you :D m  moving logic around based on suggestion...
[03:20] <anastasiamac> axw: revew on github is my mistake - i've sent u github link instead of RB
[03:20] <axw> anastasiamac: that explains it :p
[03:35] <davecheney> thumper: menn0 https://github.com/juju/juju/pull/5507
[03:35] <menn0> davecheney, thumper looking
[03:37] <axw> wallyworld: in cmd/juju/application/unexpose.go there's still "A application"
[03:37] <thumper> davecheney: how does this fix the timeout?
[03:38] <wallyworld> axw: damn, i must have had the case checkbox ticked, i'll fix in the next one
[03:38] <davecheney> thumper: 1, removes the use of the suite
[03:38] <davecheney> 2, adds buffering to the channel
[03:39] <davecheney> thumper: look at metricmanager_test, that has a buffer of 2
[03:39] <menn0> davecheney: good stuff. ship it.
[03:39] <davecheney> but when testing the sender and the cleanup, they didn't have a buffer
[03:41] <thumper> davecheney: shipit
[03:41] <davecheney> thanks
[03:41] <davecheney> i ways also pleased that I could unexport most of the code in that package
[03:41] <davecheney> which made passing the notify channel for test less gross
[03:48] <menn0> thumper: simply alias help formatting change: http://reviews.vapour.ws/r/4953/
[03:49] <thumper> shipit
[03:52] <davecheney> thumper: there's just one blocker left on the race build https://bugs.launchpad.net/juju-core/+bug/1587716
[03:52] <mup> Bug #1587716: worker: test timeout during race build <juju-core:New> <https://launchpad.net/bugs/1587716>
[03:52] <davecheney> i'll look a that after lunch
[03:52] <thumper> awesome
[03:52] <thumper> thanks davecheney
[03:52] <davecheney> no worries
[04:00] <menn0> thumper: password check now passes for controller workers using HTTP endpoints for a hosted model
[04:00] <thumper> hazaah
[04:00] <menn0> thumper: still need to sort out the nonce issue though
[04:00] <menn0> thumper: i'll be pulling this out into a separate PR
[04:00] <thumper> sounds good
[04:00] <menn0> it's reasonably terrible
[04:01] <menn0> and problematic if I screw it up
[04:06] <anastasiamac> axw: cleaned and all tests pass. here is RB link :D http://reviews.vapour.ws/r/4951/
[04:07] <anastasiamac> axw: which is to say.. thre is probably nothing that really tests this well.. i'll talk to ppl to get better ci coverage i think
[04:08] <mup> Bug #1587716 opened: worker: test timeout during race build <juju-core:New> <https://launchpad.net/bugs/1587716>
[04:11]  * thumper sighs
[04:11] <thumper> where has --upload-tools gone?
[04:11] <thumper> bugger
[04:11] <thumper> pebkac
[04:22] <axw> anastasiamac: couple more small things
[06:09] <mup> Bug #1587734 opened: worker/machiner: test failure during race build <juju-core:New> <https://launchpad.net/bugs/1587734>
[06:30] <bradm> is there anyway to kill a juju run action?
[06:30] <bradm> got some queued that will never run
[07:31] <wallyworld> axw: FFS, why is it when you really need to land something, the bot hates you
[07:31] <axw> wallyworld: :(
[07:31] <wallyworld> still trying to land branch #2
[07:32] <axw> wallyworld: I've paused my branch, started a new one to restructure instancecfg. I need to pass more bootstrap params through, and the ball of mud in there was driving me crazy
[07:32] <axw> now finding bugs in providers, woo
[07:32] <wallyworld> yay
[08:16] <dimitern> dooferlad: ping
[08:20] <dimitern> dooferlad: didn't we port the fix that puts add-juju-bridge.py in /var/tmp vs /tmp from 1.25 to master ? precise is still broken because of this..
[08:30] <wallyworld> axw: yay, branch #2 landed, will propose #3
[08:30] <dooferlad> dimitern: yes, we did. Thought I had. I am on holiday until tomorrow so can you do it / file a bug / drop me an email to remind me?
[08:31] <dimitern> dooferlad: ah, ok - sorry then, enjoy your holiday :)
[08:37] <wallyworld> axw: only 235 files this time :-/ https://github.com/juju/juju/pull/5509
[08:48] <frobware> dimitern: not sure we have
[08:48] <frobware> dimitern: re: /var/tmp
[08:50] <dimitern> frobware: we haven't
[08:50] <dimitern> frobware: I've verified this earlier, but it only affects precise fortunately
[08:51] <frobware> dimitern: I don't see it in master or 1.25 :(
[08:52] <dimitern> frobware: it's in 1.25
[08:53] <dimitern> frobware: https://github.com/juju/juju/blob/1.25/provider/maas/bridgescript.go#L11
[08:53] <frobware> dimitern: right. sorry - my ref juju clone was out of date
[08:54] <frobware> dimitern: the scripts should be identical in 1.25 and master -- at least that's my expectation
[08:56] <frobware> dimitern: so they /var/tmp is missing. the other change I think dooferlad has in fligh, or in review
[08:56] <dimitern> frobware: the script is the same AFAICS, onlt the path is different
[08:58] <frobware> dimitern: I see a difference
[08:58] <frobware> dimitern: let me propose the /var/tmp change
[09:03] <dimitern> fwereade, jam: standup?
[09:13] <axw> wallyworld: looking now
[09:14] <axw> 10000 lines fewer than last time I think? :)
[09:14] <frobware> babbageclunk, dimitern: http://reviews.vapour.ws/r/4957/
[09:16] <wallyworld> axw: yeah, on home run now. still lots of vars etc not done. but those can wait
[09:20] <dimitern> frobware: LGTM, testing on precise now
[09:21] <axw> menn0: I wonder if we should split the help text out into separate files, and compile them in afterwards. might make it easier for, say, the docs team to manage them separately?
[09:21] <axw> menn0: also would be a sensible direction if we ever want to i18n
[09:30] <fwereade> axw, menn0: +1, but, omg, i18n, /shudder
[09:33] <axw> fwereade: heh, yeah. but won't somebody please think of the enterprises!
[09:33] <menn0> axw: yeah, not a bad idea
[09:34] <menn0> fwereade: I meant to say during the call, that a potential idea for the sprint would be to run a workshop where we run through code with issues and review them as a group
[09:35] <menn0> I wouldn't use specific code from Juju but come up with code that's heavily inspired by existing code, past or current
[09:36] <menn0> it would be helpful to work through a few examples, say a new worker, some apiserver changes, a client API change and a CLI change
[09:36] <menn0> and then try to hit as many of the issues you address in your doc as possible
[09:42] <fwereade> menn0, I am writing bits of http://paste.ubuntu.com/16886293/ when I need a break from agent tests -- and it STM that, to be useful, it needs to cover juju-specific scenarios and will thus inevitably end up with content a bit like that
[09:48] <dimitern> frobware: your fix works as far as the bridges get created on precise, however it seems lxc is still broken on precise due to: 2016/06/01 09:39:17 http: TLS handshake error from 10.12.19.103:43989: tls: client offered an unsupported, maximum protocol version of 302
[09:48] <axw> wallyworld: shipit
[09:48] <wallyworld> axw: you are awesome ty
[09:50] <frobware> dimitern: I'm ignoring LXC until advised not to
[09:51] <dimitern> frobware: yeah, just saying :)
[09:54] <frobware> dimitern: so I do notice on precise that we end up with an additional eth0.cfg (not 50-cloud-init.cfg) :(
[09:54] <dimitern> frobware: that's coming from the cloud image
[09:55] <frobware> dimitern: yep, it's just different from trust, xenial
[12:02] <mup> Bug #1587689 changed: Can't upgrade controller with --upload-tools (due to rename) <juju-core:Triaged by natefinch> <https://launchpad.net/bugs/1587689>
[12:02] <mup> Bug #1587739 opened: rackspace,vsphere: firewalling in HA setup is broken <juju-core:Triaged> <https://launchpad.net/bugs/1587739>
[12:02] <mup> Bug #1587788 opened: MAAS bridge script needs to reside in /var/tmp on precise <bootstrap> <network> <juju-core:Fix Committed by frobware> <https://launchpad.net/bugs/1587788>
[12:47] <fwereade> jam, incidentally, a picture of what's happening in an agent: http://paste.ubuntu.com/16889596/
[13:16] <dimitern> fwereade: do you have ~15m for a quick chat?
[13:18] <fwereade> dimitern, sure
[13:19] <dimitern> fwereade: I'm in today's standup HO
[13:49] <alexisb> natefinch-afk, please ping when you are in
[14:12] <natefinch> alexisb: ping
[14:15] <mup> Bug #1587701 changed: worker/metricworker: test timeout under race detector <blocker> <juju-core:Fix Released by dave-cheney> <https://launchpad.net/bugs/1587701>
[15:59] <babbageclunk> dimitern: I'm tempted to have a goroutine sleep in a test. Am I a bad person?
[16:00] <dimitern> babbageclunk: :) you mean os.Sleep?
[16:00] <mup> Bug #1553292 changed: TestGoroutineProfile dial unix : no such file or directory <go1.5> <go1.6> <intermittent-failure> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1553292>
[16:00] <babbageclunk> dimitern: yup
[16:00] <dimitern> babbageclunk: that actually is a known trick to yield back to the scheduler (with os.Sleep(0) for example)
[16:00] <dimitern> babbageclunk: but otherwise it's rarely a good idea
[16:00] <natefinch> babbageclunk: sleeping is really only bad if you wait until the thing wakes up.
[16:01] <dimitern> indeed, natefinch
[16:01] <babbageclunk> dimitern: Well, it's that or the goroutine goes into a tight spin until its OplogTailer gets stopped (which should happen very shortly after).
[16:01] <natefinch> dimitern: you shouldn't ever need to yield to the scheduler manually... every function call is a yield point these days.  Unless you're doing odd math stuff in a tight loop, you probably never need to worry about yielding
[16:02] <babbageclunk> dimitern: But yeah, it seemed like it was probably a bad idea.
[16:02] <natefinch> babbageclunk: well, ideally you'd have a channel that gets closed when the tailer stops
[16:02] <dimitern> natefinch: personally, I haven't found a good use for os.Sleep(0) except "let's see what will happen" :)
[16:02] <babbageclunk> natefinch: yeah, I do
[16:03] <babbageclunk> natefinch: but between it getting running out of canned data and getting stopped by the test, it goes into a spin.
[16:04] <babbageclunk> natefinch: When running for real it's querying mongo with a timeout, but that's faked out, essentially.
[16:05] <babbageclunk> natefinch: I think I'm worrying about nothing. Or at least, putting a sleep in to mimic the timeout would be actively worse.
[16:07]  * babbageclunk deliberately adds imports out of order for the joy of seeing them pop into place.
[16:07] <natefinch> lol
[16:07] <natefinch> I copy and paste code and intentionally don't even try to make my edited code look nice, because why bother when gofmt will just fix it up?
[16:19] <mup> Bug #1553292 changed: TestGoroutineProfile dial unix : no such file or directory <go1.5> <go1.6> <intermittent-failure> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1553292>
[16:19] <mup> Bug #1568179 changed: filestorageSuite.TestRelativePath fails because s390x host /tmp is different <ci> <s390x> <test-failure> <unit-tests> <juju-core:Fix Released by reedobrien> <https://launchpad.net/bugs/1568179>
[16:28] <babbageclunk> alexisb: ping?
[16:28] <alexisb> babbageclunk, pong
[16:29] <alexisb> whats up?
[16:29] <alexisb> o you acan merge you bug
[16:29] <alexisb> just tag it
[16:29] <babbageclunk> Ok, that was it - thanks!
[16:29] <perrito666> bbl lunch
[16:37] <babbageclunk> dimitern: I've tagged my bug with blocker and ci - should it be showing up on juju.fail?
[16:38] <babbageclunk> dimitern: It isn
[16:38] <babbageclunk> gah
[16:38] <mgz> babbageclunk: only if critical
[16:38] <mgz> babbageclunk: you can just run the script locally
[16:38] <babbageclunk> mgz: Ah, that'll be it.
[16:39] <babbageclunk> mgz: Where do I get it?
[16:39] <mgz> lp:juju-ci-tools
[16:39] <babbageclunk> mgz: great, thanks
[16:39] <babbageclunk> mgz: Welcome back by the way!
[16:40] <mgz> ./check_blockers.py check master
[16:40] <mgz> and wow, big list
[16:41] <mgz> want python-launchpadlib - can install via apt or pip
[16:41] <mgz> babbageclunk: thanks :)
[16:45] <babbageclunk> mgz - ok, so as you said, needs to be critical as well.
[16:51] <alexisb> babbageclunk, you sould be good now
[16:51] <babbageclunk> alexisb: Thanks!
[19:12] <natefinch> man, just used the 1.25 local provider.. forgot how fast it bootstraps.
[19:32] <perrito666> fast to bootstrap, fast to wipe your env
[19:32] <perrito666> and your machine
[19:32] <perrito666> and your network
[19:38] <mup> Bug #1588041 opened: juju bootstrap with vsphere provider hangs with xenial <oil> <juju-core:New> <https://launchpad.net/bugs/1588041>
[19:46] <natefinch> perrito666: I know I know... I just have a need for speed.
[20:16]  * thumper dashing to Jessie's school to drop off homework
[20:32] <perrito666> shame on you, you do your daughter's homework?
[20:32] <thumper> homework delivered
[20:35] <perrito666> there was a way to bootstrap (as in bootstrap.Bootstrap) that prevents upgrade after bootstrap, is that still possible?
[20:45] <natefinch> perrito666: probably passing config in with bootstrap
[21:31] <alexisb> wallyworld, thumper will be a few minutes late
[21:31] <wallyworld> ok
[22:19] <menn0> anastasiamac: should bug 1514874 be Fix Committed now? (for 1.25 and master)
[22:19] <mup> Bug #1514874: "Invalid entity name or password" error with valid credentials. <blocker> <juju-core:In Progress by anastasia-macmood> <juju-core 1.25:In Progress by anastasia-macmood> <https://launchpad.net/bugs/1514874>
[22:20] <anastasiamac> menn0: m waiting for stakeholder to verify..
[22:21] <anastasiamac> menn0: i'll have more info soon... m not too keen to "fix commit" atm.. it's a bit of a handful bug \o/
[22:21] <mup> Bug #1559299 opened: cannot obtain provisioning script <blocker> <bootstrap> <ci> <manual-provider> <regression> <xenial> <juju-core:Incomplete> <juju-core 1.25:Triaged> <juju-core api-call-retry:Fix Released by axwalk> <https://launchpad.net/bugs/1559299>
[22:21] <anastasiamac> menn0: but fwiw i *think* it's fixed ;-P
[22:30] <menn0> anastasiamac: ok np... just thought the bug status update had been missed :)
[22:32] <anastasiamac> menn0: no :-D it's intentional. thnx for checking
[22:51] <mup> Bug #1588084 opened: "upgrade-juju --upload-tools" is hard coded to the model name "admin" <juju-core:Triaged> <https://launchpad.net/bugs/1588084>
[23:03] <menn0> thumper and anastasiamac : trivial change to change command alias formatting: http://reviews.vapour.ws/r/4961/
[23:04] <thumper> shipit
[23:04] <anastasiamac> menn0: u have 3 shipits \o/
[23:05] <menn0> anastasiamac: that should do it :)
[23:05] <anastasiamac> yes \o/ highly desired, i guess :-P
[23:09] <axw> anastasiamac: do we still need to store image metadata in simplestreams format in the blob store?
[23:09] <axw> wallyworld: ^^
[23:09] <anastasiamac> axw: for 2,0?
[23:09] <axw> anastasiamac: yes
[23:10] <anastasiamac> axw: i don't think so
[23:10] <anastasiamac> axw: i don;t think we need to read from it anymore...
[23:10] <axw> anastasiamac: I don't see why it should be there either, so I'll take it out and see what breaks
[23:10] <axw> thanks
[23:11] <anastasiamac> axw: nps :D
[23:12] <wallyworld> axw: yeah, simplestream is external to juju, we model the image/tools metadata in the blobstore
[23:12] <axw> it's nice when putting things into a good structure shakes out a lot of old and broken things
[23:12] <axw> wallyworld: yep, just wasn't sure if anything was still reading it. sounds like not, which is what I expected/hoped
[23:13] <wallyworld> yup
[23:13] <anastasiamac> axw: r u resolving an issue around it or just a cleanup while u r in the area?
[23:14] <axw> anastasiamac: I'm changing the way we transfer args to the bootstrap agent, and part of that means changing how we transfer custom image metadata
[23:14] <anastasiamac> axw: awesome \o/
[23:14] <axw> anastasiamac: I'm transferring it in the original ImageMetadata struct format, so there's no need to save to disk then read back off
[23:14] <axw> anastasiamac: and while I was there I saw that we were still writing to blobstorte
[23:14] <anastasiamac> axw: brilliant - thnx :D
[23:14] <axw> store*
[23:14] <axw> it'll be much more straight forward to parameterise the bootstrap agent now
[23:17] <wallyworld> redir: standup?
[23:21] <mup> Bug #1588084 changed: "upgrade-juju --upload-tools" is hard coded to the model name "admin" <juju-core:Triaged> <https://launchpad.net/bugs/1588084>
[23:22] <redir> sorry
[23:22] <redir> brt
[23:25] <thumper> davecheney: ping
[23:26] <thumper> davecheney: unping
[23:30] <cmars> thumper, davecheney see my comments on LP:#1581157
[23:30] <mup> Bug #1581157: github.com/juju/juju/cmd/jujud test timeout on windows <blocker> <ci> <regression> <test-failure> <unit-tests> <windows> <juju-core:Triaged by dave-cheney> <https://launchpad.net/bugs/1581157>
[23:38] <thumper> davecheney: actually if you are around, I'd like to chat quickly
[23:58] <mup> Bug #1588092 opened: juju-2.0 has no way to cancel an action <canonical-bootstack> <juju-core:Triaged> <https://launchpad.net/bugs/1588092>
[23:58] <mup> Bug #1588095 opened: help for juju run-action refers to commands that don't exist <juju-core:Triaged> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1588095>
[23:59] <alexisb> menn0, I will be a few minutes late