[14:29] <hazmat> smoser, ping.. we're seeing some issues booting instances on ec2, with apt-update failing.. was curious if you knew anything regarding http://paste.ubuntu.com/907227/ cloud-init output..
[14:30] <hazmat> looks like a hash mismatch on the update
[14:30] <smoser> utlemming, ^
[14:30] <hazmat> i thought the whole pipeline thing was default enabled
[14:30] <smoser> someone needs to open an RT on it, hazmat
[14:30] <smoser> we have to have IS kick that server
[14:30] <utlemming> hazmat: run "rm -rf /var/lib/apt/lists/*"
[14:30] <utlemming> hazmat: and then try again
[14:30]  * smoser is very looking forward to the hope of this all magically being fixed.
[14:31] <hazmat> utlemming, on the instance.. we can't per se.. we rely on cloud-init to install ssh keys etc.. but its failing pretty early.
[14:31] <smoser> cloud-init should still install keys, no ?
[14:32]  * hazmat checks
[14:32] <hazmat> smoser, it does
[14:33] <hazmat> but manual intervention is a hoser for juju
[14:33] <hazmat> smoser,  is there an RT already? i'd be happy to file it, i'm just fuzzy on the details of whats needed
[14:34] <hazmat> utlemming, smoser is this issue in relation to  the s3 apt repos, or is it just one of the ec2 pkg repos is borked
[14:34]  * hazmat tries a different a region
[14:34] <utlemming> hazmat:  its the ec2 pkg repos
[14:35] <utlemming> hazmat: so what I think is happening, is that the release file is not being updated. When "curl -I" on the Release file, the information seems stale for us-east-1. If you delete the local release file, then it should work.
[14:36] <utlemming> hazmat: if you can verify that, then I think we'll have enough information for a RT ticket
[14:36] <utlemming> hazmat: and I'll happily file it
[14:36] <smoser> utlemming, we dont really have to verify it
[14:37] <hazmat> utlemming,  i just killed the instance, but i should be able to reproduce and verify in 5m
[14:38] <utlemming> smoser: true, but verification which proves my theory makes it easier for IS to fix it
[14:44] <smoser> utlemming, check-archive (i added to make it check Sources.gz and Sources.bz2) runs successfully
[14:44] <smoser> ./check-archive http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise
[14:47] <hazmat> utlemming, confirmed fwiw
[14:48] <utlemming> hazmat: okay, thank you kindly.
[14:49] <smoser> hazmat, what did you confirm ?
[14:49] <smoser> just that removing those files and trying again fixed it ?
[14:51] <hazmat> smoser, yup
[14:53] <utlemming> smoser: in looking at the headers between the us-west-2 and us-east-1 mirrors, the HTTP headers indicate that the us-east-1 records is stale (at least when I looked yesterday). So what I think is happening, based on what I saw yesterday is that apt is using the cached release file because it appears that a newer one doesn't exist.
[14:53] <utlemming> smoseer: I'm rechecking now
[14:55] <smoser> utlemming, you're correct
[14:55] <utlemming> smoser: see http://paste.ubuntu.com/907322/
[14:55] <smoser> none of the mirrors in us-east-1 has updated since the 27th
[14:57] <smoser> utlemming, http://paste.ubuntu.com/907324/
[14:59] <utlemming> RT filed
[15:06] <smoser> utlemming, fwiw, this particular issue woudl have been (i think) fixed if we had the source lists in /var/lib/apt/lists