AWS credentials issue on stable

[jywarren]

After publishing #1712, I'm seeing a 500 upon uploading images:

I, [2022-05-24T23:21:30.624892 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] [paperclip] Trying to link /tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg to /tmp/cfcd208495d565ef66e7dff9
f98764da20220524-11541-14289da.jpg
I, [2022-05-24T23:21:30.625302 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: file -b --mime '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-14289da.jpg'
I, [2022-05-24T23:21:30.646900 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: identify -format '%wx%h,%[exif:orientation]' '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]
' 2>/dev/null
I, [2022-05-24T23:21:30.691077 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: identify -format %m '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]'
I, [2022-05-24T23:21:30.731190 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: convert '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]' -auto-orient -resize "500x375" '/tm
p/dbbcbbe5c18542c4494bf2d6c28821eb20220524-11541-1gze6a7'
I, [2022-05-24T23:21:30.783449 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] [paperclip] Trying to link /tmp/dbbcbbe5c18542c4494bf2d6c28821eb20220524-11541-1gze6a7 to /tmp/0791ea3a0e6c58e35f521430709b
a94d20220524-11541-1s41nwx
I, [2022-05-24T23:21:30.784762 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: identify -format '%wx%h,%[exif:orientation]' '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]
' 2>/dev/null
I, [2022-05-24T23:21:30.823005 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: identify -format %m '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]'
I, [2022-05-24T23:21:30.861567 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: convert '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]' -auto-orient -resize "240x180" '/tm
p/dbbcbbe5c18542c4494bf2d6c28821eb20220524-11541-1gt36q8'
I, [2022-05-24T23:21:30.908730 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] [paperclip] Trying to link /tmp/dbbcbbe5c18542c4494bf2d6c28821eb20220524-11541-1gt36q8 to /tmp/9739964993b30ba049d58390da9d
744220220524-11541-10q5r3z
I, [2022-05-24T23:21:30.909403 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: identify -format '%wx%h,%[exif:orientation]' '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]
' 2>/dev/null
I, [2022-05-24T23:21:30.949645 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: identify -format %m '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]'
I, [2022-05-24T23:21:30.987060 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: convert '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg[0]' -auto-orient -resize "100x100>" '/t
mp/dbbcbbe5c18542c4494bf2d6c28821eb20220524-11541-1w7scmn'
I, [2022-05-24T23:21:31.040184 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] [paperclip] Trying to link /tmp/dbbcbbe5c18542c4494bf2d6c28821eb20220524-11541-1w7scmn to /tmp/ce22f1f8fab173a95faf8be97e23
146220220524-11541-1tnruhd
D, [2022-05-24T23:21:31.044217 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[36mMap Load (2.0ms)ESC[0m  ESC[1mESC[34mSELECT  `maps`.* FROM `maps` WHERE `maps`.`slug` = 'test-bcccfc71-64d1-
4299-a8a1-9806a1c95ca7' LIMIT 1ESC[0m
D, [2022-05-24T23:21:31.046572 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[35m (0.9ms)ESC[0m  ESC[1mESC[35mBEGINESC[0m
D, [2022-05-24T23:21:31.049220 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[36mMap Exists (1.2ms)ESC[0m  ESC[1mESC[34mSELECT  1 AS one FROM `maps` WHERE `maps`.`slug` = BINARY 'test-bcccf
c71-64d1-4299-a8a1-9806a1c95ca7' AND `maps`.`id` != 14719 LIMIT 1ESC[0m
D, [2022-05-24T23:21:31.052562 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[36mMap Update (1.7ms)ESC[0m  ESC[1mESC[33mUPDATE `maps` SET `updated_at` = '2022-05-24 23:21:31' WHERE `maps`.`
id` = 14719ESC[0m
D, [2022-05-24T23:21:31.147755 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[35m (94.4ms)ESC[0m  ESC[1mESC[35mCOMMITESC[0m
D, [2022-05-24T23:21:31.149488 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[35m (1.3ms)ESC[0m  ESC[1mESC[35mBEGINESC[0m
I, [2022-05-24T23:21:31.150438 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] [paperclip] Trying to link /tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1d2cnrb.jpg to /tmp/cfcd208495d565ef66e7dff9
f98764da20220524-11541-1tg1dir.jpg
I, [2022-05-24T23:21:31.151240 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Command :: file -b --mime '/tmp/cfcd208495d565ef66e7dff9f98764da20220524-11541-1tg1dir.jpg'
D, [2022-05-24T23:21:31.178194 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[36mWarpable Create (2.7ms)ESC[0m  ESC[1mESC[32mINSERT INTO `warpables` (`image_content_type`, `image_file_name`
, `image_file_size`, `created_at`, `updated_at`, `map_id`, `history`) VALUES ('image/jpeg', '0.jpg', 61129, '2022-05-24 23:21:31', '2022-05-24 23:21:31', 14719, 'None')ESC[0m
D, [2022-05-24T23:21:31.183141 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[36mPaperTrail::Version Create (1.8ms)ESC[0m  ESC[1mESC[32mINSERT INTO `versions` (`item_type`, `item_id`, `event`, `whodunnit`, `created_at`) VALUES ('Warpable', 329085, 'create', '1', '2022-05-24 23:21:31')ESC[0m
D, [2022-05-24T23:21:31.186147 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[35m (1.3ms)ESC[0m  ESC[1mESC[34mSELECT COUNT(*) FROM `versions` WHERE `versions`.`item_type` = 'Warpable' AND `versions`.`item_id` = 329085 AND (event <> 'create')ESC[0m
I, [2022-05-24T23:21:31.186667 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] [paperclip] saving warpables/329085/0.jpg
I, [2022-05-24T23:21:31.367398 #11604]  INFO -- : [f62abeaa-8ec2-48ca-a573-d4a49f1f3123] Started GET "/cable" for 96.238.50.169 at 2022-05-24 23:21:31 +0000
I, [2022-05-24T23:21:31.372320 #11604]  INFO -- : [f62abeaa-8ec2-48ca-a573-d4a49f1f3123] Started GET "/cable/"[non-WebSocket] for 96.238.50.169 at 2022-05-24 23:21:31 +0000
E, [2022-05-24T23:21:31.372601 #11604] ERROR -- : [f62abeaa-8ec2-48ca-a573-d4a49f1f3123] Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: , HTTP_UPGRADE: )
I, [2022-05-24T23:21:31.372701 #11604]  INFO -- : [f62abeaa-8ec2-48ca-a573-d4a49f1f3123] Finished "/cable/"[non-WebSocket] for 96.238.50.169 at 2022-05-24 23:21:31 +0000
D, [2022-05-24T23:21:34.307572 #11541] DEBUG -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   ESC[1mESC[35m (111.2ms)ESC[0m  ESC[1mESC[31mROLLBACKESC[0m
I, [2022-05-24T23:21:34.309057 #11541]  INFO -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Completed 500 Internal Server Error in 3691ms (ActiveRecord: 220.1ms)
F, [2022-05-24T23:21:34.312677 #11541] FATAL -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   
F, [2022-05-24T23:21:34.312790 #11541] FATAL -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] Aws::Sigv4::Errors::MissingCredentialsError (missing credentials, provide credentials with one of the following options:
  - :access_key_id and :secret_access_key
  - :credentials
  - :credentials_provider):
F, [2022-05-24T23:21:34.312900 #11541] FATAL -- : [b317a5bc-142b-43c9-9346-2f6e91daf449]   
F, [2022-05-24T23:21:34.313020 #11541] FATAL -- : [b317a5bc-142b-43c9-9346-2f6e91daf449] app/controllers/images_controller.rb:34:in `block in create'

[jywarren]

We had changed creds and not updated on stable, so made new keys, let's reboot and see!

[jywarren]

Didn't work -- i do see this though, maybe we missed something? But it is working on production so i'm not sure why this wouldn't work:

Post-install message from paperclip:
##################################################
#  NOTE FOR UPGRADING FROM 4.3.0 OR EARLIER      #
##################################################

Paperclip is now compatible with aws-sdk-s3.

If you are using S3 storage, aws-sdk-s3 requires you to make a few small
changes:

* You must set the `s3_region`
* If you are explicitly setting permissions anywhere, such as in an initializer,
  note that the format of the permissions changed from using an underscore to
  using a hyphen. For example, `:public_read` needs to be changed to
  `public-read`.

For a walkthrough of upgrading from 4 to *5* (not 6) and aws-sdk >= 2.0 you can watch
http://rubythursday.com/episodes/ruby-snack-27-upgrade-paperclip-and-aws-sdk-in-prep-for-rails-5

[jywarren]

I don't think the above is relevant, we have been on aws-sdk-s3 for a few years it seems.

Aws::Sigv4::Errors::MissingCredentialsError is exact - i checked AWS logs and there was no access at all. Why isn't it picking up the keys?

[jywarren]

We just got staging working, by standardizing all references of AWS_ACCESS_KEY_ID -- it had been missing _ID in: amazon_s3.yml and mapknitter.env.

Strange because it was working in production in Google Cloud, where it drew on a stored app.yaml file that used _ID.

But now strangely AWS console says images are being uploaded (from stable, i believe, based on having done so in the timeframe successfully) but NOT using the newly made key we put in environment.sh. So is there another place it could be getting the key?

[jywarren]

OK, no, correction - the new key IS being used. the access log must have been slow to load. It shows usage within a minute of my successful upload to staging. So that seems all good.

The only question now is whether we will break things upon deploying MK, by having changed some env var names in a working system.

[jywarren]

We tried deploying to production but got:

error webgpu@0.1.16: The engine "node" is incompatible with this module. Expected version ">= 13.0.0". Got "12.22.5"

[jywarren]

Will try setting this here:

mapknitter/package.json

Line 48 in da8ec9d

"node": ">=10.0.0"