Amazon - S3

aws

http://stackoverflow.com/questions/16069784/how-can-i-mount-an-s3-volume-with-proper-permissions-using-fuse
http://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html
https://cwhite.me/avoiding-the-burden-of-file-uploads/

What is Amazon S3?

Amazon Simple Storage Service (S3) provides a highly-available and redundant object store. This is great storage solution for somewhat static or slow-changing objects such as images, videos, and other static media. S3 also support edge caching and streaming of these assets via the CloudFront service.

Files stored in Amazon S3 are called objects, and every object is stored in a location called a bucket. An object is comprised of a file and optionally any metadata that describes that file. To store an object in Amazon S3, you upload the file you want to store to the bucket. When you upload a file, you can set permission on the object as well as any metadata.

Buckets are the containers for objects. You can have one or more buckets. For each bucket, you can control access to the bucket (who can create, delete, and list objects in the bucket), view access logs for the bucket and its objects. You can also choose the geographical region where Amazon S3 will store the bucket and its contents.

When using AWS Management Console, you can create folders to group objects. You can nest folders.

What are the basic steps to get started with S3?

  1. Sign up for S3
  2. Create a bucket
  3. Add an object to a bucket
  4. View an object
  5. Move an object
  6. Delete an object and bucket

How can we sign up for Amazon S3?

  1. Go to http://aws.amazon.com/s3 and click Sign Up for Amazon S3
  2. Follow the on-screen instructions
  3. AWS will notify you by e-mail when your account is active and available for you to use

How can we create a bucket?

  1. Sign into the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3
  2. Click Create Bucket
  3. Specify a name for the bucket. The bucket name you choose must be unique across all existing bucket names in Amazon S3. One way to do that is to prefix your bucket name with your company's name. Bucket name can contain lowercase letter, numbers, periods (.), underscore (_) and dashes (-). Bucket name must start with a letter or number. It must be between 3 and 255 characters long. Bucket name must not be formatted as an IP address. Once you create a bucket, you cannot change its name. In addition, the bucket name is visible in the URL that points to the objects stored in the bucket. Make sure the bucket name you choose is appropriate.
  4. In the Region drop-down box, select a Region
  5. Click Create

How can we create a bucket and upload a file?

  1. To allocate storage or create a bucket:
    1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
    2. In the Amazon S3 console, click Create Bucket.
    3. In the Create a Bucket — Select a Bucket Name and Region dialog box, in the Bucket Name box, enter a bucket name. The bucket name you choose must be unique across all existing bucket names in Amazon S3. After you create a bucket, you cannot change its name. For this tutorial, select a bucket name between 3 and 63 characters long, composed of lowercase letters and numbers. In this tutorial we use example123, but you will need to use a unique bucket name of your own choosing.
    4. In the Region box, select a geographic region that is close to you.
    5. When the settings are as you want them, click Create. When Amazon S3 successfully creates your bucket, the console displays its name in the All Buckets pane. This bucket is where you'll upload your files.
  2. To upload a file:
    1. In the Amazon S3 console, select the bucket that you previously created to contain your files. In our example, we will select the example123 bucket.
    2. Click Actions, and then click Upload.
    3. In the Upload — Select Files dialog box, click Add Files.
    4. In the File Upload dialog box, select files to upload. In order to stay within the limits of the free usage tier, the total size of the files you upload should be less than 5 GB.
    5. Click Open.
    6. In the Upload ‐ Select Files dialog box, click Start Upload.
    7. When your files have finished uploading, they will appear as shown in the following image.

How can we make a file public?

By default, a file uploaded to Amazon S3 can be accessed only by the AWS account that was used to upload it. If you want to make the file freely accessible to anyone on the Internet to view or download from a URL, you must make the file public:

  1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
  2. In the Amazon S3 console, select the bucket that you created to contain your files. In our example, we will select the example123 bucket.
  3. Right-click the file you uploaded.
  4. Click Make Public.
  5. Click OK to confirm making the file public.

How can we view a file stored on AWS?

  1. In the Amazon S3 console, select the bucket that you created to contain your files. In our example, we will select the example123 bucket.
  2. Click the file you uploaded.
  3. Click Properties.
  4. Click the URL that appears after Link. The file will appear in your browser. If the files is in a format that your browser can't display, you will be prompted to download the file.

How can we make a file private?

  1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
  2. In the Amazon S3 console, select the bucket that you created to contain your files. In our example, we will select the example123 bucket.
  3. Click the file you uploaded.
  4. Click Properties.
  5. Click Permissions.
  6. Uncheck the Open/Download checkbox for the Grantee Everyone.
  7. Click Save.

How can we delete a file and bucket?

  1. To delete a file:
    1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
    2. In the Buckets pane, click the bucket where the file is stored.
    3. Right-click the file that you want to delete and then click Delete.
    4. In the confirmation message that appears, click Yes, Delete.
  2. To delete a bucket:
    1. Right-click the bucket you want to delete, and then click Delete.
    2. In the confirmation message that appears, click Yes, Delete.

How can we mount S3 as a local directory? What are some different ways to mount S3 as a local directory?

If have more than one set of credentials, what should be the format of your /etc/passwd-s3fs?

bucketName1:accessKeyId:secretAccessKey
bucketName2:accessKeyId:secretAccessKey

What should be the permission on /etc/passwd-s3fs?

chmod 600 ~/.passwd-s3fs
sudo chmod 640 /etc/passwd-s3fs

Each user on the system can have his or her own passwd-s3fs file in their home directory, and there can also be a system-wide passwd-s3fs in /etc folder.

How can we debug s3fs?

Add the FUSE -f option to the s3fs command and look at /var/log/messages

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License