Dear reader,
Don’t close the tab , you’ve come to the right place.
In this article, we are going to tackle a very interesting subject which is storage in the Cloud.
Nowadays, various Cloud Storage Services exists (Amazon S3, Azure, Google, etc…) among them their is GCS (Google Cloud Storage) which we are going to focus on in this article.
Files in the Cloud are stored as objects not as blocks (traditional way). An object is composed of :
- Data
- Unique identifier : which represents the access path to data.
- Metadata : contains information about the object (type, ….)
Accessing/storing files can be established via an API, which offer more easy and elegant way to manage files.
For more details, read this great article about object storage and its advantages and drawbacks.
As you have noticed in the title, we are going to interact with GCS using Symfony, so let’s go 😛
Prerequisites
- You must have PHP and Composer installed on your computer. If it’s your first time to do that, check out theses articles that will help you setup PHP and Composer.
- A Google Cloud Platform account (free trial or paid membership, it doesn’t matter).
The outcome of this article will be a REST API that exposes three endpoints:
- Listing release notes;
- Creating a release note;
- Download a release note for given id;
First things first, let’s create a new Symfony application using composer:
composer create-project symfony/framework-standard-edition sf_gcs "3.4*"After that, we are going to install a few dependencies into our project, we will tackle the configuration just after that.
Dependencies installation
The first one is Gaufrette Bundle which provides a filesystem abstraction layer.
To install it just run the following command in your terminal (in the project directory):
composer require knplabs/knp-gaufrette-bundleThe second dependency is VichUploaderBundle, this bundle makes file upload easy.
The installation is same as before :
composer require vich/uploader-bundleIn order to make some HTTP calls to our API, we are going use FOSRestBundle which provides various tools to rapidly develop RESTful API’s & applications with Symfony.
The installation is done with a simple command:
composer require friendsofsymfony/rest-bundleIn order to interact with Google Services (Cloud Storage in our case), we will need the Google APIs Client which enables you to work with Google APIs such as Google+, Drive, or YouTube on your server. The good this that is maintained by Google, so no worries ;)
The installation is straight forward :
composer require google/apiclient:"^2.0"Configuration
Now after installing the necessary dependencies, we need to enable them in our app and make some configurations.
In your app/AppKernel.php add the lines (bold ones) to your bundles array:
$bundles = [
new Symfony\Bundle\FrameworkBundle\FrameworkBundle(),
new Symfony\Bundle\SecurityBundle\SecurityBundle(),
new Symfony\Bundle\TwigBundle\TwigBundle(),
new Symfony\Bundle\MonologBundle\MonologBundle(),
new Symfony\Bundle\SwiftmailerBundle\SwiftmailerBundle(),
new Doctrine\Bundle\DoctrineBundle\DoctrineBundle(),
new Sensio\Bundle\FrameworkExtraBundle\SensioFrameworkExtraBundle(),
new FOS\RestBundle\FOSRestBundle(),
new JMS\SerializerBundle\JMSSerializerBundle(),
new Knp\Bundle\GaufretteBundle\KnpGaufretteBundle(),
new Vich\UploaderBundle\VichUploaderBundle(),
new AppBundle\AppBundle(),
];The JMSSerializerBundle comes with FOSRestBundle which helps in the serialization/deserialization process that why we enable it too.
In order to make these bundles work together, we must configure them in app/config.yml file
Here is a sample configuration :
| parameters: | |
| locale: en | |
| cloud_storage: | |
| scope: '%google_cloud_storage_scope%' | |
| key: '%google_cloud_storage_private_key_location%' | |
| #... | |
| fos_rest: | |
| param_fetcher_listener: true | |
| body_listener: true | |
| format_listener: false | |
| view: | |
| view_response_listener: 'force' | |
| formats: | |
| xml: true | |
| json : true | |
| templating_formats: | |
| html: true | |
| force_redirects: | |
| html: true | |
| failed_validation: HTTP_BAD_REQUEST | |
| default_engine: twig | |
| routing_loader: | |
| default_format: json | |
| serializer: | |
| serialize_null: true | |
| knp_gaufrette: | |
| stream_wrapper: ~ | |
| adapters: | |
| release_notes_adapter: | |
| google_cloud_storage: | |
| service_id: 'app.google_cloud_storage.service' | |
| bucket_name: '%release_notes_bucket_name%' | |
| filesystems: | |
| release_notes_filesystem: | |
| adapter: release_notes_adapter | |
| alias: release_notes_filesystem | |
| vich_uploader: | |
| db_driver: orm | |
| storage: gaufrette | |
| mappings: | |
| release_note_mapping: | |
| upload_destination: release_notes_filesystem |
The configuration of the Gaufrette bundle is divided into two parts: the adapters and the filesystems.
Configuring the Adapters
Gaufrette uses adapters to distinguish how the files are going to be persisted, in our case we are using googlecloudstorage adapter which takes as parameters :
- service_id : The service id of the
\Google_Service_Storageto use. (required) - bucket_name : The name of the GCS bucket to use. (required)
- detectcontenttype : if true will detect the content type for each file (default true)
- options : A list of additional options passed to the adapter.
- directory : A directory to operate in. (default ”)
- acl : Whether the uploaded files should be private or public. (default private)
Configuring FileSystems
The defined adapters are then used to create the filesystems.
Each defined filesystem must have an adapter with its value set to an adapter’s key. The alias parameter allows us to define an alias for it (releasenotesfilesystem in this case).
We also need to add reference to our buckets within the application, for that let’s add some parameters to app/config/parameters.yml.dist.
| parameters: | |
| database_host: 127.0.0.1 | |
| database_port: ~ | |
| database_name: symfony | |
| database_user: root | |
| database_password: ~ | |
| # You should uncomment this if you want to use pdo_sqlite | |
| #database_path: '%kernel.project_dir%/var/data/data.sqlite' | |
| mailer_transport: smtp | |
| mailer_host: 127.0.0.1 | |
| mailer_user: ~ | |
| mailer_password: ~ | |
| # A secret key that's used to generate certain security-related tokens | |
| secret: ThisTokenIsNotSoSecretChangeIt | |
| google_cloud_storage_scope: 'https://www.googleapis.com/auth/devstorage.full_control' | |
| google_cloud_storage_private_key_location: '%env(GCS_PRIVATE_KEY_LOCATION)%' | |
| release_notes_bucket_name: '%env(GCS_RELEASE_NOTES_BUCKET_NAME)%' |
google_cloud_storage_scope: Access Rights (READ, READWRITE, FULLCONTROL)google_cloud_storage_private_key_location: relative or absolute path to credentials file (P12 or JSON (Recommended way))release_notes_bucket_name: The name of the buckets we want to interact with.
P.S : Values are retrieved from environment variables.
Service configuration
As you have noticed in the adapters configuration, the serviceid is ‘app.googlecloud_storage.service’ which is an alias to the service that communicates with Cloud Storage.
| services: | |
| app.google_cloud_storage_factory: | |
| class: AppBundle\Factory\GoogleCloudStorageServiceFactory | |
| arguments: ['%cloud_storage%'] | |
| app.google_cloud_storage.service: | |
| class: \Google_Service_Storage | |
| factory: 'app.google_cloud_storage_factory:createService' |
The ServiceFactory role in simple, here is how it looks:
| <?php | |
| namespace AppBundle\Factory; | |
| use Google_Service_Storage; | |
| class GoogleCloudStorageServiceFactory | |
| { | |
| private $configuration; | |
| function __construct($cloudStorageConfig) | |
| { | |
| $this->configuration = $cloudStorageConfig; | |
| } | |
| public function createService() { | |
| $client = new \Google_Client(); | |
| $client->setAuthConfig($this->configuration['key']); | |
| $client->setScopes([$this->configuration['scope']]); | |
| return new Google_Service_Storage($client); | |
| } | |
| } |
Another important thing that your entity must have a file field annotated so that VichUploader can do the mapping in the incoming request. (line 72 on the below gist).
| <?php | |
| namespace AppBundle\Entity; | |
| use Symfony\Component\HttpFoundation\File\File as File; | |
| use Vich\UploaderBundle\Mapping\Annotation as Vich; | |
| use Doctrine\ORM\Mapping as ORM; | |
| use JMS\Serializer\Annotation as Serializer; | |
| use Symfony\Component\Validator\Constraints as Assert; | |
| /** | |
| * ReleaseNotes | |
| * | |
| * @ORM\Table(name="release_notes") | |
| * @ORM\Entity(repositoryClass="AppBundle\Repository\ReleaseNotesRepository") | |
| * @Vich\Uploadable | |
| */ | |
| class ReleaseNotes | |
| { | |
| /** | |
| * @var integer | |
| * | |
| * @ORM\Column(name="id", type="integer") | |
| * @ORM\Id | |
| * @ORM\GeneratedValue(strategy="AUTO") | |
| */ | |
| private $id; | |
| /** | |
| * @ORM\Column(type="string", length=255, name="version", nullable=true) | |
| * | |
| * @var string $version | |
| */ | |
| private $version; | |
| /** | |
| * @ORM\Column(type="string", length=255, name="document", nullable=true) | |
| * | |
| * @var string $document | |
| */ | |
| private $document; | |
| /** | |
| * @var string | |
| * | |
| * @ORM\Column(name="message", type="text") | |
| */ | |
| private $message; | |
| /** | |
| * @var \DateTime | |
| * Release date | |
| * @ORM\Column(type="datetime", name="release_date", nullable=true) | |
| */ | |
| private $releaseDate; | |
| /** | |
| * @Assert\File( | |
| * mimeTypes={ | |
| * "application/pdf", | |
| * "image/jpeg", | |
| * "image/pjpeg", | |
| * "application/msword", | |
| * "image/png" | |
| * }, | |
| * mimeTypesMessage="The file format is not correct" | |
| * ) | |
| * @Vich\UploadableField(mapping="release_note_mapping", fileNameProperty="document", nullable=true) | |
| * | |
| * @var File $file | |
| * @Serializer\Exclude | |
| */ | |
| private $file; | |
| public function __construct() | |
| { | |
| $this->releaseDate = new \DateTime(); | |
| } | |
| /** | |
| * @param int $id | |
| */ | |
| public function setId($id) | |
| { | |
| $this->id = $id; | |
| } | |
| /** | |
| * @return int | |
| */ | |
| public function getId() | |
| { | |
| return $this->id; | |
| } | |
| /** | |
| * @return string | |
| */ | |
| public function getVersion() | |
| { | |
| return $this->version; | |
| } | |
| /** | |
| * @param string $version | |
| */ | |
| public function setVersion($version) | |
| { | |
| $this->version = $version; | |
| } | |
| /** | |
| * @return mixed | |
| */ | |
| public function getDocument() | |
| { | |
| return $this->document; | |
| } | |
| /** | |
| * @param mixed $document | |
| */ | |
| public function setDocument($document) | |
| { | |
| $this->document = $document; | |
| } | |
| /** | |
| * @return string | |
| */ | |
| public function getMessage() | |
| { | |
| return $this->message; | |
| } | |
| /** | |
| * @param string $message | |
| */ | |
| public function setMessage($message) | |
| { | |
| $this->message = $message; | |
| } | |
| /** | |
| * @return File | |
| */ | |
| public function getFile() | |
| { | |
| return $this->file; | |
| } | |
| /** | |
| * @param File $file | |
| */ | |
| public function setFile($file) | |
| { | |
| $this->file = $file; | |
| } | |
| /** | |
| * @return \DateTime | |
| */ | |
| public function getReleaseDate() | |
| { | |
| return $this->releaseDate; | |
| } | |
| /** | |
| * @param \DateTime $releaseDate | |
| */ | |
| public function setReleaseDate($releaseDate) | |
| { | |
| $this->releaseDate = $releaseDate; | |
| } | |
| } |
After this, the only thing left is to create a repository, service and a controller where you will put your logic to store and fetch data.
Let’s not forget that you will need a database with a release_notes table.
You can use DoctrineMigrationsBundle to manage database changes in your application.
That’s it, full source code is available in my github repository.